Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM

Abstract

As an important pillar of underwater intelligence, Marine Animal Segmentation (MAS) involves segmenting ani-mals within marine environments. Previous methods don't excel in extracting long-range contextual features and over-look the connectivity between discrete pixels. Recently, Segment Anything Model (SAM) offers a universal frame-workfor general segmentation tasks. Unfortunately, trained with natural images, SAM does not obtain the prior knowl-edge from marine images. In addition, the single-position prompt of SAM is very insufficient for prior guidance. To address these issues, we propose a novel feature learning framework, named Dual-SAM for high-performance MAS. To this end, we first introduce a dual structure with SAM's paradigm to enhance feature learning of marine images. Then, we propose a Multi-level Coupled Prompt (MCP) strategy to instruct comprehensive underwater prior infor-mation, and enhance the multi-level features of SAM's en-coder with adapters. Subsequently, we design a Dilated Fusion Attention Module (DFAM) to progressively inte-grate multi-level features from SAM's encoder. Finally, in-stead of directly predicting the masks of marine animals, we propose a Criss-Cross Connectivity Prediction (C3 P) paradigm to capture the inter-connectivity between discrete pixels. With dual decoders, it generates pseudo-labels and achieves mutual supervision for complementary feature rep-resentations, resulting in considerable improvements over previous techniques. Extensive experiments verify that our proposed method achieves state-of-the-art performances on five widely-used MAS datasets. The code is available at https://github.con1IDrchip61IDual_SAM.