AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder

Tal Shaharabany, Aviad Dahan, Raja Giryes, Lior Wolf
6/10/2023
cs.CV

No official code found. Best implementation: talshaharabany/AutoSAM (55 stars, last commit 2025-07-13).

Abstract

The recently introduced Segment Anything Model (SAM) combines a clever architecture and large quantities of training data to obtain remarkable image segmentation capabilities. However, it fails to reproduce such results for Out-Of-Distribution (OOD) domains such as medical images. Moreover, while SAM is conditioned on either a mask or a set of points, it may be desirable to have a fully automatic solution. In this work, we replace SAM's conditioning with an encoder that operates on the same input image. By adding this encoder and without further fine-tuning SAM, we obtain state-of-the-art results on multiple medical images and video benchmarks. This new encoder is trained via gradients provided by a frozen SAM. For inspecting the knowledge within it, and providing a lightweight segmentation solution, we also learn to decode it into a mask by a shallow deconvolution network.

View on arXivView PDF

Code Implementations(1)

talshaharabany/AutoSAMCommunity72%

AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder

5517PythonSep 15, 20231 years ago

Similar Papers

Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation

88% match1 repo352025

GMISeg: General Medical Image Segmentation without Re-Training

87% match2023

How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images

87% match1 repo1392023

Dilated-UNet: A Fast and Accurate Medical Image Segmentation Approach using a Dilated Transformer and U-Net Architecture

87% match1 repo82023

DeSAM: Decoupled Segment Anything Model for Generalizable Medical Image Segmentation

87% match1 repo1342023

SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3

86% match1 repo2892025

Cite this paper

@article{shaharabany2023autosam,
  title  = {AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder},
  author = {Tal Shaharabany and Aviad Dahan and Raja Giryes and Lior Wolf},
  year   = {2023},
  eprint = {2306.06370},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.06370v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.06370.svg)](https://codeofpaper.com/paper/2306.06370)

Discussion