DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
Tal Grossman, Noa Cahan, Lev Ayzenberg, Hayit Greenspan
TLDR
DiffuSAM adapts SAM2 for prompt-free medical image segmentation using a diffusion prior, achieving strong performance with efficient training.
Key contributions
- Proposes DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation.
- Synthesizes SAM2-compatible mask embeddings using a lightweight diffusion prior from frozen SAM2 features.
- Integrates generated embeddings into SAM2's mask decoder, eliminating the need for user prompts.
- Conditions the diffusion prior on previous slices to enforce spatial consistency across medical volumes.
Why it matters
SAM/SAM2 struggle with medical data due to natural image training and require extensive fine-tuning and prompts. DiffuSAM addresses this by enabling accurate, prompt-free medical image segmentation. This significantly improves domain transfer and efficiency for clinical applications.
Original Abstract
Segmentation models such as Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation typically requires extensive fine-tuning and expert-designed prompts. We propose DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation. Our framework synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion-prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across volumes. Evaluated on the BTCV and CHAOS datasets for CT and MRI under Source-Free Unsupervised Domain Adaptation (SF-UDA) and Few-Shot settings, DiffuSAM achieves competitive performance with efficient training and inference. Code is available upon request from the corresponding author.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.