MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation

[🏠 Sa2VA] [📜 arXiv] [🤗 HuggingFace] [Gradio Demo] [Gradio Demo (By HuggingFace Offical)]

Opensource progress

• Release Demo.
• Release evaluation code for each dataset.
• Release model weight.
• Release training code for InterVL-4b model.
• Release inference and test code.

Overview

This repository contains the code for the paper "MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation".

Model Zoo

We provide the following models weights on huggingface:

🤗 Gradio Demos

We provide a script that implements interactive chat using gradio, which requires installing gradio. You can try it to build a local chat interface quickly.

PYTHONPATH=. python projects/llava_sam2/gradio/app.py PATH/TO/Model

Environment

Use uv to manage dependencies. Run uv sync to install everything, choosing the extra based on your model family:

• uv sync --extra=legacy for InternVL2.5 or earlier models (legacy Transformers).
• uv sync --extra=latest for newer models (latest Transformers).

🚀 Quick Start

Training

Evaluation

References

If you find this repository useful, please consider referring to the following paper:

@article{medvlsam2unified3dmedical,
  title={MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation},
  author={Yang Xing and Jiong Wu and Savas Ozdemir and Ying Zhang and Yang Yang and Wei Shao and Kuang Gong},
  journal={arXiv pre-print},
  year={2026}
}