Tong Wang¹², Yaolei Qi¹, Siwen Wang², Imran Razzak², Guanyu Yang¹✉, Yutong Xie²✉
¹ Southeast University, China
² Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE
✉ Corresponding Author
- [May, 2026] CMSA-Net has been accepted as a MICCAI 2026 Early Accept paper.
- [Feb, 2026] Released prediction results for both PVT-V2-B2 and Res2Net50 backbones.
- [Feb, 2026] CMSA-Net is available on arXiv.
Video polyp segmentation (VPS) is an important task in computer-aided colonoscopy, as it helps doctors accurately locate and track polyps during examinations. However, VPS remains challenging because polyps often look similar to surrounding mucosa, leading to weak semantic discrimination. In addition, large changes in polyp position and scale across video frames make stable and accurate segmentation difficult.
To address these challenges, we propose a robust VPS framework named CMSA-Net. The proposed network introduces a Causal Multi-scale Aggregation (CMA) module to effectively gather semantic information from multiple historical frames at different scales. By using causal attention, CMA ensures that temporal feature propagation follows strict time order, which helps reduce noise and improve feature reliability.
Furthermore, we design a Dynamic Multi-source Reference (DMR) strategy that adaptively selects informative and reliable reference frames based on semantic separability and prediction confidence. This strategy provides strong multi-frame guidance while keeping the model efficient for real-time inference.
Extensive experiments on the SUN-SEG dataset demonstrate that CMSA-Net achieves state-of-the-art performance, offering a favorable balance between segmentation accuracy and real-time clinical applicability.
Figure: Framework Overview.
Figure: Qualitative comparison of CMSA-Net against other SOTA methods.
Complete quantitative results on the SUN-SEG benchmark. Bold indicates the best performance.
| Method | Publication | Backbone | Sα↑ | Eφ↑ | Fβ↑ | Dice↑ | IoU↑ | MAE↓ |
|---|---|---|---|---|---|---|---|---|
| COSNet | TPAMI'19 | - | 84.5 | 83.6 | 72.7 | 73.0 | 64.8 | 3.4 |
| PCSA | AAAI'20 | - | 85.2 | 83.5 | 68.1 | 70.9 | 60.4 | 3.9 |
| 2/3D | MICCAI'20 | - | 89.5 | 90.9 | 81.9 | 82.9 | 75.6 | 2.1 |
| PraNet | MICCAI'20 | R2-50 | 91.8 | 94.2 | 87.7 | 88.3 | 82.5 | 2.0 |
| ACSNet | MICCAI'20 | R-34 | 92.0 | 94.2 | 87.4 | 88.2 | 82.8 | 1.7 |
| SANet | MICCAI'21 | R2-50 | 91.6 | 93.3 | 86.6 | 87.2 | 82.0 | 1.8 |
| SEPNet | TCSVT'24 | PVTv2-B2 | 93.1 | 96.2 | 88.3 | 89.6 | 83.4 | 1.7 |
| PNS | MICCAI'21 | R2-50 | 90.6 | 91.0 | 83.6 | 84.1 | 78.3 | 2.0 |
| PNS+ | MIR'22 | R2-50 | 91.7 | 92.5 | 84.8 | 85.5 | 78.7 | 2.1 |
| MAST | Arxiv'24 | PVTv2-B2 | 92.5 | 96.2 | 87.8 | 89.3 | 82.7 | 1.6 |
| SALI | MICCAI'24 | PVTv2-B2 | 90.2 | 93.2 | 84.9 | 85.8 | 78.9 | 2.4 |
| SALI | MICCAI'24 | PVTv2-B5 | 90.7 | 93.7 | 85.1 | 86.2 | 79.6 | 2.2 |
| STDDNet | ICCV'25 | R2-50 | 93.5 | 96.0 | 89.7 | 90.5 | 85.0 | 1.5 |
| STDDNet | ICCV'25 | PVTv2-B2 | 94.1 | 96.9 | 90.5 | 91.5 | 86.1 | 1.4 |
| Ours | - | R2-50 | 94.5 | 97.3 | 90.5 | 91.9 | 86.5 | 1.2 |
| Ours | - | PVTv2-B2 | 95.1 | 97.5 | 91.6 | 92.6 | 87.6 | 1.1 |
| Method | Publication | Backbone | Sα↑ | Eφ↑ | Fβ↑ | Dice↑ | IoU↑ | MAE↓ |
|---|---|---|---|---|---|---|---|---|
| COSNet | TPAMI'19 | - | 65.4 | 60.0 | 43.1 | 42.3 | 34.2 | 7.3 |
| PCSA | AAAI'20 | - | 68.0 | 66.0 | 45.1 | 45.0 | 35.3 | 7.8 |
| 2/3D | MICCAI'20 | - | 78.6 | 77.7 | 65.2 | 65.6 | 57.0 | 4.4 |
| PraNet | MICCAI'20 | R2-50 | 78.1 | 78.8 | 66.3 | 66.5 | 58.2 | 5.2 |
| ACSNet | MICCAI'20 | R-34 | 77.2 | 76.6 | 63.0 | 63.8 | 56.4 | 4.6 |
| SANet | MICCAI'21 | R2-50 | 75.0 | 72.8 | 59.0 | 59.3 | 52.4 | 5.2 |
| SEPNet | TCSVT'24 | PVTv2-B2 | 82.9 | 88.3 | 73.5 | 75.1 | 66.6 | 4.2 |
| PNS | MICCAI'21 | R2-50 | 76.7 | 74.4 | 61.6 | 61.8 | 54.5 | 4.8 |
| PNS+ | MIR'22 | R2-50 | 80.6 | 79.8 | 67.6 | 67.8 | 59.1 | 4.4 |
| MAST | Arxiv'24 | PVTv2-B2 | 83.2 | 89.4 | 74.9 | 77.0 | 67.4 | 3.7 |
| SALI | MICCAI'24 | PVTv2-B2 | 73.1 | 75.2 | 58.7 | 59.2 | 50.2 | 6.3 |
| SALI | MICCAI'24 | PVTv2-B5 | 77.1 | 82.1 | 64.6 | 65.6 | 56.8 | 5.5 |
| STDDNet | ICCV'25 | R2-50 | 81.7 | 83.0 | 72.1 | 72.4 | 64.3 | 3.7 |
| STDDNet | ICCV'25 | PVTv2-B2 | 86.0 | 90.3 | 78.6 | 80.1 | 72.4 | 3.4 |
| Ours | - | R2-50 | 84.4 | 90.1 | 75.3 | 77.5 | 69.2 | 3.5 |
| Ours | - | PVTv2-B2 | 86.7 | 90.3 | 79.3 | 80.3 | 72.6 | 2.9 |
| Method | Publication | Backbone | Sα↑ | Eφ↑ | Fβ↑ | Dice↑ | IoU↑ | MAE↓ |
|---|---|---|---|---|---|---|---|---|
| COSNet | TPAMI'19 | - | 78.5 | 77.2 | 62.6 | 63.3 | 54.1 | 4.6 |
| PCSA | AAAI'20 | - | 77.2 | 75.9 | 56.6 | 58.5 | 47.9 | 5.7 |
| 2/3D | MICCAI'20 | - | 84.9 | 86.9 | 75.3 | 76.4 | 67.1 | 3.5 |
| PraNet | MICCAI'20 | R2-50 | 88.4 | 91.9 | 83.1 | 83.9 | 76.6 | 3.1 |
| ACSNet | MICCAI'20 | R-34 | 87.2 | 91.0 | 80.6 | 82.0 | 74.8 | 3.6 |
| SANet | MICCAI'21 | R2-50 | 87.4 | 90.5 | 81.0 | 82.0 | 74.8 | 3.3 |
| SEPNet | TCSVT'24 | PVTv2-B2 | 89.4 | 94.0 | 83.5 | 85.7 | 77.6 | 3.4 |
| PNS | MICCAI'21 | R2-50 | 87.0 | 89.2 | 78.7 | 79.6 | 72.1 | 3.3 |
| PNS+ | MIR'22 | R2-50 | 88.7 | 90.2 | 80.6 | 81.3 | 72.8 | 3.0 |
| MAST | Arxiv'24 | PVTv2-B2 | 89.2 | 94.2 | 83.2 | 85.3 | 76.7 | 2.6 |
| SALI | MICCAI'24 | PVTv2-B2 | 86.8 | 90.9 | 79.9 | 81.0 | 72.6 | 3.4 |
| SALI | MICCAI'24 | PVTv2-B5 | 86.6 | 91.0 | 79.7 | 81.0 | 72.9 | 3.8 |
| STDDNet | ICCV'25 | R2-50 | 91.3 | 95.2 | 86.9 | 88.1 | 81.0 | 2.3 |
| STDDNet | ICCV'25 | PVTv2-B2 | 91.1 | 95.0 | 86.0 | 87.8 | 80.6 | 2.8 |
| Ours | - | R2-50 | 92.7 | 96.1 | 87.1 | 89.8 | 83.0 | 1.8 |
| Ours | - | PVTv2-B2 | 92.3 | 95.6 | 87.1 | 88.9 | 82.1 | 1.9 |
| Method | Publication | Backbone | Sα↑ | Eφ↑ | Fβ↑ | Dice↑ | IoU↑ | MAE↓ |
|---|---|---|---|---|---|---|---|---|
| COSNet | TPAMI'19 | - | 67.0 | 62.7 | 44.3 | 43.8 | 35.3 | 7.0 |
| PCSA | AAAI'20 | - | 68.2 | 66.0 | 44.2 | 45.0 | 35.1 | 8.0 |
| 2/3D | MICCAI'20 | - | 78.6 | 77.5 | 63.4 | 64.4 | 55.8 | 4.4 |
| PraNet | MICCAI'20 | R2-50 | 78.7 | 80.2 | 66.7 | 67.5 | 58.7 | 5.3 |
| ACSNet | MICCAI'20 | R-34 | 76.2 | 77.6 | 61.0 | 62.4 | 54.7 | 5.3 |
| SANet | MICCAI'21 | R2-50 | 75.3 | 73.6 | 59.0 | 59.5 | 52.7 | 5.5 |
| SEPNet | TCSVT'24 | PVTv2-B2 | 84.7 | 89.5 | 74.5 | 77.4 | 68.4 | 3.9 |
| PNS | MICCAI'21 | R2-50 | 76.7 | 75.5 | 60.9 | 61.5 | 53.9 | 5.0 |
| PNS+ | MIR'22 | R2-50 | 79.8 | 79.3 | 65.4 | 66.1 | 57.1 | 5.0 |
| MAST | Arxiv'24 | PVTv2-B2 | 85.6 | 91.3 | 77.2 | 79.9 | 70.8 | 3.1 |
| SALI | MICCAI'24 | PVTv2-B2 | 72.8 | 75.9 | 56.9 | 57.9 | 48.7 | 6.8 |
| SALI | MICCAI'24 | PVTv2-B5 | 76.5 | 81.3 | 62.0 | 63.6 | 54.7 | 5.7 |
| STDDNet | ICCV'25 | R2-50 | 83.4 | 85.6 | 74.1 | 75.0 | 67.3 | 3.7 |
| STDDNet | ICCV'25 | PVTv2-B2 | 86.3 | 90.2 | 78.1 | 80.2 | 72.2 | 3.5 |
| Ours | - | R2-50 | 85.1 | 89.8 | 75.0 | 78.0 | 69.5 | 3.6 |
| Ours | - | PVTv2-B2 | 87.3 | 91.0 | 79.6 | 81.3 | 73.7 | 2.9 |
All scores are in percentage (%).
We provide pretrained backbone weights, CMSA-Net model checkpoints, and predicted maps for reproducibility and comparison.
| Resource | PVT-V2-B2 | Res2Net50 |
|---|---|---|
| Backbone weights | Download | Download |
| Model checkpoints | Download | Download |
| Predicted maps | Download | Download |
| Resource | PVT-V2-B2 | Res2Net50 |
|---|---|---|
| Backbone weights | Download | Download |
| Model checkpoints | Download | Download |
| Predicted maps | Download | Download |
Following are the steps to set up the environment and use the project code.
We recommend using Anaconda to manage the environment. The code has been tested with Python 3.10, CUDA 11.8, PyTorch 2.1.1, and Torchvision 0.16.1.
- Create and activate the Conda environment:
conda create -n CMSANet python=3.10
conda activate CMSANet- Install PyTorch and Torchvision with CUDA 11.8:
pip install torch==2.1.1+cu118 torchvision==0.16.1+cu118 --extra-index-url https://download.pytorch.org/whl/cu118The corresponding wheel files are:
torch-2.1.1+cu118-cp310-cp310-linux_x86_64.whl
torchvision-0.16.1+cu118-cp310-cp310-linux_x86_64.whl
- Install other dependencies:
pip install numpy==1.26.0 \
timm==0.5.4 \
opencv-python==4.7.0.72 \
scipy \
tqdm \
scikit-learn \
tensorboard \
six \
PillowDownload the SUN-SEG dataset and place it under the project data directory, for example:
cmsa_project/data/SUN-SEG/
Download pretrained backbone weights and CMSA-Net checkpoints from the links above. By default, the scripts expect checkpoints under:
cmsa_project/snapshot/
To train CMSA-Net, run:
python cmsa_project/scripts/my_train.pyBefore training, please check cmsa_project/scripts/config.py and adjust dataset paths, GPU id, batch size, and output directory as needed.
To test CMSA-Net with a pretrained checkpoint, run:
python cmsa_project/scripts/my_test.pyBy default, the testing script uses:
cmsa_project/snapshot/pvtv2b2/ckpt_epoch_005.pth
You can edit model_path and save_dir in cmsa_project/scripts/my_test.py to evaluate different checkpoints or save predictions to a different folder.
To evaluate prediction maps, run:
python cmsa_project/eval/vps_evaluator.pyPlease make sure the ground-truth path and prediction path are correctly configured before running evaluation.
The main structure of the project is as follows.
- README.md
- cmsa_project
- scripts
- config.py
- my_train.py
- my_test.py
- lib
- backbone
- CompressEncoder.py
- Decoder.py
- LightRFB.py
- Res2Net_v1b.py
- pvt_v2.py
- dataloader
- __init__.py
- dataloader.py
- preprocess.py
- statistics.pth
- model_arch
- cmsa_net.py
- misc_blocks.py
- utils
- __init__.py
- utils.py
- eval
- dice_score.py
- metrics.py
- vps_evaluator.py
- snapshot
- pvtv2b2
- r250If you find this work useful, please consider citing:
@article{wang2026cmsanet,
title={CMSA-Net: Causal Multi-scale Aggregation with Adaptive Multi-source Reference for Video Polyp Segmentation},
author={Tong Wang and Yaolei Qi and Siwen Wang and Imran Razzak and Guanyu Yang and Yutong Xie},
journal={arXiv preprint arXiv:2602.22821},
year={2026}
}