Seeing Through Touch:
Tactile-Driven Visual Localization of Material Regions

The official Pytorch implementation for "Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions", CVPR 2026
Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak

   

1. Environment

# Create and activate conda environment
conda create -n stt python=3.10 -y
conda activate stt

# Install PyTorch
# The command below uses CUDA 11.8. Adjust the version according to your CUDA driver.
pip install torch==2.7.1+cu118 torchvision==0.22.1+cu118 torchaudio==2.7.1+cu118 --index-url https://download.pytorch.org/whl/cu118

# Install dependencies
# --upgrade-strategy only-if-needed prevents other packages from overriding the installed PyTorch version.
pip install -r requirements.txt --upgrade-strategy only-if-needed

# Install KNN_CUDA
pip install https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl

2. Data Preparation

For dataset preparation and directory structure, please refer to datasets/README.md.

3. Evaluation

Checkpoint

Download the model checkpoints from the links and place them in the checkpoints/ directory.

Model	Checkpoint
STT-Local	STT-Local.pth
STT-Indomain	STT-Indomain.pth
SeeingThroughTouch (STT)	STT.pth

TG-test

To evaluate on Touch and Go dataset, run:

bash shell/test_TG.sh

WebMaterial-test

To evaluate on WebMaterial dataset, run:

bash shell/test_WebMaterial.sh

OpenSurfaces-test

To evaluate on OpenSurfaces dataset, run:

bash shell/test_OS.sh

Interactive Localization

To evaluate on Interative Localization test set of WebMaterial, run:

bash shell/test_IIoU.sh

4. Training

Download dinov3_vits16 pretrained model onto your working server.
To train the Seeing Through Touch model, run:

bash shell/train_STT.sh

5. Citation

If you find this work useful, please cite it as:

@inproceedings{kim2026seeingthroughtouch,
  author    = {Seongyu Kim and Seungwoo Lee and Hyeonggon Ryu and Joon Son Chung and Arda Senocak},
  title     = {Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
}

6. Acknowledgments

This codebase is built upon TVL (ICML2024). We thank the authors for their excellent work.