Dynamic Resolution Routing for Efficient Egocentric Grounding

arXiv  ·  Installation  ·  Quick Start  ·  Training  ·  Evaluation

---

Introduction

Motivation. Egocentric grounding localizes target objects from natural-language queries in first-person video. Targets are often small and observed under rapid viewpoint changes, so high-resolution inputs are essential for preserving the fine-grained details required for perception. That makes scaling MLLMs prohibitively expensive: a 3780×1920 frame can yield 9.3k visual tokens, and visual encoding consumes up to 66.5% of the end-to-end inference budget.

Architecture. SmartRes shifts efficiency optimization from post-hoc token pruning to proactive pixel-space selection. A low-resolution branch provides spatial guidance, enabling a lightweight router to activate high-resolution patches only in object-centric regions.

Release

• Environment: LLaMA-Factory fork with the SmartRes integration
• Training: router trained jointly with the backbone
• Inference: EgoIntention context and uncommon splits
• Checkpoint: SmartRes-Lite
• Analysis: per-scale accuracy
• Comparisons: down-scaling, FastV
• Data: annotations at 100% and 10% → 50%, and a script for other budgets

Installation

Step 1. Environment and package.

git clone --recursive https://github.com/HuixinSun/SmartRes.git && cd SmartRes
conda create -n smartres python=3.10 -y && conda activate smartres

pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
pip install -r env/requirements.txt
pip install -e .

Step 2. Training and generation loop. Provided by our LLaMA-Factory fork, included as a submodule:

git submodule update --init          # only if you cloned without --recursive
pip install -e third_party/LLaMA-Factory -c env/constraints.txt

bash env/setup.sh smartres runs both steps in one go.

Quick Start

Step 1. Unpack the checkpoint.

sha256sum -c checkpoints/smartres-lite.tar.gz.sha256
tar -xzf checkpoints/smartres-lite.tar.gz -C checkpoints/

Step 2. Install the router onto a Qwen2.5-VL model.

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

from smartres import install_smartres
from smartres.preprocess import build_dual_resolution

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", torch_dtype=torch.bfloat16, device_map="cuda"
)
model = PeftModel.from_pretrained(model, "checkpoints/smartres-lite")
install_smartres(model, tau=0.5, router_layer=30, encode_snap="window")

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
views = build_dual_resolution(Image.open("frame.jpg"), processor.image_processor, hr_scale=0.2)

Use the same tau the adapter was trained with.

Data

Frames. EgoIntention uses the Ego4D split of PACO. Point the images field of the JSONs at your copy.

Labels. Boxes are stored in the coordinate frame of the resolution they were rendered at, so the frame is part of the setting. 10to50 means the image is stored at 50% of the original token budget while the boxes live in the 10% coordinate frame.

Setting	Files
100%	mllm_rec_egoint.json, egointention_{context,uncommon}_test.json
10% → 50% (Lite)	mllm_rec_egoint_10to50.json, egointention_{context,uncommon}_test_10to50.json

Tools. Build a label set at another budget with:

python tools/prepare_labels.py --input data/egointention_context_test.json \
    --output data/egointention_context_test_10to50.json \
    --image-ratio 0.5 --label-ratio 0.2 --write-images /path/to/frames_50pct

--image-ratio resizes relative to the original; --label-ratio picks the box coordinate frame relative to that resized image.

Training

bash scripts/train.sh                       # SmartRes-Lite
bash scripts/train.sh --hr-budget 1.00      # SmartRes-Pro

Configs. Set in configs/qwen2_5vl_3b_lora_sft_egoint_lite.yaml; the matching flag overrides it.

use_smartres: true
tau: 0.5             # routing threshold, M = STE(S > tau)
router_layer: 30     # vision block the router reads
encode_snap: window  # encode-set granularity: window | unit
lr_budget: 0.10      # r_LR, low-resolution token budget
hr_budget: 0.50      # r_HR, high-resolution token budget
lambda_route: 0.01   # weight of the routing BCE term
lambda_hinge: 0.05   # weight of the margin regulariser

Use the same encode_snap for training and evaluation.

Evaluation

bash scripts/eval.sh context     # also: uncommon

Configs. Set in configs/qwen2_5vl_3b_lora_predict_egoint_lite.yaml; use the same values the checkpoint was trained with.

use_smartres: true
tau: 0.5                        # routing threshold, M = STE(S > tau)
router_layer: 30                # vision block the router reads
encode_snap: window             # encode-set granularity: window | unit
lr_budget: 0.10                 # r_LR, low-resolution token budget
hr_budget: 0.50                 # r_HR, high-resolution token budget
per_device_eval_batch_size: 1   # must stay 1

Analysis

Scores the boxes written by scripts/eval.sh.

python tools/score_per_scale.py \
    --predictions outputs/eval_context/generated_predictions.jsonl \
    --dataset data/egointention_context_test_10to50.json

Reports P@0.3, P@0.5 and mIoU, overall and per object scale. Objects are grouped by relative box area S into small (S<0.005), medium (0.005≤S<0.05) and large (S≥0.05), reported as P_s, P_m and P_l.

Comparisons

python comparisons/downscale.py --input data/egointention_context_test.json \
    --output data/egointention_context_test_32pct.json --ratio 0.32

Qualitative comparison. FastV's (b) pruning mask and (c) prediction against SmartRes' (d) routing mask and (e) prediction, with IoU on each.

Citation

@article{sun2026smartres,
  title   = {Dynamic Resolution Routing for Efficient Egocentric Grounding},
  author  = {Sun, Huixin and Zhao, Wangbo and Wei, Fanyue and Lin, Qiuxia and
             Sun, Pengzhan and Yao, Angela},
  journal = {arXiv preprint arXiv:2608.01638},
  year    = {2026}
}

Acknowledgements

Built on Qwen2.5-VL, LLaMA-Factory and FastV. Evaluation data comes from PACO and Ego4D.

License

Apache 2.0. See LICENSE.