NanoVLM SpeedRun Basic Ruleset Spec

1. Motivation

The goal of this spec is to evaluate training efficiency in compact vision-language models.

This benchmark measures methods by the shortest wall clock time required to reach a target level of multimodal performance under a shared recipe and dataset setting.

This benchmark answers the question:

Given a fixed NanoVLM recipe and fixed training data, what is the shortest wall clock time needed to train a compact VLM to a useful evaluation target?

The main task in this initial version is image-text understanding and instruction following with a minimalist VLM pipeline.

Note:

• The exact benchmark set may evolve with community discussion.
• The initial version focuses on a practical and reproducible baseline rather than a maximally scaled setup.

---

2. Core Metric

Primary Leaderboard

Each method must report a single leaderboard entry that includes both benchmark scores and training compute.

For the initial NanoVLM release, the leaderboard should contain:

Method	Train Time	Total FLOPs	MME	MMMU_Val	MMStar	GQA	ChartQA	DocVQA	OCRBench	POPE
NanoVLM Baseline	23.75 GPU hours (8 x H100)	335.02 PFLOPS	1204.46 (948.75/255.71)	0.3022	0.3273	0.4184	0.1084	0.1018	0.165	0.7724

Reported benchmark metrics include:

• MME (Perception/Cognition)
• MMBench (EN Dev)
• MMMU_Val
• MMStar (Average)
• GQA (Exact Match)
• ChartQA (Relaxed Overall)
• DocVQA (ANLS)
• OCRBench
• POPE (Accuracy)

---

3. Dataset and Task

Item	Specification
Stage 1 data	LMMs-Lab-Speedrun/Data_NanoVLM/Stage1-LLaVA-Pretrain
Stage 2 data	LMMs-Lab-Speedrun/Data_NanoVLM/Stage2-LLaVA-NeXT-Data

The initial task is to train a compact VLM with the released two-stage recipe:

• Stage 1: projector-only alignment between vision and language
• Stage 2: instruction tuning on the released multimodal data recipe

Note:

• Local YAML path configuration is allowed, but the underlying dataset content and split policy should remain unchanged.
• Any major dataset change should be proposed as a separate track rather than silently changing the baseline.

---

4. Model Scaling Policy

Baseline Recipe

The initial NanoVLM SpeedRun baseline uses:

• LLM: Qwen/Qwen3-0.6B
• Vision Encoder: google/siglip2-so400m-patch16-naflex
• Projector: LLaVA-style 2-layer MLP
• Training Paradigm: 2-stage training

Model Changes

• The baseline recipe is the default comparison point
• Participants may propose architecture or training changes
• Any deviation from the baseline recipe must be clearly reported
• Total trainable parameter count must be reported

Note:

• Future versions may split submissions into a strict baseline track and a more open innovation track.

---

5. Training Protocol

Training Axis Definition

Permitted:

• Any optimizer
• Any precision
• Any batch size
• Any distributed training strategy
• Systems optimization that does not change the underlying task definition

Prohibited:

• Additional supervised multimodal data outside the released NanoVLM data recipe
• External pretrained VLM checkpoints beyond the released initialization path
• Evaluation-time finetuning
• Changing the benchmark task into a different problem setting while claiming the same submission track

Allowed initialization:

• The released NanoVLM_Init initialization for Stage 1
• The baseline pretrained language and vision backbones specified in the recipe

---

6. Evaluation Protocol

Submissions should evaluate the merged Stage 2 checkpoint using the released evaluation flow.

At minimum, each submission should report:

1. Final MME score
2. Total wall clock training time
3. Stage 1 and Stage 2 time breakdown
4. Hardware used

Optional but encouraged:

• Intermediate evaluations over time
• Additional benchmark results
• Energy, CO2, and GPU-hour statistics

---

7. Required Plots and Artifacts

Each submission must include:

1. Wall clock time
2. Model architecture changes relative to baseline
3. Total parameter count
4. Evaluation command or evaluation logs
5. Final benchmark scores

Optional but encouraged:

• Checkpoint merge details
• Training loss curves
• Number of samples seen
• WandB or other experiment tracking links
• Example qualitative outputs or failure cases

---

8. Reference Implementation

The current reference implementation and training walkthrough live in:

• NanoVLM training guide

This ruleset is intended to align with that released recipe while leaving room for future track refinement through community discussion.