[Zhehao-Hu]

[5/8] Day-0 speculative-decoding readiness flow. Merge in order 1 → 8. Base: [4/8] #2932. New files only, so it can be reviewed in parallel with [1/8]–[4/8].
Series: [1/8] #2839 · [2/8] #2930 · [3/8] #2931 · [4/8] #2932 · [5/8] #2933 (this) · [6/8] #2934 · [7/8] #2935 · [8/8] #2936

What does this PR do?

Type of change: New feature

A drafter's speedup and correctness need the target alone (arm A) and the target plus drafter (arm B) measured on the same node, image, prompts and sampling. Run as separate jobs, node, image and load differences leak into the speedup. An earlier M0 run did exactly that: its two arms ran in separate jobs, arm A happened to be slow, and the C=32 speedup came out as 1.39. Matched in one job, it measures 1.27.

This PR adds a matched A|B runner and its comparator.

• examples/specdec_bench/compare_ab.py summarizes run.py outputs laid out as [r<repeat>/]{A,B}_c<N>/ into ab_summary.json. Per row (repeat × concurrency) it records:

  • Output TPS of both arms, speedup, acceptance length (AL), step time and TTFT;
  • the temperature, and result-entry and turn counts;
  • each arm's algorithm, drafter, speculative-token count, request count and engine version, from configuration.json.

  A-vs-B greedy exact match (all turns and first turns, plus unpaired turns) is reported only at concurrency 1 and temperature 0. The same row's a_self_match compares arm A with arm A at the next concurrency: how far the target alone drifts with batch shape, the noise floor for the A-vs-B rate. The temperature follows run.py's precedence: --temperature, then runtime_params sampling_kwargs, else the engine default. A row lists problems (missing arm, other than one result entry, no responses, turn-count mismatch), and any problem makes the CLI exit 1.

• tools/launcher/common/specdec_bench/ab_compare.sh runs both arms per repeat and concurrency in one job:

  • A first on odd repeats and B first on even ones, so run order doesn't favor an arm;
  • refuses a non-empty SAVE_ROOT (run.py appends to existing results), and a local DRAFT_MODEL path that doesn't exist (HF repo IDs pass through);
  • records the host and GPUs;
  • fails on a requirements install error, at the first failed arm, and when the comparator finds a problem.

• examples/Qwen/Qwen3-8B/specdec_ab_dflash2.yaml: MT-Bench A|B sweep at concurrency 1/8/32 on vLLM v0.31.0, with draft_model passed per run.

Usage

uv run launch.py --yaml examples/Qwen/Qwen3-8B/specdec_ab_dflash2.yaml \
    pipeline.global_vars.draft_model=/cicd/drafts/Qwen3-8B/dflash2/<run_id> \
    pipeline.global_vars.output_dir=/scratchspace/ab --yes
# REPEATS / CONCURRENCIES / NUM_SPEC_TOKENS are task environment entries:
#   'pipeline.task_0.environment+=[{"REPEATS":"2"}]'

python examples/specdec_bench/compare_ab.py <save_root> --json <save_root>/ab_summary.json

Testing

• tests/examples/specdec_bench/test_compare_ab.py: speedup, match and a_self_match reported only at concurrency 1 and temperature 0; no match under sampling; the CLI exit code reflects each problem type. 11 passed (pytest tests/examples/specdec_bench/test_compare_ab.py, CPU only). Launcher suite on this commit: 192 passed.
• GPU (OCI-JHB, GB300, vLLM v0.31.0, TP 1, MT-Bench 80 × 2 turns, temperature 0, 7 speculative tokens, C = 1/8/32), 7 jobs on at least 4 nodes:
  • Every job wrote ab_summary.json with no problem rows and ended COMPLETED. The four jobs whose exit we timed ended 7–13 s after writing the summary.
  • Measurement noise (one drafter, 2 nodes × 2 repeats): AL CV ≤ 0.4%. Speedup CV is 0.2% at C=1 and 2–3% at C=8 and C=32. Means are 1.42 at C=1, 1.39 at C=8 and 1.27 at C=32.
  • Greedy exact match is 0.956–0.963 over all turns and 1.0 on first turns, in every run.
• Why a_self_match: on Qwen3.8-27B (hybrid GDN + full attention) A-vs-B matched on only 21% of first turns, which looked like a speculative-decoding defect. Computed offline from that job's responses, the target alone matched itself across C=1/8/32 on 20–22% of first turns: the target's batch-variant numerics, not the drafter.
• Not run: the TRT-LLM and SGLang engines through this runner.

Before your PR is "Ready for review"

Make sure you read and follow Contributor guidelines and your commits are signed (git commit -s -S).

Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded trust_remote_code=True, torch.load(..., weights_only=False), pickle, etc.).

• Is this change backward compatible?: ✅ (new files only)
• If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A (stdlib only)
• Did you write any new necessary tests?: ✅
• Did you update Changelog?: ✅ (Speculative Decoding entry)
• Did you get Claude approval on this PR?: ❌ - not yet run

Additional Information

• The specdec-release skill's gate_bench.py ([7/8]) decides whether an ab_summary.json is a valid matched measurement. This PR only measures and flags structural problems.
• At C=8 and C=32 a single run can differ from the mean by up to 6%. Use ≥ 2 repeats on ≥ 2 nodes before reading a throughput difference of under ~5%.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}