Skip to content

[4/8] launcher: clean up the Qwen3-8B DFlash2 launcher examples - #2932

Draft
Zhehao-Hu wants to merge 2 commits into
zhehaoh/launcher-dflash-reexportfrom
zhehaoh/launcher-dflash2-examples
Draft

Zhehao-Hu wants to merge 2 commits into
zhehaoh/launcher-dflash-reexportfrom
zhehaoh/launcher-dflash2-examples

Conversation

@Zhehao-Hu

@Zhehao-Hu Zhehao-Hu commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

[4/8] Day-0 speculative-decoding readiness flow. Merge in order 1 → 8. Base: [3/8] #2931. Uses latest_checkpoint and SMOKE_JSON from [2/8]; edits hf_online_dflash2.yaml after [1/8].
Series: [1/8] #2839 · [2/8] #2930 · [3/8] #2931 · [4/8] #2932 (this) · [5/8] #2933 · [6/8] #2934 · [7/8] #2935 · [8/8] #2936

What does this PR do?

Type of change: New example

Cleans up the two Qwen3-8B DFlash2 launcher examples so one submission trains, exports and serving-checks a drafter, and every artifact path moves with one override.

hf_online_dflash2.yaml:

  • Adds task_2, a vLLM smoke test of the newest export (DRAFT_CKPT_DIR, final preferred), gated at MIN_ACCEPTANCE_LENGTH 1.5 with 7 speculative tokens. A healthy 2000-step run of this config clears it (1.68–1.79 measured).
  • Runs task_0 and task_1 on the same TRT-LLM image (1.3.0rc20). task_0 was pinned to rc10 while task_1 took the factory default.
  • Drops dflash_loss_decay_factor: the recipe's dpace objective ignores it, as the training log warns.

Both examples:

  • Route every artifact path through global_vars.output_dir (training output, exports, SMOKE_JSON). One override moves producer and consumer together, e.g. to rerun the smoke task against an earlier experiment.
  • Pin vllm/vllm-openai:v0.31.0 (multi-arch; contains vllm#52816 for DFlash2) instead of latest/nightly.
  • Set TRITON_CACHE_DIR node-local, so parallel jobs on different nodes never share a Lustre cache folder.
  • hf_streaming_dflash2.yaml exports to <output_dir>/export via EXPORT_PATH.
  • Train on the block vLLM serves: dflash_block_size=8 (was 16, the recipe's value). vLLM serves DFlash2 with a block of 1 + NUM_SPEC_TOKENS = 8 and ignores the checkpoint's block_size, so the examples trained on a block they never serve. Block 8 also matches the library default and the published DFlash2 drafters.

Usage

uv run launch.py --yaml examples/Qwen/Qwen3-8B/hf_online_dflash2.yaml --yes
# rerun only the smoke test against an earlier experiment's export
uv run launch.py --yaml examples/Qwen/Qwen3-8B/hf_online_dflash2.yaml \
    pipeline.global_vars.output_dir=/cicd/<experiment_id>/dflash2_bs16 ... --yes

Testing

  • uv run launch.py --yaml … --to-yaml resolves both examples. Launcher unit suite on this commit: 191 passed.
  • GPU (OCI-JHB, GB300, 4 GPUs per node): five runs of the full three-task pipeline (data → train + export → smoke) completed with no manual step. They used gpus_per_node=4 and training.gradient_accumulation_steps=2 to keep the global batch at 8. Each took about 40 minutes end to end, plus queue time. Results: loss 6.37–6.71, accuracy 0.167–0.177, smoke AL 1.68–1.79.
  • Block-size ablation (same cluster, dev budget, same data): the online example at block 8 against block 16. Bench AL 1.751/1.746/1.747 vs 1.753/1.759/1.758 at C=1/8/32; speedup 1.40/1.41/1.18 vs 1.41/1.39/1.33 (one repeat; the C=32 gap comes from both arms' TPS moving); greedy exact match 0.956 vs 0.963. Loss and accuracy aren't comparable across block sizes (block 8 averages over fewer, easier positions).
  • Not run: hf_streaming_dflash2.yaml on GPU (path changes only, checked with --to-yaml).

Before your PR is "Ready for review"

Make sure you read and follow Contributor guidelines and your commits are signed (git commit -s -S).

Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded trust_remote_code=True, torch.load(..., weights_only=False), pickle, etc.).

  • Is this change backward compatible?: ✅ (example configs only. Training output paths are unchanged; the streaming example's export moves from /scratchspace/export to /scratchspace/dflash2/export.)
  • If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A
  • Did you write any new necessary tests?: N/A (example configs)
  • Did you update Changelog?: N/A
  • Did you get Claude approval on this PR?: ❌ - not yet run

Additional Information

  • Clusters that only accept whole nodes (e.g. GB300 with 4 GPUs) reject the 1-GPU tasks as written. The speculative-decoding skill's configure stage ([8/8]) covers fitting gpus_per_node and the accumulation steps.

@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from 3ac8822 to 9a9b3cc Compare October 22, 2026 17:59
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash2-examples branch 2 times, most recently from 2370f99 to efc7bce Compare October 22, 2026 22:09
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch 2 times, most recently from 390e821 to a2d6478 Compare October 23, 2026 00:09
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash2-examples branch from efc7bce to 82d5b56 Compare October 23, 2026 00:09
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from a2d6478 to dbddd39 Compare October 23, 2026 04:51
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash2-examples branch from 82d5b56 to 8a2bd57 Compare October 23, 2026 04:51
Zhehao-Hu added a commit that referenced this pull request Oct 23, 2026
…strict (#2839)

> **[1/8] Day-0 speculative-decoding readiness flow.** Merge in order 1
→ 8. Base: `main`.
> Series: [1/8] #2839 (this) · [2/8] #2930 · [3/8] #2931 · [4/8] #2932 ·
[5/8] #2933 · [6/8] #2934 · [7/8] #2935 · [8/8] #2936

### What does this PR do?

Type of change: Bug fix


[common/check_regression.py](https://github.com/NVIDIA/Model-Optimizer/blob/main/tools/launcher/common/check_regression.py)
gates DFlash-family training runs on final loss and accuracy. Four
things stopped that gate from catching a bad run:

1. **It read the wrong checkpoint on long runs.** `find_trainer_state()`
sorted `checkpoint-*` as strings, so `checkpoint-8000` ranked after
`checkpoint-12000` and an older checkpoint's metrics were checked. Any
run whose saved steps differ in digit count was affected; the ~92K-step
full DFlash2 recipe in `hf_online_dflash2.yaml` saves every 4000 steps.
It now sorts by step number.
2. **An unreadable run passed.** With thresholds set, a missing
`trainer_state.json`, or one with no training logs, only warned and
exited 0. An empty threshold variable (`MAX_FINAL_LOSS=""`) crashed on
`float('')`.
3. **A regression never failed the job.** `dflash_online_training.sh`
runs the check as `check_regression.py … || true`, so Slurm reports
`COMPLETED` even when the log says `REGRESSION`. The verdict also
existed only as log text.
4. **A crashed run was still graded.** The script's `ERR` trap only
records a failure, so after a failed `accelerate launch` it went on to
export and run the check, and a checkpoint that survived the crash could
pass.

Changes:

- `check_regression.py`:
  - Sorts checkpoints by step number.
- Fails when thresholds are set but there are no metrics to check,
including a `trainer_state.json` of the wrong shape. An empty threshold
variable counts as unset. A threshold that isn't a finite number (`7,0`,
`nan`) fails as `METRICS_MISSING`.
- A NaN or infinite value for a metric that has a threshold fails as
`TRAIN_REGRESSION`.
- Takes `--json <path>` and writes `{pass, failure_class, trainer_state,
metrics, thresholds, messages}` as strict JSON (non-finite metrics as
`null`). `failure_class` is `null`, `TRAIN_REGRESSION` or
`METRICS_MISSING`. The verdict is serialized before the file is opened,
so a failure can't leave a truncated file.
- Its existing log lines (`=== Regression Check (…) ===`, `PASS:` /
`REGRESSION:`, `Regression check PASSED`) are unchanged, so existing log
readers keep working. A failure now also prints `Regression check
FAILED`.
- `dflash_online_training.sh` writes the verdict to
`${OUTPUT_DIR}/regression_check.json`, next to the exports.
`REGRESSION_STRICT=1` drops the `|| true`, so a regression fails the
task through the existing `ERR` trap and `exit_handler`. **The default
(`0` or unset) is unchanged (warn only)**, so the other examples with
possibly uncalibrated thresholds don't start failing. Any other value
fails the task at startup, before installing or training.
- `dflash_online_training.sh` also skips export and the check when
training fails, still reporting the failure like any `ERR`-trap failure,
and removes an old `regression_check.json` first thing, so no early exit
can leave an earlier run's verdict behind.
- `examples/Qwen/Qwen3-8B/hf_online_dflash2.yaml`:
  - Sets `REGRESSION_STRICT: "1"`.
- Recalibrates `MAX_FINAL_LOSS` from 5.0 to 7.0. The 5.0 was inherited
from `hf_online_dflash.yaml`, whose own reference curve only drops below
5.0 at step ~2500 of a 12.5K-step run. DFlash2 adds a selector loss term
and this example stops at 2000 steps, so 5.0 fails every healthy run.
`MIN_FINAL_ACC` stays at 0.15.
- The `speculative-decoding` skill's `dflash.md` and `dflash2.md`
describe the new gate: judge a run by `regression_check.json`, the two
failure classes, strict mode, and DFlash2's 7.0 threshold.

### Usage

```bash
MAX_FINAL_LOSS=7.0 MIN_FINAL_ACC=0.15 \
  python3 common/check_regression.py /scratchspace/dflash2_bs16 --json /scratchspace/dflash2_bs16/regression_check.json
echo $?   # 0 pass, 1 regression or nothing to check
```

```yaml
environment:
  - MAX_FINAL_LOSS: "7.0"
  - REGRESSION_STRICT: "1"   # fail the training task on a regression
```

### Testing

- `tools/launcher/tests/test_check_regression.py`: 4 new test functions
(13 cases). Run against `main`'s `check_regression.py`, 17 of the file's
18 cases fail:
- `test_latest_checkpoint_is_chosen_by_step_number` guards the
string-sort bug.
  - `test_empty_threshold_var_is_unset` guards the `float('')` crash.
- `test_cli_exit_code_and_json_verdict` is parametrized over a missing
`trainer_state.json`, a truncated one, two of the wrong shape, a NaN
loss, an unchecked infinite metric, a regression and a pass. Each case
checks the exit code and parses the verdict as strict JSON, so it covers
the missing-metrics hole and the `--json` contract.
- `test_invalid_threshold_writes_failed_verdict` covers `7,0`, `nan` and
`inf`.
- `tools/launcher/tests/test_dflash_online_training.py` (5 cases) runs
the real script in a sandbox: stubbed `pip`, `nvidia-smi`, `accelerate`,
exporter and `torch`, a checkpoint, and an old passing verdict. Each
case asserts the exact exit code. Against `main`'s script, all 5 fail:
- A failing launch leaves no verdict and exits 1; a succeeding one
grades the checkpoint and exits 0 with no `ERR`-trap error.
- With a checkpoint over the threshold, `REGRESSION_STRICT=1` exits 1
and `0` exits 0, both writing `TRAIN_REGRESSION`.
- `REGRESSION_STRICT=true` exits 1 without launching training and leaves
no verdict.
- Full launcher suite on this commit: `uv run --with pytest python3 -m
pytest tests/`, 183 passed.
- `dflash_online_training.sh`: `bash -n` passes.
- **GPU (OCI-JHB, GB300, global batch 8):** five 2000-step runs of this
example with this change passed the strict gate. Final loss was
6.37–6.71 and accuracy 0.167–0.177, against 7.0 / 0.15. The calibration
run that motivated 7.0 (before this change) finished at 6.51 / 0.181.
The seed-to-seed spread puts the loss margin at about 4%; see the
variance note under Additional Information.
- Not run: the strict-failure path and a real training crash on GPU
(covered by the sandbox test above).

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ (strict mode is opt-in; log
output unchanged)
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A
- Did you get Claude approval on this PR?: ❌ - not yet run

### Additional Information

- Part of the Day-0 speculative-decoding workflow: the `specdec-release`
skill ([8/8]) gates training on `regression_check.json`, so it needs a
verdict it can read and a job state that reflects it.
- **Why not reuse `day0-release/scripts/gate_run.py`?** The two sit at
different layers and check different things, so their schemas stay
independent on purpose:
- `gate_run.py` is an agent-side gate for an *evaluation* run. It reads
a run summary the agent builds from NEL output, and checks per-task
status, sample counts and judge errors. None of that exists for a
training run.
- `check_regression.py` runs *inside* the Slurm training task, on rank
0, and its verdict decides the task's exit status in strict mode. It
reads `trainer_state.json` against thresholds from the launcher YAML,
and has to work for launcher users who never load a skill. A launcher
script importing a skill script would also invert the dependency: skills
drive the launcher, not the other way round.
- The skill-side counterpart of `gate_run.py` for training is
`specdec-release/scripts/gate_train.py` ([6/8] #2934). It reads
`regression_check.json` and returns the day-0 gate shape (`{pass,
failure_class, detail, …}`), with `failure_class` from the shared triage
vocabulary. That is the adapter layer, so `check_regression.py` doesn't
need to match `gate_run.py`.
- What the two share is the convention: a boolean `pass`, and a
`failure_class` that is `null` on a pass. The launcher emits only
`METRICS_MISSING` and `TRAIN_REGRESSION`, and `gate_train.py` treats any
other value as `TRAIN_REGRESSION`, so the launcher can add detail fields
without breaking the gate.
- Other Qwen3-8B examples share the inherited `MAX_FINAL_LOSS: "5.0"`
(`hf_online_domino.yaml`, `hf_online_lilicorr.yaml`). They are left
unchanged and non-strict because there is no calibration run for them.
- Variance: a seed-1 retrain landed at loss 6.705, 4.2% under the 7.0
floor. Accuracy has more headroom (11%). Recalibrating from more seeds
is planned for the release gates, not this PR.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Training runs with an output directory now produce a regression-check
verdict file reporting whether configured quality thresholds passed.
* Regression checks account for missing or invalid metrics and select
checkpoints by numeric step.

* **Bug Fixes**
* Failed training no longer proceeds to checkpoint export or regression
checks.
* Strict mode fails the task when regression checks fail; invalid
strict-mode values stop the run before training starts.

* **Documentation**
* Updated DFlash2 guidance and example settings for the final-loss
threshold and strict-mode behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from dbddd39 to 6d46dab Compare October 23, 2026 21:24
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash2-examples branch from 8a2bd57 to 5b20c8d Compare October 23, 2026 21:24
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from 6d46dab to d217cd2 Compare October 23, 2026 21:35
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash2-examples branch from 5b20c8d to 406058d Compare October 23, 2026 21:35
Zhehao-Hu added a commit that referenced this pull request Oct 23, 2026
…ction (#2930)

> **[2/8] Day-0 speculative-decoding readiness flow.** Merge in order 1
→ 8. Base: `main` ([1/8] has merged).
> Series: [1/8] #2839 (merged) · [2/8] #2930 (this) · [3/8] #2931 ·
[4/8] #2932 · [5/8] #2933 · [6/8] #2934 · [7/8] #2935 · [8/8] #2936

### What does this PR do?

Type of change: Bug fix

The vLLM smoke test (`common/specdec/vllm_smoke_test.sh`) is the serving
check after draft training. It had six problems:

1. **It could serve the wrong checkpoint.** `ar_eval_mtbench.sh` and
`vllm_smoke_test.sh` picked the newest checkpoint by sorting on a
dash-separated field of the **whole path**, so a dash in a parent
directory (e.g. `/data/my-drafts/...`) picked the wrong one.
`quantize_drafter.sh` had a third, different sort. The smoke test also
ignored `exported-checkpoint-final`.
2. **Every throughput figure printed `?`.** Timing used `bc`, which the
vLLM image doesn't have.
3. **A drafter that never speculated passed.** With
`MIN_ACCEPTANCE_LENGTH` set, a vLLM log with no acceptance length only
warned.
4. **The acceptance length covered only the last few seconds.** It came
from the log's "Mean acceptance length" line, which covers only vLLM's
last logging interval.
5. **The verdict existed only as log text.**
6. **The server wait was fixed at 180 s.** Qwen3.8-27B with a DFlash2
drafter takes longer to start on GB300 (torch.compile, kernel JIT, CUDA
graph capture, GDN warmups), so every smoke attempt stopped at
`server_timeout`. The script's exit then tore the container down
mid-warmup, and the engine's next exec failed with a ptxas ENOENT that
read like the cause.

Changes:

- `service_utils.sh`: new `latest_checkpoint <dir> <prefix>`. It returns
`<dir>/<prefix>final` if present, else the highest numeric step, keyed
on the directory's own name. It replaces the three ad hoc sorts in
`ar_eval_mtbench.sh`, `vllm_smoke_test.sh` and `quantize_drafter.sh`.
- `vllm_smoke_test.sh`:
- Prefers `exported-checkpoint-final`, as `quantize_drafter.sh` already
did.
  - Times prompts with shell arithmetic and `python3` instead of `bc`.
- Computes the mean acceptance length over the whole run from vLLM's
cumulative `/metrics` counters: 1 +
`spec_decode_num_accepted_tokens_total` /
`spec_decode_num_drafts_total`.
- With `MIN_ACCEPTANCE_LENGTH` set, fails when no speculative drafts
were recorded.
- Fails loudly if `service_utils.sh` is missing (it used to be optional,
so the failure surfaced as a misleading `draft_missing`). Only a missing
file is fatal: `source` returns the status of the file's last command, a
rank-0 append that fails outside a launcher working directory.
- `SERVER_TIMEOUT` sets the server wait in seconds (default 180) and
appears in the timeout reason.
- Checks `MIN_ACCEPTANCE_LENGTH` (a finite number), and
`NUM_SPEC_TOKENS` and `SERVER_TIMEOUT` (positive integers without
leading zeros, since `NUM_SPEC_TOKENS` goes into vLLM's JSON config),
before any work, and fails with `stage: invalid_config` and a reason
naming the value. Both checks run after the `EXIT` trap is set, so even
these failures replace any earlier `SMOKE_JSON`.
- Optional `SMOKE_JSON=<path>` writes the verdict on every exit:
`{stage, pass, reason, method, target, draft, num_speculative_tokens,
prompts passed/failed, tokens, average tok/s, mean_acceptance_length,
min_acceptance_length}`. `stage` says how far it got (`invalid_config`,
`starting`, `draft_missing`, `server_died`, `server_timeout`, `serving`,
`completed`), so a downstream gate can tell "not servable" from "below
the floor". The file's parent directory is created if needed, and it is
strict JSON (a non-numeric value is written as `null`). The mean
acceptance length is checked against the floor and written at full
precision; only the log rounds it.

### Usage

```yaml
task_2:
  script: common/specdec/vllm_smoke_test.sh
  environment:
    - HF_MODEL_CKPT: <<global_vars.hf_model>>
    - DRAFT_CKPT_DIR: <<global_vars.output_dir>>      # newest export, final preferred
    - SPEC_METHOD: "dflash"
    - NUM_SPEC_TOKENS: "7"
    - MIN_ACCEPTANCE_LENGTH: "1.5"
    - SMOKE_JSON: <<global_vars.output_dir>>/smoke_check.json
    - SERVER_TIMEOUT: "900"                            # optional; default 180 s
```

### Testing

- `tools/launcher/tests/test_service_utils.py`: `latest_checkpoint` with
a dash in a parent directory, `final` winning over steps, numeric rather
than string order, and no checkpoint at all (3 cases).
- `tools/launcher/tests/test_vllm_smoke_test.py` runs the script as rank
0 from a directory with no `modules/` (stubbed `pip` and `nvidia-smi`),
up to the paths that stop before vLLM starts (6 cases). Against the
script without these fixes, all 6 fail:
  - It reaches `draft_missing` rather than stopping at `source`.
- `MIN_ACCEPTANCE_LENGTH` of `1,5` or `nan`, `NUM_SPEC_TOKENS` of `x` or
`08` and `SERVER_TIMEOUT=15m` each fail as `invalid_config` with a
strict-JSON `SMOKE_JSON`. Before, `NUM_SPEC_TOKENS=x` crashed in the
`EXIT` trap and wrote no JSON.
- `bash -n` on all four scripts. Launcher unit suite on this commit: 192
passed.
- **GPU (OCI-JHB, GB300, vLLM v0.31.0)**, through
`hf_online_dflash2.yaml`'s smoke task ([4/8]) and one smoke-only
submission, six runs in all:
- `latest_checkpoint` picked `exported-checkpoint-final` in every
training run.
- Per-prompt tok/s printed (e.g. 335 tok/s average), 8/8 prompts passed.
- Mean acceptance length 1.68–1.79 against the 1.5 floor;
`smoke_check.json` written each time.
- **GPU, Qwen3.8-27B + DFlash2 (same cluster and image):** four smoke
runs at the old fixed 180 s each wrote `smoke_check.json` with `stage:
server_timeout`. A fifth with `SERVER_TIMEOUT=900` passed at AL 1.76.
- Not run on GPU:
- the `server_died` and no-drafts failure paths. They exit through the
same `write_smoke_json` trap as the passing path;
- creating `SMOKE_JSON`'s parent directory. A smoke-only run hit the
missing directory before this was added; the rerun used an existing
directory.

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ (`SMOKE_JSON` is optional. With
`MIN_ACCEPTANCE_LENGTH` set, a run with no speculative drafts now fails
instead of passing; that was the bug.)
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅ (`latest_checkpoint`; the
smoke script itself is validated on GPU as above)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A
- Did you get Claude approval on this PR?: ❌ - not yet run

### Additional Information

- The `specdec-release` skill's `gate_smoke.py` ([6/8]) reads
`SMOKE_JSON`.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Checkpoint selection now prefers a `final` checkpoint, or otherwise
chooses the highest-numbered checkpoint.
* Smoke tests validate numeric settings, use a configurable server
timeout, and report run results, failure reasons, and performance
metrics in an optional JSON file.
* Acceptance-length checks now use cumulative draft and accepted-token
counts; prompt failures and unmet acceptance thresholds result in a
failed test.
* **Tests**
* Added coverage for checkpoint selection and smoke-test configuration
failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Zhehao-Hu and others added 2 commits October 28, 2026 00:09
hf_online_dflash2.yaml:
- Add task_2: vLLM smoke test of the newest export (DRAFT_CKPT_DIR), gated
  on MIN_ACCEPTANCE_LENGTH 1.5 with 7 spec tokens, a floor that a healthy
  2000-step run of this config clears.
- Run task_0 and task_1 on the same TRT-LLM image (1.3.0rc20); task_0 was
  pinned to rc10 while task_1 took the factory default.

Both examples:
- Route every artifact path through global_vars.output_dir (training output,
  exports, SMOKE_JSON), so one override moves the producer and its consumer
  together, e.g. to rerun the smoke task against an earlier experiment.
- Pin vllm/vllm-openai:v0.31.0 (multi-arch, contains vllm#52816 for DFlash2)
  instead of latest/nightly, and set TRITON_CACHE_DIR node-local.
- hf_streaming_dflash2.yaml exports to <output_dir>/export via EXPORT_PATH.

Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>
vLLM serves DFlash2 with a block of 1 + num_speculative_tokens (8 with the
examples' NUM_SPEC_TOKENS 7) and ignores the checkpoint's block_size, while
both examples trained on 16-token blocks. A dev-budget rerun of the online
example with block 8 gave the same acceptance length as block 16 (1.75 at
concurrency 1/8/32), so the served block costs nothing and removes a
train/serve mismatch. It also matches the library default and the published
DFlash2 drafters.

The online example drops dflash_loss_decay_factor: the recipe's dpace
objective ignores it, as the training log warns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from d217cd2 to 84467d2 Compare October 28, 2026 00:21
@Zhehao-Hu
Zhehao-Hu force-pushed the zhehaoh/launcher-dflash2-examples branch from 406058d to c81e5eb Compare October 28, 2026 00:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant