[3/8] launcher: re-export partial or stale DFlash checkpoints after training by Zhehao-Hu · Pull Request #2931 · NVIDIA/Model-Optimizer

Zhehao-Hu · 2026-10-22T04:30:37Z

[3/8] Day-0 speculative-decoding readiness flow. Merge in order 1 → 8. Base: [2/8] #2930. Touches dflash_online_training.sh after [1/8].
Series: [1/8] #2839 · [2/8] #2930 · [3/8] #2931 (this) · [4/8] #2932 · [5/8] #2933 · [6/8] #2934 · [7/8] #2935 · [8/8] #2936

What does this PR do?

Type of change: Bug fix

After training, dflash_online_training.sh exports each saved checkpoint, but it skipped any exported-checkpoint-* directory that already existed. Two cases left a bad export in place:

An export interrupted mid-write was never redone. A preempted or timed-out job left a directory with weights and no config.json, and every later run skipped it.
A resumed run kept the previous exported-checkpoint-final, even though training had moved on.

Changes (dflash_online_training.sh only):

An export counts as done only once its config.json exists; the exporters write the weights first. Otherwise the directory is removed and exported again.
exported-checkpoint-final is also redone when modelopt_state.pth is newer than its config.json.

Usage

No new options. Re-running or resuming a training task now repairs partial or stale exports.

Testing

bash -n passes. Launcher unit suite on this commit: 183 passed.
GPU (OCI-JHB, GB300): four fresh 2000-step hf_online_dflash2.yaml runs exported checkpoint-2000 and final through this path, and the smoke test served the result each time.
Not run: the repair paths themselves (a partial export, or a stale final after a resume). Reviewers may want a reproduction: kill a job mid-export, or touch modelopt_state.pth, then rerun the task.

Before your PR is "Ready for review"

Make sure you read and follow Contributor guidelines and your commits are signed (git commit -s -S).

Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded trust_remote_code=True, torch.load(..., weights_only=False), pickle, etc.).

Is this change backward compatible?: ✅
If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A
Did you write any new necessary tests?: ❌ (the repair paths need a real, interrupted export)
Did you update Changelog?: N/A
Did you get Claude approval on this PR?: ❌ - not yet run

Additional Information

The specdec-release skill ([8/8]) freezes and benchmarks exported-checkpoint-final, so a stale or partial final would be measured as the new drafter.

copy-pr-bot · 2026-10-22T04:30:41Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-10-22T04:30:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

…strict (#2839) > **[1/8] Day-0 speculative-decoding readiness flow.** Merge in order 1 → 8. Base: `main`. > Series: [1/8] #2839 (this) · [2/8] #2930 · [3/8] #2931 · [4/8] #2932 · [5/8] #2933 · [6/8] #2934 · [7/8] #2935 · [8/8] #2936 ### What does this PR do? Type of change: Bug fix [common/check_regression.py](https://github.com/NVIDIA/Model-Optimizer/blob/main/tools/launcher/common/check_regression.py) gates DFlash-family training runs on final loss and accuracy. Four things stopped that gate from catching a bad run: 1. **It read the wrong checkpoint on long runs.** `find_trainer_state()` sorted `checkpoint-*` as strings, so `checkpoint-8000` ranked after `checkpoint-12000` and an older checkpoint's metrics were checked. Any run whose saved steps differ in digit count was affected; the ~92K-step full DFlash2 recipe in `hf_online_dflash2.yaml` saves every 4000 steps. It now sorts by step number. 2. **An unreadable run passed.** With thresholds set, a missing `trainer_state.json`, or one with no training logs, only warned and exited 0. An empty threshold variable (`MAX_FINAL_LOSS=""`) crashed on `float('')`. 3. **A regression never failed the job.** `dflash_online_training.sh` runs the check as `check_regression.py … || true`, so Slurm reports `COMPLETED` even when the log says `REGRESSION`. The verdict also existed only as log text. 4. **A crashed run was still graded.** The script's `ERR` trap only records a failure, so after a failed `accelerate launch` it went on to export and run the check, and a checkpoint that survived the crash could pass. Changes: - `check_regression.py`: - Sorts checkpoints by step number. - Fails when thresholds are set but there are no metrics to check, including a `trainer_state.json` of the wrong shape. An empty threshold variable counts as unset. A threshold that isn't a finite number (`7,0`, `nan`) fails as `METRICS_MISSING`. - A NaN or infinite value for a metric that has a threshold fails as `TRAIN_REGRESSION`. - Takes `--json <path>` and writes `{pass, failure_class, trainer_state, metrics, thresholds, messages}` as strict JSON (non-finite metrics as `null`). `failure_class` is `null`, `TRAIN_REGRESSION` or `METRICS_MISSING`. The verdict is serialized before the file is opened, so a failure can't leave a truncated file. - Its existing log lines (`=== Regression Check (…) ===`, `PASS:` / `REGRESSION:`, `Regression check PASSED`) are unchanged, so existing log readers keep working. A failure now also prints `Regression check FAILED`. - `dflash_online_training.sh` writes the verdict to `${OUTPUT_DIR}/regression_check.json`, next to the exports. `REGRESSION_STRICT=1` drops the `|| true`, so a regression fails the task through the existing `ERR` trap and `exit_handler`. **The default (`0` or unset) is unchanged (warn only)**, so the other examples with possibly uncalibrated thresholds don't start failing. Any other value fails the task at startup, before installing or training. - `dflash_online_training.sh` also skips export and the check when training fails, still reporting the failure like any `ERR`-trap failure, and removes an old `regression_check.json` first thing, so no early exit can leave an earlier run's verdict behind. - `examples/Qwen/Qwen3-8B/hf_online_dflash2.yaml`: - Sets `REGRESSION_STRICT: "1"`. - Recalibrates `MAX_FINAL_LOSS` from 5.0 to 7.0. The 5.0 was inherited from `hf_online_dflash.yaml`, whose own reference curve only drops below 5.0 at step ~2500 of a 12.5K-step run. DFlash2 adds a selector loss term and this example stops at 2000 steps, so 5.0 fails every healthy run. `MIN_FINAL_ACC` stays at 0.15. - The `speculative-decoding` skill's `dflash.md` and `dflash2.md` describe the new gate: judge a run by `regression_check.json`, the two failure classes, strict mode, and DFlash2's 7.0 threshold. ### Usage ```bash MAX_FINAL_LOSS=7.0 MIN_FINAL_ACC=0.15 \ python3 common/check_regression.py /scratchspace/dflash2_bs16 --json /scratchspace/dflash2_bs16/regression_check.json echo $? # 0 pass, 1 regression or nothing to check ``` ```yaml environment: - MAX_FINAL_LOSS: "7.0" - REGRESSION_STRICT: "1" # fail the training task on a regression ``` ### Testing - `tools/launcher/tests/test_check_regression.py`: 4 new test functions (13 cases). Run against `main`'s `check_regression.py`, 17 of the file's 18 cases fail: - `test_latest_checkpoint_is_chosen_by_step_number` guards the string-sort bug. - `test_empty_threshold_var_is_unset` guards the `float('')` crash. - `test_cli_exit_code_and_json_verdict` is parametrized over a missing `trainer_state.json`, a truncated one, two of the wrong shape, a NaN loss, an unchecked infinite metric, a regression and a pass. Each case checks the exit code and parses the verdict as strict JSON, so it covers the missing-metrics hole and the `--json` contract. - `test_invalid_threshold_writes_failed_verdict` covers `7,0`, `nan` and `inf`. - `tools/launcher/tests/test_dflash_online_training.py` (5 cases) runs the real script in a sandbox: stubbed `pip`, `nvidia-smi`, `accelerate`, exporter and `torch`, a checkpoint, and an old passing verdict. Each case asserts the exact exit code. Against `main`'s script, all 5 fail: - A failing launch leaves no verdict and exits 1; a succeeding one grades the checkpoint and exits 0 with no `ERR`-trap error. - With a checkpoint over the threshold, `REGRESSION_STRICT=1` exits 1 and `0` exits 0, both writing `TRAIN_REGRESSION`. - `REGRESSION_STRICT=true` exits 1 without launching training and leaves no verdict. - Full launcher suite on this commit: `uv run --with pytest python3 -m pytest tests/`, 183 passed. - `dflash_online_training.sh`: `bash -n` passes. - **GPU (OCI-JHB, GB300, global batch 8):** five 2000-step runs of this example with this change passed the strict gate. Final loss was 6.37–6.71 and accuracy 0.167–0.177, against 7.0 / 0.15. The calibration run that motivated 7.0 (before this change) finished at 6.51 / 0.181. The seed-to-seed spread puts the loss margin at about 4%; see the variance note under Additional Information. - Not run: the strict-failure path and a real training crash on GPU (covered by the sandbox test above). ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ (strict mode is opt-in; log output unchanged) - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: ❌ - not yet run ### Additional Information - Part of the Day-0 speculative-decoding workflow: the `specdec-release` skill ([8/8]) gates training on `regression_check.json`, so it needs a verdict it can read and a job state that reflects it. - **Why not reuse `day0-release/scripts/gate_run.py`?** The two sit at different layers and check different things, so their schemas stay independent on purpose: - `gate_run.py` is an agent-side gate for an *evaluation* run. It reads a run summary the agent builds from NEL output, and checks per-task status, sample counts and judge errors. None of that exists for a training run. - `check_regression.py` runs *inside* the Slurm training task, on rank 0, and its verdict decides the task's exit status in strict mode. It reads `trainer_state.json` against thresholds from the launcher YAML, and has to work for launcher users who never load a skill. A launcher script importing a skill script would also invert the dependency: skills drive the launcher, not the other way round. - The skill-side counterpart of `gate_run.py` for training is `specdec-release/scripts/gate_train.py` ([6/8] #2934). It reads `regression_check.json` and returns the day-0 gate shape (`{pass, failure_class, detail, …}`), with `failure_class` from the shared triage vocabulary. That is the adapter layer, so `check_regression.py` doesn't need to match `gate_run.py`. - What the two share is the convention: a boolean `pass`, and a `failure_class` that is `null` on a pass. The launcher emits only `METRICS_MISSING` and `TRAIN_REGRESSION`, and `gate_train.py` treats any other value as `TRAIN_REGRESSION`, so the launcher can add detail fields without breaking the gate. - Other Qwen3-8B examples share the inherited `MAX_FINAL_LOSS: "5.0"` (`hf_online_domino.yaml`, `hf_online_lilicorr.yaml`). They are left unchanged and non-strict because there is no calibration run for them. - Variance: a seed-1 retrain landed at loss 6.705, 4.2% under the 7.0 floor. Accuracy has more headroom (11%). Recalibrating from more seeds is planned for the release gates, not this PR.   ## Summary by CodeRabbit * **New Features** * Training runs with an output directory now produce a regression-check verdict file reporting whether configured quality thresholds passed. * Regression checks account for missing or invalid metrics and select checkpoints by numeric step. * **Bug Fixes** * Failed training no longer proceeds to checkpoint export or regression checks. * Strict mode fails the task when regression checks fail; invalid strict-mode values stop the run before training starts. * **Documentation** * Updated DFlash2 guidance and example settings for the final-loss threshold and strict-mode behavior.  --------- Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

- Add latest_checkpoint() to service_utils.sh: picks <dir>/<prefix>final if present, else the highest numeric step, keyed on the directory's own name. It replaces three ad hoc sorts; ar_eval_mtbench.sh and vllm_smoke_test.sh keyed on a dash-separated field of the whole path, so a dash in a parent directory (e.g. /data/my-drafts/...) picked the wrong checkpoint. vllm_smoke_test.sh now prefers exported-checkpoint-final, as quantize_drafter.sh already did. - vllm_smoke_test.sh: compute the mean acceptance length over the whole run from vLLM's /metrics counters (1 + accepted / drafts). The log's "Mean acceptance length" line covers only the last logging interval. - vllm_smoke_test.sh: with MIN_ACCEPTANCE_LENGTH set, fail when no speculative drafts were recorded instead of warning and passing. - vllm_smoke_test.sh: time prompts with shell arithmetic and python3 instead of bc, which the vLLM image lacks (every tok/s figure printed "?"). - vllm_smoke_test.sh: optional SMOKE_JSON writes the verdict, acceptance length and throughput as JSON for downstream gates, creating its parent directory if needed. Sourcing service_utils.sh is now required; it runs after the EXIT trap, so a failure there still writes this run's verdict instead of leaving an earlier run's SMOKE_JSON in place. Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>

…d numbers Sourcing service_utils.sh returns the status of its last command, a rank-0 append to modules/Model-Optimizer/modelopt/__init__.py. Outside a launcher working directory that append fails, so the smoke test stopped with "cannot source service_utils.sh". Only a missing file is fatal now. A non-integer NUM_SPEC_TOKENS crashed write_smoke_json inside the EXIT trap, so no SMOKE_JSON was written; a non-numeric MIN_ACCEPTANCE_LENGTH was reported as "REGRESSION: X < Y". Both are checked before any work and fail with the new stage invalid_config. The writer also turns a non-integer or non-finite value into null, so the file is always written and is strict JSON. The prompt-failure reason drops its "ERROR:" prefix and gives the count, and the prompt counters are renamed PROMPTS_OK/PROMPTS_FAILED so they no longer share FAIL with service_utils.sh. cleanup waits for the vLLM server to stop only when one was started, so an early exit no longer sleeps 2 seconds. Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>

vllm_smoke_test.sh waited a fixed 180 s for /health. Qwen3.8-27B with a DFlash2 drafter needs longer on GB300 (torch.compile of target and draft head, FlashInfer CuTe-DSL JIT, CUDA graph capture, GDN/Mamba Triton warmups), so every smoke attempt stopped at server_timeout. The script's exit then tore the container down mid-warmup, and the engine's next exec failed with ENOENT on ptxas, which read like the cause. SERVER_TIMEOUT sets the wait (default 180) and appears in the timeout reason. A non-integer or zero value fails up front as invalid_config, like NUM_SPEC_TOKENS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>

…nded NUM_SPEC_TOKENS=08 passed validation, then went into SPEC_CONFIG as a bare JSON number, which is invalid JSON: vLLM died at startup and the run read as server_died instead of invalid_config. NUM_SPEC_TOKENS and SERVER_TIMEOUT now have to be canonical positive integers. The mean acceptance length was rounded to 4 decimals before the MIN_ACCEPTANCE_LENGTH check, so 1.16666... passed a floor of 1.1667, and the rounded value went into SMOKE_JSON, where gate_smoke repeats the check. The check and SMOKE_JSON now use full precision; only the log is rounded. Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>

dflash_online_training.sh skipped any exported-checkpoint-* directory that existed, so an export interrupted mid-write was never redone, and a resumed run kept the previous exported-checkpoint-final. An export now counts as done only once its config.json exists (exporters write weights first); otherwise the directory is removed and exported again. exported-checkpoint-final is also redone when modelopt_state.pth is newer than its config.json. Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com>

…ction (#2930) > **[2/8] Day-0 speculative-decoding readiness flow.** Merge in order 1 → 8. Base: `main` ([1/8] has merged). > Series: [1/8] #2839 (merged) · [2/8] #2930 (this) · [3/8] #2931 · [4/8] #2932 · [5/8] #2933 · [6/8] #2934 · [7/8] #2935 · [8/8] #2936 ### What does this PR do? Type of change: Bug fix The vLLM smoke test (`common/specdec/vllm_smoke_test.sh`) is the serving check after draft training. It had six problems: 1. **It could serve the wrong checkpoint.** `ar_eval_mtbench.sh` and `vllm_smoke_test.sh` picked the newest checkpoint by sorting on a dash-separated field of the **whole path**, so a dash in a parent directory (e.g. `/data/my-drafts/...`) picked the wrong one. `quantize_drafter.sh` had a third, different sort. The smoke test also ignored `exported-checkpoint-final`. 2. **Every throughput figure printed `?`.** Timing used `bc`, which the vLLM image doesn't have. 3. **A drafter that never speculated passed.** With `MIN_ACCEPTANCE_LENGTH` set, a vLLM log with no acceptance length only warned. 4. **The acceptance length covered only the last few seconds.** It came from the log's "Mean acceptance length" line, which covers only vLLM's last logging interval. 5. **The verdict existed only as log text.** 6. **The server wait was fixed at 180 s.** Qwen3.8-27B with a DFlash2 drafter takes longer to start on GB300 (torch.compile, kernel JIT, CUDA graph capture, GDN warmups), so every smoke attempt stopped at `server_timeout`. The script's exit then tore the container down mid-warmup, and the engine's next exec failed with a ptxas ENOENT that read like the cause. Changes: - `service_utils.sh`: new `latest_checkpoint <dir> <prefix>`. It returns `<dir>/<prefix>final` if present, else the highest numeric step, keyed on the directory's own name. It replaces the three ad hoc sorts in `ar_eval_mtbench.sh`, `vllm_smoke_test.sh` and `quantize_drafter.sh`. - `vllm_smoke_test.sh`: - Prefers `exported-checkpoint-final`, as `quantize_drafter.sh` already did. - Times prompts with shell arithmetic and `python3` instead of `bc`. - Computes the mean acceptance length over the whole run from vLLM's cumulative `/metrics` counters: 1 + `spec_decode_num_accepted_tokens_total` / `spec_decode_num_drafts_total`. - With `MIN_ACCEPTANCE_LENGTH` set, fails when no speculative drafts were recorded. - Fails loudly if `service_utils.sh` is missing (it used to be optional, so the failure surfaced as a misleading `draft_missing`). Only a missing file is fatal: `source` returns the status of the file's last command, a rank-0 append that fails outside a launcher working directory. - `SERVER_TIMEOUT` sets the server wait in seconds (default 180) and appears in the timeout reason. - Checks `MIN_ACCEPTANCE_LENGTH` (a finite number), and `NUM_SPEC_TOKENS` and `SERVER_TIMEOUT` (positive integers without leading zeros, since `NUM_SPEC_TOKENS` goes into vLLM's JSON config), before any work, and fails with `stage: invalid_config` and a reason naming the value. Both checks run after the `EXIT` trap is set, so even these failures replace any earlier `SMOKE_JSON`. - Optional `SMOKE_JSON=<path>` writes the verdict on every exit: `{stage, pass, reason, method, target, draft, num_speculative_tokens, prompts passed/failed, tokens, average tok/s, mean_acceptance_length, min_acceptance_length}`. `stage` says how far it got (`invalid_config`, `starting`, `draft_missing`, `server_died`, `server_timeout`, `serving`, `completed`), so a downstream gate can tell "not servable" from "below the floor". The file's parent directory is created if needed, and it is strict JSON (a non-numeric value is written as `null`). The mean acceptance length is checked against the floor and written at full precision; only the log rounds it. ### Usage ```yaml task_2: script: common/specdec/vllm_smoke_test.sh environment: - HF_MODEL_CKPT: <<global_vars.hf_model>> - DRAFT_CKPT_DIR: <<global_vars.output_dir>> # newest export, final preferred - SPEC_METHOD: "dflash" - NUM_SPEC_TOKENS: "7" - MIN_ACCEPTANCE_LENGTH: "1.5" - SMOKE_JSON: <<global_vars.output_dir>>/smoke_check.json - SERVER_TIMEOUT: "900" # optional; default 180 s ``` ### Testing - `tools/launcher/tests/test_service_utils.py`: `latest_checkpoint` with a dash in a parent directory, `final` winning over steps, numeric rather than string order, and no checkpoint at all (3 cases). - `tools/launcher/tests/test_vllm_smoke_test.py` runs the script as rank 0 from a directory with no `modules/` (stubbed `pip` and `nvidia-smi`), up to the paths that stop before vLLM starts (6 cases). Against the script without these fixes, all 6 fail: - It reaches `draft_missing` rather than stopping at `source`. - `MIN_ACCEPTANCE_LENGTH` of `1,5` or `nan`, `NUM_SPEC_TOKENS` of `x` or `08` and `SERVER_TIMEOUT=15m` each fail as `invalid_config` with a strict-JSON `SMOKE_JSON`. Before, `NUM_SPEC_TOKENS=x` crashed in the `EXIT` trap and wrote no JSON. - `bash -n` on all four scripts. Launcher unit suite on this commit: 192 passed. - **GPU (OCI-JHB, GB300, vLLM v0.31.0)**, through `hf_online_dflash2.yaml`'s smoke task ([4/8]) and one smoke-only submission, six runs in all: - `latest_checkpoint` picked `exported-checkpoint-final` in every training run. - Per-prompt tok/s printed (e.g. 335 tok/s average), 8/8 prompts passed. - Mean acceptance length 1.68–1.79 against the 1.5 floor; `smoke_check.json` written each time. - **GPU, Qwen3.8-27B + DFlash2 (same cluster and image):** four smoke runs at the old fixed 180 s each wrote `smoke_check.json` with `stage: server_timeout`. A fifth with `SERVER_TIMEOUT=900` passed at AL 1.76. - Not run on GPU: - the `server_died` and no-drafts failure paths. They exit through the same `write_smoke_json` trap as the passing path; - creating `SMOKE_JSON`'s parent directory. A smoke-only run hit the missing directory before this was added; the rerun used an existing directory. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ (`SMOKE_JSON` is optional. With `MIN_ACCEPTANCE_LENGTH` set, a run with no speculative drafts now fails instead of passing; that was the bug.) - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ (`latest_checkpoint`; the smoke script itself is validated on GPU as above) - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: ❌ - not yet run ### Additional Information - The `specdec-release` skill's `gate_smoke.py` ([6/8]) reads `SMOKE_JSON`.   ## Summary by CodeRabbit * **Improvements** * Checkpoint selection now prefers a `final` checkpoint, or otherwise chooses the highest-numbered checkpoint. * Smoke tests validate numeric settings, use a configurable server timeout, and report run results, failure reasons, and performance metrics in an optional JSON file. * Acceptance-length checks now use cumulative draft and accepted-token counts; prompt failures and unmet acceptance thresholds result in a failed test. * **Tests** * Added coverage for checkpoint selection and smoke-test configuration failures.  --------- Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Zhehao-Hu force-pushed the zhehaoh/launcher-smoke-test branch from 6b6b999 to 56f75fc Compare October 22, 2026 17:59

Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from 3ac8822 to 9a9b3cc Compare October 22, 2026 17:59

Zhehao-Hu force-pushed the zhehaoh/launcher-smoke-test branch from 56f75fc to 66c70e1 Compare October 22, 2026 22:09

Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch 3 times, most recently from a2d6478 to dbddd39 Compare October 23, 2026 04:51

Zhehao-Hu and others added 3 commits October 23, 2026 21:22

Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from dbddd39 to 6d46dab Compare October 23, 2026 21:24

Zhehao-Hu force-pushed the zhehaoh/launcher-smoke-test branch from a2bec7b to 6941971 Compare October 23, 2026 21:24

Zhehao-Hu added 2 commits October 23, 2026 21:33

Zhehao-Hu force-pushed the zhehaoh/launcher-dflash-reexport branch from 6d46dab to d217cd2 Compare October 23, 2026 21:35

Base automatically changed from zhehaoh/launcher-smoke-test to main October 23, 2026 23:25

Navigation Menu

Repository navigation

[3/8] launcher: re-export partial or stale DFlash checkpoints after training - #2931

[3/8] launcher: re-export partial or stale DFlash checkpoints after training#2931
Zhehao-Hu wants to merge 5 commits into
mainfrom
zhehaoh/launcher-dflash-reexport

Zhehao-Hu commented Oct 22, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 22, 2026

Uh oh!

coderabbitai Bot commented Oct 22, 2026

Draft PR not reviewed

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Zhehao-Hu commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Usage

Testing

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Oct 22, 2026

Uh oh!

coderabbitai Bot commented Oct 22, 2026

Draft PR not reviewed

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Zhehao-Hu commented Oct 22, 2026 •

edited

Loading