[6/8] specdec-release: add training and serving readiness gates for drafters

.claude/skills/specdec-release

@@ -0,0 +1 @@
+../../.agents/skills/specdec-release

plugins/modelopt/skills/specdec-release/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: specdec-release
+description: Deterministic readiness gates for a speculative-decoding drafter — training/export (gate_train), and serving canary (gate_smoke). Use when the user asks whether a trained drafter (EAGLE3, DFlash, DFlash2, DSpark, Domino, LiLiCorr) is ready for release review, or to gate the artifacts of a draft-training pipeline run. Do NOT use to configure, debug or triage a draft-training run (use speculative-decoding), to quantize (use ptq), or to serve a model (use deployment).
+license: Apache-2.0
+---
+
+# Spec-Dec Release
+
+Gates that decide whether a drafter is **ready for release review**: trained cleanly,
+exported to fit its target, and servable with speculation active. They set no per-model
+performance bar; AL is returned as evidence. The
+checkpoint release criteria (Addendum A: AL and throughput bars) are a later stage and are
+not implemented here.
+
+## Gate contract
+
+Same as `day0-release`. Each script prints JSON `{pass, failure_class, detail, notes,
+inputs, ...}` and exits `0` pass, `1` gate failed, `2` unreadable or invalid input
+(`USER_CONFIG_ERROR`). Run the gates on local copies of the artifacts or on the cluster
+login node; they are stdlib-only. **Never read a verdict from a log or a Slurm state.**
+
+| Stage | Command | Reads |
+| --- | --- | --- |
+| Train + export | `python "$SKILL_DIR/scripts/gate_train.py" --regression <out>/regression_check.json --export <out>/exported-checkpoint-final --target-config <target>/config.json --algorithm <algo>` | `check_regression.py --json` output, export dir, target config |
+| Smoke | `python "$SKILL_DIR/scripts/gate_smoke.py" --smoke <SMOKE_JSON> --draft <drafter> --target <target>` | `vllm_smoke_test.sh` `SMOKE_JSON` |
+
+A gate never passes a run its producing script failed, classifies the failure itself only
+when the script cannot (so `gate_smoke` reads raw values; `gate_train` keeps
+`check_regression.py`'s verdict), and never re-implements a script's comparison.
+
+Artifact files are read
+from host paths (`/cicd/<exp_id>/...`), but `--draft`/`--target` are compared to the paths the
+job recorded, so pass them as the job saw them (e.g. `/scratchspace/...`). `gate_smoke`
+classifies failures from the raw values and its `stage`; a SMOKE_JSON without `stage` comes
+from an older script, so re-run the smoke test.
+
+## Failure classes
+
+| `failure_class` | Gate | Action |
+| --- | --- | --- |
+| `USER_CONFIG_ERROR` | any | Fix the invocation (paths, `--algorithm`, unknown family → `--expect-architecture`); re-run the gate only |
+| `METRICS_MISSING` | train | No regression verdict, or the recipe set no thresholds. Check the training log; set `MAX_FINAL_LOSS`/`MIN_FINAL_ACC` in the launcher YAML |
+| `TRAIN_REGRESSION` | train | The recipe's own floors failed. Triage with the **speculative-decoding** skill; do not serve |
+| `EXPORT_MISSING` | train | Re-run the training task to re-export (an export counts only once `config.json` and weights exist) |
+| `DRAFT_CONFIG_MISMATCH` | train | Export does not fit the target (architecture, hidden/vocab size, mask token, layer ids). Fix the recipe overrides per the algorithm sheet; retrain |
+| `CHECKPOINT_NOT_SERVABLE` | smoke | Same as `day0-release`: fix the export or serving flags; do not benchmark |
+| `SPEC_NOT_ACTIVE` | smoke | AL ≤ 1.05: drafter not wired up or useless. Check `SPEC_METHOD`, token count, drafter path |
+| `ACCEPTANCE_REGRESSION` | smoke | Serves and speculates, but AL < the recipe's `MIN_ACCEPTANCE_LENGTH`. Triage training with **speculative-decoding**; do not benchmark |
+| `SPEC_CONFIG_MISMATCH` | smoke | Served the wrong drafter or target. Fix the submission; re-run |
+
+Verdict: both gates passed → a servable drafter, ready for a matched benchmark;
+training, export or serving failed for good → `INFEASIBLE`.

plugins/modelopt/skills/specdec-release/scripts/gate_smoke.py

@@ -0,0 +1,148 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Spec-dec serving canary gate: target + drafter load, generate, and speculate.
+
+Reads the SMOKE_JSON written by tools/launcher/common/specdec/vllm_smoke_test.sh and
+classifies the failure from its raw values: the script's own ``pass`` cannot tell a serving
+failure from an acceptance-length regression. A failing ``pass`` the values do not explain
+still fails the gate. Pure decision logic in ``evaluate_smoke``.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import sys
+
+# Mean acceptance length counts the bonus token, so exactly 1.0 means no draft token was
+# ever accepted. Anything at or below this is a drafter that is wired up but useless, or
+# not wired up at all; generic, not a performance bar.
+SPEC_ON_MIN_AL = 1.05
+
+_RANKING = (
+    "CHECKPOINT_NOT_SERVABLE",
+    "SPEC_CONFIG_MISMATCH",
+    "SPEC_NOT_ACTIVE",
+    "ACCEPTANCE_REGRESSION",
+)
+
+
+def evaluate_smoke(smoke, draft=None, target=None):
+    """Validate a smoke-test result.
+
+    Args:
+        smoke: dict from SMOKE_JSON.
+        draft: expected drafter path or HF id, as the smoke job saw it; optional.
+        target: expected target model path or HF id, as the smoke job saw it; optional.
+
+    Returns:
+        dict ``{pass, failure_class, detail, notes, mean_acceptance_length}``.
+    """
+    if "stage" not in smoke:
+        # Written by a vllm_smoke_test.sh that predates "stage": it cannot say how far the
+        # run got, so re-run the smoke test rather than guess.
+        return {
+            "pass": False,
+            "failure_class": "USER_CONFIG_ERROR",
+            "detail": "SMOKE_JSON has no 'stage'; re-run the smoke test with the current script",
+            "notes": [],
+            "mean_acceptance_length": smoke.get("mean_acceptance_length"),
+        }
+    stage = smoke["stage"]
+    if stage == "invalid_config":
+        # The script rejected a recipe value before serving anything: nothing about the
+        # drafter is known, and re-running unchanged fails the same way.
+        return {
+            "pass": False,
+            "failure_class": "USER_CONFIG_ERROR",
+            "detail": f"smoke task config: {smoke.get('reason')}; fix it in the recipe and "
+            "re-run the smoke task",
+            "notes": [],
+            "mean_acceptance_length": smoke.get("mean_acceptance_length"),
+        }
+    problems = []
+    passed, failed = smoke.get("prompts_passed"), smoke.get("prompts_failed")
+    if stage != "completed":
+        problems.append(("CHECKPOINT_NOT_SERVABLE", f"stopped at {stage}: {smoke.get('reason')}"))
+    elif failed or not passed:
+        problems.append(("CHECKPOINT_NOT_SERVABLE", f"{passed} prompts passed, {failed} failed"))
+    for label, expected in (("draft", draft), ("target", target)):
+        if expected is not None and smoke.get(label) != expected:
+            problems.append(
+                ("SPEC_CONFIG_MISMATCH", f"{label} {smoke.get(label)!r} != expected {expected!r}")
+            )
+    al = smoke.get("mean_acceptance_length")
+    valid = isinstance(al, (int, float)) and not isinstance(al, bool) and math.isfinite(al)
+    if not valid:
+        problems.append(("SPEC_NOT_ACTIVE", f"no acceptance length reported ({al!r})"))
+    elif al <= SPEC_ON_MIN_AL:
+        problems.append(("SPEC_NOT_ACTIVE", f"acceptance length {al} <= {SPEC_ON_MIN_AL}"))
+    min_al = smoke.get("min_acceptance_length")
+    if valid and min_al is not None and al < min_al:
+        problems.append(
+            ("ACCEPTANCE_REGRESSION", f"acceptance length {al} < recipe floor {min_al}")
+        )
+    if not problems and smoke.get("pass") is not True:
+        # Every failure the script knows is re-derived above; one this gate does not
+        # recognize must still not pass a run whose launcher task failed.
+        problems.append(
+            ("CHECKPOINT_NOT_SERVABLE", f"smoke test failed: {smoke.get('reason') or 'no reason'}")
+        )
+
+    notes = []
+    if min_al is None:
+        notes.append("recipe set no MIN_ACCEPTANCE_LENGTH; only the generic floor applied")
+    result = {"mean_acceptance_length": al, "notes": notes}
+    if not problems:
+        return {"pass": True, "failure_class": None, "detail": "servable, speculating", **result}
+    fc = min((c for c, _ in problems), key=_RANKING.index)
+    return {
+        "pass": False,
+        "failure_class": fc,
+        "detail": "; ".join(f"{c}: {m}" for c, m in problems),
+        **result,
+    }
+
+
+def main(argv=None):
+    """CLI entry point: read SMOKE_JSON and print the verdict."""
+    p = argparse.ArgumentParser(description="Spec-dec serving canary gate")
+    p.add_argument("--smoke", required=True, help="SMOKE_JSON from vllm_smoke_test.sh")
+    p.add_argument("--draft", help="expected drafter path or HF id")
+    p.add_argument("--target", help="expected target model path or HF id")
+    args = p.parse_args(argv)
+
+    try:
+        with open(args.smoke) as f:
+            smoke = json.load(f)
+        if not isinstance(smoke, dict):
+            raise ValueError(f"{args.smoke} does not hold a JSON object")
+    except (OSError, ValueError) as e:
+        error = {"pass": False, "failure_class": "USER_CONFIG_ERROR", "detail": str(e)}
+        print(json.dumps({**error, "notes": [], "inputs": {"smoke": args.smoke}}, indent=2))
+        return 2
+
+    result = evaluate_smoke(smoke, args.draft, args.target)
+    result["inputs"] = {"smoke": args.smoke}
+    print(json.dumps(result, indent=2))
+    if result["failure_class"] == "USER_CONFIG_ERROR":
+        return 2
+    return 0 if result["pass"] else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())