From 3d6893222a552fcff38cc8fc56a0e6c25a8dfe97 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Wed, 23 Sep 2026 15:48:42 +0000 Subject: [PATCH 1/8] docs(eval): tolerate limited model-output faults Signed-off-by: Chad Voegele --- .codex/agents/modelopt_model_evaluator.toml | 2 +- AGENTS.md | 10 +++ .../agents/modelopt-model-evaluator.md | 2 +- plugins/modelopt/skills/evaluation/SKILL.md | 13 ++- .../evaluation/references/run-validation.md | 79 ++++++++++++++++--- .../skills/evaluation/tests/evals.json | 48 +++++++++++ 6 files changed, 136 insertions(+), 18 deletions(-) diff --git a/.codex/agents/modelopt_model_evaluator.toml b/.codex/agents/modelopt_model_evaluator.toml index ad99a342d4b..02887306b55 100644 --- a/.codex/agents/modelopt_model_evaluator.toml +++ b/.codex/agents/modelopt_model_evaluator.toml @@ -11,7 +11,7 @@ Before acting, load these Model Optimizer instructions: - `accessing-mlflow/SKILL.md` when runs or artifacts are in MLflow - `common/workspace-management.md` -Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. +Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Aggregate Model-Output-Fault Policy per benchmark/run: report category counts, their deduplicated union, the verified unique-successful-response denominator, and the unrounded rate. A rate ≤2.0% is a visible warning only when affected raw responses are preserved and scored incorrect, coverage is complete, and every independent gate passes. Above tolerance, return findings and a recommendation to the parent without automatically resubmitting. Never tune token limits merely to pass. Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. """ diff --git a/AGENTS.md b/AGENTS.md index c2cebc9a399..8fb0422dfa1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,16 @@ These instructions apply to AI-assisted work in this repository. those skills through relative symlinks. Shared agent config and scripts remain under `.agents/`. See `.agents/README.md` for the convention. +## Evaluation analysis + +Before analyzing completed evaluations or reporting scores, including standalone +`launching-evals` sessions, read +`plugins/modelopt/skills/evaluation/references/run-validation.md`. +Its aggregate model-output-fault policy supersedes vendored `launching-evals` +advice to uncap reasoning output (`max_tokens=null` or payload field removal), +use fixed token/context limits, or automatically invalidate a preserved, +successful response that is retained and scored incorrect. + ## Coding guidelines - **Coding guide:** Code development and review require reading and following diff --git a/plugins/modelopt/agents/modelopt-model-evaluator.md b/plugins/modelopt/agents/modelopt-model-evaluator.md index 592770f11f2..9291c6f4f7e 100644 --- a/plugins/modelopt/agents/modelopt-model-evaluator.md +++ b/plugins/modelopt/agents/modelopt-model-evaluator.md @@ -16,6 +16,6 @@ Before acting, load these Model Optimizer instructions: - `accessing-mlflow/SKILL.md` when runs or artifacts are in MLflow - `common/workspace-management.md` -Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. +Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Aggregate Model-Output-Fault Policy per benchmark/run: report category counts, their deduplicated union, the verified unique-successful-response denominator, and the unrounded rate. A rate ≤2.0% is a visible warning only when affected raw responses are preserved and scored incorrect, coverage is complete, and every independent gate passes. Above tolerance, return findings and a recommendation to the parent without automatically resubmitting. Never tune token limits merely to pass. Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index bf107f68e07..6b86ebe6c2f 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -131,9 +131,16 @@ Read [Step 8](references/launcher-workflow.md#step-8--run-evaluation-gated-dry-r ### Step 9 — Verify completed run Read [run-validation.md](references/run-validation.md) before reporting scores: -validate logs and sample coverage, complete **Timeout and Output-Limit Accounting** -for every task, and report missing telemetry as unknown. For comparisons, also -apply its **External Baseline Sanity Check**, then use `compare-results`. +validate logs and coverage, complete **Timeout and Output-Limit Accounting** for +every task, and report missing telemetry as unknown. Parents and evaluators must +also apply its **Aggregate Model-Output-Fault Policy** per benchmark/run: report +category counts, their deduplicated union, the verified unique-successful-response +denominator, and the unrounded rate. At ≤2.0%, warn rather than invalidate or +retry only when raw responses are preserved and scored incorrect, coverage is +complete, and all independent gates pass. Above tolerance, return findings and +a recommendation without automatically resubmitting. Never tune token limits +merely to pass. For comparisons, also apply its **External Baseline Sanity +Check**, then use `compare-results`. --- diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index 2d3122dc60b..604e358cff0 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -25,18 +25,71 @@ For each completed invocation/run directory, whether baseline, quantized, or a single-model run: 1. Inspect client, server/deployment, SLURM, judge, and task-specific/code-execution logs as applicable. Search for `Traceback`, `Exception`, `ERROR`, `FAILED`, `OOM`, `Killed`, `timeout`, `rate limit`, `unauthorized`, `connection refused/reset`, `health check`, `sandbox`, `container`, `judge`, `parse`, `scoring`, and task-specific failure strings. -2. Confirm the inference server loaded the intended checkpoint/model and stayed healthy through the run: no startup failure, mid-run crash/restart, OOM, request validation failure, max-context truncation, quantization load error, or repeated 4xx/5xx responses. +2. Confirm the inference server loaded the intended checkpoint/model and stayed healthy through the run: no startup failure, mid-run crash/restart, OOM, request validation failure, input/context clipping, quantization load error, or repeated 4xx/5xx responses. 3. For judge-backed tasks, confirm judge calls succeeded and were parsed/scored correctly: no auth/rate-limit failures, malformed judge responses, invalid JSON, missing scores, or fallback/default scores. 4. For code-execution tasks, inspect executor/sandbox/container logs for setup failures, package install failures, timeouts, thread/process exhaustion, permission errors, harness crashes, or skipped tests that would make scores non-comparable. -5. Confirm sample accounting: expected samples/repeats match completed, scored samples; no unexpected dropped/skipped/failed samples, `unknown_agent_error`, `failed_samples_policy` aborts, empty outputs, or partial result files. -6. If reasoning traces are present, confirm they are parsed/stripped/ignored before scoring consistently. Assess output-limit termination such as `finish_reason: length` using the accounting below; it is not by itself a parsing failure. Check for parser errors, unmatched reasoning delimiters, reasoning text leaked into answers, answers stripped with the reasoning, or reasoning disabled when the config intended it to be active. +5. Confirm coverage separately at every available level: expected, selected, evaluated, and scored samples/repeats/trajectories must match. No response may be missing or unscored, and no unexpected dropped/skipped/failed sample, `unknown_agent_error`, `failed_samples_policy` abort, or partial result file is allowed. +6. If reasoning traces are present, confirm they are parsed/stripped/ignored before scoring consistently. Assess unusable model outputs under the policy below. Check for parser or fallback errors, unmatched reasoning delimiters, reasoning text leaked into answers, answers stripped with the reasoning, or reasoning disabled when the config intended it to be active. 7. Complete the **Timeout and Output-Limit Accounting** below for every task, including non-reasoning models and successful runs. -Report the run-validation summary before any score: log scan status, sample -accounting, reasoning/answer parsing status, and any errors or warnings found. -If any validation item fails, either rerun/fix it or label the result as -incomplete or invalid. +Report the run-validation summary before any score: log scan status, coverage, +reasoning/answer parsing status, and any errors or warnings found. If an +independent validation item fails, label the result incomplete or invalid and +return findings and a recommendation to the parent (or user); do not +automatically resubmit a completed run. + +### Aggregate Model-Output-Fault Policy (Parent and Evaluator) + +For each benchmark/run, report category counts, their deduplicated union, the +verified denominator, and `rate = 100 × union / denominator`. Calculate with +unrounded counts; round only the displayed percentage. + +The denominator is the deduplicated set of unique, successful raw evaluated-model +responses selected for evaluation. Count a cached response reused by multiple +trajectories once; exclude duplicate log/cache records, failed request attempts, +judge calls, and unrelated runs. Verify the denominator from response identity +and provenance rather than dataset size, especially for repeated or multi-turn +tasks. An unknown or zero denominator leaves the rate unverified, not 0%. + +The numerator is the set union of denominator responses unusable because of +model output behavior: + +- length or token-budget truncation, including reasoning that consumes the budget; +- an empty final answer, including a nonempty reasoning trace with no final answer; +- a malformed or otherwise unusable final output. + +Count a response in every applicable category for reporting, but once in the +union. To qualify, the successful raw response must exist, remain preserved in +the artifacts, and be deterministically retained and scored incorrect. A parser +exception, parser fallback/default score, missing parsed record, or scorer/harness +failure is not automatically a model-output fault; it remains invalid unless the +preserved raw response's malformed answer is explicitly retained and scored +incorrect under the benchmark protocol. + +- **0 < rate ≤ 2.0% (inclusive, before rounding):** valid with a visible warning + when expected/selected/evaluated/scored coverage is complete and all independent + gates pass. Retain the affected responses and their incorrect scores; do not + invalidate or retry solely for these faults. A low rate does **not** imply + negligible score impact. +- **rate > 2.0%:** return category counts, union, rate, findings, and a + recommendation to the parent (or user); do not automatically retry or declare + success. +- The tolerance never waives missing or unscored responses; exhausted request, + transport, server, or authentication failures; incomplete coverage; parser, + harness, sandbox, or scorer failures; wrong benchmark, model, configuration, or + version; secret issues; input/context clipping; benchmark-specific validity + rules; or an unverified denominator. A score above a reference does not establish + validity. +- Check configured and effective output limits against the reference evaluation + protocol and deployed context capacity, including prompt/history plus output + space. Report mismatches or unavailable reference settings. Do not remove or + tune token limits merely to pass; propose protocol-justified changes for + parent/user approval instead. + +The parent must preserve the category and rate warning and apply the same policy +to evaluator handoffs rather than treating any nonzero model-output fault as a +failed run. ## Timeout and Output-Limit Accounting @@ -58,12 +111,12 @@ Missing termination metadata, sampled-only artifacts, or omitted failures make full-run rates **unknown**, not zero. Report coverage and what evidence is missing; do not extrapolate sampled rates. -- **Valid with warnings:** small fractions of output-limit or benchmark-timeout - events can pass when sample/repeat/scoring coverage is complete, limits match - the protocol, and other checks pass. Apply explicit task/user tolerances, - retain failures per benchmark scoring, and report observed impact. A nonzero - rate alone must not trigger failure or automatic rerun; a low rate does not - prove negligible impact or excuse infrastructure failures. +- **Model-output faults:** apply the aggregate policy above. Output-limit stops + qualify only when the response is preserved and scored incorrect; terminal + request, task, judge, sandbox, or harness timeouts remain invalid. +- **Recovered retries:** report them separately. They do not enter the fault + numerator or denominator as extra responses, and they do not excuse a terminal + failure or missing coverage. - **Provisional/inconclusive:** unknown accounting leaves validation incomplete. Investigate infrastructure failures, unexpected exclusions, or mismatched limits before a model-quality verdict. A valid run alone does not establish diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 3a33a6af053..9670479c63d 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -1,4 +1,52 @@ [ + { + "name": "standalone-kimi-scicode-model-output-fault", + "skills": ["launching-evals"], + "query": "Analyze Kimi SciCode attempt 15 in a fresh repository session, without an evaluator role or prior evaluation skill context. Coverage is complete: 65/65 samples and 338/338 selected, evaluated, and scored trajectories. Provenance verifies 337 unique successful HTTP model responses as the response denominator. One preserved response is both finish_reason=length and empty-final after reasoning consumed its token budget; it was retained and scored incorrect. There are zero HTTP, infrastructure, parser, scorer, sandbox, or harness failures. The measured score is 53.25443786982248. Is this run valid, or should it be rerun? Do not launch jobs.", + "files": [], + "expected_behavior": [ + "Follows AGENTS.md (also exposed via CLAUDE.md) to read evaluation/references/run-validation.md before reporting the score, without preloading the evaluation skill", + "Reports complete 65/65 sample and 338/338 selected/evaluated/scored trajectory coverage plus 337 verified unique successful responses", + "Reports length-capped=1 and empty-final=1 but a deduplicated union of 1/337 = approximately 0.2967%", + "Declares the run valid with a visible model-output-fault warning, retains score 53.25443786982248, and does not rerun", + "Does not infer negligible score impact from the low rate or tune/remove token limits" + ] + }, + { + "name": "model-output-fault-overlap-and-two-percent-boundary", + "skills": ["evaluation", "launching-evals"], + "query": "Classify two independent runs with complete expected/selected/evaluated/scored coverage and 10000 verified unique successful raw model responses each. In A, preserved responses 1-120 are length-capped, 81-170 have empty finals, and 151-200 have malformed unusable finals; every affected response is retained and scored incorrect. In B, 201 distinct preserved responses have malformed finals and are retained and scored incorrect. Each log also repeats 5000 cache records and includes 100 failed transport attempts that were retried successfully plus 10000 judge responses. All independent gates pass. Report category and union accounting; do not launch jobs.", + "files": [], + "expected_behavior": [ + "Uses 10000 as each denominator, excluding duplicate records, retry attempts, and judge calls; does not pool runs", + "Reports A's category counts as length-capped=120, empty-final=90, malformed-final=50 and deduplicates overlaps to response IDs 1-200, a union of 200/10000 = exactly 2.0%", + "Treats A as valid with a visible warning because the inclusive unrounded boundary passes", + "Reports B's union as 201/10000 = 2.01%, above tolerance; returns findings and a recommendation without automatic retry or a success verdict" + ] + }, + { + "name": "model-output-tolerance-does-not-waive-independent-failures", + "skills": ["evaluation", "launching-evals"], + "query": "Review five separate runs whose apparent model-output-fault rate is below 2%. A selected 1000 responses but evaluated and scored only 999. B exhausted an HTTP/server request and has no successful raw response for one item. C raised a parser exception and assigned a fallback score without retaining the malformed answer as an explicit incorrect result; its harness also failed. D used the wrong benchmark version, model, and config. E has an authentication/secret failure and no verifiable response denominator. All reported scores exceed the reference. Can the parent accept any under the tolerance?", + "files": [], + "expected_behavior": [ + "Marks every run invalid rather than using the model-output tolerance", + "Separately identifies missing/unscored coverage, exhausted HTTP/server request, parser fallback and harness failure, wrong protocol identity, authentication/secret failure, and unverified denominator", + "Does not reclassify C as a model-output fault because no preserved successful raw malformed answer was deterministically retained and scored incorrect", + "Returns findings and recommendations without automatically retrying; exceeding the reference is not proof of validity" + ] + }, + { + "name": "model-output-fault-unverified-and-token-budget", + "skills": ["evaluation", "launching-evals"], + "query": "A run has no reliable unique-successful-response denominator. Another has 5/1000 preserved length-capped responses retained and scored incorrect, max_new_tokens=131072, deployed context=131072, and 4096-token prompts; its reference protocol uses a 65536 output budget. Should I remove the output cap or keep increasing it until the reference score passes?", + "files": [], + "expected_behavior": [ + "Marks the first run's model-output-fault rate unverified, not 0% or within tolerance", + "Reports the second run's union as 5/1000 = 0.5% but flags reference-budget mismatch and insufficient context for prompt plus configured output", + "Does not uncap or tune limits to pass the target; returns a protocol-justified recommendation for approval without automatically resubmitting" + ] + }, { "name": "nemotron3-nano-bf16-reasoning", "skills": ["evaluation"], From e63d0d53c0697017506c6b6e95df2dd4cf3b42d0 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Thu, 1 Oct 2026 15:29:12 -0500 Subject: [PATCH 2/8] docs(eval): extend bounded failure tolerance to benchmark trials Signed-off-by: Chad Voegele --- .codex/agents/modelopt_model_evaluator.toml | 2 +- AGENTS.md | 6 +- .../agents/modelopt-model-evaluator.md | 2 +- .../skills/day0-release/scripts/gate_run.py | 6 +- .../skills/day0-release/tests/test_gates.py | 15 +++ plugins/modelopt/skills/evaluation/SKILL.md | 21 ++--- .../recipes/examples/example_eval_next.yaml | 6 +- .../evaluation/recipes/tasks/aa/scicode.md | 3 +- .../tasks/aa_next/terminal_bench_2_1.md | 20 +++- .../skills/evaluation/references/nel-next.md | 8 +- .../evaluation/references/run-validation.md | 92 +++++++++++++------ .../skills/evaluation/tests/evals.json | 65 ++++++++++++- .../evaluation/tests/test_failure_guidance.py | 63 +++++++++++++ 13 files changed, 250 insertions(+), 59 deletions(-) create mode 100644 plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py diff --git a/.codex/agents/modelopt_model_evaluator.toml b/.codex/agents/modelopt_model_evaluator.toml index 02887306b55..86478b8cff7 100644 --- a/.codex/agents/modelopt_model_evaluator.toml +++ b/.codex/agents/modelopt_model_evaluator.toml @@ -11,7 +11,7 @@ Before acting, load these Model Optimizer instructions: - `accessing-mlflow/SKILL.md` when runs or artifacts are in MLflow - `common/workspace-management.md` -Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Aggregate Model-Output-Fault Policy per benchmark/run: report category counts, their deduplicated union, the verified unique-successful-response denominator, and the unrounded rate. A rate ≤2.0% is a visible warning only when affected raw responses are preserved and scored incorrect, coverage is complete, and every independent gate passes. Above tolerance, return findings and a recommendation to the parent without automatically resubmitting. Never tune token limits merely to pass. +Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Bounded Evaluation-Failure Policy per benchmark/run: verify complete coverage and protocol-valid failed scoring, and report separate deduplicated response/trial counts, denominators, and unrounded rates. Both applicable gates must be ≤2.0%; retain nonzero warnings and raw evidence privately. Missing/unscored trials and independent security, identity, provenance, or scoring failures remain blockers. Above tolerance, return findings for parent review without automatic retry or tuning. Never change benchmark limits merely to pass. Keep score, validation, and MLflow delivery separate. Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. """ diff --git a/AGENTS.md b/AGENTS.md index 8fb0422dfa1..eda2e28e5f9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,10 +17,10 @@ These instructions apply to AI-assisted work in this repository. Before analyzing completed evaluations or reporting scores, including standalone `launching-evals` sessions, read `plugins/modelopt/skills/evaluation/references/run-validation.md`. -Its aggregate model-output-fault policy supersedes vendored `launching-evals` +Its bounded evaluation-failure policy supersedes vendored `launching-evals` advice to uncap reasoning output (`max_tokens=null` or payload field removal), -use fixed token/context limits, or automatically invalidate a preserved, -successful response that is retained and scored incorrect. +use fixed token/context limits, or automatically invalidate a preserved +response or protocol-valid failed trial that is retained and scored incorrect. ## Coding guidelines diff --git a/plugins/modelopt/agents/modelopt-model-evaluator.md b/plugins/modelopt/agents/modelopt-model-evaluator.md index 9291c6f4f7e..396512b0cd3 100644 --- a/plugins/modelopt/agents/modelopt-model-evaluator.md +++ b/plugins/modelopt/agents/modelopt-model-evaluator.md @@ -16,6 +16,6 @@ Before acting, load these Model Optimizer instructions: - `accessing-mlflow/SKILL.md` when runs or artifacts are in MLflow - `common/workspace-management.md` -Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Aggregate Model-Output-Fault Policy per benchmark/run: report category counts, their deduplicated union, the verified unique-successful-response denominator, and the unrounded rate. A rate ≤2.0% is a visible warning only when affected raw responses are preserved and scored incorrect, coverage is complete, and every independent gate passes. Above tolerance, return findings and a recommendation to the parent without automatically resubmitting. Never tune token limits merely to pass. +Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Bounded Evaluation-Failure Policy per benchmark/run: verify complete coverage and protocol-valid failed scoring, and report separate deduplicated response/trial counts, denominators, and unrounded rates. Both applicable gates must be ≤2.0%; retain nonzero warnings and raw evidence privately. Missing/unscored trials and independent security, identity, provenance, or scoring failures remain blockers. Above tolerance, return findings for parent review without automatic retry or tuning. Never change benchmark limits merely to pass. Keep score, validation, and MLflow delivery separate. Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. diff --git a/plugins/modelopt/skills/day0-release/scripts/gate_run.py b/plugins/modelopt/skills/day0-release/scripts/gate_run.py index d5dcbe94a70..408383819aa 100644 --- a/plugins/modelopt/skills/day0-release/scripts/gate_run.py +++ b/plugins/modelopt/skills/day0-release/scripts/gate_run.py @@ -28,10 +28,14 @@ "expected_samples": int, "scored_samples": int, "score": float | null, # canonical score, if extracted - "errors": [str, ...] # judge/parse/sample errors, if any + "errors": [str, ...], # unresolved validation blockers + "warnings": [str, ...] # policy-validated failure diagnostics } } } +Apply run-validation.md's bounded failure policy before constructing this +summary; this gate does not verify failure rates or raw evidence. Preserve +accepted failures in warnings and their original artifacts, never erase blockers. Only a terminal SUCCESS with complete, numeric scores passes. Non-terminal statuses (RUNNING/PENDING/TIMEOUT/RESUMING) do NOT pass — the run hasn't finished — but they classify as INFRA_TRANSIENT (wait for NEL to resume/finish; diff --git a/plugins/modelopt/skills/day0-release/tests/test_gates.py b/plugins/modelopt/skills/day0-release/tests/test_gates.py index b8217883e17..de88fc971ea 100644 --- a/plugins/modelopt/skills/day0-release/tests/test_gates.py +++ b/plugins/modelopt/skills/day0-release/tests/test_gates.py @@ -151,6 +151,21 @@ def test_run_judge_error(): assert not r["pass"] and r["failure_class"] == "EVAL_JUDGE_FAILED" +def test_run_validated_warnings_do_not_waive_blockers(): + task = _task(warnings=["protocol-valid scored timeout; trial rate 1/100 = 1%"]) + assert evaluate_run({"tasks": {"t": task}})["pass"] + for blockers in ( + {"scored_samples": 99}, + {"score": None}, + {"errors": ["unscored harness crash"]}, + {"errors": ["secret leak"]}, + {"errors": ["wrong model/task/version/config"]}, + {"errors": ["cross-invocation contamination; no provenance"]}, + {"errors": ["systemic broken scoring"]}, + ): + assert not evaluate_run({"tasks": {"t": task | blockers}})["pass"] + + def test_run_missing_score(): r = evaluate_run({"tasks": {"gpqa": _task(score=None)}}) assert not r["pass"] and r["failure_class"] == "SAMPLE_ACCOUNTING_FAILED" diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 6b86ebe6c2f..5477a81502b 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -130,17 +130,16 @@ Read [Step 8](references/launcher-workflow.md#step-8--run-evaluation-gated-dry-r ### Step 9 — Verify completed run -Read [run-validation.md](references/run-validation.md) before reporting scores: -validate logs and coverage, complete **Timeout and Output-Limit Accounting** for -every task, and report missing telemetry as unknown. Parents and evaluators must -also apply its **Aggregate Model-Output-Fault Policy** per benchmark/run: report -category counts, their deduplicated union, the verified unique-successful-response -denominator, and the unrounded rate. At ≤2.0%, warn rather than invalidate or -retry only when raw responses are preserved and scored incorrect, coverage is -complete, and all independent gates pass. Above tolerance, return findings and -a recommendation without automatically resubmitting. Never tune token limits -merely to pass. For comparisons, also apply its **External Baseline Sanity -Check**, then use `compare-results`. +Read [run-validation.md](references/run-validation.md) before reporting scores. +Parents and evaluators must apply its **Bounded Evaluation-Failure Policy**: +verify complete coverage, protocol-valid failed scoring, and separate deduplicated +response/trial rates. Both applicable gates must be ≤2.0% before rounding; warn +for nonzero rates. Missing/unscored trials and independent integrity failures +remain blockers. Above tolerance, return findings for parent review without +automatic retry or tuning. Preserve raw evidence privately and report timeout, +limit, and concurrency settings. Keep score, validation, and MLflow delivery +separate. For comparisons, apply its **External Baseline Sanity Check**, then +use `compare-results`. --- diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index 9c58bb50e38..de7bf94e833 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -6,7 +6,7 @@ # "$SKILL_DIR/scripts/nel-next.sh" --setup-only # set -a && source .env && set +a # HF_TOKEN, AWS_*, NEL_NEXT_EVAL_IMAGE, HARBOR_*_ECR_REPOSITORY (from modelopttools:eval-config) # "$SKILL_DIR/scripts/nel-next.sh" eval run "$SKILL_DIR/recipes/examples/example_eval_next.yaml" --dry-run -# ... --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 # canary +# ... --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 -O benchmarks.0.sandbox.concurrency=2 # canary # ... --submit # full # Internal harbor infra (eval_image + ECR) comes from .env via ${VAR}; all blocks # validate against the pinned nel-next build (extra="forbid"). `???` = fill. @@ -68,7 +68,7 @@ services: benchmarks: - playbook: terminal_bench_2_1 # SWE-bench: swebench_verified (see recipe) repeats: 8 # AA count — keep for a scored run (canary: -O benchmarks.0.repeats=1) - max_concurrent: 50 # canonical TB2.1 (nano-class; larger models 15); keep == sandbox.concurrency + max_concurrent: 8 # conservative per shard; keep == sandbox.concurrency solver: service: model timeout_strategy: max # canonical TB2.1; "task" = leaderboard-comparable @@ -78,7 +78,7 @@ benchmarks: max_task_lifetime_sec: 21600 # ModelOpt 6h: agent budget plus setup/verification region: ${HARBOR_ECS_REGION:-us-east-1} # SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY} ecr_repository: ${HARBOR_ECR_REPOSITORY} # from modelopttools:eval-config - concurrency: 50 + concurrency: 8 log_stream_prefix: terminalbench-21-??? cluster: diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa/scicode.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa/scicode.md index 29f2afce54b..d581ec7974e 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa/scicode.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa/scicode.md @@ -44,7 +44,8 @@ when you have them, never fewer. - **Report the mean** of the n pooled runs, with the **sample** stdev (n-1 denominator) over `sqrt(n)` as the standard error, and n itself. Validate each run (`references/run-validation.md`) before averaging it in; - resubmit to replace invalid runs rather than pooling a sandbox-crashed one. + bounded protocol-valid sandbox/verifier failures may pass with warnings. Do not + pool incomplete/unscored runs; return blockers for parent review before reruns. - **Expect an export to time out.** The exports land on the CPU partition together and each reinstalls the launcher against a fixed 30 min sbatch limit. The score survives in the run artifacts — re-submit that run's diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index 5d0289f54c9..4ede582e71a 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -21,7 +21,7 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-1}` | | `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by `modelopttools:eval-config`; repo name tracks the region (`harbor-`) | | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | -| `max_concurrent` / `sandbox.concurrency` | `50` (canonical, nano-class); **`15` for larger models** (upstream non-nano leaves). Keep both equal and fixed across baseline/candidate (see Sharding) | +| `max_concurrent` / `sandbox.concurrency` | `8` per shard (conservative starting point); keep both equal and fixed across comparisons | | timeout_strategy | `max` + `run_timeout: 14400` (ModelOpt default); use `task` for leaderboard-comparable | | `cluster.eval_image` | **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) *(shared — see `references/nel-next.md`)* | | `proxy.request_timeout` | `3600` — set explicitly (TB2.1 has no benchmark key for it); must be **≥** `agent_kwargs.llm_kwargs.timeout` *(shared — see `references/nel-next.md`)* | @@ -32,7 +32,7 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | `output.export_config.mlflow.tags` | `task_name: terminal-bench-2.1` | | scope | 89 tasks × `repeats: 8` | -Except for the ModelOpt timeout/lifetime overrides below, these values follow +Except for concurrency and the ModelOpt timeout/lifetime overrides below, these values follow the canonical TB2.1 config — re-check it before a scored run: `configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the image pin in @@ -44,7 +44,7 @@ nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the benchmarks: - playbook: terminal_bench_2_1 repeats: 8 - max_concurrent: 50 # nano-class; LARGE models use 15. Keep == sandbox.concurrency + max_concurrent: 8 # conservative per shard; keep == sandbox.concurrency solver: service: timeout_strategy: max # canonical bench.yaml; use "task" for leaderboard-comparable @@ -56,7 +56,7 @@ benchmarks: max_task_lifetime_sec: 21600 # 6h: allow agent execution plus setup/verification region: ${HARBOR_ECS_REGION:-us-east-1} # repo name below tracks it ecr_repository: ${HARBOR_ECR_REPOSITORY} # from eval-config (internal harbor account/region) - concurrency: 50 + concurrency: 8 log_stream_prefix: terminalbench-21-- ``` @@ -81,6 +81,18 @@ intended trial set. Sharding or concurrency changes can still affect scores when serving speed or queueing changes timeout rates. `shards: 4` suits 89 × r8 = 712 trials. Check `N × concurrency` against the Fargate quota and `N × gpus_per_node` against your allocation. +Use `8` as a conservative starting point, not a benchmark-mandated value. Model +endpoints are shared by trials within each shard. Reduce concurrency for capacity +only with latency/queueing evidence; match concurrency and effective settings +across comparisons. Keep benchmark-prescribed limits, temperature, top_p, and +output settings fixed; never raise timeouts merely to improve scores. This default +change has not been benchmarked and does not establish an improvement. + +Apply `references/run-validation.md`'s bounded policy before reporting scores: +solver/harness timeouts and terminal transport/action errors may count only when +recorded and scored as protocol-valid failures. Keep all expected trials and +repeats, including zero scores; incomplete coverage remains a blocker. + ## Score Extraction Before reporting `pass@1`, complete the evaluation skill's diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 14f4420aa06..de2bb3cbfe5 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -225,7 +225,7 @@ with its own `run_id`, copying the shared `services:` block. ```bash set -a && source .env && set +a; NEL="$SKILL_DIR/scripts/nel-next.sh" "$NEL" eval run .yaml --dry-run # validate/render (no SSH) -"$NEL" eval run .yaml --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 # canary +"$NEL" eval run .yaml --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 -O benchmarks.0.sandbox.concurrency=2 # canary "$NEL" eval run .yaml --submit # full "$NEL" eval {status|logs -f|report -f markdown|merge} -r # lifecycle "$NEL" mlflow-push -r -c .yaml # post-run: push merged bundle(s) to MLflow @@ -243,7 +243,9 @@ kwargs after `--` (e.g. `-- -o copy_artifacts=true`). ## Non-fatal noise - AWS `RegisterTaskDefinition: ThrottlingException` at high concurrency — SDK - retries (≤8); lower `max_concurrent` if frequent. + retries (≤8); lower matched `max_concurrent`/`sandbox.concurrency` only with capacity evidence. - `solver failed — grading 0.0` agent crashes are rare but real (e.g. terminus-2 tmux `send-keys`) — distinguish from genuine misses + task timeouts; the final - report breaks them out. + report breaks them out. Apply `run-validation.md`'s bounded failure policy: + verify protocol-valid scored failures and complete trial accounting. A harness + crash leaving missing/unscored trials is a blocker, not a zero score. diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index 604e358cff0..e59f4351159 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -25,10 +25,10 @@ For each completed invocation/run directory, whether baseline, quantized, or a single-model run: 1. Inspect client, server/deployment, SLURM, judge, and task-specific/code-execution logs as applicable. Search for `Traceback`, `Exception`, `ERROR`, `FAILED`, `OOM`, `Killed`, `timeout`, `rate limit`, `unauthorized`, `connection refused/reset`, `health check`, `sandbox`, `container`, `judge`, `parse`, `scoring`, and task-specific failure strings. -2. Confirm the inference server loaded the intended checkpoint/model and stayed healthy through the run: no startup failure, mid-run crash/restart, OOM, request validation failure, input/context clipping, quantization load error, or repeated 4xx/5xx responses. -3. For judge-backed tasks, confirm judge calls succeeded and were parsed/scored correctly: no auth/rate-limit failures, malformed judge responses, invalid JSON, missing scores, or fallback/default scores. -4. For code-execution tasks, inspect executor/sandbox/container logs for setup failures, package install failures, timeouts, thread/process exhaustion, permission errors, harness crashes, or skipped tests that would make scores non-comparable. -5. Confirm coverage separately at every available level: expected, selected, evaluated, and scored samples/repeats/trajectories must match. No response may be missing or unscored, and no unexpected dropped/skipped/failed sample, `unknown_agent_error`, `failed_samples_policy` abort, or partial result file is allowed. +2. Verify the intended checkpoint loaded; inspect startup failures, crashes/restarts, OOMs, request errors, input/context clipping, and quantization load errors. Classify attributable per-trial failures under the policy below; systemic serving failures remain blockers. +3. For judge-backed tasks, inspect authentication, rate limits, malformed responses, parsing, and scoring. Distinguish protocol-valid failed scores from missing scores or undocumented fallback/default scores. +4. For code-execution tasks, inspect executor/sandbox/container setup, installation, timeout, resource, permission, harness, and skipped-test diagnostics. Verify each failed trial's protocol-defined outcome rather than treating every error as invalid. +5. Confirm coverage at every available level: expected, selected, accounted-for, and scored samples/repeats/trajectories must match. Explicit protocol-valid failed trials count as accounted-for; missing/unscored trials, dropped submissions, `failed_samples_policy` aborts leaving gaps, and partial results do not. 6. If reasoning traces are present, confirm they are parsed/stripped/ignored before scoring consistently. Assess unusable model outputs under the policy below. Check for parser or fallback errors, unmatched reasoning delimiters, reasoning text leaked into answers, answers stripped with the reasoning, or reasoning disabled when the config intended it to be active. 7. Complete the **Timeout and Output-Limit Accounting** below for every task, including non-reasoning models and successful runs. @@ -39,11 +39,13 @@ independent validation item fails, label the result incomplete or invalid and return findings and a recommendation to the parent (or user); do not automatically resubmit a completed run. -### Aggregate Model-Output-Fault Policy (Parent and Evaluator) +### Bounded Evaluation-Failure Policy (Parent and Evaluator) For each benchmark/run, report category counts, their deduplicated union, the -verified denominator, and `rate = 100 × union / denominator`. Calculate with -unrounded counts; round only the displayed percentage. +verified denominators, and `rate = 100 × union / denominator` for each applicable +gate below. Calculate with unrounded counts; round only the displayed percentage. + +**Response gate (model-output faults).** Preserve unique-response accounting: The denominator is the deduplicated set of unique, successful raw evaluated-model responses selected for evaluation. Count a cached response reused by multiple @@ -62,34 +64,65 @@ model output behavior: Count a response in every applicable category for reporting, but once in the union. To qualify, the successful raw response must exist, remain preserved in the artifacts, and be deterministically retained and scored incorrect. A parser -exception, parser fallback/default score, missing parsed record, or scorer/harness -failure is not automatically a model-output fault; it remains invalid unless the -preserved raw response's malformed answer is explicitly retained and scored -incorrect under the benchmark protocol. - +exception, parser fallback/default score, or missing parsed record is not +automatically a model-output fault. A malformed answer qualifies only if the +preserved raw response is explicitly retained and scored incorrect under the +benchmark protocol. Runtime scorer/harness failures require trial-gate assessment +and protocol-valid scoring instead. + +**Trial gate (runtime failures).** Use the expected benchmark trial/repeat count, +not successful trials, HTTP attempts, or the number of records found. Its numerator +is the deduplicated union of attributable terminal request/transport/server, +judge, executor/sandbox/verifier, terminal action, solver, and harness failures, +including timeouts, that the benchmark explicitly records and scores incorrect +or zero under its existing protocol. A failed request without a raw response may +qualify here only through such a scored trial; it cannot enter the response gate. +Ordinary wrong answers are not runtime failures. + +Preserve diagnostics and original raw evidence privately, including failed +submissions. Verify failure type and scoring from structured artifacts and logs; +an exception string or a summary zero alone is insufficient. Deduplicate overlaps +and resumes by invocation, benchmark, task, trial, and repeat identity. Count +overlapping categories once within each gate; never add response and trial rates +or pool benchmarks. Both applicable gates must pass. A shared fault may affect +both gates, but is counted once within each. Mark an inapplicable gate explicitly; +an unknown applicable denominator is not an exemption. + +Continue other trials after a harness crash only when the harness can record the +affected trial as a protocol-valid scored failure. If it cannot, report a blocker. +Do not fabricate zeros, modify scoring semantics, treat missing/unscored trials +as completed, select passing repetitions, or silently regrade old runs. Keep the +original trial and score denominators; omitted zero-scored failures leave coverage +incomplete. Never average incomplete data. + +- **rate = 0%:** no failure warning for that gate; independent checks still apply. - **0 < rate ≤ 2.0% (inclusive, before rounding):** valid with a visible warning - when expected/selected/evaluated/scored coverage is complete and all independent - gates pass. Retain the affected responses and their incorrect scores; do not - invalidate or retry solely for these faults. A low rate does **not** imply - negligible score impact. + only when coverage is complete and all independent gates pass. Retain failures + and their protocol-defined scores; do not invalidate, abort the invocation, or + retry solely for bounded failures. A low rate does **not** imply negligible + score impact or leaderboard comparability. - **rate > 2.0%:** return category counts, union, rate, findings, and a recommendation to the parent (or user); do not automatically retry or declare success. -- The tolerance never waives missing or unscored responses; exhausted request, - transport, server, or authentication failures; incomplete coverage; parser, - harness, sandbox, or scorer failures; wrong benchmark, model, configuration, or - version; secret issues; input/context clipping; benchmark-specific validity - rules; or an unverified denominator. A score above a reference does not establish - validity. +- The tolerance never waives secret leaks (scanning remains fail-closed), wrong + model/task/version/configuration, falsified or provenance-less output, + cross-invocation contamination, input/context clipping, systemic serving or + broken scoring, incomplete coverage, or benchmark-specific validity rules. + These are independent integrity failures, not ordinary per-trial failures. + Undocumented parser/judge fallback scores remain invalid. A score above a + reference does not establish validity. - Check configured and effective output limits against the reference evaluation protocol and deployed context capacity, including prompt/history plus output space. Report mismatches or unavailable reference settings. Do not remove or tune token limits merely to pass; propose protocol-justified changes for parent/user approval instead. -The parent must preserve the category and rate warning and apply the same policy -to evaluator handoffs rather than treating any nonzero model-output fault as a -failed run. +Keep score, run validation, and MLflow delivery separate: a valid score does not +prove export succeeded, and export cannot validate scoring. In a day0 run summary, +keep policy-validated failure diagnostics in `warnings`; `errors` contains +unresolved blockers and still fails `gate_run.py`. That summary gate does not +compute these rates or verify raw evidence; perform this policy check first. +The parent must retain both gates' counts, rates, and warnings in handoffs. ## Timeout and Output-Limit Accounting @@ -111,12 +144,13 @@ Missing termination metadata, sampled-only artifacts, or omitted failures make full-run rates **unknown**, not zero. Report coverage and what evidence is missing; do not extrapolate sampled rates. -- **Model-output faults:** apply the aggregate policy above. Output-limit stops - qualify only when the response is preserved and scored incorrect; terminal - request, task, judge, sandbox, or harness timeouts remain invalid. +- **Terminal failures:** apply the bounded policy above to output faults and + protocol-valid scored runtime/trial failures, including harness timeouts. + Report timeout type, effective limits, concurrency/queueing, and protocol + differences; tolerance alone does not establish comparability. - **Recovered retries:** report them separately. They do not enter the fault numerator or denominator as extra responses, and they do not excuse a terminal - failure or missing coverage. + unscored failure or missing coverage. - **Provisional/inconclusive:** unknown accounting leaves validation incomplete. Investigate infrastructure failures, unexpected exclusions, or mismatched limits before a model-quality verdict. A valid run alone does not establish diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 9670479c63d..7d8b965f041 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -1,4 +1,65 @@ [ + { + "name": "runtime-trial-failure-boundaries", + "skills": ["evaluation"], + "query": "Classify four independent synthetic benchmark runs with 10000 expected and protocol-scored trials each. Their attributable terminal failures are respectively 0, 100, 200, and 201 unique trials; the benchmark records and scores each failure zero. Failures include solver/harness timeouts, terminal transport/action errors, exhausted requests, judge and sandbox/verifier errors. Original artifacts and diagnostics are preserved privately. Coverage and independent gates pass; the applicable response gate is 0/12000 for each run. A fifth complete run has 20001 failures of 1000000 expected scored trials; its rate displays as 2.00% after rounding. Do not launch jobs or change settings.", + "files": [], + "expected_behavior": [ + "Reports separate trial rates 0%, 1%, 2%, and 2.01% using expected trials, without pooling benchmarks or adding response and trial rates", + "Accepts zero without a failure warning and 1% and exactly 2% with visible warnings, retaining all failures and original score denominators", + "Above 2% returns findings for parent review without automatic retry, tuning, or success; 20001/1000000 = 2.0001% fails before rounding", + "Does not equate bounded failure rates with negligible score impact or leaderboard comparability" + ] + }, + { + "name": "runtime-overlap-resume-and-both-gates", + "skills": ["evaluation"], + "query": "Review runs A and B, each with 1000 expected scored trials and 10000 unique successful model responses. In A, solver timeouts affect trial IDs 1-12, harness timeouts 9-16, and terminal action errors 15-20. Duplicate resumed records repeat those IDs. All failures are protocol-valid zeros, retained in the merged score and raw artifacts. A also has 201 unique malformed model responses scored incorrect. B has no model-output faults but 21 unique protocol-valid runtime failures. Both have complete coverage and independent gates pass. Should either pass? Do not launch jobs.", + "files": [], + "expected_behavior": [ + "A category counts are 12, 8, 6; resumed overlaps deduplicate to 20/1000 = 2% trial failures, with response rate 201/10000 = 2.01%", + "B trial rate is 21/1000 = 2.1% even though its response rate is zero", + "Neither passes because both applicable gates must pass; rates with different denominators are never added", + "Retains warnings and returns above-tolerance findings to the parent without retry" + ] + }, + { + "name": "runtime-coverage-and-integrity-blockers", + "skills": ["evaluation"], + "query": "Six runs claim only 1% timeout failures. A expected 1000 trials but a harness crash left 999 scored and one unscored. B has 1000 records but its report omitted ten zero-scored failures and averaged only passing repetitions. C inserted fabricated zeros for missing trials. D has a secret leak. E used the wrong model/task/version/config and cross-invocation records. F has falsified or provenance-less output and systemic broken scoring. Can any pass? Do not rerun or silently regrade.", + "files": [], + "expected_behavior": [ + "Blocks every run independently of apparent rate: missing/unscored coverage, omitted zeros/selected repetitions, fabricated scoring, secrets, identity/contamination, and provenance/systemic scoring", + "Keeps secret scanning fail-closed and original raw evidence private", + "Requires original expected trial and score denominators, not observed records or successful-only trials", + "Reports blockers rather than fabricating scores, averaging incomplete data, or retroactively declaring success" + ] + }, + { + "name": "harness-continuation-and-delivery-separation", + "skills": ["evaluation"], + "query": "In synthetic run A a harness action crashed on one trial of 100 expected; the harness retained diagnostics and recorded a protocol-valid zero. It can continue the other trials safely under unchanged scoring. Run B has the same crash but no protocol-valid failure record or score. After A completes all 100 scored trials its MLflow export fails. What should the evaluator and parent do? No job submissions authorized.", + "files": [], + "expected_behavior": [ + "Allows continuation of A without aborting solely for a 1% failure, conditional on eventual complete scored coverage and both applicable tolerance gates passing", + "B is blocked; does not invent zero or patch scoring semantics", + "A completed score may be valid with a warning while MLflow delivery remains failed; export success is an independent handoff requirement", + "For day0 summaries retains accepted diagnostics in warnings, keeps unresolved blockers in errors, and does not treat gate_run.py as raw-evidence/rate validation" + ] + }, + { + "name": "terminal-bench-concurrency-eight-and-protocol", + "skills": ["evaluation"], + "query": "Prepare guidance only for a future Terminal-Bench 2.1 comparison with four shards. The former template used concurrency 50 and a later run used 15 per shard; lower the starting concurrency to 8. Can this be called a proven score improvement, or should timeouts also be raised? No canary, Harbor, or benchmark rerun is authorized.", + "files": [], + "expected_behavior": [ + "Reads the canonical recipe/example and sets max_concurrent and sandbox.concurrency equally to 8 per shard; four shards permit 32 simultaneous trials", + "Calls 8 a conservative starting point, not a benchmark mandate or proven improvement; endpoints are shared within each shard", + "Requires latency/queueing evidence for capacity changes and matches effective concurrency, timeout type/limits, temperature/top_p/output settings across comparisons", + "Keeps benchmark-prescribed limits fixed, does not raise timeouts to improve scores, and surfaces protocol/leaderboard differences", + "Does not submit a rerun or claim live validation" + ] + }, { "name": "standalone-kimi-scicode-model-output-fault", "skills": ["launching-evals"], @@ -27,11 +88,11 @@ { "name": "model-output-tolerance-does-not-waive-independent-failures", "skills": ["evaluation", "launching-evals"], - "query": "Review five separate runs whose apparent model-output-fault rate is below 2%. A selected 1000 responses but evaluated and scored only 999. B exhausted an HTTP/server request and has no successful raw response for one item. C raised a parser exception and assigned a fallback score without retaining the malformed answer as an explicit incorrect result; its harness also failed. D used the wrong benchmark version, model, and config. E has an authentication/secret failure and no verifiable response denominator. All reported scores exceed the reference. Can the parent accept any under the tolerance?", + "query": "Review five separate runs whose apparent model-output-fault rate is below 2%. A selected 1000 responses but evaluated and scored only 999. B exhausted an HTTP/server request and has no successful raw response for one item and no protocol-valid scored trial. C raised a parser exception and assigned a fallback score without retaining the malformed answer as an explicit incorrect result; its harness also failed without recording a protocol-valid scored trial. D used the wrong benchmark version, model, and config. E has a secret leak and no verifiable response denominator. All reported scores exceed the reference. Can the parent accept any under the tolerance?", "files": [], "expected_behavior": [ "Marks every run invalid rather than using the model-output tolerance", - "Separately identifies missing/unscored coverage, exhausted HTTP/server request, parser fallback and harness failure, wrong protocol identity, authentication/secret failure, and unverified denominator", + "Separately identifies missing/unscored coverage, exhausted HTTP/server request, parser fallback and harness failure, wrong protocol identity, secret leak, and unverified denominator", "Does not reclassify C as a model-output fault because no preserved successful raw malformed answer was deterministically retained and scored incorrect", "Returns findings and recommendations without automatically retrying; exceeding the reference is not proof of validity" ] diff --git a/plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py b/plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py new file mode 100644 index 00000000000..2827eb5ba87 --- /dev/null +++ b/plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Check guidance/config consistency; these tests do not run agents or scoring.""" + +import json +import re +from pathlib import Path + +import yaml + +_SKILL = Path(__file__).resolve().parents[1] +_ROOT = _SKILL.parents[3] + + +def test_terminal_bench_concurrency_matches_recipe(): + template = yaml.safe_load((_SKILL / "recipes/examples/example_eval_next.yaml").read_text()) + recipe = (_SKILL / "recipes/tasks/aa_next/terminal_bench_2_1.md").read_text() + yaml_block = re.search(r"```yaml\n(.*?)```", recipe, re.DOTALL) + assert yaml_block is not None + fragment = yaml.safe_load(yaml_block.group(1)) + for config in (template, fragment): + benchmark = config["benchmarks"][0] + assert benchmark["max_concurrent"] == benchmark["sandbox"]["concurrency"] == 8 + assert benchmark["repeats"] == 8 + + +def test_failure_policy_is_reachable_from_parent_and_evaluators(): + policy_path = "evaluation/references/run-validation.md" + policy = (_SKILL / "references/run-validation.md").read_text() + title = "Bounded Evaluation-Failure Policy" + assert f"### {title}" in policy + assert "**Response gate" in policy and "**Trial gate" in policy + assert "Both applicable gates must pass" in policy + assert policy_path in (_ROOT / "AGENTS.md").read_text() + for path in ( + _SKILL / "SKILL.md", + _ROOT / ".codex/agents/modelopt_model_evaluator.toml", + _ROOT / "plugins/modelopt/agents/modelopt-model-evaluator.md", + ): + assert title in path.read_text() + + +def test_evaluation_scenarios_reference_existing_skills(): + scenarios = json.loads((_SKILL / "tests/evals.json").read_text()) + names = [scenario["name"] for scenario in scenarios] + assert len(names) == len(set(names)) + for scenario in scenarios: + assert scenario["query"] and scenario["expected_behavior"] + for skill in scenario["skills"]: + assert (_SKILL.parent / skill / "SKILL.md").is_file() From e4522f21653684a1b7b3d479625d4953529e5310 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Fri, 2 Oct 2026 20:43:42 +0000 Subject: [PATCH 3/8] docs(eval): isolate bounded failure policy from concurrency changes Signed-off-by: Chad Voegele --- .codex/agents/modelopt_model_evaluator.toml | 2 +- .../agents/modelopt-model-evaluator.md | 2 +- .../recipes/examples/example_eval_next.yaml | 6 +- .../tasks/aa_next/terminal_bench_2_1.md | 15 +- .../skills/evaluation/references/nel-next.md | 4 +- .../skills/evaluation/tests/evals.json | 198 ------------------ .../evaluation/tests/test_failure_guidance.py | 63 ------ 7 files changed, 11 insertions(+), 279 deletions(-) delete mode 100644 plugins/modelopt/skills/evaluation/tests/evals.json delete mode 100644 plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py diff --git a/.codex/agents/modelopt_model_evaluator.toml b/.codex/agents/modelopt_model_evaluator.toml index 86478b8cff7..ad99a342d4b 100644 --- a/.codex/agents/modelopt_model_evaluator.toml +++ b/.codex/agents/modelopt_model_evaluator.toml @@ -11,7 +11,7 @@ Before acting, load these Model Optimizer instructions: - `accessing-mlflow/SKILL.md` when runs or artifacts are in MLflow - `common/workspace-management.md` -Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Bounded Evaluation-Failure Policy per benchmark/run: verify complete coverage and protocol-valid failed scoring, and report separate deduplicated response/trial counts, denominators, and unrounded rates. Both applicable gates must be ≤2.0%; retain nonzero warnings and raw evidence privately. Missing/unscored trials and independent security, identity, provenance, or scoring failures remain blockers. Above tolerance, return findings for parent review without automatic retry or tuning. Never change benchmark limits merely to pass. Keep score, validation, and MLflow delivery separate. +Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. """ diff --git a/plugins/modelopt/agents/modelopt-model-evaluator.md b/plugins/modelopt/agents/modelopt-model-evaluator.md index 396512b0cd3..592770f11f2 100644 --- a/plugins/modelopt/agents/modelopt-model-evaluator.md +++ b/plugins/modelopt/agents/modelopt-model-evaluator.md @@ -16,6 +16,6 @@ Before acting, load these Model Optimizer instructions: - `accessing-mlflow/SKILL.md` when runs or artifacts are in MLflow - `common/workspace-management.md` -Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Apply `evaluation/references/run-validation.md`'s Bounded Evaluation-Failure Policy per benchmark/run: verify complete coverage and protocol-valid failed scoring, and report separate deduplicated response/trial counts, denominators, and unrounded rates. Both applicable gates must be ≤2.0%; retain nonzero warnings and raw evidence privately. Missing/unscored trials and independent security, identity, provenance, or scoring failures remain blockers. Above tolerance, return findings for parent review without automatic retry or tuning. Never change benchmark limits merely to pass. Keep score, validation, and MLflow delivery separate. +Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. diff --git a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml index de7bf94e833..9c58bb50e38 100644 --- a/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml +++ b/plugins/modelopt/skills/evaluation/recipes/examples/example_eval_next.yaml @@ -6,7 +6,7 @@ # "$SKILL_DIR/scripts/nel-next.sh" --setup-only # set -a && source .env && set +a # HF_TOKEN, AWS_*, NEL_NEXT_EVAL_IMAGE, HARBOR_*_ECR_REPOSITORY (from modelopttools:eval-config) # "$SKILL_DIR/scripts/nel-next.sh" eval run "$SKILL_DIR/recipes/examples/example_eval_next.yaml" --dry-run -# ... --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 -O benchmarks.0.sandbox.concurrency=2 # canary +# ... --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 # canary # ... --submit # full # Internal harbor infra (eval_image + ECR) comes from .env via ${VAR}; all blocks # validate against the pinned nel-next build (extra="forbid"). `???` = fill. @@ -68,7 +68,7 @@ services: benchmarks: - playbook: terminal_bench_2_1 # SWE-bench: swebench_verified (see recipe) repeats: 8 # AA count — keep for a scored run (canary: -O benchmarks.0.repeats=1) - max_concurrent: 8 # conservative per shard; keep == sandbox.concurrency + max_concurrent: 50 # canonical TB2.1 (nano-class; larger models 15); keep == sandbox.concurrency solver: service: model timeout_strategy: max # canonical TB2.1; "task" = leaderboard-comparable @@ -78,7 +78,7 @@ benchmarks: max_task_lifetime_sec: 21600 # ModelOpt 6h: agent budget plus setup/verification region: ${HARBOR_ECS_REGION:-us-east-1} # SWE-bench: us-east-2 + ${HARBOR_SWEBENCH_ECR_REPOSITORY} ecr_repository: ${HARBOR_ECR_REPOSITORY} # from modelopttools:eval-config - concurrency: 8 + concurrency: 50 log_stream_prefix: terminalbench-21-??? cluster: diff --git a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md index 4ede582e71a..08e60b1f6bc 100644 --- a/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md +++ b/plugins/modelopt/skills/evaluation/recipes/tasks/aa_next/terminal_bench_2_1.md @@ -21,7 +21,7 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | `sandbox.region` | `${HARBOR_ECS_REGION:-us-east-1}` | | `sandbox.ecr_repository` | `${HARBOR_ECR_REPOSITORY}` — set by `modelopttools:eval-config`; repo name tracks the region (`harbor-`) | | `cluster.container_env.AWS_DEFAULT_REGION` | match `sandbox.region` | -| `max_concurrent` / `sandbox.concurrency` | `8` per shard (conservative starting point); keep both equal and fixed across comparisons | +| `max_concurrent` / `sandbox.concurrency` | `50` (canonical, nano-class); **`15` for larger models** (upstream non-nano leaves). Keep both equal and fixed across baseline/candidate (see Sharding) | | timeout_strategy | `max` + `run_timeout: 14400` (ModelOpt default); use `task` for leaderboard-comparable | | `cluster.eval_image` | **`0.5.0.1-harbor`** (`${NEL_NEXT_EVAL_IMAGE}`, multi-arch) *(shared — see `references/nel-next.md`)* | | `proxy.request_timeout` | `3600` — set explicitly (TB2.1 has no benchmark key for it); must be **≥** `agent_kwargs.llm_kwargs.timeout` *(shared — see `references/nel-next.md`)* | @@ -32,7 +32,7 @@ pinned via a vendored registry override shipped in the `nemo-evaluator` package. | `output.export_config.mlflow.tags` | `task_name: terminal-bench-2.1` | | scope | 89 tasks × `repeats: 8` | -Except for concurrency and the ModelOpt timeout/lifetime overrides below, these values follow +Except for the ModelOpt timeout/lifetime overrides below, these values follow the canonical TB2.1 config — re-check it before a scored run: `configs/benchmarks/terminal-bench-2.1/bench.yaml` (+ `manifest.yaml`) in nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the image pin in @@ -44,7 +44,7 @@ nvidia-eval-factory-benchmarking (`dl/JoC/competitive_evaluation/…`), with the benchmarks: - playbook: terminal_bench_2_1 repeats: 8 - max_concurrent: 8 # conservative per shard; keep == sandbox.concurrency + max_concurrent: 50 # nano-class; LARGE models use 15. Keep == sandbox.concurrency solver: service: timeout_strategy: max # canonical bench.yaml; use "task" for leaderboard-comparable @@ -56,7 +56,7 @@ benchmarks: max_task_lifetime_sec: 21600 # 6h: allow agent execution plus setup/verification region: ${HARBOR_ECS_REGION:-us-east-1} # repo name below tracks it ecr_repository: ${HARBOR_ECR_REPOSITORY} # from eval-config (internal harbor account/region) - concurrency: 8 + concurrency: 50 log_stream_prefix: terminalbench-21-- ``` @@ -81,13 +81,6 @@ intended trial set. Sharding or concurrency changes can still affect scores when serving speed or queueing changes timeout rates. `shards: 4` suits 89 × r8 = 712 trials. Check `N × concurrency` against the Fargate quota and `N × gpus_per_node` against your allocation. -Use `8` as a conservative starting point, not a benchmark-mandated value. Model -endpoints are shared by trials within each shard. Reduce concurrency for capacity -only with latency/queueing evidence; match concurrency and effective settings -across comparisons. Keep benchmark-prescribed limits, temperature, top_p, and -output settings fixed; never raise timeouts merely to improve scores. This default -change has not been benchmarked and does not establish an improvement. - Apply `references/run-validation.md`'s bounded policy before reporting scores: solver/harness timeouts and terminal transport/action errors may count only when recorded and scored as protocol-valid failures. Keep all expected trials and diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index de2bb3cbfe5..ee9884eb3a7 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -225,7 +225,7 @@ with its own `run_id`, copying the shared `services:` block. ```bash set -a && source .env && set +a; NEL="$SKILL_DIR/scripts/nel-next.sh" "$NEL" eval run .yaml --dry-run # validate/render (no SSH) -"$NEL" eval run .yaml --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 -O benchmarks.0.sandbox.concurrency=2 # canary +"$NEL" eval run .yaml --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 # canary "$NEL" eval run .yaml --submit # full "$NEL" eval {status|logs -f|report -f markdown|merge} -r # lifecycle "$NEL" mlflow-push -r -c .yaml # post-run: push merged bundle(s) to MLflow @@ -243,7 +243,7 @@ kwargs after `--` (e.g. `-- -o copy_artifacts=true`). ## Non-fatal noise - AWS `RegisterTaskDefinition: ThrottlingException` at high concurrency — SDK - retries (≤8); lower matched `max_concurrent`/`sandbox.concurrency` only with capacity evidence. + retries (≤8); lower `max_concurrent` if frequent. - `solver failed — grading 0.0` agent crashes are rare but real (e.g. terminus-2 tmux `send-keys`) — distinguish from genuine misses + task timeouts; the final report breaks them out. Apply `run-validation.md`'s bounded failure policy: diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json deleted file mode 100644 index 7d8b965f041..00000000000 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ /dev/null @@ -1,198 +0,0 @@ -[ - { - "name": "runtime-trial-failure-boundaries", - "skills": ["evaluation"], - "query": "Classify four independent synthetic benchmark runs with 10000 expected and protocol-scored trials each. Their attributable terminal failures are respectively 0, 100, 200, and 201 unique trials; the benchmark records and scores each failure zero. Failures include solver/harness timeouts, terminal transport/action errors, exhausted requests, judge and sandbox/verifier errors. Original artifacts and diagnostics are preserved privately. Coverage and independent gates pass; the applicable response gate is 0/12000 for each run. A fifth complete run has 20001 failures of 1000000 expected scored trials; its rate displays as 2.00% after rounding. Do not launch jobs or change settings.", - "files": [], - "expected_behavior": [ - "Reports separate trial rates 0%, 1%, 2%, and 2.01% using expected trials, without pooling benchmarks or adding response and trial rates", - "Accepts zero without a failure warning and 1% and exactly 2% with visible warnings, retaining all failures and original score denominators", - "Above 2% returns findings for parent review without automatic retry, tuning, or success; 20001/1000000 = 2.0001% fails before rounding", - "Does not equate bounded failure rates with negligible score impact or leaderboard comparability" - ] - }, - { - "name": "runtime-overlap-resume-and-both-gates", - "skills": ["evaluation"], - "query": "Review runs A and B, each with 1000 expected scored trials and 10000 unique successful model responses. In A, solver timeouts affect trial IDs 1-12, harness timeouts 9-16, and terminal action errors 15-20. Duplicate resumed records repeat those IDs. All failures are protocol-valid zeros, retained in the merged score and raw artifacts. A also has 201 unique malformed model responses scored incorrect. B has no model-output faults but 21 unique protocol-valid runtime failures. Both have complete coverage and independent gates pass. Should either pass? Do not launch jobs.", - "files": [], - "expected_behavior": [ - "A category counts are 12, 8, 6; resumed overlaps deduplicate to 20/1000 = 2% trial failures, with response rate 201/10000 = 2.01%", - "B trial rate is 21/1000 = 2.1% even though its response rate is zero", - "Neither passes because both applicable gates must pass; rates with different denominators are never added", - "Retains warnings and returns above-tolerance findings to the parent without retry" - ] - }, - { - "name": "runtime-coverage-and-integrity-blockers", - "skills": ["evaluation"], - "query": "Six runs claim only 1% timeout failures. A expected 1000 trials but a harness crash left 999 scored and one unscored. B has 1000 records but its report omitted ten zero-scored failures and averaged only passing repetitions. C inserted fabricated zeros for missing trials. D has a secret leak. E used the wrong model/task/version/config and cross-invocation records. F has falsified or provenance-less output and systemic broken scoring. Can any pass? Do not rerun or silently regrade.", - "files": [], - "expected_behavior": [ - "Blocks every run independently of apparent rate: missing/unscored coverage, omitted zeros/selected repetitions, fabricated scoring, secrets, identity/contamination, and provenance/systemic scoring", - "Keeps secret scanning fail-closed and original raw evidence private", - "Requires original expected trial and score denominators, not observed records or successful-only trials", - "Reports blockers rather than fabricating scores, averaging incomplete data, or retroactively declaring success" - ] - }, - { - "name": "harness-continuation-and-delivery-separation", - "skills": ["evaluation"], - "query": "In synthetic run A a harness action crashed on one trial of 100 expected; the harness retained diagnostics and recorded a protocol-valid zero. It can continue the other trials safely under unchanged scoring. Run B has the same crash but no protocol-valid failure record or score. After A completes all 100 scored trials its MLflow export fails. What should the evaluator and parent do? No job submissions authorized.", - "files": [], - "expected_behavior": [ - "Allows continuation of A without aborting solely for a 1% failure, conditional on eventual complete scored coverage and both applicable tolerance gates passing", - "B is blocked; does not invent zero or patch scoring semantics", - "A completed score may be valid with a warning while MLflow delivery remains failed; export success is an independent handoff requirement", - "For day0 summaries retains accepted diagnostics in warnings, keeps unresolved blockers in errors, and does not treat gate_run.py as raw-evidence/rate validation" - ] - }, - { - "name": "terminal-bench-concurrency-eight-and-protocol", - "skills": ["evaluation"], - "query": "Prepare guidance only for a future Terminal-Bench 2.1 comparison with four shards. The former template used concurrency 50 and a later run used 15 per shard; lower the starting concurrency to 8. Can this be called a proven score improvement, or should timeouts also be raised? No canary, Harbor, or benchmark rerun is authorized.", - "files": [], - "expected_behavior": [ - "Reads the canonical recipe/example and sets max_concurrent and sandbox.concurrency equally to 8 per shard; four shards permit 32 simultaneous trials", - "Calls 8 a conservative starting point, not a benchmark mandate or proven improvement; endpoints are shared within each shard", - "Requires latency/queueing evidence for capacity changes and matches effective concurrency, timeout type/limits, temperature/top_p/output settings across comparisons", - "Keeps benchmark-prescribed limits fixed, does not raise timeouts to improve scores, and surfaces protocol/leaderboard differences", - "Does not submit a rerun or claim live validation" - ] - }, - { - "name": "standalone-kimi-scicode-model-output-fault", - "skills": ["launching-evals"], - "query": "Analyze Kimi SciCode attempt 15 in a fresh repository session, without an evaluator role or prior evaluation skill context. Coverage is complete: 65/65 samples and 338/338 selected, evaluated, and scored trajectories. Provenance verifies 337 unique successful HTTP model responses as the response denominator. One preserved response is both finish_reason=length and empty-final after reasoning consumed its token budget; it was retained and scored incorrect. There are zero HTTP, infrastructure, parser, scorer, sandbox, or harness failures. The measured score is 53.25443786982248. Is this run valid, or should it be rerun? Do not launch jobs.", - "files": [], - "expected_behavior": [ - "Follows AGENTS.md (also exposed via CLAUDE.md) to read evaluation/references/run-validation.md before reporting the score, without preloading the evaluation skill", - "Reports complete 65/65 sample and 338/338 selected/evaluated/scored trajectory coverage plus 337 verified unique successful responses", - "Reports length-capped=1 and empty-final=1 but a deduplicated union of 1/337 = approximately 0.2967%", - "Declares the run valid with a visible model-output-fault warning, retains score 53.25443786982248, and does not rerun", - "Does not infer negligible score impact from the low rate or tune/remove token limits" - ] - }, - { - "name": "model-output-fault-overlap-and-two-percent-boundary", - "skills": ["evaluation", "launching-evals"], - "query": "Classify two independent runs with complete expected/selected/evaluated/scored coverage and 10000 verified unique successful raw model responses each. In A, preserved responses 1-120 are length-capped, 81-170 have empty finals, and 151-200 have malformed unusable finals; every affected response is retained and scored incorrect. In B, 201 distinct preserved responses have malformed finals and are retained and scored incorrect. Each log also repeats 5000 cache records and includes 100 failed transport attempts that were retried successfully plus 10000 judge responses. All independent gates pass. Report category and union accounting; do not launch jobs.", - "files": [], - "expected_behavior": [ - "Uses 10000 as each denominator, excluding duplicate records, retry attempts, and judge calls; does not pool runs", - "Reports A's category counts as length-capped=120, empty-final=90, malformed-final=50 and deduplicates overlaps to response IDs 1-200, a union of 200/10000 = exactly 2.0%", - "Treats A as valid with a visible warning because the inclusive unrounded boundary passes", - "Reports B's union as 201/10000 = 2.01%, above tolerance; returns findings and a recommendation without automatic retry or a success verdict" - ] - }, - { - "name": "model-output-tolerance-does-not-waive-independent-failures", - "skills": ["evaluation", "launching-evals"], - "query": "Review five separate runs whose apparent model-output-fault rate is below 2%. A selected 1000 responses but evaluated and scored only 999. B exhausted an HTTP/server request and has no successful raw response for one item and no protocol-valid scored trial. C raised a parser exception and assigned a fallback score without retaining the malformed answer as an explicit incorrect result; its harness also failed without recording a protocol-valid scored trial. D used the wrong benchmark version, model, and config. E has a secret leak and no verifiable response denominator. All reported scores exceed the reference. Can the parent accept any under the tolerance?", - "files": [], - "expected_behavior": [ - "Marks every run invalid rather than using the model-output tolerance", - "Separately identifies missing/unscored coverage, exhausted HTTP/server request, parser fallback and harness failure, wrong protocol identity, secret leak, and unverified denominator", - "Does not reclassify C as a model-output fault because no preserved successful raw malformed answer was deterministically retained and scored incorrect", - "Returns findings and recommendations without automatically retrying; exceeding the reference is not proof of validity" - ] - }, - { - "name": "model-output-fault-unverified-and-token-budget", - "skills": ["evaluation", "launching-evals"], - "query": "A run has no reliable unique-successful-response denominator. Another has 5/1000 preserved length-capped responses retained and scored incorrect, max_new_tokens=131072, deployed context=131072, and 4096-token prompts; its reference protocol uses a 65536 output budget. Should I remove the output cap or keep increasing it until the reference score passes?", - "files": [], - "expected_behavior": [ - "Marks the first run's model-output-fault rate unverified, not 0% or within tolerance", - "Reports the second run's union as 5/1000 = 0.5% but flags reference-budget mismatch and insufficient context for prompt plus configured output", - "Does not uncap or tune limits to pass the target; returns a protocol-justified recommendation for approval without automatically resubmitting" - ] - }, - { - "name": "nemotron3-nano-bf16-reasoning", - "skills": ["evaluation"], - "query": "Help me evaluate Nemotron 3 Nano BF16 from NVIDIA", - "files": [], - "expected_behavior": [ - "Verifies nel is installed by running 'nel --version'", - "Asks all 5 base config questions (execution, deployment, auto-export, model type, benchmarks) before generating the config", - "Runs 'nel skills build-config' with correct flags matching user answers, using NEL's current model_type terminology for the chat/reasoning template family", - "Searches the web for the model card on HuggingFace and extracts model-specific settings", - "Sets correct HF handle: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", - "Sets reasoning sampling params from model card: temperature=1.0, top_p=1.0", - "Configures reasoning toggle via params_to_add with chat_template_kwargs.enable_thinking (not via system prompt)", - "Disables reasoning for IFEval task using enable_thinking: false with use_system_prompt: false", - "Adds deployment.pre_cmd using curl (not wget) to download nano_v3_reasoning_parser.py from HuggingFace", - "Adds vLLM extra_args including --trust-remote-code, --reasoning-parser-plugin, --reasoning-parser nano_v3, --max-num-seqs 8", - "Pins vLLM image to v0.12.0 or later as required by model card", - "Adds target.api_endpoint.api_key_name: DUMMY_API_KEY for nemo_skills tasks with self-deployment", - "Fills in all ??? placeholders after asking the user for SLURM hostname, account, output_dir, MLflow tracking_uri, and experiment_name", - "Applies user-requested SLURM customizations: partition batch_short, walltime 00:20:00, MLflow tag scenario: demo", - "Presents task list and waits for user confirmation before proceeding", - "Configures request and response logging interceptors under evaluation.nemo_evaluator_config.config.target.api_endpoint.adapter_config using correct field names (max_logged_requests/max_logged_responses, not max_saved_*)", - "Handles dry-run failure for missing HF_TOKEN_FOR_GPQA_DIAMOND by offering to fix the config", - "Successfully submits a limited-samples canary with limit_samples=10 after dry-run passes", - "Inspects canary logs for judge, evaluation container, deployment, and code-execution errors before allowing the full run", - "Tunes parallelism from canary results and avoids over-raising it for judge-backed or code-execution tasks", - "Before reporting final scores, inspects full-run logs for judge, inference server, code-execution, reasoning parsing, dropped-sample, and scoring errors", - "Routes completed baseline-vs-quantized delta checks to the compare-results skill after final-run validation", - "Provides monitoring commands (nel status, nel info --logs) and inspects server logs via SSH when asked" - ] - }, - { - "name": "quantized-checkpoint-local-vllm", - "skills": ["evaluation"], - "query": "evaluate my FP8 quantized Llama checkpoint at ./llama-3.1-8b-fp8 on MMLU and GSM8K", - "files": [], - "expected_behavior": [ - "Verifies nel is installed by running nel --version", - "Asks all 5 base config questions (execution, deployment, auto-export, model type, benchmarks)", - "Runs nel skills build-config with correct flags matching user answers", - "Sets deployment.checkpoint_path to ./llama-3.1-8b-fp8 and deployment.hf_model_handle to null", - "Auto-detects quantization format by reading ./llama-3.1-8b-fp8/config.json first, then ./llama-3.1-8b-fp8/hf_quant_config.json if needed", - "Finds quant_algo=FP8 and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", - "Recommends accuracy-sensitive benchmarks from references/quantization-benchmarks.md", - "Searches web for Llama-3.1-8B model card and extracts sampling params, context length, TP settings", - "Fills in remaining missing values by asking user", - "Runs dry-run, then a limited-samples canary with limit_samples=10, then full evaluation only after canary log checks pass", - "Tunes parallelism from the canary and avoids over-raising it when benchmarks depend on inference judges or code execution containers", - "Before reporting accuracy, scans final run logs and sample accounting for runtime errors, dropped/skipped samples, reasoning parsing problems, and wrong score-field usage", - "Routes baseline-vs-quantized comparability verification to the compare-results skill after final-run validation", - "Reports accuracy results per benchmark only after log validation passes" - ] - }, - { - "name": "slurm-quantized-model", - "skills": ["evaluation"], - "query": "Evaluate my quantized Llama-3.1-8B-FP8 checkpoint on mmlu and gsm8k on the SLURM cluster", - "files": [], - "expected_behavior": [ - "Verifies nel is installed by running nel --version", - "Asks 5 base config questions with execution=slurm pre-selected based on user request", - "Runs nel skills build-config with --execution slurm --deployment vllm --benchmarks standard", - "Detects FP8 quantization from config.json or hf_quant_config.json and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", - "Reads references/quantization-benchmarks.md and recommends accuracy-sensitive benchmarks", - "Uses WebSearch to research model card for sampling params and context length", - "Fills in SLURM-specific values: hostname, account, partition from user input", - "Runs dry-run validation and a limited-samples canary before full evaluation", - "Checks canary logs for judge, container, deployment, and code-execution failures before treating the run as ready", - "Checks final run logs and sample accounting before accepting result files as complete and comparable", - "Routes baseline-vs-quantized comparability verification to the compare-results skill before presenting an accuracy delta", - "Provides SSH-based log monitoring commands for SLURM execution" - ] - }, - { - "name": "scicode-eight-run-average", - "skills": ["evaluation"], - "query": "Evaluate my NVFP4 checkpoint on the AA suite including SciCode on SLURM", - "files": [], - "expected_behavior": [ - "Keeps num_repeats at 1 and plans at least 8 independent SciCode submissions instead", - "Re-submits SciCode alone with -t ns_scicode rather than re-running the whole suite", - "Reports the SciCode score as the mean of the pooled runs with standard error and run count", - "Confirms cache replay from run provenance rather than equal scores before excluding a run", - "Re-submits replacements until at least 8 valid independent runs exist before reporting a mean", - "Reports SciCode as INDETERMINATE when fewer than 8 valid runs are available" - ] - } -] diff --git a/plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py b/plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py deleted file mode 100644 index 2827eb5ba87..00000000000 --- a/plugins/modelopt/skills/evaluation/tests/test_failure_guidance.py +++ /dev/null @@ -1,63 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Check guidance/config consistency; these tests do not run agents or scoring.""" - -import json -import re -from pathlib import Path - -import yaml - -_SKILL = Path(__file__).resolve().parents[1] -_ROOT = _SKILL.parents[3] - - -def test_terminal_bench_concurrency_matches_recipe(): - template = yaml.safe_load((_SKILL / "recipes/examples/example_eval_next.yaml").read_text()) - recipe = (_SKILL / "recipes/tasks/aa_next/terminal_bench_2_1.md").read_text() - yaml_block = re.search(r"```yaml\n(.*?)```", recipe, re.DOTALL) - assert yaml_block is not None - fragment = yaml.safe_load(yaml_block.group(1)) - for config in (template, fragment): - benchmark = config["benchmarks"][0] - assert benchmark["max_concurrent"] == benchmark["sandbox"]["concurrency"] == 8 - assert benchmark["repeats"] == 8 - - -def test_failure_policy_is_reachable_from_parent_and_evaluators(): - policy_path = "evaluation/references/run-validation.md" - policy = (_SKILL / "references/run-validation.md").read_text() - title = "Bounded Evaluation-Failure Policy" - assert f"### {title}" in policy - assert "**Response gate" in policy and "**Trial gate" in policy - assert "Both applicable gates must pass" in policy - assert policy_path in (_ROOT / "AGENTS.md").read_text() - for path in ( - _SKILL / "SKILL.md", - _ROOT / ".codex/agents/modelopt_model_evaluator.toml", - _ROOT / "plugins/modelopt/agents/modelopt-model-evaluator.md", - ): - assert title in path.read_text() - - -def test_evaluation_scenarios_reference_existing_skills(): - scenarios = json.loads((_SKILL / "tests/evals.json").read_text()) - names = [scenario["name"] for scenario in scenarios] - assert len(names) == len(set(names)) - for scenario in scenarios: - assert scenario["query"] and scenario["expected_behavior"] - for skill in scenario["skills"]: - assert (_SKILL.parent / skill / "SKILL.md").is_file() From b93e539e7a230576927ea569058f10f20d476711 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Fri, 2 Oct 2026 20:54:37 +0000 Subject: [PATCH 4/8] docs(eval): shorten bounded failure policy and restore unchanged guidance Signed-off-by: Chad Voegele --- AGENTS.md | 10 -- .../evaluation/references/run-validation.md | 119 ++++++------------ .../skills/evaluation/tests/evals.json | 89 +++++++++++++ 3 files changed, 126 insertions(+), 92 deletions(-) create mode 100644 plugins/modelopt/skills/evaluation/tests/evals.json diff --git a/AGENTS.md b/AGENTS.md index eda2e28e5f9..c2cebc9a399 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,16 +12,6 @@ These instructions apply to AI-assisted work in this repository. those skills through relative symlinks. Shared agent config and scripts remain under `.agents/`. See `.agents/README.md` for the convention. -## Evaluation analysis - -Before analyzing completed evaluations or reporting scores, including standalone -`launching-evals` sessions, read -`plugins/modelopt/skills/evaluation/references/run-validation.md`. -Its bounded evaluation-failure policy supersedes vendored `launching-evals` -advice to uncap reasoning output (`max_tokens=null` or payload field removal), -use fixed token/context limits, or automatically invalidate a preserved -response or protocol-valid failed trial that is retained and scored incorrect. - ## Coding guidelines - **Coding guide:** Code development and review require reading and following diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index e59f4351159..3d8b230d1e1 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -41,88 +41,43 @@ automatically resubmit a completed run. ### Bounded Evaluation-Failure Policy (Parent and Evaluator) -For each benchmark/run, report category counts, their deduplicated union, the -verified denominators, and `rate = 100 × union / denominator` for each applicable -gate below. Calculate with unrounded counts; round only the displayed percentage. - -**Response gate (model-output faults).** Preserve unique-response accounting: - -The denominator is the deduplicated set of unique, successful raw evaluated-model -responses selected for evaluation. Count a cached response reused by multiple -trajectories once; exclude duplicate log/cache records, failed request attempts, -judge calls, and unrelated runs. Verify the denominator from response identity -and provenance rather than dataset size, especially for repeated or multi-turn -tasks. An unknown or zero denominator leaves the rate unverified, not 0%. - -The numerator is the set union of denominator responses unusable because of -model output behavior: - -- length or token-budget truncation, including reasoning that consumes the budget; -- an empty final answer, including a nonempty reasoning trace with no final answer; -- a malformed or otherwise unusable final output. - -Count a response in every applicable category for reporting, but once in the -union. To qualify, the successful raw response must exist, remain preserved in -the artifacts, and be deterministically retained and scored incorrect. A parser -exception, parser fallback/default score, or missing parsed record is not -automatically a model-output fault. A malformed answer qualifies only if the -preserved raw response is explicitly retained and scored incorrect under the -benchmark protocol. Runtime scorer/harness failures require trial-gate assessment -and protocol-valid scoring instead. - -**Trial gate (runtime failures).** Use the expected benchmark trial/repeat count, -not successful trials, HTTP attempts, or the number of records found. Its numerator -is the deduplicated union of attributable terminal request/transport/server, -judge, executor/sandbox/verifier, terminal action, solver, and harness failures, -including timeouts, that the benchmark explicitly records and scores incorrect -or zero under its existing protocol. A failed request without a raw response may -qualify here only through such a scored trial; it cannot enter the response gate. -Ordinary wrong answers are not runtime failures. - -Preserve diagnostics and original raw evidence privately, including failed -submissions. Verify failure type and scoring from structured artifacts and logs; -an exception string or a summary zero alone is insufficient. Deduplicate overlaps -and resumes by invocation, benchmark, task, trial, and repeat identity. Count -overlapping categories once within each gate; never add response and trial rates -or pool benchmarks. Both applicable gates must pass. A shared fault may affect -both gates, but is counted once within each. Mark an inapplicable gate explicitly; -an unknown applicable denominator is not an exemption. - -Continue other trials after a harness crash only when the harness can record the -affected trial as a protocol-valid scored failure. If it cannot, report a blocker. -Do not fabricate zeros, modify scoring semantics, treat missing/unscored trials -as completed, select passing repetitions, or silently regrade old runs. Keep the -original trial and score denominators; omitted zero-scored failures leave coverage -incomplete. Never average incomplete data. - -- **rate = 0%:** no failure warning for that gate; independent checks still apply. -- **0 < rate ≤ 2.0% (inclusive, before rounding):** valid with a visible warning - only when coverage is complete and all independent gates pass. Retain failures - and their protocol-defined scores; do not invalidate, abort the invocation, or - retry solely for bounded failures. A low rate does **not** imply negligible - score impact or leaderboard comparability. -- **rate > 2.0%:** return category counts, union, rate, findings, and a - recommendation to the parent (or user); do not automatically retry or declare - success. -- The tolerance never waives secret leaks (scanning remains fail-closed), wrong - model/task/version/configuration, falsified or provenance-less output, - cross-invocation contamination, input/context clipping, systemic serving or - broken scoring, incomplete coverage, or benchmark-specific validity rules. - These are independent integrity failures, not ordinary per-trial failures. - Undocumented parser/judge fallback scores remain invalid. A score above a - reference does not establish validity. -- Check configured and effective output limits against the reference evaluation - protocol and deployed context capacity, including prompt/history plus output - space. Report mismatches or unavailable reference settings. Do not remove or - tune token limits merely to pass; propose protocol-justified changes for - parent/user approval instead. - -Keep score, run validation, and MLflow delivery separate: a valid score does not -prove export succeeded, and export cannot validate scoring. In a day0 run summary, -keep policy-validated failure diagnostics in `warnings`; `errors` contains -unresolved blockers and still fails `gate_run.py`. That summary gate does not -compute these rates or verify raw evidence; perform this policy check first. -The parent must retain both gates' counts, rates, and warnings in handoffs. +Accept complete runs with **≤2% failures in each applicable gate**, measured +before rounding. Warn for nonzero rates; do not abort, invalidate, or retry solely +for bounded failures. Above 2%, return findings to the parent/user without +automatic retry or a success verdict. + +| Gate | Failures | Denominator | +|---|---|---| +| Response | Truncated, empty-final, or malformed/unusable model outputs | Verified unique successful raw model responses selected for evaluation | +| Trial | Terminal request/server/transport, judge, sandbox/verifier, terminal action, solver, or harness errors/timeouts | Expected trials including repeats | + +Failures qualify only when preserved and recorded as incorrect/zero under the +existing benchmark protocol. Verify this from artifacts and logs, not exception +strings or summary zeros alone. Failed requests without raw responses belong only +in the trial gate; ordinary wrong answers are not runtime failures. + +Report category counts, deduplicated unions, denominators, and rates per +benchmark/run. Deduplicate overlaps, caches, and resumes by response/trial identity; +exclude judge responses and failed attempts from the response denominator. Never +add the two rates or pool benchmarks. Unknown/zero applicable denominators leave +validation incomplete; identify inapplicable gates explicitly. + +Retain all trials, zero scores, original score denominators, and private raw +evidence. Continue after a harness crash only if it records a protocol-valid +scored failure. Missing/unscored trials, fabricated or undocumented fallback +scores, secret leaks, wrong identity/configuration, contamination, provenance +failures, input/context clipping, systemic serving/scoring failures, and violated +benchmark rules remain blockers. Never select passing repeats or silently regrade. + +Keep protocol limits fixed, verify prompt/history plus output fits deployed +context, and report mismatches or unknown reference settings; proposed changes +need parent/user approval. Tolerance does not prove negligible score impact or +leaderboard comparability. + +Keep score, validation, and MLflow delivery separate. Carry counts, rates, and +warnings into handoffs. In day0 summaries, accepted diagnostics go in `warnings`; +unresolved blockers stay in `errors`. Apply this policy before `gate_run.py`, +which does not verify rates or raw evidence. ## Timeout and Output-Limit Accounting diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json new file mode 100644 index 00000000000..3a33a6af053 --- /dev/null +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -0,0 +1,89 @@ +[ + { + "name": "nemotron3-nano-bf16-reasoning", + "skills": ["evaluation"], + "query": "Help me evaluate Nemotron 3 Nano BF16 from NVIDIA", + "files": [], + "expected_behavior": [ + "Verifies nel is installed by running 'nel --version'", + "Asks all 5 base config questions (execution, deployment, auto-export, model type, benchmarks) before generating the config", + "Runs 'nel skills build-config' with correct flags matching user answers, using NEL's current model_type terminology for the chat/reasoning template family", + "Searches the web for the model card on HuggingFace and extracts model-specific settings", + "Sets correct HF handle: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", + "Sets reasoning sampling params from model card: temperature=1.0, top_p=1.0", + "Configures reasoning toggle via params_to_add with chat_template_kwargs.enable_thinking (not via system prompt)", + "Disables reasoning for IFEval task using enable_thinking: false with use_system_prompt: false", + "Adds deployment.pre_cmd using curl (not wget) to download nano_v3_reasoning_parser.py from HuggingFace", + "Adds vLLM extra_args including --trust-remote-code, --reasoning-parser-plugin, --reasoning-parser nano_v3, --max-num-seqs 8", + "Pins vLLM image to v0.12.0 or later as required by model card", + "Adds target.api_endpoint.api_key_name: DUMMY_API_KEY for nemo_skills tasks with self-deployment", + "Fills in all ??? placeholders after asking the user for SLURM hostname, account, output_dir, MLflow tracking_uri, and experiment_name", + "Applies user-requested SLURM customizations: partition batch_short, walltime 00:20:00, MLflow tag scenario: demo", + "Presents task list and waits for user confirmation before proceeding", + "Configures request and response logging interceptors under evaluation.nemo_evaluator_config.config.target.api_endpoint.adapter_config using correct field names (max_logged_requests/max_logged_responses, not max_saved_*)", + "Handles dry-run failure for missing HF_TOKEN_FOR_GPQA_DIAMOND by offering to fix the config", + "Successfully submits a limited-samples canary with limit_samples=10 after dry-run passes", + "Inspects canary logs for judge, evaluation container, deployment, and code-execution errors before allowing the full run", + "Tunes parallelism from canary results and avoids over-raising it for judge-backed or code-execution tasks", + "Before reporting final scores, inspects full-run logs for judge, inference server, code-execution, reasoning parsing, dropped-sample, and scoring errors", + "Routes completed baseline-vs-quantized delta checks to the compare-results skill after final-run validation", + "Provides monitoring commands (nel status, nel info --logs) and inspects server logs via SSH when asked" + ] + }, + { + "name": "quantized-checkpoint-local-vllm", + "skills": ["evaluation"], + "query": "evaluate my FP8 quantized Llama checkpoint at ./llama-3.1-8b-fp8 on MMLU and GSM8K", + "files": [], + "expected_behavior": [ + "Verifies nel is installed by running nel --version", + "Asks all 5 base config questions (execution, deployment, auto-export, model type, benchmarks)", + "Runs nel skills build-config with correct flags matching user answers", + "Sets deployment.checkpoint_path to ./llama-3.1-8b-fp8 and deployment.hf_model_handle to null", + "Auto-detects quantization format by reading ./llama-3.1-8b-fp8/config.json first, then ./llama-3.1-8b-fp8/hf_quant_config.json if needed", + "Finds quant_algo=FP8 and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", + "Recommends accuracy-sensitive benchmarks from references/quantization-benchmarks.md", + "Searches web for Llama-3.1-8B model card and extracts sampling params, context length, TP settings", + "Fills in remaining missing values by asking user", + "Runs dry-run, then a limited-samples canary with limit_samples=10, then full evaluation only after canary log checks pass", + "Tunes parallelism from the canary and avoids over-raising it when benchmarks depend on inference judges or code execution containers", + "Before reporting accuracy, scans final run logs and sample accounting for runtime errors, dropped/skipped samples, reasoning parsing problems, and wrong score-field usage", + "Routes baseline-vs-quantized comparability verification to the compare-results skill after final-run validation", + "Reports accuracy results per benchmark only after log validation passes" + ] + }, + { + "name": "slurm-quantized-model", + "skills": ["evaluation"], + "query": "Evaluate my quantized Llama-3.1-8B-FP8 checkpoint on mmlu and gsm8k on the SLURM cluster", + "files": [], + "expected_behavior": [ + "Verifies nel is installed by running nel --version", + "Asks 5 base config questions with execution=slurm pre-selected based on user request", + "Runs nel skills build-config with --execution slurm --deployment vllm --benchmarks standard", + "Detects FP8 quantization from config.json or hf_quant_config.json and does not add a vLLM --quantization flag unless the vLLM version, model card, or dry-run error requires one", + "Reads references/quantization-benchmarks.md and recommends accuracy-sensitive benchmarks", + "Uses WebSearch to research model card for sampling params and context length", + "Fills in SLURM-specific values: hostname, account, partition from user input", + "Runs dry-run validation and a limited-samples canary before full evaluation", + "Checks canary logs for judge, container, deployment, and code-execution failures before treating the run as ready", + "Checks final run logs and sample accounting before accepting result files as complete and comparable", + "Routes baseline-vs-quantized comparability verification to the compare-results skill before presenting an accuracy delta", + "Provides SSH-based log monitoring commands for SLURM execution" + ] + }, + { + "name": "scicode-eight-run-average", + "skills": ["evaluation"], + "query": "Evaluate my NVFP4 checkpoint on the AA suite including SciCode on SLURM", + "files": [], + "expected_behavior": [ + "Keeps num_repeats at 1 and plans at least 8 independent SciCode submissions instead", + "Re-submits SciCode alone with -t ns_scicode rather than re-running the whole suite", + "Reports the SciCode score as the mean of the pooled runs with standard error and run count", + "Confirms cache replay from run provenance rather than equal scores before excluding a run", + "Re-submits replacements until at least 8 valid independent runs exist before reporting a mean", + "Reports SciCode as INDETERMINATE when fewer than 8 valid runs are available" + ] + } +] From a6618eea6b2bcbc9d760a6fcab413e8644ba073f Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Fri, 2 Oct 2026 21:07:04 +0000 Subject: [PATCH 5/8] docs(eval): route standalone analysis through run validation Signed-off-by: Chad Voegele --- plugins/modelopt/skills/day0-release/tests/test_gates.py | 1 + plugins/modelopt/skills/evaluation/SKILL.md | 9 ++++++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/plugins/modelopt/skills/day0-release/tests/test_gates.py b/plugins/modelopt/skills/day0-release/tests/test_gates.py index de88fc971ea..ddb576dd38d 100644 --- a/plugins/modelopt/skills/day0-release/tests/test_gates.py +++ b/plugins/modelopt/skills/day0-release/tests/test_gates.py @@ -152,6 +152,7 @@ def test_run_judge_error(): def test_run_validated_warnings_do_not_waive_blockers(): + """Accept validated warnings without overlooking coverage or integrity blockers.""" task = _task(warnings=["protocol-valid scored timeout; trial rate 1/100 = 1%"]) assert evaluate_run({"tasks": {"t": task}})["pass"] for blockers in ( diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 5477a81502b..760abb6b697 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -1,6 +1,6 @@ --- name: evaluation -description: Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying/serving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results). +description: Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Always load when validating completed runs or reporting evaluation scores, including standalone launching-evals analysis. Not for quantizing models (use ptq), deploying/serving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results). license: Apache-2.0 # Based on nel-assistant skill from NeMo Evaluator Launcher (commit f1fa073). # https://github.com/NVIDIA-NeMo/Evaluator/tree/f1fa073/packages/nemo-evaluator-launcher/.claude/skills/nel-assistant @@ -10,6 +10,13 @@ license: Apache-2.0 Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress. +### Completed-run analysis + +For existing runs, skip launch/config steps and read +[run-validation.md](references/run-validation.md) before reporting scores; use +`launching-evals` for artifact collection and benchmark analysis. This policy +supersedes its blanket output-uncapping and fixed token/context-limit advice. + ### Workspace integration If `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md` and reuse existing workspaces (this skill is usually the final stage of PTQ → Deploy → Eval; carry any deployment-time patches into `deployment.command`). From bf83506f903b43c5d5b1c3a26151ded155b80afa Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Fri, 2 Oct 2026 21:13:48 +0000 Subject: [PATCH 6/8] docs(eval): clarify strict failure boundary and add review scenarios Signed-off-by: Chad Voegele --- plugins/modelopt/skills/evaluation/SKILL.md | 14 +++----- .../evaluation/references/run-validation.md | 4 +-- .../skills/evaluation/tests/evals.json | 35 ++++++++++++++++++- 3 files changed, 40 insertions(+), 13 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index 760abb6b697..c4ede3a35e3 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -137,16 +137,10 @@ Read [Step 8](references/launcher-workflow.md#step-8--run-evaluation-gated-dry-r ### Step 9 — Verify completed run -Read [run-validation.md](references/run-validation.md) before reporting scores. -Parents and evaluators must apply its **Bounded Evaluation-Failure Policy**: -verify complete coverage, protocol-valid failed scoring, and separate deduplicated -response/trial rates. Both applicable gates must be ≤2.0% before rounding; warn -for nonzero rates. Missing/unscored trials and independent integrity failures -remain blockers. Above tolerance, return findings for parent review without -automatic retry or tuning. Preserve raw evidence privately and report timeout, -limit, and concurrency settings. Keep score, validation, and MLflow delivery -separate. For comparisons, apply its **External Baseline Sanity Check**, then -use `compare-results`. +Apply [run-validation.md](references/run-validation.md)'s **Bounded Evaluation-Failure +Policy** and **Timeout and Output-Limit Accounting** before reporting scores. For +comparisons, also apply its **External Baseline Sanity Check**, then use +`compare-results`. --- diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index 3d8b230d1e1..7e2713b3925 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -41,9 +41,9 @@ automatically resubmit a completed run. ### Bounded Evaluation-Failure Policy (Parent and Evaluator) -Accept complete runs with **≤2% failures in each applicable gate**, measured +Accept complete runs with **<2% failures in each applicable gate**, measured before rounding. Warn for nonzero rates; do not abort, invalidate, or retry solely -for bounded failures. Above 2%, return findings to the parent/user without +for bounded failures. At or above 2%, return findings to the parent/user without automatic retry or a success verdict. | Gate | Failures | Denominator | diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 3a33a6af053..90947a46ba8 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -1,4 +1,37 @@ [ + { + "name": "bounded-failures-strict-boundary", + "skills": ["evaluation"], + "query": "Classify independent complete runs with 10000 expected scored trials and 10000 verified unique successful model responses each. Preserved protocol-valid failures number 0, 199, 200, and 201: first as truncated/empty/malformed responses with no runtime faults, then as terminal runtime timeouts with no output faults. Independent checks pass. Also classify 19999/1000000 failures, which displays as 2.00%. Do not launch jobs.", + "files": [], + "expected_behavior": [ + "Applies each gate separately: 0% passes without a failure warning; 1.99% passes with a warning; exactly 2% and 2.01% require parent review without success or automatic retry", + "Accepts 1.9999% with a warning despite its rounded 2.00% display", + "Retains failed scores and original denominators; does not infer negligible score impact" + ] + }, + { + "name": "bounded-failures-overlap-and-resume", + "skills": ["evaluation"], + "query": "A complete run has 1000 expected scored trials and 10000 verified unique successful model responses. Solver timeouts affect trials 1-12, harness errors 9-16, and terminal action failures 15-19. Resumed logs duplicate these records. Model response faults affect IDs 1-120 (truncated), 81-170 (empty-final), and 151-190 (malformed). All failures are preserved protocol-valid incorrect/zero outcomes; other checks pass. A second run has the same response faults but 20 runtime-failed trials. Do not launch jobs.", + "files": [], + "expected_behavior": [ + "Reports runtime categories 12, 8, 5, deduplicated to 19/1000 = 1.9%; response categories 120, 90, 40, deduplicated to 190/10000 = 1.9%", + "Accepts the first run with warnings without counting resumed records twice, pooling benchmarks, or adding gate rates", + "Does not accept the second run: its trial gate is exactly 2%, even though its response gate passes" + ] + }, + { + "name": "bounded-failures-scored-zeros-versus-missing", + "skills": ["evaluation"], + "query": "Run A has 100 expected trials, 100 scored, and one preserved harness timeout recorded zero under the benchmark protocol; its response gate is verified fault-free. Run B has 100 expected but only 99 scored after a crash. Run C fabricates a zero for that missing trial. Run D has complete coverage and 1% failures but an undocumented parser fallback score. Can these pass? Explain how to build day0 summaries; do not launch jobs or regrade.", + "files": [], + "expected_behavior": [ + "Accepts A with a 1% trial warning; blocks B, C, and D independently of apparent failure rate", + "Verifies coverage, identities, failure outcomes, and denominators from artifacts/logs before constructing summaries; a zero or exception string alone is insufficient", + "Keeps accepted diagnostics in warnings and unresolved blockers in errors; gate_run.py does not enforce failure rates or inspect raw evidence" + ] + }, { "name": "nemotron3-nano-bf16-reasoning", "skills": ["evaluation"], @@ -82,7 +115,7 @@ "Re-submits SciCode alone with -t ns_scicode rather than re-running the whole suite", "Reports the SciCode score as the mean of the pooled runs with standard error and run count", "Confirms cache replay from run provenance rather than equal scores before excluding a run", - "Re-submits replacements until at least 8 valid independent runs exist before reporting a mean", + "Requests parent/user approval to replace invalid runs until at least 8 valid independent runs exist; does not retry solely for bounded failures", "Reports SciCode as INDETERMINATE when fewer than 8 valid runs are available" ] } From 83d7915b4fffac5dd9a8b28a7d84311fa3d03153 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Fri, 2 Oct 2026 21:55:49 +0000 Subject: [PATCH 7/8] docs(eval): compress completed-run guidance Signed-off-by: Chad Voegele --- plugins/modelopt/skills/evaluation/SKILL.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index c4ede3a35e3..50c2b8eb66f 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -12,10 +12,7 @@ Guide the user through creating NEL YAML configs, running evaluations, and monit ### Completed-run analysis -For existing runs, skip launch/config steps and read -[run-validation.md](references/run-validation.md) before reporting scores; use -`launching-evals` for artifact collection and benchmark analysis. This policy -supersedes its blanket output-uncapping and fixed token/context-limit advice. +For completed runs, skip launch/config; use launching-evals for artifacts/analysis and apply [run-validation.md](references/run-validation.md) before scores. Its policy overrides blanket uncapping and fixed token/context limits. ### Workspace integration From fa8759ab8e50c3fe8206b827606ad82633fc3c12 Mon Sep 17 00:00:00 2001 From: Chad Voegele Date: Fri, 2 Oct 2026 22:02:02 +0000 Subject: [PATCH 8/8] docs(day0): defer completed eval retries to validation policy Signed-off-by: Chad Voegele --- plugins/modelopt/skills/day0-release/SKILL.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/plugins/modelopt/skills/day0-release/SKILL.md b/plugins/modelopt/skills/day0-release/SKILL.md index ee58d078219..9b740512be2 100644 --- a/plugins/modelopt/skills/day0-release/SKILL.md +++ b/plugins/modelopt/skills/day0-release/SKILL.md @@ -270,6 +270,13 @@ the final path after any move. ## Triage (gate failure → decision) +For completed evaluations, apply +[evaluation run validation](../evaluation/references/run-validation.md) before +triage; its policy overrides retry actions below. Warn but do not retry solely +for failures below tolerance. For runs that fail tolerance or an independent +validation check, return findings and a recommendation to the parent/user for +review; do not automatically resubmit. + Map a gate's `failure_class` to the next action: | `failure_class` | Action | @@ -281,7 +288,7 @@ Map a gate's `failure_class` to the next action: | `CHECKPOINT_NOT_SERVABLE` | The Step 2b canary could not load/generate. Usually a tensor-naming or config-schema mismatch between the exporter and the serving stack, or missing/dangling auxiliary files (tokenizer). Fix the export; do not evaluate. | | `VERBOSITY_EXCEEDED` | Re-check run hygiene first (mixed reasoning effort, partial runs, unequal sample counts) — that has explained every false positive so far. If the delta survives matched, complete runs, it is a real behavioural change; do not publish on accuracy alone. | | `DEPLOYMENT_HEALTH_FAILED` | Drop to the **deployment** skill: reproduce serving standalone (`/health` + one generation), debug flags / image / TP / env, then carry the working command into NEL's `deployment.command` and retry the eval. If it can't serve, `POINT_INFEASIBLE`. | -| `EVAL_JUDGE_FAILED` | Usually transient (auth / rate limit) — wait and retry. | +| `EVAL_JUDGE_FAILED` | For completed runs, apply evaluation run validation above; return tolerance failures to the parent/user without automatic retry. Otherwise investigate auth/rate limits before retrying. | | `SAMPLE_ACCOUNTING_FAILED` | Investigate dropped/failed samples before trusting scores. | | `EXTERNAL_BASELINE_MISMATCH` | Investigate baseline configuration, correct it, rerun the baseline, and repeat external sanity before comparison. | | `USER_CONFIG_ERROR` | Correct it from the request, workspace, or model/config metadata and retry; if irrecoverable, return `ANOMALOUS` with evidence. |