diff --git a/CHANGELOG.rst b/CHANGELOG.rst index eb4102dd116..a262d05d1b6 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -15,6 +15,10 @@ Changelog - Add the **external draft** speculative-decoding mode: train a pretrained standalone draft model against a cached base model, instead of grafting a draft head onto the target. Select it with ``mtsp.convert(draft, [("external", cfg)])`` or the ``speculative_external`` recipe; the draft keeps its own embeddings and lm_head and exports as an ordinary HF checkpoint. +*Misc* + +- The ``evaluation`` agent skill can stop looping ("doom loop") reasoning early: ask it to avoid doom loops and it adds vLLM's ``repetition_detection`` request field (vLLM 0.17.0 or later) to every task, ending a request once its output repeats. Use the same setting for baseline and candidate runs. + **Backward Breaking Changes** **Deprecations** diff --git a/plugins/modelopt/skills/compare-results/SKILL.md b/plugins/modelopt/skills/compare-results/SKILL.md index 8b7e40f6b87..1cb573e64a8 100644 --- a/plugins/modelopt/skills/compare-results/SKILL.md +++ b/plugins/modelopt/skills/compare-results/SKILL.md @@ -54,8 +54,8 @@ the validated runs are comparable: 2. Task name, benchmark version, dataset split, container, harness, and task fragment match. 3. Generation settings match, including temperature, top_p, top_k, max tokens, - stop strings, chat-template kwargs, reasoning mode/budget, and task-specific - overrides. + stop strings, repetition detection, chat-template kwargs, reasoning + mode/budget, and task-specific overrides. 4. Reasoning traces are enabled, disabled, parsed, stripped, or ignored consistently between runs. 5. The number of evaluated and scored samples/repeats matches for each task and diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index ba1d33500c8..b147dac670a 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -128,6 +128,9 @@ Read [Step 6](references/launcher-workflow.md#step-6--multi-node). Read [Step 7](references/launcher-workflow.md#step-7--interceptors). +If the user asks to avoid doom loops (looping reasoning), or a canary under a long +`max_new_tokens` runs far longer than expected, read [doom_loops.md](references/doom_loops.md). + ### Step 7.5 — Container registry auth (SLURM private images only) Read [Step 7.5](references/launcher-workflow.md#step-75--container-registry-auth-slurm-private-images-only). diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md new file mode 100644 index 00000000000..c18123af9ae --- /dev/null +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -0,0 +1,60 @@ +# Doom Loops: Stop Repeating Reasoning Early + +Damaged (often quantized) reasoning models can repeat one block verbatim until +`max_new_tokens`, burning the full budget on a sample that scores wrong anyway. +vLLM ≥ 0.17.0 ends such a request through the per-request `repetition_detection` field. + +- **User asks** to avoid doom loops or looping reasoning: add it to every task. +- **Suspected** (under a long `max_new_tokens` a canary runs far longer than its + sample count implies, a few requests stay `Running` after the rest finish, or + `finish_reason: length` responses end in one block repeated verbatim): show the + evidence and ask first, since it changes the eval protocol. Long but varied + reasoning, e.g. from a high reasoning effort, is not a loop. + +## Config + +Add beside the existing entries in +`evaluation.nemo_evaluator_config.target.api_endpoint.adapter_config.params_to_add` +(nel-next: `services..proxy.extra_body`): + +```yaml + repetition_detection: + max_pattern_size: 256 # longest chunk checked, in tokens; sets the CPU cost + min_pattern_size: 32 # shortest chunk; lower fires on text like 1,000,000,000 + min_count: 3 # back-to-back copies that end the request +``` + +AA-LCR loops restate paragraphs up to ~4K tokens long, so on `ns_aa_lcr` set only +`repetition_detection.max_pattern_size: 1024` at the same path in the task's +`nemo_evaluator_config`. NEL deep-merges task config over the top level, so the +other entries survive; check the dry-run's `config_ef.yaml` block. A hit ends the +request with `finish_reason: "repetition"`, keeping the text so far; a loop inside +the reasoning leaves no final answer. + +**Keep these values**; larger sizes add scheduler CPU (~1 ms per request per decode +step at 4096) and cut more escaped-loop answers. + +## Guards + +- **Backend.** vLLM < 0.17.0 silently ignores the field. Check the image's vLLM + version; for an external endpoint, run this probe. `logit_bias` forces one token + every step, so the output repeats at once: `repetition` means the field works, + `length` means it was ignored. + + ```bash + curl -s "$BASE_URL/v1/completions" -H 'Content-Type: application/json' -d '{ + "model": "", "prompt": "Hi", "max_tokens": 16, + "temperature": 0, "logit_bias": {"1000": 100}, "repetition_detection": + {"max_pattern_size": 4, "min_pattern_size": 1, "min_count": 2}}' \ + | jq -r '.choices[0].finish_reason' + ``` + + SGLang and TRT-LLM don't support it: tell the user and don't add the field. +- **Matched and recorded.** Use identical values for baseline and candidate, and add + them to `export.mlflow.description` and the tags. +- **Report scores as detector-on.** Ending the request also cuts loops a damaged + model would have escaped: IQ1 lost up to 0.75 pt per benchmark at 512 (1.5 pt on + AA-LCR at 4096), IQ2 at most 0.13 pt, BF16 nothing. Published scores ran without it. + +Ending only the reasoning (unmerged vLLM PR #52677) lets the model still answer but +can inflate damaged-model scores: use it only when the user asks, matched on both sides. diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index 03f1160056f..57b1859faa3 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -92,7 +92,7 @@ percentages, explicit denominators, and artifact paths before the score: |---|---| | Coverage | Expected trials including repeats; completed/scored, failed/skipped/missing; score denominator and whether failures receive zero or are excluded. | | Timeouts | Timed-out / all request attempts; affected unique trials / expected trials; terminal-timeout trials / expected trials. Separate recovered retries and terminal failures by layer: client/proxy, agent/task, judge, sandbox/verifier. | -| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length`; distinguish output caps, context exhaustion, and agent step/total-token limits where possible. | +| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length` (and [doom-loop](doom_loops.md) `repetition` stops, counted separately); distinguish output caps, context exhaustion, and agent step/total-token limits where possible. | | Limits | Effective request/proxy, agent/task, judge/verifier, output-token and context limits, timeout strategy, overrides, concurrency, sharding, and serving setup. | Deduplicate resumed records by sample/trial/repeat ID; keep request attempts diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 5f7789d8660..9207e78b080 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -188,5 +188,18 @@ "Sets NEMO_EVALUATOR_TRUST_PRE_CMD=1 before launching", "Grades baseline and candidate with the same pinned judge rather than reusing an older LiveCodeBench score" ] + }, + { + "name": "doom-loop-repetition-stop", + "skills": ["evaluation"], + "query": "Run GPQA and AA-LCR on my NVFP4 Qwen3 checkpoint and its BF16 baseline on SLURM with vLLM, max_new_tokens 65536. Please avoid doom loops.", + "files": [], + "expected_behavior": [ + "Adds repetition_detection {max_pattern_size: 256, min_pattern_size: 32, min_count: 3} under the top-level adapter_config.params_to_add beside the existing entries, not a smaller min_pattern_size", + "Overrides max_pattern_size to 1024 on ns_aa_lcr only, not 4096 and not on every task", + "Requires a vLLM >= 0.17.0 deploy image, since older vLLM silently ignores the field", + "Uses the identical setting for baseline and candidate and records it in the MLflow description and tags", + "Counts finish_reason repetition stops alongside length stops during run validation and reports them separately" + ] } ]