From e161fa7706a0201827cc448bc9e0718cbdf6bc17 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Fri, 16 Oct 2026 21:43:07 +0000 Subject: [PATCH 1/6] Add doom-loop repetition stop to the evaluation skill When the user asks to avoid doom loops, or a canary under a long max_new_tokens runs far longer than expected, the evaluation skill now adds vLLM's per-request repetition_detection field (512/32/3) through adapter_config.params_to_add, with guards for the vLLM version, matched baseline/candidate settings, MLflow recording and run validation. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- CHANGELOG.rst | 4 + .../modelopt/skills/compare-results/SKILL.md | 4 +- plugins/modelopt/skills/evaluation/SKILL.md | 3 + .../evaluation/references/doom_loops.md | 78 +++++++++++++++++++ .../references/launcher-workflow.md | 2 +- .../evaluation/references/run-validation.md | 2 +- .../skills/evaluation/tests/evals.json | 12 +++ 7 files changed, 101 insertions(+), 4 deletions(-) create mode 100644 plugins/modelopt/skills/evaluation/references/doom_loops.md diff --git a/CHANGELOG.rst b/CHANGELOG.rst index eb4102dd116..a262d05d1b6 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -15,6 +15,10 @@ Changelog - Add the **external draft** speculative-decoding mode: train a pretrained standalone draft model against a cached base model, instead of grafting a draft head onto the target. Select it with ``mtsp.convert(draft, [("external", cfg)])`` or the ``speculative_external`` recipe; the draft keeps its own embeddings and lm_head and exports as an ordinary HF checkpoint. +*Misc* + +- The ``evaluation`` agent skill can stop looping ("doom loop") reasoning early: ask it to avoid doom loops and it adds vLLM's ``repetition_detection`` request field (vLLM 0.17.0 or later) to every task, ending a request once its output repeats. Use the same setting for baseline and candidate runs. + **Backward Breaking Changes** **Deprecations** diff --git a/plugins/modelopt/skills/compare-results/SKILL.md b/plugins/modelopt/skills/compare-results/SKILL.md index 8b7e40f6b87..1cb573e64a8 100644 --- a/plugins/modelopt/skills/compare-results/SKILL.md +++ b/plugins/modelopt/skills/compare-results/SKILL.md @@ -54,8 +54,8 @@ the validated runs are comparable: 2. Task name, benchmark version, dataset split, container, harness, and task fragment match. 3. Generation settings match, including temperature, top_p, top_k, max tokens, - stop strings, chat-template kwargs, reasoning mode/budget, and task-specific - overrides. + stop strings, repetition detection, chat-template kwargs, reasoning + mode/budget, and task-specific overrides. 4. Reasoning traces are enabled, disabled, parsed, stripped, or ignored consistently between runs. 5. The number of evaluated and scored samples/repeats matches for each task and diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index ba1d33500c8..b147dac670a 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -128,6 +128,9 @@ Read [Step 6](references/launcher-workflow.md#step-6--multi-node). Read [Step 7](references/launcher-workflow.md#step-7--interceptors). +If the user asks to avoid doom loops (looping reasoning), or a canary under a long +`max_new_tokens` runs far longer than expected, read [doom_loops.md](references/doom_loops.md). + ### Step 7.5 — Container registry auth (SLURM private images only) Read [Step 7.5](references/launcher-workflow.md#step-75--container-registry-auth-slurm-private-images-only). diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md new file mode 100644 index 00000000000..6ac19842e6f --- /dev/null +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -0,0 +1,78 @@ +# Doom Loops: Stop Repeating Reasoning Early + +A damaged (often quantized) reasoning model can repeat one block verbatim until +`max_new_tokens`. Each such sample burns the full output budget and scores wrong +anyway. vLLM can end the request as soon as its output tail repeats. + +## When to add it + +- **User asks** ("avoid doom loops", "stop looping reasoning", "repetition + detection"): add it to every task they asked to run. +- **Suspected only**: under a long `max_new_tokens` (≥32K) a canary or short task + runs far longer than its sample count implies, the vLLM log keeps a few requests + `Running` after the rest finish, or many responses stop with + `finish_reason: length` and logged ones end in one block repeated verbatim (long + but varied reasoning is not a loop). Show the evidence and ask first: it changes + the eval protocol. + +## Config + +vLLM ≥ 0.17.0 accepts a per-request `repetition_detection` field. Add it beside the +existing entries in the top-level `target.api_endpoint.adapter_config.params_to_add`, +and confirm every task keeps it in the dry-run's resolved config: + +```yaml + adapter_config: + params_to_add: + repetition_detection: + max_pattern_size: 512 # longest repeating chunk checked, in tokens + min_pattern_size: 32 # shortest chunk; the false-fire lever + min_count: 3 # back-to-back copies that stop the request +``` + +nel-next: the same dict under `proxy.extra_body.repetition_detection`. + +On a hit vLLM ends the whole request with `finish_reason: "repetition"`. No final +answer is produced, so the sample scores as it would after looping to the cap. + +**Keep 512/32/3.** Replaying vLLM's detector on 1,152 healthy DeepSeek-R1 traces, +512/32/3 fired on none and still caught the degenerate loops; 512/4/3 fired on 4.2% of +correct math and 12.2% of complete code traces (`1,000,000,000`, `0.333333333333`, +`5 \times 5 \times 5`), each a correct answer turned wrong. Don't lower +`min_pattern_size` to catch more. Tokenizers that group digits (Llama 3) shift +these numbers. The check costs ~30 ms of scheduler time per decode step at 256 +running requests; `max_pattern_size: 256` halves it if throughput drops. + +## Guards + +- **Backend.** vLLM < 0.17.0 silently ignores the field (debug log only), so the + run looks protected but isn't. Check the deploy image's vLLM version; probe an + external endpoint (expect `repetition`; `length` means ignored): + + ```bash + curl -s "$BASE_URL/v1/completions" -H 'Content-Type: application/json' -d '{ + "model": "", "prompt": "ha ha ha ha ha ha", "max_tokens": 64, + "temperature": 0, "repetition_detection": + {"max_pattern_size": 4, "min_pattern_size": 1, "min_count": 2}}' \ + | jq -r '.choices[0].finish_reason' + ``` + + SGLang and TRT-LLM don't support it: tell the user and don't add the field. +- **Matched runs.** Baseline and candidate use the identical setting. Published + scores ran without it; say so when comparing against them. +- **Record it.** Append `repetition_detection=512/32/3` to + `export.mlflow.description` and add it as a tag; it is not otherwise queryable. +- **Validation.** In run-validation's Output limits row and Response gate, count + `repetition` stops with `length` stops but report them separately. A candidate + rate well above the baseline's is a quantization finding. + +## End only the reasoning (unmerged vLLM PR) + +vLLM PR #52677 (Model Runner V2 only; check whether it has merged) instead forces +`reasoning_end_str` on a loop so the model still answers, configured server-side +with `--reasoning-config` keys `loop_break_max_pattern_size`, +`loop_break_min_pattern_size` and `loop_break_min_count`. That can lift a damaged +model above its real score. Use it only when the user asks and their vLLM build +carries the PR, matched on both sides. Put a short transition phrase before +`` in `reasoning_end_str`: a bare `` sent Qwen3.8-27B-NVFP4 back +into the loop inside the answer. diff --git a/plugins/modelopt/skills/evaluation/references/launcher-workflow.md b/plugins/modelopt/skills/evaluation/references/launcher-workflow.md index 75c61e3c641..a40098d22a7 100644 --- a/plugins/modelopt/skills/evaluation/references/launcher-workflow.md +++ b/plugins/modelopt/skills/evaluation/references/launcher-workflow.md @@ -444,7 +444,7 @@ Fix unresolved `???`, bad Hydra overrides, missing env vars, invalid mounts, ima nel run --config -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10 ``` -Catches judge auth/rate-limits, container failures, sandbox issues, OOM, bad request formatting, low evaluated counts. Always inspect logs: +Catches judge auth/rate-limits, container failures, sandbox issues, OOM, bad request formatting, low evaluated counts. A canary that runs far longer than its sample count implies may be doom-looping — see `references/doom_loops.md`. Always inspect logs: ```bash nel status diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index 03f1160056f..9cfa059a9c5 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -92,7 +92,7 @@ percentages, explicit denominators, and artifact paths before the score: |---|---| | Coverage | Expected trials including repeats; completed/scored, failed/skipped/missing; score denominator and whether failures receive zero or are excluded. | | Timeouts | Timed-out / all request attempts; affected unique trials / expected trials; terminal-timeout trials / expected trials. Separate recovered retries and terminal failures by layer: client/proxy, agent/task, judge, sandbox/verifier. | -| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length`; distinguish output caps, context exhaustion, and agent step/total-token limits where possible. | +| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length` (and `repetition` under [doom-loop stops](doom_loops.md)); distinguish output caps, context exhaustion, and agent step/total-token limits where possible. | | Limits | Effective request/proxy, agent/task, judge/verifier, output-token and context limits, timeout strategy, overrides, concurrency, sharding, and serving setup. | Deduplicate resumed records by sample/trial/repeat ID; keep request attempts diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 5f7789d8660..84da0a622ae 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -188,5 +188,17 @@ "Sets NEMO_EVALUATOR_TRUST_PRE_CMD=1 before launching", "Grades baseline and candidate with the same pinned judge rather than reusing an older LiveCodeBench score" ] + }, + { + "name": "doom-loop-repetition-stop", + "skills": ["evaluation"], + "query": "Run GPQA and AIME 2025 on my NVFP4 Qwen3 checkpoint and its BF16 baseline on SLURM with vLLM, max_new_tokens 65536. Please avoid doom loops.", + "files": [], + "expected_behavior": [ + "Adds repetition_detection {max_pattern_size: 512, min_pattern_size: 32, min_count: 3} under adapter_config.params_to_add beside the existing entries, not a smaller min_pattern_size", + "Requires a vLLM >= 0.17.0 deploy image, since older vLLM silently ignores the field", + "Uses the identical setting for baseline and candidate and records it in the MLflow description and tags", + "Counts finish_reason repetition stops alongside length stops during run validation and reports them separately" + ] } ] From 75dd93dac54c1b503cef2eb8fa76962267993f46 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Sat, 17 Oct 2026 16:32:16 +0000 Subject: [PATCH 2/6] Add quantized-model replay evidence to the doom-loop page A replay of vLLM's detector on Qwen3.8-Flash-Next BF16 and IQ-quantized eval outputs keeps 512/32/3 as the default: it catches 99-100% of IQ1 SciCode/GPQA loops but 85-90% of AA-LCR paragraph cycles, and ending the request also cuts loops a damaged model would have escaped, so IQ1 scores drop by up to 0.75 pt with it on. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- .../evaluation/references/doom_loops.md | 31 +++++++++++++------ 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md index 6ac19842e6f..9b74d6f045b 100644 --- a/plugins/modelopt/skills/evaluation/references/doom_loops.md +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -32,16 +32,25 @@ and confirm every task keeps it in the dry-run's resolved config: nel-next: the same dict under `proxy.extra_body.repetition_detection`. -On a hit vLLM ends the whole request with `finish_reason: "repetition"`. No final -answer is produced, so the sample scores as it would after looping to the cap. +On a hit vLLM ends the whole request with `finish_reason: "repetition"`, so the +sample gets no final answer. Most loops run to the cap anyway, but a badly damaged +model sometimes escapes one and answers (see Guards). -**Keep 512/32/3.** Replaying vLLM's detector on 1,152 healthy DeepSeek-R1 traces, -512/32/3 fired on none and still caught the degenerate loops; 512/4/3 fired on 4.2% of -correct math and 12.2% of complete code traces (`1,000,000,000`, `0.333333333333`, -`5 \times 5 \times 5`), each a correct answer turned wrong. Don't lower -`min_pattern_size` to catch more. Tokenizers that group digits (Llama 3) shift -these numbers. The check costs ~30 ms of scheduler time per decode step at 256 -running requests; `max_pattern_size: 256` halves it if throughput drops. +**Keep 512/32/3.** Replays of vLLM's detector on finished eval outputs: + +- **Healthy models:** one fire across 1,152 healthy DeepSeek-R1 traces and 7,471 + non-looping Qwen3.8-Flash-Next BF16 generations, on an answer that was wrong + anyway. 512/4/3 fired on 4.2% of correct R1 math and 12.2% of complete code + traces (`1,000,000,000`, `0.333333333333`, `5 \times 5 \times 5`), each a correct + answer turned wrong. Don't lower `min_pattern_size` to catch more. +- **IQ1-quantized Qwen3.8-Flash-Next:** caught 99-100% of SciCode and GPQA loops + and saved 18-67% of generated tokens. AA-LCR loops cycle whole paragraphs + (periods up to ~4K tokens), so 512 caught 85-90%; 1024 catches 91-93% and 4096 + all, at the cost of more lost answers. +- **Cost:** scheduler time grows with `max_pattern_size`, ~30 ms per decode step at + 256 running requests for 512 (256 halves it, 1024 doubles it). + +Tokenizers that group digits (Llama 3) shift these numbers. ## Guards @@ -60,6 +69,10 @@ running requests; `max_pattern_size: 256` halves it if throughput drops. SGLang and TRT-LLM don't support it: tell the user and don't add the field. - **Matched runs.** Baseline and candidate use the identical setting. Published scores ran without it; say so when comparing against them. +- **Damaged checkpoints score lower with it.** Ending the request also cuts loops + the model would have escaped and answered: at 512/32/3 this cost IQ1 checkpoints + 0.1-0.75 pt on GPQA and AA-LCR (1.5 pt on AA-LCR at 4096), IQ2 at most 0.13 pt + and BF16 nothing. Report such scores as detector-on. - **Record it.** Append `repetition_detection=512/32/3` to `export.mlflow.description` and add it as a tag; it is not otherwise queryable. - **Validation.** In run-validation's Output limits row and Response gate, count From cc63ab78bdab193d7cee2236e78ab91f014a715a Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Sat, 17 Oct 2026 22:27:51 +0000 Subject: [PATCH 3/6] Recommend 256/32/3 for doom-loop stops, 1024/32/3 on AA-LCR The IQ-quantized Qwen3.8-Flash-Next replay shows SciCode/GPQA loops are short (99th-percentile period under 160 tokens): 256 saves the same tokens as 512-4096 and fires on the same non-loop generations, at under half 512's scheduler cost (49 vs 117 us per request per step, measured on vLLM's own check). AA-LCR loops restate paragraphs with periods up to ~4K tokens, so it gets 1024 (91-93% caught, same BF16/IQ2 fires as 512); 4096 catches all but cuts at least twice as many IQ1 answers and costs ~1 ms per request per step. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- .../evaluation/references/doom_loops.md | 40 +++++++++++-------- .../skills/evaluation/tests/evals.json | 5 ++- 2 files changed, 27 insertions(+), 18 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md index 9b74d6f045b..efe66398816 100644 --- a/plugins/modelopt/skills/evaluation/references/doom_loops.md +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -18,37 +18,43 @@ anyway. vLLM can end the request as soon as its output tail repeats. ## Config vLLM ≥ 0.17.0 accepts a per-request `repetition_detection` field. Add it beside the -existing entries in the top-level `target.api_endpoint.adapter_config.params_to_add`, -and confirm every task keeps it in the dry-run's resolved config: +existing entries in the top-level `target.api_endpoint.adapter_config.params_to_add`: ```yaml adapter_config: params_to_add: repetition_detection: - max_pattern_size: 512 # longest repeating chunk checked, in tokens + max_pattern_size: 256 # longest repeating chunk checked, in tokens; sets the CPU cost min_pattern_size: 32 # shortest chunk; the false-fire lever min_count: 3 # back-to-back copies that stop the request ``` +On `ns_aa_lcr`, set `max_pattern_size: 1024` under the same path in the task's own +`nemo_evaluator_config`. Confirm each task's values in the dry-run's resolved config. nel-next: the same dict under `proxy.extra_body.repetition_detection`. On a hit vLLM ends the whole request with `finish_reason: "repetition"`, so the sample gets no final answer. Most loops run to the cap anyway, but a badly damaged model sometimes escapes one and answers (see Guards). -**Keep 512/32/3.** Replays of vLLM's detector on finished eval outputs: +**Use 256/32/3, and 1024/32/3 on AA-LCR.** From replays of vLLM's detector on +finished eval outputs, including IQ1/IQ2-quantized Qwen3.8-Flash-Next: -- **Healthy models:** one fire across 1,152 healthy DeepSeek-R1 traces and 7,471 +- **Healthy models:** no fire on 1,152 healthy DeepSeek-R1 traces and one on 7,471 non-looping Qwen3.8-Flash-Next BF16 generations, on an answer that was wrong anyway. 512/4/3 fired on 4.2% of correct R1 math and 12.2% of complete code traces (`1,000,000,000`, `0.333333333333`, `5 \times 5 \times 5`), each a correct answer turned wrong. Don't lower `min_pattern_size` to catch more. -- **IQ1-quantized Qwen3.8-Flash-Next:** caught 99-100% of SciCode and GPQA loops - and saved 18-67% of generated tokens. AA-LCR loops cycle whole paragraphs - (periods up to ~4K tokens), so 512 caught 85-90%; 1024 catches 91-93% and 4096 - all, at the cost of more lost answers. -- **Cost:** scheduler time grows with `max_pattern_size`, ~30 ms per decode step at - 256 running requests for 512 (256 halves it, 1024 doubles it). +- **Most loops are short.** Over half have a period of 1-4 tokens; on SciCode and + GPQA the 99th percentile is under 160. There 256 catches 98-100% of IQ1 loops + and saves the same 18-64% of generated tokens as 512-4096 (within 0.2 pt). +- **AA-LCR loops restate whole paragraphs**, with periods up to ~4K tokens: 256 + catches 77-86%, 512 85-90%, 1024 91-93%, saving 57-73% of IQ1's tokens. At 1024, + BF16 and IQ2 fire on exactly the generations they fire on at 512. 4096 catches + all, but cuts at least twice as many correct IQ1 answers as 512. +- **Cost** is vLLM scheduler CPU time per running request per decode step: ~50 µs + at 256, 120 µs at 512, 250 µs at 1024, 1 ms at 4096. The default adds ~13 ms + per step at 256 requests; 1024 adds 4-8 ms at AA-LCR's 16-32. Tokenizers that group digits (Llama 3) shift these numbers. @@ -70,11 +76,13 @@ Tokenizers that group digits (Llama 3) shift these numbers. - **Matched runs.** Baseline and candidate use the identical setting. Published scores ran without it; say so when comparing against them. - **Damaged checkpoints score lower with it.** Ending the request also cuts loops - the model would have escaped and answered: at 512/32/3 this cost IQ1 checkpoints - 0.1-0.75 pt on GPQA and AA-LCR (1.5 pt on AA-LCR at 4096), IQ2 at most 0.13 pt - and BF16 nothing. Report such scores as detector-on. -- **Record it.** Append `repetition_detection=512/32/3` to - `export.mlflow.description` and add it as a tag; it is not otherwise queryable. + the model would have escaped and answered. IQ1 checkpoints lost up to 0.75 pt per + benchmark (GPQA is the same at every size; AA-LCR at 1024 falls between 512's + 0.75 pt and 4096's 1.5 pt), IQ2 at most 0.13 pt and BF16 nothing. Report such + scores as detector-on. +- **Record it.** Append `repetition_detection=256/32/3, AA-LCR 1024/32/3` (the + values used) to `export.mlflow.description` and add it as a tag; it is not + otherwise queryable. - **Validation.** In run-validation's Output limits row and Response gate, count `repetition` stops with `length` stops but report them separately. A candidate rate well above the baseline's is a quantization finding. diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 84da0a622ae..9207e78b080 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -192,10 +192,11 @@ { "name": "doom-loop-repetition-stop", "skills": ["evaluation"], - "query": "Run GPQA and AIME 2025 on my NVFP4 Qwen3 checkpoint and its BF16 baseline on SLURM with vLLM, max_new_tokens 65536. Please avoid doom loops.", + "query": "Run GPQA and AA-LCR on my NVFP4 Qwen3 checkpoint and its BF16 baseline on SLURM with vLLM, max_new_tokens 65536. Please avoid doom loops.", "files": [], "expected_behavior": [ - "Adds repetition_detection {max_pattern_size: 512, min_pattern_size: 32, min_count: 3} under adapter_config.params_to_add beside the existing entries, not a smaller min_pattern_size", + "Adds repetition_detection {max_pattern_size: 256, min_pattern_size: 32, min_count: 3} under the top-level adapter_config.params_to_add beside the existing entries, not a smaller min_pattern_size", + "Overrides max_pattern_size to 1024 on ns_aa_lcr only, not 4096 and not on every task", "Requires a vLLM >= 0.17.0 deploy image, since older vLLM silently ignores the field", "Uses the identical setting for baseline and candidate and records it in the MLflow description and tags", "Counts finish_reason repetition stops alongside length stops during run validation and reports them separately" From 38f934dbece38139e513f39ed84ff24669060b8d Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Sun, 18 Oct 2026 06:33:02 +0000 Subject: [PATCH 4/6] Compress the doom-loop skill page Keep only what changes agent behavior: the trigger, config with the AA-LCR override, one line of evidence for the values, and the guards. The full replay numbers stay in the PR description. Move the validation rule into run-validation.md and drop the launcher-workflow canary note that SKILL.md's trigger already covers. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- .../evaluation/references/doom_loops.md | 108 ++++++------------ .../references/launcher-workflow.md | 2 +- .../evaluation/references/run-validation.md | 2 +- 3 files changed, 36 insertions(+), 76 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md index efe66398816..fcb9e56fbcf 100644 --- a/plugins/modelopt/skills/evaluation/references/doom_loops.md +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -1,68 +1,41 @@ # Doom Loops: Stop Repeating Reasoning Early -A damaged (often quantized) reasoning model can repeat one block verbatim until -`max_new_tokens`. Each such sample burns the full output budget and scores wrong -anyway. vLLM can end the request as soon as its output tail repeats. +Damaged (often quantized) reasoning models can repeat one block verbatim until +`max_new_tokens`, burning the full budget on a sample that scores wrong anyway. +vLLM ≥ 0.17.0 ends such a request through the per-request `repetition_detection` field. -## When to add it - -- **User asks** ("avoid doom loops", "stop looping reasoning", "repetition - detection"): add it to every task they asked to run. -- **Suspected only**: under a long `max_new_tokens` (≥32K) a canary or short task - runs far longer than its sample count implies, the vLLM log keeps a few requests - `Running` after the rest finish, or many responses stop with - `finish_reason: length` and logged ones end in one block repeated verbatim (long - but varied reasoning is not a loop). Show the evidence and ask first: it changes - the eval protocol. +- **User asks** to avoid doom loops or looping reasoning: add it to every task. +- **Suspected** (under a long `max_new_tokens` a canary runs far longer than its + sample count implies, a few requests stay `Running` after the rest finish, or + `finish_reason: length` responses end in one block repeated verbatim): show the + evidence and ask first, since it changes the eval protocol. ## Config -vLLM ≥ 0.17.0 accepts a per-request `repetition_detection` field. Add it beside the -existing entries in the top-level `target.api_endpoint.adapter_config.params_to_add`: +Add beside the existing entries in the top-level +`target.api_endpoint.adapter_config.params_to_add` (nel-next: `proxy.extra_body`): ```yaml - adapter_config: - params_to_add: repetition_detection: - max_pattern_size: 256 # longest repeating chunk checked, in tokens; sets the CPU cost - min_pattern_size: 32 # shortest chunk; the false-fire lever - min_count: 3 # back-to-back copies that stop the request + max_pattern_size: 256 # longest chunk checked, in tokens; sets the CPU cost + min_pattern_size: 32 # shortest chunk; lower fires on text like 1,000,000,000 + min_count: 3 # back-to-back copies that end the request ``` -On `ns_aa_lcr`, set `max_pattern_size: 1024` under the same path in the task's own -`nemo_evaluator_config`. Confirm each task's values in the dry-run's resolved config. -nel-next: the same dict under `proxy.extra_body.repetition_detection`. - -On a hit vLLM ends the whole request with `finish_reason: "repetition"`, so the -sample gets no final answer. Most loops run to the cap anyway, but a badly damaged -model sometimes escapes one and answers (see Guards). - -**Use 256/32/3, and 1024/32/3 on AA-LCR.** From replays of vLLM's detector on -finished eval outputs, including IQ1/IQ2-quantized Qwen3.8-Flash-Next: +On `ns_aa_lcr`, override `max_pattern_size: 1024` in the task's own `adapter_config`: +its loops restate paragraphs up to ~4K tokens long. Confirm every task's values in +the dry-run's resolved config. A hit ends the whole request with +`finish_reason: "repetition"` and no answer. -- **Healthy models:** no fire on 1,152 healthy DeepSeek-R1 traces and one on 7,471 - non-looping Qwen3.8-Flash-Next BF16 generations, on an answer that was wrong - anyway. 512/4/3 fired on 4.2% of correct R1 math and 12.2% of complete code - traces (`1,000,000,000`, `0.333333333333`, `5 \times 5 \times 5`), each a correct - answer turned wrong. Don't lower `min_pattern_size` to catch more. -- **Most loops are short.** Over half have a period of 1-4 tokens; on SciCode and - GPQA the 99th percentile is under 160. There 256 catches 98-100% of IQ1 loops - and saves the same 18-64% of generated tokens as 512-4096 (within 0.2 pt). -- **AA-LCR loops restate whole paragraphs**, with periods up to ~4K tokens: 256 - catches 77-86%, 512 85-90%, 1024 91-93%, saving 57-73% of IQ1's tokens. At 1024, - BF16 and IQ2 fire on exactly the generations they fire on at 512. 4096 catches - all, but cuts at least twice as many correct IQ1 answers as 512. -- **Cost** is vLLM scheduler CPU time per running request per decode step: ~50 µs - at 256, 120 µs at 512, 250 µs at 1024, 1 ms at 4096. The default adds ~13 ms - per step at 256 requests; 1024 adds 4-8 ms at AA-LCR's 16-32. - -Tokenizers that group digits (Llama 3) shift these numbers. +**Keep these values.** In replays on BF16 and IQ1/IQ2 Qwen3.8-Flash-Next outputs, +256 caught 98-100% of SciCode/GPQA loops and 1024 caught 91-93% of AA-LCR's, with +one fire on 7,471 non-looping BF16 generations. Larger sizes add scheduler CPU +(~1 ms per request per decode step at 4096) and cut more escaped-loop answers. ## Guards -- **Backend.** vLLM < 0.17.0 silently ignores the field (debug log only), so the - run looks protected but isn't. Check the deploy image's vLLM version; probe an - external endpoint (expect `repetition`; `length` means ignored): +- **Backend.** vLLM < 0.17.0 silently ignores the field. Check the image's vLLM + version; probe an external endpoint (expect `repetition`, not `length`): ```bash curl -s "$BASE_URL/v1/completions" -H 'Content-Type: application/json' -d '{ @@ -73,27 +46,14 @@ Tokenizers that group digits (Llama 3) shift these numbers. ``` SGLang and TRT-LLM don't support it: tell the user and don't add the field. -- **Matched runs.** Baseline and candidate use the identical setting. Published - scores ran without it; say so when comparing against them. -- **Damaged checkpoints score lower with it.** Ending the request also cuts loops - the model would have escaped and answered. IQ1 checkpoints lost up to 0.75 pt per - benchmark (GPQA is the same at every size; AA-LCR at 1024 falls between 512's - 0.75 pt and 4096's 1.5 pt), IQ2 at most 0.13 pt and BF16 nothing. Report such - scores as detector-on. -- **Record it.** Append `repetition_detection=256/32/3, AA-LCR 1024/32/3` (the - values used) to `export.mlflow.description` and add it as a tag; it is not - otherwise queryable. -- **Validation.** In run-validation's Output limits row and Response gate, count - `repetition` stops with `length` stops but report them separately. A candidate - rate well above the baseline's is a quantization finding. - -## End only the reasoning (unmerged vLLM PR) - -vLLM PR #52677 (Model Runner V2 only; check whether it has merged) instead forces -`reasoning_end_str` on a loop so the model still answers, configured server-side -with `--reasoning-config` keys `loop_break_max_pattern_size`, -`loop_break_min_pattern_size` and `loop_break_min_count`. That can lift a damaged -model above its real score. Use it only when the user asks and their vLLM build -carries the PR, matched on both sides. Put a short transition phrase before -`` in `reasoning_end_str`: a bare `` sent Qwen3.8-27B-NVFP4 back -into the loop inside the answer. +- **Matched and recorded.** Use identical values for baseline and candidate, and add + them to `export.mlflow.description` and the tags. +- **Report scores as detector-on.** Ending the request also cuts loops a damaged + model would have escaped: IQ1 lost up to 0.75 pt per benchmark at 512 (1.5 pt on + AA-LCR at 4096), IQ2 at most 0.13 pt, BF16 nothing. Published scores ran without it. + +Ending only the reasoning, so the model still answers, needs unmerged vLLM PR #52677 +(server-side `--reasoning-config` `loop_break_*` keys, Model Runner V2 only). It can +lift a damaged model above its real score: use it only when asked, matched on both +sides, with a transition phrase before `` in `reasoning_end_str` (a bare +`` re-entered the loop). diff --git a/plugins/modelopt/skills/evaluation/references/launcher-workflow.md b/plugins/modelopt/skills/evaluation/references/launcher-workflow.md index a40098d22a7..75c61e3c641 100644 --- a/plugins/modelopt/skills/evaluation/references/launcher-workflow.md +++ b/plugins/modelopt/skills/evaluation/references/launcher-workflow.md @@ -444,7 +444,7 @@ Fix unresolved `???`, bad Hydra overrides, missing env vars, invalid mounts, ima nel run --config -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10 ``` -Catches judge auth/rate-limits, container failures, sandbox issues, OOM, bad request formatting, low evaluated counts. A canary that runs far longer than its sample count implies may be doom-looping — see `references/doom_loops.md`. Always inspect logs: +Catches judge auth/rate-limits, container failures, sandbox issues, OOM, bad request formatting, low evaluated counts. Always inspect logs: ```bash nel status diff --git a/plugins/modelopt/skills/evaluation/references/run-validation.md b/plugins/modelopt/skills/evaluation/references/run-validation.md index 9cfa059a9c5..57b1859faa3 100644 --- a/plugins/modelopt/skills/evaluation/references/run-validation.md +++ b/plugins/modelopt/skills/evaluation/references/run-validation.md @@ -92,7 +92,7 @@ percentages, explicit denominators, and artifact paths before the score: |---|---| | Coverage | Expected trials including repeats; completed/scored, failed/skipped/missing; score denominator and whether failures receive zero or are excluded. | | Timeouts | Timed-out / all request attempts; affected unique trials / expected trials; terminal-timeout trials / expected trials. Separate recovered retries and terminal failures by layer: client/proxy, agent/task, judge, sandbox/verifier. | -| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length` (and `repetition` under [doom-loop stops](doom_loops.md)); distinguish output caps, context exhaustion, and agent step/total-token limits where possible. | +| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length` (and [doom-loop](doom_loops.md) `repetition` stops, counted separately); distinguish output caps, context exhaustion, and agent step/total-token limits where possible. | | Limits | Effective request/proxy, agent/task, judge/verifier, output-token and context limits, timeout strategy, overrides, concurrency, sharding, and serving setup. | Deduplicate resumed records by sample/trial/repeat ID; keep request attempts From ad025e138d9db44bdaf8a73b55f81e4e0ca045f9 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Sun, 18 Oct 2026 20:31:07 +0000 Subject: [PATCH 5/6] Address doom-loop review feedback - Probe forces one token with logit_bias, so its output repeats at once: repetition means the field works and length means it was ignored, with no inconclusive stop. - Give the full config paths, and state that NEL deep-merges the AA-LCR override (checked in NEL 0.2.6's get_eval_factory_config). - Note that long, varied reasoning such as from a high reasoning effort is not a loop. - Cut the replay evidence and the unmerged-PR paragraph to one line each. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- .../evaluation/references/doom_loops.md | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md index fcb9e56fbcf..a701c1a2aad 100644 --- a/plugins/modelopt/skills/evaluation/references/doom_loops.md +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -8,12 +8,14 @@ vLLM ≥ 0.17.0 ends such a request through the per-request `repetition_detectio - **Suspected** (under a long `max_new_tokens` a canary runs far longer than its sample count implies, a few requests stay `Running` after the rest finish, or `finish_reason: length` responses end in one block repeated verbatim): show the - evidence and ask first, since it changes the eval protocol. + evidence and ask first, since it changes the eval protocol. Long but varied + reasoning, e.g. from a high reasoning effort, is not a loop. ## Config -Add beside the existing entries in the top-level -`target.api_endpoint.adapter_config.params_to_add` (nel-next: `proxy.extra_body`): +Add beside the existing entries in +`evaluation.nemo_evaluator_config.target.api_endpoint.adapter_config.params_to_add` +(nel-next: `services..proxy.extra_body`): ```yaml repetition_detection: @@ -22,25 +24,26 @@ Add beside the existing entries in the top-level min_count: 3 # back-to-back copies that end the request ``` -On `ns_aa_lcr`, override `max_pattern_size: 1024` in the task's own `adapter_config`: -its loops restate paragraphs up to ~4K tokens long. Confirm every task's values in -the dry-run's resolved config. A hit ends the whole request with -`finish_reason: "repetition"` and no answer. +AA-LCR loops restate paragraphs up to ~4K tokens long, so on `ns_aa_lcr` set only +`repetition_detection.max_pattern_size: 1024` at the same path in the task's +`nemo_evaluator_config`. NEL deep-merges task config over the top level, so the +other entries survive; check the dry-run's `config_ef.yaml` block. A hit ends the +whole request with `finish_reason: "repetition"` and no answer. -**Keep these values.** In replays on BF16 and IQ1/IQ2 Qwen3.8-Flash-Next outputs, -256 caught 98-100% of SciCode/GPQA loops and 1024 caught 91-93% of AA-LCR's, with -one fire on 7,471 non-looping BF16 generations. Larger sizes add scheduler CPU -(~1 ms per request per decode step at 4096) and cut more escaped-loop answers. +**Keep these values**; larger sizes add scheduler CPU (~1 ms per request per decode +step at 4096) and cut more escaped-loop answers. ## Guards - **Backend.** vLLM < 0.17.0 silently ignores the field. Check the image's vLLM - version; probe an external endpoint (expect `repetition`, not `length`): + version; for an external endpoint, run this probe. `logit_bias` forces one token + every step, so the output repeats at once: `repetition` means the field works, + `length` means it was ignored. ```bash curl -s "$BASE_URL/v1/completions" -H 'Content-Type: application/json' -d '{ - "model": "", "prompt": "ha ha ha ha ha ha", "max_tokens": 64, - "temperature": 0, "repetition_detection": + "model": "", "prompt": "Hi", "max_tokens": 16, + "temperature": 0, "logit_bias": {"1000": 100}, "repetition_detection": {"max_pattern_size": 4, "min_pattern_size": 1, "min_count": 2}}' \ | jq -r '.choices[0].finish_reason' ``` @@ -52,8 +55,5 @@ one fire on 7,471 non-looping BF16 generations. Larger sizes add scheduler CPU model would have escaped: IQ1 lost up to 0.75 pt per benchmark at 512 (1.5 pt on AA-LCR at 4096), IQ2 at most 0.13 pt, BF16 nothing. Published scores ran without it. -Ending only the reasoning, so the model still answers, needs unmerged vLLM PR #52677 -(server-side `--reasoning-config` `loop_break_*` keys, Model Runner V2 only). It can -lift a damaged model above its real score: use it only when asked, matched on both -sides, with a transition phrase before `` in `reasoning_end_str` (a bare -`` re-entered the loop). +Ending only the reasoning (unmerged vLLM PR #52677) lets the model still answer but +can inflate damaged-model scores: use it only when the user asks, matched on both sides. From 728e4365a37da7e5ac0c3e0f7970d8f75b9a90f2 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Mon, 19 Oct 2026 17:38:23 +0000 Subject: [PATCH 6/6] Say a repetition stop keeps the text generated so far vLLM keeps the tokens generated before the stop, so a loop in the answer can leave a partial answer that still scores; only a loop inside the reasoning leaves no final answer. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- plugins/modelopt/skills/evaluation/references/doom_loops.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plugins/modelopt/skills/evaluation/references/doom_loops.md b/plugins/modelopt/skills/evaluation/references/doom_loops.md index a701c1a2aad..c18123af9ae 100644 --- a/plugins/modelopt/skills/evaluation/references/doom_loops.md +++ b/plugins/modelopt/skills/evaluation/references/doom_loops.md @@ -28,7 +28,8 @@ AA-LCR loops restate paragraphs up to ~4K tokens long, so on `ns_aa_lcr` set onl `repetition_detection.max_pattern_size: 1024` at the same path in the task's `nemo_evaluator_config`. NEL deep-merges task config over the top level, so the other entries survive; check the dry-run's `config_ef.yaml` block. A hit ends the -whole request with `finish_reason: "repetition"` and no answer. +request with `finish_reason: "repetition"`, keeping the text so far; a loop inside +the reasoning leaves no final answer. **Keep these values**; larger sizes add scheduler CPU (~1 ms per request per decode step at 4096) and cut more escaped-loop answers.