Add doom-loop repetition stop to the evaluation skill

CHANGELOG.rst

@@ -15,6 +15,10 @@ Changelog
 
 - Add the **external draft** speculative-decoding mode: train a pretrained standalone draft model against a cached base model, instead of grafting a draft head onto the target. Select it with ``mtsp.convert(draft, [("external", cfg)])`` or the ``speculative_external`` recipe; the draft keeps its own embeddings and lm_head and exports as an ordinary HF checkpoint.
 
+*Misc*
+
+- The ``evaluation`` agent skill can stop looping ("doom loop") reasoning early: ask it to avoid doom loops and it adds vLLM's ``repetition_detection`` request field (vLLM 0.17.0 or later) to every task, ending a request once its output repeats. Use the same setting for baseline and candidate runs.
+
 **Backward Breaking Changes**
 
 **Deprecations**

plugins/modelopt/skills/compare-results/SKILL.md

@@ -54,8 +54,8 @@ the validated runs are comparable:
 2. Task name, benchmark version, dataset split, container, harness, and task
    fragment match.
 3. Generation settings match, including temperature, top_p, top_k, max tokens,
-   stop strings, chat-template kwargs, reasoning mode/budget, and task-specific
-   overrides.
+   stop strings, repetition detection, chat-template kwargs, reasoning
+   mode/budget, and task-specific overrides.
 4. Reasoning traces are enabled, disabled, parsed, stripped, or ignored
    consistently between runs.
 5. The number of evaluated and scored samples/repeats matches for each task and

plugins/modelopt/skills/evaluation/SKILL.md

@@ -128,6 +128,9 @@ Read [Step 6](references/launcher-workflow.md#step-6--multi-node).
 
 Read [Step 7](references/launcher-workflow.md#step-7--interceptors).
 
+If the user asks to avoid doom loops (looping reasoning), or a canary under a long
+`max_new_tokens` runs far longer than expected, read [doom_loops.md](references/doom_loops.md).
+
 ### Step 7.5 — Container registry auth (SLURM private images only)
 
 Read [Step 7.5](references/launcher-workflow.md#step-75--container-registry-auth-slurm-private-images-only).

plugins/modelopt/skills/evaluation/references/doom_loops.md

@@ -0,0 +1,60 @@
+# Doom Loops: Stop Repeating Reasoning Early
+
+Damaged (often quantized) reasoning models can repeat one block verbatim until
+`max_new_tokens`, burning the full budget on a sample that scores wrong anyway.
+vLLM ≥ 0.17.0 ends such a request through the per-request `repetition_detection` field.
+
+- **User asks** to avoid doom loops or looping reasoning: add it to every task.
+- **Suspected** (under a long `max_new_tokens` a canary runs far longer than its
+  sample count implies, a few requests stay `Running` after the rest finish, or
+  `finish_reason: length` responses end in one block repeated verbatim): show the
+  evidence and ask first, since it changes the eval protocol. Long but varied
+  reasoning, e.g. from a high reasoning effort, is not a loop.
+
+## Config
+
+Add beside the existing entries in
+`evaluation.nemo_evaluator_config.target.api_endpoint.adapter_config.params_to_add`
+(nel-next: `services.<service>.proxy.extra_body`):
+
+```yaml
+            repetition_detection:
+              max_pattern_size: 256   # longest chunk checked, in tokens; sets the CPU cost
+              min_pattern_size: 32    # shortest chunk; lower fires on text like 1,000,000,000
+              min_count: 3            # back-to-back copies that end the request
+```
+
+AA-LCR loops restate paragraphs up to ~4K tokens long, so on `ns_aa_lcr` set only
+`repetition_detection.max_pattern_size: 1024` at the same path in the task's
+`nemo_evaluator_config`. NEL deep-merges task config over the top level, so the
+other entries survive; check the dry-run's `config_ef.yaml` block. A hit ends the
+request with `finish_reason: "repetition"`, keeping the text so far; a loop inside
+the reasoning leaves no final answer.
+
+**Keep these values**; larger sizes add scheduler CPU (~1 ms per request per decode
+step at 4096) and cut more escaped-loop answers.
+
+## Guards
+
+- **Backend.** vLLM < 0.17.0 silently ignores the field. Check the image's vLLM
+  version; for an external endpoint, run this probe. `logit_bias` forces one token
+  every step, so the output repeats at once: `repetition` means the field works,
+  `length` means it was ignored.
+
+  ```bash
+  curl -s "$BASE_URL/v1/completions" -H 'Content-Type: application/json' -d '{
+    "model": "<served_model_name>", "prompt": "Hi", "max_tokens": 16,
+    "temperature": 0, "logit_bias": {"1000": 100}, "repetition_detection":
+      {"max_pattern_size": 4, "min_pattern_size": 1, "min_count": 2}}' \
+    | jq -r '.choices[0].finish_reason'
+  ```
+
+  SGLang and TRT-LLM don't support it: tell the user and don't add the field.
+- **Matched and recorded.** Use identical values for baseline and candidate, and add
+  them to `export.mlflow.description` and the tags.
+- **Report scores as detector-on.** Ending the request also cuts loops a damaged
+  model would have escaped: IQ1 lost up to 0.75 pt per benchmark at 512 (1.5 pt on
+  AA-LCR at 4096), IQ2 at most 0.13 pt, BF16 nothing. Published scores ran without it.
+
+Ending only the reasoning (unmerged vLLM PR #52677) lets the model still answer but
+can inflate damaged-model scores: use it only when the user asks, matched on both sides.

plugins/modelopt/skills/evaluation/references/run-validation.md

@@ -92,7 +92,7 @@ percentages, explicit denominators, and artifact paths before the score:
 |---|---|
 | Coverage | Expected trials including repeats; completed/scored, failed/skipped/missing; score denominator and whether failures receive zero or are excluded. |
 | Timeouts | Timed-out / all request attempts; affected unique trials / expected trials; terminal-timeout trials / expected trials. Separate recovered retries and terminal failures by layer: client/proxy, agent/task, judge, sandbox/verifier. |
-| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length`; distinguish output caps, context exhaustion, and agent step/total-token limits where possible. |
+| Output limits | Limit-stopped / observed responses; affected unique trials / expected trials. Use termination metadata such as `finish_reason: length` (and [doom-loop](doom_loops.md) `repetition` stops, counted separately); distinguish output caps, context exhaustion, and agent step/total-token limits where possible. |
 | Limits | Effective request/proxy, agent/task, judge/verifier, output-token and context limits, timeout strategy, overrides, concurrency, sharding, and serving setup. |
 
 Deduplicate resumed records by sample/trial/repeat ID; keep request attempts

plugins/modelopt/skills/evaluation/tests/evals.json

@@ -188,5 +188,18 @@
       "Sets NEMO_EVALUATOR_TRUST_PRE_CMD=1 before launching",
       "Grades baseline and candidate with the same pinned judge rather than reusing an older LiveCodeBench score"
     ]
+  },
+  {
+    "name": "doom-loop-repetition-stop",
+    "skills": ["evaluation"],
+    "query": "Run GPQA and AA-LCR on my NVFP4 Qwen3 checkpoint and its BF16 baseline on SLURM with vLLM, max_new_tokens 65536. Please avoid doom loops.",
+    "files": [],
+    "expected_behavior": [
+      "Adds repetition_detection {max_pattern_size: 256, min_pattern_size: 32, min_count: 3} under the top-level adapter_config.params_to_add beside the existing entries, not a smaller min_pattern_size",
+      "Overrides max_pattern_size to 1024 on ns_aa_lcr only, not 4096 and not on every task",
+      "Requires a vLLM >= 0.17.0 deploy image, since older vLLM silently ignores the field",
+      "Uses the identical setting for baseline and candidate and records it in the MLflow description and tags",
+      "Counts finish_reason repetition stops alongside length stops during run validation and reports them separately"
+    ]
   }
 ]