[zoroyihan7]

• Description: what and why

  In a synthetic-benchmark session, a speculative-decoding variant whose draft copies tokens from the context (vLLM ngram) is KEPT on throughput that does not carry over to real text. In a 12h vLLM run (Qwen3-14B-FP8), ngram k=7 accepted 6.7-7.9 tokens per step on the random-prompt benchmark against 2.3-2.7 on the GSM8K eval the same server ran. It was reported at +95.67% when the gain that holds on real text is about +40%, which comes from fp8 KV. Details are in Speculative decoding (vLLM ngram) KEEPs on inflated acceptance in synthetic benchmark mode #1901.

  The benchmark seed is fixed, so the random prompts are reproducible; what inflates the draft is the workload itself. Random-token prompts held to their full output length (ignore_eos) drive the model into repetition loops, and a draft that copies from the context accepts almost every token of a loop.

  Change of approach. The first version of this PR compared the benchmark's acceptance length with the GSM8K eval's and refused a candidate whose benchmark acceptance exceeded the eval's by more than the 5% noise band. Review pointed out that acceptance length differs across datasets by design, so a threshold against GSM8K is not principled: no fixed ratio separates "inflated" from "this dataset is easier to draft". The reviewer's alternative is simpler and reproducible: refuse the methods that copy from the context on synthetic data, and leave model-based drafts alone. This PR now does that.

  1. Refuse a context-copying draft before it runs (_spec_decode_context_copy.py, called from explore's variant loop just before each launch, against the stack as it stands then, since a KEEP earlier in the round may have changed it). In a synthetic session, a candidate is refused with reason speculative_context_copy_on_synthetic when the launch it materializes runs a context-copying draft and its speculative settings differ from its base's, i.e. it introduces such a draft, switches to one, or tunes one the base already runs. The methods, checked against the pinned engines (vLLM 0.27.0 SpeculativeConfig, SGLang v0.5.21 SpeculativeAlgorithm):

     • vLLM --speculative-config method ngram, ngram_gpu (prompt lookup) or suffix (suffix decoding), or a model of ngram / [ngram] with no method, which vLLM resolves to ngram. The JSON object and its dotted --speculative-config.<key> flags are both read, the last setting winning like the engine.
     • SGLang --speculative-algorithm NGRAM (case-insensitive).
     • ATOM's native drafts are EAGLE/MTP/DSpark only.

     Both launches are built with _build_variant_yaml, the way the no-op probe and run_grid build them, so a draft reached through args, replace mode, envs or a server-args env (EXTRA_VLLM_ARGS) counts. A launch that cannot be materialized is judged on the variant's own args and envs. The refusal is recorded in skipped_dup / per_variant_outcomes with the method named, and the server is never launched for it.

     Not refused:

     • model-based drafts (MTP, EAGLE/EAGLE3, a draft model, DFlash/DSpark);
     • a base that already runs a copying draft, and a candidate that leaves its speculative settings as they are or drops the draft. The base is the operator's choice, and both sides of such a comparison carry the same draft. Only candidates that introduce or change a copying draft are refused;
     • AgentX sessions, which benchmark real datasets, and stack revalidations, which re-measure the stack as it stands.

  2. Open the eval phase (_subprocess_kill.py, unchanged from the first version): the watchdog now scans Magpie's nested benchmark_*/benchmark_stderr.log for HYPERLOOM_EVAL_START. It used to reach that file only through a nested server.log, so when the engine logged to the round's own server.log (SERVER_LOG) the eval was folded into measured. In the run above, the warmup round's whole GSM8K eval was recorded as measured, so its kv_metrics.json KV/GPU-power figures included eval traffic.

  Removed from the first version: the vLLM draft/accept counters in kv_metrics.json, the acceptance comparison and its speculative_acceptance_inflated / speculative_acceptance_unvalidated refusals, the extra noise band on speculative changes, and the rule that held a speculative KEEP unvalidated until a stack rebench. The noise band and the rebench guarded against picking the best of several noisy copying variants; with those variants refused, model-based drafts are graded like any other lever.

  Relation to fix(agentx): pin the draft and its acceptance when the baseline simulates acceptance #1885: complementary, no file overlap. fix(agentx): pin the draft and its acceptance when the baseline simulates acceptance #1885 pins the draft when the base simulates acceptance. This PR refuses a real copying draft on synthetic data.

• Linked issue(s): fixes Speculative decoding (vLLM ngram) KEEPs on inflated acceptance in synthetic benchmark mode #1901

• Tests: src/hyperloom/inference_optimizer/tests/test_spec_decode_context_copy.py (40 tests):

  • Methods: vLLM ngram / ngram_gpu / suffix, model: [ngram], quoted, attached, unquoted-JSON and dotted spellings, last-wins and a later --speculative-config replacing the object; SGLang NGRAM in either case; EAGLE3, MTP, a draft model, SGLang EAGLE3/NEXTN and no draft are not copying; a copying draft inside a server-args env is found.
  • Rule: introducing, switching to, tuning (including a dotted key or a speculative env) a copying draft is refused; a lever on a copying base, dropping the draft, switching to or introducing a model draft are not.
  • Explore end to end: the measured case (ngram k=7) is refused and never launched while an EAGLE3 variant and a lever in the same round run; a copying draft set through EXTRA_VLLM_ARGS is refused; a lever on a copying base runs; after a KEEP drops the base's ngram, a later variant restoring it in the same round is refused; AgentX and a stack revalidation are not filtered; a variant that cannot be materialized is judged on its own args.
  • Eval marker: the nested-stderr marker opens the eval phase, and a stale nested stderr from an earlier run stays silent.
  • Failing before: on main, the explore refusal tests and the eval-marker test fail. The same-round test also fails on this PR's previous revision, which checked every variant against the round's opening base.
  • Mutations, each killed: no AgentX guard; no stack-revalidation guard; no base comparison; suffix dropped; no model: [ngram] fallback; dotted keys ignored; unmaterializable variant read as empty; env values not scanned; SGLang case-sensitive; --speculative-config merging instead of replacing; no unquoted-JSON fallback; filter not applied; judged against the round's opening base instead of the current stack; nested stderr not scanned.
  • ruff check / ruff format --check (0.16.2) pass.

• Size/complexity triggers crossed: the rule lives in a new module; explore.py (already over the module-lines limit) gains the filter function and its call, and ExploreExecutor.__call__ gains the per-variant check. The baseline-raise label stays for that growth.

• If this simplifies or refactors: not a refactor.

• Observable effect: in a synthetic session, an explore variant that introduces or changes a vLLM ngram/ngram_gpu/suffix or SGLang NGRAM draft is skipped before launch with reason speculative_context_copy_on_synthetic and a message naming the method. kv_metrics.json now has an eval phase on rounds whose eval marker sat in Magpie's nested stderr. Everything else is graded as before.

• Breaking changes: no

• PR addresses single concern: yes

• Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: no. The inflation comes from the synthetic workload interacting with a copying draft, and the rule belongs to Hyperloom's candidate selection.

🤖 Generated with Claude Code

[github-actions]

Code metrics gate: PASSED with waived findings (baseline-raise)

Judged: the 4 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Worse	Baseline out of date	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	1	0	72	+0
Cognitive complexity	fails > 30	0	1	0	284	+0
Function length (lines)	fails > 80	0	1	0	449	+0
Nested block depth	fails > 5	0	0	0	27	+0
Module length (lines)	warns > 800, fails > 1200	0	1	0	53	+0
Duplicated lines per file	fails > 0	0	0	0	118	+0
Dead code (vulture)	fails > 0	0	0	0	2	+0

Check	Findings	Effect
Comments	0
English only	0
Production code imports test code	0
Repeated literals	0
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	0

Worse than the baseline (waived by the baseline-raise label): 4

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/actions/executors/explore.py:759 ExploreExecutor.__call__	Cognitive complexity	369	377
src/hyperloom/orchestrator/actions/executors/explore.py:759 ExploreExecutor.__call__	Cyclomatic complexity	76	78
src/hyperloom/orchestrator/actions/executors/explore.py:759 ExploreExecutor.__call__	Function length (lines)	1124	1156
src/hyperloom/orchestrator/actions/executors/explore.py:1	Module length (lines)	1880	1965

How to fix

• New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
• Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
• Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
• Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1

Module	Lines	Limit
src/hyperloom/orchestrator/actions/executors/_subprocess_kill.py	1173	warns > 800

Already over at the base (backlog, not this change's): 1

Unit	Dimension	Base	Now
src/hyperloom/orchestrator/actions/executors/explore.py:576 filter_baseline_noop_variants	Function length (lines)	112	112

93 findings in files this change does not touch

Outside this change (informational): 93

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/enablement/recipe/projections.py project_launch_evidence	Cognitive complexity	absent	36
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison	Cognitive complexity	absent	34
src/hyperloom_kb/collect/mapping.py _experience	Cognitive complexity	absent	31
src/hyperloom_kb/schema.py FieldDeclaration.__post_init__	Cognitive complexity	absent	32
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver	Cognitive complexity	absent	36
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start	Cognitive complexity	absent	40
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate	Cognitive complexity	absent	44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt	Cognitive complexity	absent	38
src/hyperloom_kb/tests/test_cli.py	Duplicated lines per file	absent	18
src/hyperloom_kb/tests/test_global_identity_and_pull.py	Duplicated lines per file	absent	13
src/hyperloom_kb/tests/test_local_state.py	Duplicated lines per file	absent	18
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment	Function length (lines)	absent	112
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup	Function length (lines)	absent	89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology	Function length (lines)	absent	92
src/hyperloom/inference_optimizer/cli/__init__.py _resume_session	Function length (lines)	absent	171
src/hyperloom/inference_optimizer/cli/__init__.py _seed_fresh_session	Function length (lines)	absent	183
src/hyperloom/inference_optimizer/multi_node/cli.py _restart_multinode	Function length (lines)	absent	107
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser	Function length (lines)	absent	84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors	Function length (lines)	absent	90
src/hyperloom_kb/http_service.py RequestHandler._dispatch	Function length (lines)	absent	91
src/hyperloom_kb/sync.py GlobalSync.pull	Function length (lines)	absent	86
src/kernelforge/cli/forge_loop_command.py _RunReporter.build_result	Function length (lines)	absent	91
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver	Function length (lines)	absent	140
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start	Function length (lines)	absent	154
src/kernelforge/loop/knowledge_integration.py _measure_warmstart_field	Function length (lines)	absent	83

...and 68 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

[xiaofei-zheng]

PR #1904 -- fix(explore): validate speculative decoding on a synthetic benchmark against its real-text acceptance

What it does: on the synthetic benchmark a vLLM ngram draft copies the model's repetition loops and is measured at an acceptance it never reaches on real text, so explore KEEPs it on a throughput that measures the workload rather than the draft (+95.67% reported where about +40% holds). The PR brackets vLLM's draft/accept counters per phase, opens the eval phase by scanning Magpie's nested benchmark_stderr.log for the eval marker, and makes explore refuse a candidate whose materialized launch changes speculative decoding when its benchmark acceptance exceeds its own GSM8K eval acceptance beyond the session noise band or when either figure cannot be read; such a KEEP counts toward cumulative gain only after a stack rebench. Variants that change nothing about the draft, AgentX sessions and stack revalidations are graded exactly as before.

Blocking issues: none

Checked: _spec_decode_acceptance.py (speculative_delta, _speculative_settings, variant_acceptance, _phase_counts, judge_speculative_variant); explore.py:1391-1498,1585,1672 against the decision-round run_grid call at :1345 and all three keep_threshold_pct uses; _grid_runner._build_variant_yaml / SERVER_LOG pinning at :703; _workload_envs JSON compaction; _subprocess_kill._resolve_scan_logs / _stale_scan_log_sizes / _communicate_with_watchdog phase marking; _kv_metrics._spec_decode_window / acceptance_length / _adopt_aiperf_samples; writeback.validate + all five call sites and lift_to_current_best's entry_extra filter; grading.resolved_grading / perf_metric.parse_intvty_noise_pct / cli/bootstrap grading seed; the executor registry for sibling grading paths; prompts/references/speculative_decoding.md.
Ran: the new test file minus the three Windows-only fcntl collection failures (44 passed, same failures on main); _scan_logs_increment eval-marker repro on both trees (base saw_eval_start=False, head True, stale nested stderr silent on both); a materialization repro through _build_variant_yaml + config_server_argv for a spaced --speculative-config JSON (compacted, delta sets); ruff --select C901 at head and at the merge base.
SKIPPED: V4 -- openprs.txt carries # PARTIAL: lines for #1032, #1154, #1276, #1675, #1869 (>100 files each), so overlap with those PRs was not compared.
Base: 14e127d | Head: 4dd5797

[xiaofei-zheng]

LGTM

[xiaofei-zheng]

No blocking issues. Verification depth is in the review comment above (checked paths, repros run, base/head comparisons).

[zoroyihan7]

@xiaofei-zheng thanks for the feedback. You're right that acceptance length differs across datasets by design, so comparing against GSM8K with a 5% threshold had no principled basis. I've reworked the PR along the lines you suggested.

• Synthetic sessions now refuse, before launch, any candidate that introduces or changes a draft that copies from the context. That covers vLLM ngram / ngram_gpu / suffix and SGLang NGRAM, with method names checked against vLLM 0.27.0 and SGLang v0.5.21. The refusal reason is speculative_context_copy_on_synthetic.
• MTP, EAGLE and draft-model drafts are graded like any other lever. A base that already copies is the operator's choice, so it is not refused.
• Removed: the acceptance counters, the GSM8K comparison and its two refusals, the extra noise band, and the stack-rebench hold.
• Kept: the eval-start marker fix for Magpie's nested benchmark_stderr.log.

The PR description and #1901 are updated. Could you take another look?

[zoroyihan7]

@xiaofei-zheng Your approval was on the earlier acceptance-comparison version. The PR now carries the agreed approach instead: on a synthetic benchmark a variant that introduces or changes a context-copying draft (vLLM ngram/ngram_gpu/suffix, SGLang NGRAM) is refused before launch; the 5% acceptance comparison, noise band and re-measure rule are gone; the eval-start marker fix is kept. It is rebased onto current main as a single commit (2d80895) and CI is green. Could you take another look and merge if it looks right? Thanks.

[xiaofei-zheng]

What this PR does

On the synthetic benchmark, random-token prompts held to their full output length drive the model into repetition loops, and a speculative draft that proposes tokens by copying from the context accepts almost all of them — so explore KEPT a vLLM ngram k=7 variant at +95.7% when the gain that survives on real text is about +40%. This adds a rule module that reads the materialized launch (_spec_decode_context_copy.py) and a filter in explore's per-variant loop that refuses, before the server is launched, any candidate that introduces, switches to, or tunes a vLLM ngram/ngram_gpu/suffix or SGLang NGRAM draft in a synthetic session. Model-based drafts, a lever on a base that already copies, AgentX sessions and stack revalidations are left alone. It also makes the watchdog scan Magpie's nested benchmark_*/benchmark_stderr.log for the eval-start marker, so a round whose engine logged to the round's own server.log no longer folds its whole accuracy eval into the measured phase of kv_metrics.json.

Blocking issues: 1

1. The description's test count does not match the head.
   Problem: the body states Tests: src/hyperloom/inference_optimizer/tests/test_spec_decode_context_copy.py (41 tests); at head 2d80895 that file collects 40 (pytest --collect-only -q → 40 tests collected; by hand: 20 + 9 parametrized cases plus 11 single tests).
   Impact: a factual claim in the description is wrong at the current head, and the release cut aggregates these descriptions.
   Action: change 41 to 40.

Checked

_spec_decode_context_copy.py (context_copy_method, context_copy_refusal, _speculative_settings, _method_in, _config_keys, _LOOSE_PAIR); explore.py (filter_context_copy_drafts, _materialized_launch, the refuse_context_copy guard, the per-variant call, both run_grid calls, the skipped_dup → per_variant_outcomes mapping, _NOOP_FILTER_ALWAYS_RUNS / _CONFIG_REPLAY_PROVENANCE, explore_grid_exhausted); _subprocess_kill.py (_resolve_scan_logs, _stale_scan_log_sizes, _scan_logs_increment, _scan_server_log_increment, the watchdog loop); _grid_runner._build_variant_yaml; grid_server_args._shell_safe_dedupe / compact_json_server_args; perf_metric.agentx_active / is_agentx_mode; framework_registry.FRAMEWORKS; writeback.py:2951; cli/parser.py:480; the other run_grid callers (integrate_patch, conc_sweep, rebench).

Ran

• pytest --collect-only -q on the new test file; ruff check on the three changed sources (clean).
• ruff check --select C901 --config lint.mccabe.max-complexity=20 at head and at base: ExploreExecutor.__call__ 78 vs 76 — over the ceiling and raised, but the baseline-raise label is present and the description gives the reason, so this is waived rather than reported; _communicate_with_watchdog is 26 on both sides.
• Direct calls to context_copy_refusal and filter_context_copy_drafts for the spaced vLLM-documented --speculative-config '{"method": "ngram", ...}' spelling via extra_server_args, via base_extra_args and via a variant-supplied EXTRA_VLLM_ARGS. The raw predicate does mis-compare that spelling (_speculative_settings splits on whitespace while _method_in uses shlex), but _build_variant_yaml → _shell_safe_dedupe compacts every JSON blob before the filter reads it back, so all four production cases refuse correctly. Not reported for that reason.
• _resolve_scan_logs / _stale_scan_log_sizes / _scan_logs_increment for both eval-marker cases at head and at merge base 0198f43: head gives saw_eval_start True for a nested stderr created after the stale snapshot and False for a pre-existing one; base gives False for both. The marker fix is real and fails on base.
• Confirmed explore is the only run_grid caller that proposes speculative settings, that no accept-set keys on the new skipped_dup reason, and that the two _resolve_scan_logs consumers are order-insensitive.

Not checked: #1032, #1154, #1276, #1675, #1869 and #1922 are too large to diff fully for overlap, so conflicts with those six were not compared. Of the overlaps that were, none touches the lines this PR adds.

[xiaofei-zheng]

The description now states 40 tests, matching what the file collects at head. No blocking issues remain.

Verified on re-check: pytest --collect-only -q on test_spec_decode_context_copy.py reports 40 collected, and the description's other factual claims trace to the head tree (the method frozensets in _spec_decode_context_copy.py, the speculative_context_copy_on_synthetic reason string, and the skip happening before either run_grid call).

Earlier review covered: the refusal rule against the materialized launch on all three routes (extra_server_args, base_extra_args, a variant-supplied EXTRA_VLLM_ARGS), including the spaced vLLM-documented JSON spelling, which _shell_safe_dedupe compacts before the filter reads it; the eval-start marker reproduced at head and at merge base 0198f43 (True/False at head, False/False at base); ruff check clean on the three changed sources; C901 at head vs base (ExploreExecutor.__call__ 78 vs 76, waived by the baseline-raise label the description justifies). All 41 checks green at 2d80895.