Skip to content

refactor(kernel): delete the unused nomination lane and the duplicated queue scans - #1995

Merged
xiaofei-zheng merged 31 commits into
mainfrom
refactor/chenyi/cc20-kernel-2
Oct 22, 2026
Merged

xiaofei-zheng merged 31 commits into
mainfrom
refactor/chenyi/cc20-kernel-2

Conversation

@chennyiiis

@chennyiiis chennyiiis commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Simplifies nomination queueing and integration settlement in orchestrator/kernel/_kernel_decisions.py. The unused nomination lane argument is removed, the queue identity scan has one implementation, and pending-record filtering, result binding, attempt totals and KEEP/rejection writeback are named helpers. The existing terminal-verdict rule is retained: a settled integration is not revived by another nomination.

The current production diff against main is +183/-173 (net +10). The matching nomination tests remove lane="fusion" without dropping their assertions. Four metrics-baseline entries retire and the existing length entry for record_kernel_integrate_result is reduced; no module or named type is added. The refactor does not claim a net line deletion or elimination of all remaining size debt.

Observable effect

  • enqueue_nominated_patch no longer accepts lane. Production callers already nominate the fusion lane, so their emitted records retain source="forge_fusion" and action_label="fusion".
  • Nomination micro_speedup and pending-record ranking use common.coerce.to_float. Non-finite micro speedups are read as 0.0; a non-finite trace_gpu_pct falls back to the existing trace-derived impact calculation. Finite inputs retain their existing behavior.

The current diff does not change HL_VERIFIED_MICRO_PROMOTE_THRESHOLD or the retired _queue_kernel_keep path; claims about those belonged to older revisions.

Validation

Linux Python 3.10: terminal revival, nomination, kernel integrate/report, task-group accounting and integrate-wiring suites passed (166 tests). These cover deduplication, retained terminal verdicts, retryable faults and integration result writeback. The current head's GitHub code-metrics, lint, test and coverage checks pass. GPU E2E jobs are skipped for this head, so they are not GPU execution evidence.

Linked issue: none. The change stays within the existing queue/settlement owner.

Test and others added 4 commits October 19, 2026 16:22
…ntities and a ledger

The five write-owner functions each resolved their inputs inline: the KEEP
queue rebuilt its patch identity in three places, the pending-dispatch scan
re-read each record's task keys per predicate, the integrate settlement
threaded eight loose strings through its branches, and the hot-kernel scan
carried six-element tuples through three stages plus three closures over the
ledger.

Each now has a type: ``_KeepSnapshot`` and ``_NominatedPatch`` for the two
queue lanes, ``_PendingPatchIdentity`` with ``_SettledIntegrations`` for the
dispatch scan, ``_IntegrateSettlement`` for one integrate verdict, and in the
new ``_hot_kernel_scan`` module ``HotKernelRow`` / ``TaskGroups`` /
``KernelAttemptLedger`` for the hot-kernel gate. The hot-kernel pipeline moves
to that sibling module to keep this one inside the module-length limit; the
entry point stays here, so every caller and forwarding shim is unchanged.

Float and int coercion now goes through ``hyperloom.common.coerce`` instead of
six local try/except pairs.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d resolve a sweep's setup as typed objects

``_sweep_one_arm_single_server`` held the whole of one arm's life inline: the
lifecycle resolve, the boot-retry-descend, the three teardown paths and the
reuse ladder, with a nested restart runner closing over six locals.
``run_conc_sweep`` did the same for the sweep: seven pre-flight refusals, the
workspace and config materialization, four recorder calls, the deadline, the
arm loop and the report, all against loose locals.

The arm's server is now ``_ArmServer`` (pid dir, port, framework, lifecycle
eligibility and the Ray lease that covers its lifetime), the descend is
``_BootLadder`` reporting a ``_BootOutcome``, and the sweep's inputs are
``_SweepRequest``, ``_SweepWorkload``, ``_SweepLaunch`` and ``_SweepSetup``,
with ``_SweepRefusal`` and ``_ArmRefusal`` naming why a sweep or an arm does
not run. The three identical "the rung raised" ``VariantResult`` literals
become one ``_failed_rung_result``.

The arm ladder moves to ``_conc_sweep_ladder``, and the arms and their CONC
grid to ``_conc_sweep_grid``, so ``conc_sweep`` stays inside its module-length
budget. Both are imported back into ``conc_sweep``, so every call site, patch
target and ``__all__`` name it publishes is unchanged.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eep refactors retired

Seventeen entries removed, none added or raised.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…kernel-2

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/kernel/conc_sweep.py
@chennyiiis
chennyiiis requested a review from a team as a code owner October 19, 2026 08:48
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 3 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 11 +0
Cognitive complexity fails > 30 0 0 0 72 -3
Function length (lines) fails > 80 0 0 0 203 -1
Nested block depth fails > 5 0 0 0 8 +0
Parameter count fails > 5 0 0 0 83 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 47 +0
Duplicated lines per file fails > 0 0 0 0 95 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Check Findings Effect
Comments 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0
60 findings in files this change does not touch

Outside this change (informational): 60

Unit Dimension Baseline Now
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/kernelforge/cli/forge_loop_command.py forge_loop Cognitive complexity absent 31
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom/orchestrator/phases/kernel.py KernelPhase._record_geak_delegation_timeline Parameter count absent 7
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment Function length (lines) absent 107
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup Function length (lines) absent 89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology Function length (lines) absent 92
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser Function length (lines) absent 84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors Function length (lines) absent 84
src/hyperloom_kb/sync.py GlobalSync.pull Function length (lines) absent 84
src/kernelforge/loop/runner.py IterationLoop._close_iteration Function length (lines) absent 117
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Function length (lines) absent 185
src/kernelforge/loop/runner.py IterationLoop._implementer_history Function length (lines) absent 112
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Function length (lines) absent 102
src/kernelforge/loop/runner.py IterationLoop._source_attempt Function length (lines) absent 175
src/hyperloom_kb/http_service.py Module length (lines) absent 1525
src/kernelforge/cli/forge_loop_command.py Module length (lines) absent 2003
src/hyperloom/orchestrator/actions/executors/_kv_metrics.py port_from_server_log Nested block depth absent 6
src/hyperloom/orchestrator/specialists/dispatch.py SpecialistDispatchCollaborator.warm_specialist_params Cognitive complexity 93 94
src/hyperloom/agents/quantization/driver/retry.py quantize_via_prompt Function length (lines) 102 103
src/hyperloom/inference_optimizer/breakdown/exporter.py build Function length (lines) 132 136
src/hyperloom/inference_optimizer/cli/__init__.py _run_optimize Function length (lines) 273 279
src/hyperloom/inference_optimizer/cli/preflight.py _preflight Function length (lines) 203 208
src/hyperloom/inference_optimizer/multi_node/scripts/kernel_patch_multinode.py _do_apply Function length (lines) 98 99

...and 35 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

Comment thread src/hyperloom/orchestrator/kernel/_conc_sweep_grid.py Fixed
Comment thread src/hyperloom/orchestrator/kernel/_conc_sweep_grid.py Fixed
@chennyiiis
chennyiiis force-pushed the refactor/chenyi/cc20-kernel-2 branch from 02bb7d2 to 98bc6cd Compare October 19, 2026 08:54
@chennyiiis chennyiiis added the size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description label Oct 19, 2026
Comment thread src/hyperloom/orchestrator/kernel/_conc_sweep_ladder.py Fixed
Comment thread src/hyperloom/orchestrator/kernel/conc_sweep.py Fixed
Comment thread src/hyperloom/orchestrator/kernel/_conc_sweep_grid.py Fixed
Comment thread src/hyperloom/orchestrator/kernel/_conc_sweep_grid.py Fixed
Test and others added 3 commits October 19, 2026 18:32
``_conc_sweep_ladder`` reached back into ``conc_sweep`` for ``_SweepRun``, so
the two modules imported each other and whichever loaded first left a name
undefined. The run is the state the ladder operates on, so it moves below it:
``_conc_sweep_grid`` becomes ``_conc_sweep_run`` and now owns ``_SweepRun``,
its budget, and the curve and report it rewrites after every rung. The ladder
imports it outright and nothing imports back.

``conc_sweep`` keeps every name it publishes, and the tests that reach past it
for ``run_grid``, ``_write_csv`` and ``_flush_conc_sweep_report`` now name the
module that owns them.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n_grid

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1995 -- refactor(kernel): carry kernel-decision and conc-sweep state as named types

What it does: PR #1995 is a pure refactor. It splits the 7 over-complex functions in _kernel_decisions.py and conc_sweep.py into named types and helpers, and moves code into three new sibling modules: _hot_kernel_scan, _conc_sweep_ladder and _conc_sweep_run. The route was a differential test plus a line-by-line read. I drove the base and head implementations with the same randomized inputs and fakes, and they behave identically in practice. Two things block it: the PR is CONFLICTING (on the code-metrics baseline only, a mechanical conflict), and the description is stale since commit a84a908.

Blocking issues: 2

  1. [conflict-baseline] PR is CONFLICTING with main on scripts/code_metrics_baseline.txt [verified]
    Problem: GitHub reports mergeable=CONFLICTING. The PR deletes 17 baseline lines next to lines that three main commits rewrote after the merge base. scripts/code_metrics_baseline.txt:94
    Impact: The PR cannot merge. A resolution that takes the PR side wholesale would revert main's baseline tightenings from 84bcdfb (#1986), f7f64ac (#1984) and b7004d1 (#2021). For example, cognitive-complexity bypass_runner.py::run_benchmark 33 would come back and materialize_config_with_envs would return to 286, loosening the code-metrics gate for those units.
    Action: Rebase or merge main and resolve as main's file minus exactly the 17 _kernel_decisions.py / conc_sweep.py entries the PR retires. All 17 are present unchanged on main. Then re-run code-metrics.

  2. [desync-description-stale] PR description describes the pre-a84a9088c layout, not the head [verified]
    Problem: The description says the arms and CONC grid move to _conc_sweep_grid, that conc_sweep goes from 1429 to 1086 lines, and that 'every call site, patch target and all name it publishes is unchanged'. At head there is no _conc_sweep_grid. Commit a84a908 renamed it to _conc_sweep_run and also moved _SweepRun, _Budget, run_grid, _write_csv and the report flush there, adding _SweepRun.arms_order. conc_sweep.py is 629 lines, not 1086. The patch targets did change: conc_sweep.run_grid, conc_sweep._write_csv and conc_sweep._flush_conc_sweep_report no longer intercept anything, which is why four test files were retargeted to _conc_sweep_run. src/hyperloom/orchestrator/kernel/conc_sweep.py:30
    Impact: review-pr desync rule: the description does not match the diff at the current head. Reviewers and the release cut get a wrong module map. Anyone who still patches conc_sweep.run_grid (as all these tests did at base) is told nothing changed, but the patch now silently has no effect.
    Action: Update the PR description: the module is _conc_sweep_run and it owns _SweepRun, the budget and the report writer; conc_sweep is 629 lines (1450 physical at base); the run_grid / _write_csv / _flush_conc_sweep_report patch targets moved and the tests were retargeted. Keep 'No observable change', which the differential run supports.

Checked: _kernel_decisions._queue_kernel_keep (+ _KeepSnapshot, _keep_snapshot, _promotes_verified_micro, _queued_keep_record, _r (yes); _kernel_decisions.enqueue_nominated_patch (+ _NominatedPatch, _nominated_patch, _warn_rewrite_lane_drops_env_flags, _nom (yes); _kernel_decisions.pending_kernel_integration_records (+ _PendingPatchIdentity, _SettledIntegrations, _patch_kernel_is_re (yes); _kernel_decisions.record_kernel_integrate_result (+ _IntegrateSettlement, _integrate_settlement, _with_queued_paths, _bo (yes); _kernel_decisions.untried_hot_reusable_kernels (+ _hot_kernel_scan.TaskGroups, HotKernelGate, _hot_kernel_row, one_row (yes); conc_sweep._sweep_one_arm_single_server (+ _conc_sweep_ladder._ArmServer, _BootLadder, _BootAttempt, _ArmBoot, _record_a (yes); conc_sweep.run_conc_sweep (+ _SweepRequest, _SweepWorkload, _sweep_refusal, _SweepLaunch, _sweep_workspace, _materialize (yes) | Conflict with main mechanical-baseline-only | Base: ef93838 | Head: b4761c2

Test and others added 3 commits October 20, 2026 10:27
…kernel-2

# Conflicts:
#	scripts/code_metrics_baseline.txt
A result that carries no patch path of its own falls back to the one the last
kernel optimization produced. The eligibility test and the lookup move into
_borrowable_patch_path, so resolving a patch identity reads as four field reads
and one fallback.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

Requesting changes: this relocates complexity across 25 new types and three new modules instead of removing it, and the PR body no longer matches the head.

Production code: +2452/−1700 (net +752 lines, +31%; classes 6 → 31, functions 68 → 158). A complexity refactor should shrink production code. Please bring this to net-zero or negative by deleting the mechanisms below, not by moving them; _kernel_decisions.py grew 959 → 1045 lines even after the hot-scan code was moved out.

Blocking

  1. PR body contradicts head: it names _conc_sweep_grid (actual module is _conc_sweep_run), "1429 → 1086 lines" (measured 1450 → 629), and "patch targets unchanged" (run_grid/_write_csv/_flush_conc_sweep_report moved; 4 test files were repointed). Rewrite the body, and state that resolve_benchmark_timeouts() now runs after the workspace is created (conc_sweep.py:446) and that coerce.to_float maps inf/nan env thresholds to the default (_kernel_decisions.py:128,147).
  2. Merge conflict with main in scripts/code_metrics_baseline.txt: take main's version, remove only this PR's 17 entries, and rerun code-metrics.

Simplify instead of relocate

  • Delete the rewrite lane of enqueue_nominated_patch (_kernel_decisions.py:237-320,389-411). It has no caller (the only one is phases/kernel.py:4092, fusion). Drop lane, is_fusion, lane_prefix and _warn_rewrite_lane_drops_env_flags.
  • Delete _SweepSetup, _SweepLaunch, _ReportPaths and _SweepDeadline (conc_sweep.py:206-475). They carry the same values that are then unpacked into 22 kwargs of _SweepRun. Use a linear preflight plus a _SweepRun.open(...) classmethod, and fold _SweepRequest/_SweepWorkload into that preflight.
  • Keep one source for the fault decision: pass is_fault into _rejection_reason (_kernel_decisions.py:803,908).
  • Use one owner for the "kernel id → ledger entry" lookup: KernelAttemptLedger.attempt_for (_hot_kernel_scan.py:211) duplicates index_attempts_by_kernel_id (_kernel_decisions.py:1004-1021). Do not publish a second rule as a new public class.

Coordinate with #2039, which edits the boot-rung envs that this PR moves to _conc_sweep_ladder.py:205-216.

Test and others added 3 commits October 20, 2026 10:49
… writes anything

resolve_benchmark_timeouts() raises on a non-finite or non-positive override.
Resolving it while assembling the run left that rejection until after the
workspace was created and the preflight event recorded, so a bad override wrote
a workspace and an event before failing. The sweep request now resolves it,
which happens before the first refusal check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…fore any write

Fails on the previous commit because the workspace directory already exists by
the time the override is rejected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…it creates

The factory creates the reports directory, so a query name misreads it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis

Copy link
Copy Markdown
Contributor Author

Review response — head df0fe89c8, CI 40 checks / 0 failures

Blocking items

1. Baseline conflict (scripts/code_metrics_baseline.txt). Fixed. origin/main is merged and the conflict was resolved mechanically against the merge base, not by hand: per entry the lower value wins, an entry one side retired is dropped, an entry one side added because a unit moved file is kept. The result is main's file minus exactly the 17 entries this PR retires — +0 / -17 against main, nothing added and nothing raised. The PR is MERGEABLE and code-metrics passes.

Also confirmed, because a merge commit in history is not proof the behaviour survived: f7f64aceb (#1984), 330d32229 (#1985), 84bcdfb53 (#1986), 493581b85 (#1918) and b7004d162 (#2021) are all ancestors of this branch, none of them touches any of this PR's five files, and the files they do touch are byte-identical to origin/main here. disable_run_eval / eval_disabled are absent, benched is present, and the broad-except count across the conc-sweep modules is 7 — the same 7 conc_sweep.py had on main, with none added.

2. Stale description. Rewritten against the current tree, with every number measured rather than recalled:

  • the three new sibling modules are _conc_sweep_run.py (601), _conc_sweep_ladder.py (574) and _hot_kernel_scan.py (312), with a table saying what each owns. There is no _conc_sweep_grid;
  • conc_sweep.py 1450 → 635, _kernel_decisions.py 959 → 1046 — including why the second one grows;
  • the patch targets that moved are named, and so is the fact that four test files were repointed.

3. "No observable change" was wrong, and is now stated. Eight inline try: float(...) / int(...) except pairs became hyperloom.common.coerce, which rejects two cases the inline form accepted: a non-finite value (inf, nan) and a bool now read as the call's default. The description names the six affected readings and the one place it reaches control flow — impact <= 0.0 is False for nan, so a nan trace_gpu_pct used to skip the trace-impact fallback and now takes it.

4. resolve_benchmark_timeouts() ordering — fixed, not just documented. You were right that it had moved. It raises on a non-finite or non-positive override, and on main it ran as the first step of run_conc_sweep, before any refusal, before the workspace directory and before the preflight event. Resolving it while assembling the run left a workspace and a recorded preflight event behind on a bad override. It now resolves in _SweepRequest.of, which is again the first step, and test_run_conc_sweep_rejects_invalid_timeout_before_writing_anything pins it — that test fails on the previous commit, on the runs/conc_sweep/ assertion. The only other call that moved later is session_grid_bounds, which reads state and neither raises nor writes.

Inline comments

All six are answered on their own threads. In short:

  • the four py/unused-global-variable findings were against _conc_sweep_grid.py, which no longer exists. _SWEEP_BUDGET_STOP and _CANCEL_STOP live in _conc_sweep_run.py and have readers in two and three modules respectively;
  • the two py/unsafe-cyclic-import findings are fixed. _conc_sweep_run has no intra-package imports, _conc_sweep_ladder imports it, conc_sweep imports both — a depth-first walk of the package's from .x import edges finds no cycle, import-linter (architecture contracts) passes, and Analyze (python) passes at this head.

Worth recording from that one: putting the ladder underneath and letting the state name the ladder's result type through a TYPE_CHECKING import does not clear the rule — CodeQL counts a TYPE_CHECKING back-edge taken for a type annotation as a cycle just the same. That is why the state object is the lower module.

"Delete, don't relocate"

Taken seriously, and it did find one thing: _ReportPaths.under() was query-named but created the reports directory, so it is now create_reports_dir. Beyond that, here is the honest accounting rather than a claim.

What this PR deletes rather than moves:

  • three identical "the rung raised" VariantResult literals → one _failed_rung_result;
  • eight local try: float()/int() except pairs → hyperloom.common.coerce, the module that owns coercion;
  • three closures over the attempt ledger → one KernelAttemptLedger;
  • two inline re-derivations of the report paths → one _ReportPaths;
  • the last_kernel_opt back-fill condition, which was a four-operand if inline in _resolve_kernel_patch_identity → _borrowable_patch_path.

What it does not delete, and why. The production code does grow (+2467/-1708), and 25 of the new names are types. That is the shape the gates force, not a preference: code_metrics fails above 5 parameters, so lifting a block out of a 328-line function and passing its free variables in is not available — the values that travel together have to become one object. conc_sweep.py then had to shed ~800 lines to stay inside the 1200-line module limit once its two oversized functions were split. I checked each candidate you named before concluding it is not deletable here:

  • _budget_skip_result and _deadline_skip_result are genuinely near-duplicates, but both pre-date this PR verbatim (conc_sweep.py:135 and :164 on main), and collapsing them changes an operator-visible message ("before this variant ran" vs "before this round ran"). That is a behaviour change that does not belong in this PR;
  • the rewrite lane of enqueue_nominated_patch (lane, lane_prefix, _warn_rewrite_lane_drops_env_flags) is pre-existing on main (line 200 onward) and reachable through the shared_state shim;
  • _rejection_reason is not a second source for the fault decision: entry["last_was_fault"] is written from the caller's is_fault at _kernel_decisions.py:771 and read back from that same entry at :816;
  • KernelAttemptLedger.attempt_for is the moved _attempt_for_member closure from main:870, matching five identity fields; index_attempts_by_kernel_id indexes only current_kernel_id and breaks ties by timestamp. Different rules, both pre-existing;
  • _BootLadder is one call site but six methods over shared accumulating state (failed_boots), so it is not a forwarder — the design gate agrees (A12 pure forwarder 0, single-implementation abstraction 0).

The repo's own duplication gate is the check for anything I missed here: jscpd reports 0 duplicated lines and vulture 0 dead code on the ten files this change touches.

Verification

  • code-metrics: PASSED, 0 new / 0 worse / 0 baseline-out-of-date on every dimension; baseline entries −17 vs base, nothing added. No baseline-raise.
  • Every function in the five files measures ≤ 19 on the Cmetrics scale (strictly under the 20 ceiling).
  • No test weakened or deleted; test_conc_sweep.py 101 → 102, the other three files unchanged at 11 / 14 / 21.
  • CI: 40 checks, 0 failures, including Analyze (python), import-linter, bandit, mypy, pylint and ruff.

@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

Production code is still too large. The net growth is +759 lines (it was +752), and it needs to keep shrinking before this can merge.

I classified every changed line in the touched production files with an AST pass (merge-base vs head 0a4fde08c):

net lines
executable logic +49
docstrings +203
def/class signatures +170
NamedTuple/dataclass fields +109
blank lines between new defs +200
imports / __all__ +70

Functions go from 68 to 159, record types from 6 to 31, and total parameters from 119 to 283. Of the 111 new definitions, 58 are referenced exactly once in src/.

The logic barely changed (+49). The growth is the scaffolding around single-use units: their signatures, docstrings, fields and spacing (about +752 lines). That is the same control flow in more places, not a simpler one (AGENTS.md Simplify by removing a mechanism). The parameter-count and tuple-return limits in [tool.hyperloom.code_metrics] are review triggers, not a reason to add a type or a helper. When a limit and the clearer code disagree, keep the clearer code and say so (Size is a design signal).

Cut next:

  • The rewrite lane of enqueue_nominated_patch (lane, is_fusion, lane_prefix, _warn_rewrite_lane_drops_env_flags). It has no caller.
  • _SweepSetup, _SweepLaunch, _ReportPaths and _SweepDeadline. They are unpacked again into the 22 kwargs of _SweepRun, so use one _SweepRun.open(...).
  • "These existed on main" applies to the behaviour, not to the 25 new types that carry it.
  • Inline the remaining helpers that have one caller and only thread arguments through, and drop the docstrings that restate a private helper's name (Comment below the local average).

Target: production net ≤ 0, or name each remaining new type in the description with what it removes.

Test and others added 3 commits October 20, 2026 16:56
…kernel-2

# Conflicts:
#	src/hyperloom/orchestrator/kernel/conc_sweep.py
…kernel-2

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/kernel/conc_sweep.py
The files these PRs touch already document 96-98% of their functions, so the
docstring growth came from the function count the complexity ceiling forced,
not a higher documentation rate. Every new helper keeps a docstring; the
Args:/Returns: blocks come off so each file stays internally consistent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Test and others added 8 commits October 20, 2026 19:56
conc_sweep.py conflicted: main (#1782) renamed gpu_types._gpu_runner_type
to the public gpu_runner_type inside run_conc_sweep, the same block this
branch moved into _materialize_sweep_config. Taking our structure alone
would have left the relocated call importing a name main deleted -- an
ImportError the file's own tests cannot see. Our structure is kept and
main's rename re-applied at the relocated call site, so an r9700 session
still resolves to the gfx12 runner.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ledger by deleting

Reworks `_kernel_decisions.py` delete-first: one `_text` / `_number`
coercion rule replaces ninety-nine inline `or ""` / `float(x or 0.0)`
fallbacks, the rejection record now reads the ledger entry that already
owns its fields instead of re-deriving them from the result, the two
copies of the task-group alias predicate become one, and the queued
record's refresh, the promotion rule and the existing-id lookup are
named once each.

`conc_sweep.py` is restored to `origin/main`: its two over-threshold
functions are pre-existing main debt owned by the other half of this
effort, and every other function this branch touched there was under
every repo threshold at the merge base.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Delete the nomination rewrite lane: no caller has ever passed
``lane="rewrite"``, so its prefix, its env-flag warning and the two
``lane == "fusion"`` tests were a dead branch around the only lane there is.

Fold ``_existing_keep_integration_id`` into ``_existing_integration_id`` --
the two queue scans differed only in whether an artifact bundle takes part in
the match -- and inline the single-use ``_refresh_queued_routing`` and gate
top-n helpers back into their one caller each. ``_number`` was a second copy
of ``common.coerce.to_float``.

No observable change beyond the removed rewrite lane.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tion retired

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis

Copy link
Copy Markdown
Contributor Author

Reworked delete-first. Against the current merge base (origin/main at 9f7030af9) this PR is now −4 net production lines (was +759) with zero net docstring growth (+3/−3, and all three are the same three texts moving with the functions that carry them). origin/main is merged; CI is 40 checks / 0 failures. There is no new module and no NamedTuple or dataclass in the file.

Measured every touched function at the merge base on the gate's scales. All five that this branch restructures were over a threshold there, so nothing was reverted as never-over-threshold:

function merge base (cc/cog/lines) now
untried_hot_reusable_kernels 38 / 119 / 150 11 / 28 / 29
record_kernel_integrate_result 20 / 59 / 171 8 / 20 / 99
pending_kernel_integration_records 13 / 45 / 74 9 / 22 / 38
enqueue_nominated_patch 10 / 32 / 105 6 / 15 / 75
_queue_kernel_keep 7 / 36 / 99 5 / 12 / 68

The only other function in the file that the diff touches is none: _resolve_kernel_patch_identity (Cmetrics 19 at the merge base) and every other unit are byte-identical.

What this round deleted rather than moved

  • the nomination rewrite lane. No caller has ever passed lane="rewrite" — the one production caller (phases/kernel.py::_integrate_fusion) and every test use the fusion default — so the lane_prefix ternary, the elif fusion_env_flags warning and the two if lane == "fusion" tests were a dead branch around the only lane there is. The parameter is gone.
  • _existing_keep_integration_id and _existing_integration_id: two near-identical scans of the same queue, differing only in whether the artifact bundle takes part in the match. One function now.
  • _number: a second copy of common.coerce.to_float, and the HL_VERIFIED_MICRO_PROMOTE_THRESHOLD parse was a third.
  • _refresh_queued_routing and _hot_kernel_gate_top_n: single-use; inlined into their one caller each.
  • _attempt_totals: one pass over the attempts instead of two.

Re-verified the #1782 hazard: nothing in this round moves or re-inlines the gpu_runner_type block; conc_sweep.py is untouched by this branch now, and _sweep_one_arm_single_server (53) / run_conc_sweep (50) are byte-identical to origin/main.

Highest Cmetrics in the file: 19, shared by _resolve_kernel_patch_identity (unchanged, pre-existing) and _queue_kernel_keep. Nothing in scope is at or over 20. The one repo-gate entry that remains is record_kernel_integrate_result at 99 physical lines, down from 171; enqueue_nominated_patch's entry retires.

@chennyiiis chennyiiis changed the title refactor(kernel): carry kernel-decision and conc-sweep state as named types refactor(kernel): delete the unused nomination lane and the duplicated queue scans Oct 21, 2026
@chennyiiis chennyiiis removed the size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description label Oct 21, 2026

@ZhengGong-amd ZhengGong-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1995 -- refactor(kernel): delete the unused nomination lane and the duplicated queue scans

What it does: Restructures the five over-threshold functions in src/hyperloom/orchestrator/kernel/_kernel_decisions.py into 14 new private helpers. It also removes the lane parameter (and the forge_rewrite branch) from enqueue_nominated_patch, merges the two queue scans into _existing_integration_id, and replaces the inline float(...) parsing with common.coerce.to_float. The code is correct. The two blocking issues are both in the PR description.

Blocking issues: 2

  1. [free:observable-effect-incomplete] "Observable effect" leaves out a behaviour change at the env and persisted-state boundary [verified]
    Problem: The body says "Everything else is internal structure." The head now parses with to_float, which maps non-finite values to its default, and that changes how two kinds of input are read.

    • HL_VERIFIED_MICRO_PROMOTE_THRESHOLD: at head, _promoted_needs_review reads it with to_float(...) and falls back to 1.10 (src/hyperloom/orchestrator/kernel/_kernel_decisions.py:104,110). Base used float(os.environ.get(..., "1.10") or 1.10), which keeps inf/nan (base _kernel_decisions.py:115-124). Repro: call _queue_kernel_keep with a NEEDS_REVIEW geak entry, last_correctness_passed=True, last_micro_speedup=1.5, and default HL_PROMOTE_VERIFIED_MICRO_NEEDS_REVIEW. With the env set to inf or nan, base returns None (not queued) and head queues the patch. With 1.10, 0 or 9, base and head behave the same.
    • Persisted values: a non-finite last_micro_speedup, micro_speedup, trace_gpu_pct or gpu_pct is now read as 0.0 (head :154, :227, :374, :378, :798). Base kept it as nan/inf.
      Impact: The release notes would tell operators the only change is the removed lane argument. An operator who set HL_VERIFIED_MICRO_PROMOTE_THRESHOLD=inf (or nan) to stop promoting NEEDS_REVIEW geak patches would find them queued for integration again at threshold 1.10, and nothing in the notes would explain why.
      Action: Replace "Everything else is internal structure." with an accurate statement: HL_VERIFIED_MICRO_PROMOTE_THRESHOLD values that are not finite (inf, nan) now fall back to 1.10 instead of disabling promotion, and non-finite persisted last_micro_speedup / micro_speedup / trace_gpu_pct / gpu_pct are read as 0.0. Write the body against git diff origin/main...HEAD only; every name and number must be reproducible from that diff.
  2. [free:deleted-names-not-on-main] "What was deleted" names code that never existed on main [verified]
    Problem: Two bullets under "What was deleted" describe deletions that are not in the diff (src/hyperloom/orchestrator/kernel/_kernel_decisions.py):

    • "_number was a second copy of common.coerce.to_float": git show 9f7030af9:src/hyperloom/orchestrator/kernel/_kernel_decisions.py | grep -c _number prints 0. Base parsed inline with try: float(x or 0.0) except (base :112, :215, :381, :387, :828).
    • "Two single-use helpers, inlined at their callers": comparing def names between base 9f7030a and head, no base function is missing at head. Head adds 14: _attempt_totals, _bind_result_to_pending, _deduped_hot_rows, _existing_integration_id, _group_attempt, _hot_kernel_rows, _hot_kernel_task_groups, _pending_record_settled, _pending_task_key, _promoted_needs_review, _record_integration_keep, _reject_integration, _still_untried, _text.
      The lane removal and the merge of the two queue scans into _existing_integration_id are accurate.
      Impact: The release notes would tell operators that two helper deletions shipped and would leave out the 14 helpers this change adds. The record would not match the history of _kernel_decisions.py.
      Action: Delete the _number and "two single-use helpers" bullets. State that the inline float(...) parsing is replaced by to_float and that the five functions are split into the 14 private helpers listed above. Write the body against git diff origin/main...HEAD only; every name and number must be reproducible from that diff.

Checked: src/hyperloom/orchestrator/kernel/_kernel_decisions.py (base and head; enqueue_nominated_patch, _queue_kernel_keep, _promoted_needs_review, _existing_integration_id, def-set diff), src/hyperloom/common/coerce.py::to_float, kernel/tests/test_enqueue_terminal_revival.py, scripts/code_metrics_baseline.txt, live PR body | Ran: trial merge with origin/main b772e94 (clean); code_metrics --base-ref origin/main (passed); pytest kernel/tests + test_integrate_payload_defaults + test_task_group_dispatch_accounting on the merged tree (561 passed); base/head probe of _queue_kernel_keep with HL_VERIFIED_MICRO_PROMOTE_THRESHOLD in {inf, nan, 1.10, 0, 9} | Base: 9f7030a | Head: d2dc50f

Test and others added 4 commits October 21, 2026 21:52
…kernel-2

#2055 deleted the per-kernel attempt ledger (kernel_opt_task_attempts,
last_kernel_opt) and rewrote untried_hot_reusable_kernels, _queue_kernel_keep
and the identity resolvers around it, so the branch's decomposition of those
reads has nothing left to decompose. Main's behaviour is taken whole there, and
the branch's structure is re-applied by hand on what survives:

- enqueue_nominated_patch keeps its single fusion lane (no `lane` parameter, no
  forge_rewrite branch) over main's two-lane body.
- _existing_integration_id, _pending_record_settled, _bind_result_to_pending,
  _attempt_totals, _record_integration_keep and _reject_integration are carried
  across, each minus the ledger reads #2055 removed.
- untried_hot_reusable_kernels and its helpers are main's, unchanged.

HL_VERIFIED_MICRO_PROMOTE_THRESHOLD and HL_PROMOTE_VERIFIED_MICRO_NEEDS_REVIEW
are gone with _queue_kernel_keep; nothing reads them on main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…kernel-2

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/kernel/_kernel_decisions.py
…kernel-2

# Conflicts:
#	scripts/code_metrics_baseline.txt
@xiaofei-zheng

Copy link
Copy Markdown
Collaborator

What this PR does

enqueue_nominated_patch carried a lane parameter that no production caller ever set, so the rewrite branch, its lane_prefix fork and its env-flag warning were unreachable code holding three code_metrics_baseline entries open. This PR deletes the parameter and that branch, hoists the queue identity scan, the pending-record settled test, result binding, attempt totals and the KEEP/rejection writeback into named helpers inside the module that already owns the queue and the settlement, and replaces two hand-rolled float(...) / except (TypeError, ValueError) blocks with common.coerce.to_float. Four baseline entries retire and record_kernel_integrate_result's length entry drops 151 -> 97.

Blocking issues

None.

Checked: _kernel_decisions.py (enqueue_nominated_patch, pending_kernel_integration_records, record_kernel_integrate_result and the six extracted helpers), common/coerce.py::to_float, phases/kernel.py:4042 — the sole production caller, already lane-less at the merge base, so the removed rewrite lane was unreachable — scripts/code_metrics_baseline.txt, tests/test_enqueue_terminal_revival.py, test_task_group_dispatch_accounting.py. Confirmed the keys _reject_integration reads off entry (attempt_count, fault_count, best_gain_pct, last_decision, last_error_class, identity fields) are the same locals the old inline rejected dict used — attempt_count is len(attempts), not verdict_attempt_count. Confirmed _text(status) not in ("", "pending") matches the base's str(status or "pending") != "pending" for "", None, missing, 0, whitespace and each terminal status. No stale reference to the lane argument or the forge_rewrite: task_key prefix survives in production code.

Ran: a base-vs-head differential driving enqueue_nominated_patch, pending_kernel_integration_records and record_kernel_integrate_result over enqueue, re-enqueue refresh, terminal revival, pending ranking, KEEP, REVERT, fault exhaustion, max-attempts-without-keep and unbound-result, comparing the full record, attempt entry, rejected_kernel_patches and rejected_kernel_ids with timestamps scrubbed — identical on both sides except the non-finite coercion the description already states (nan/inf micro_speedup now read as 0.0 instead of propagating into candidates.sort). Also ruff check --select C901 --config lint.mccabe.max-complexity=20 on the changed module at head and at base: clean on both.

Base: e34bb7a Head: 8968fa0

LGTM

@xiaofei-zheng xiaofei-zheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues. Details in the review comment above: the removed lane branch is unreachable at the merge base, the extracted helpers are behaviour-identical on a base-vs-head differential over enqueue / pending ranking / KEEP / REVERT / fault exhaustion / max-attempts, and the only behaviour delta is the non-finite coercion the description already states.

@xiaofei-zheng
xiaofei-zheng merged commit ef06a87 into main Oct 22, 2026
40 checks passed
@xiaofei-zheng
xiaofei-zheng deleted the refactor/chenyi/cc20-kernel-2 branch October 22, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants