Skip to content

refactor(grid): drive the grid over named variant phases - #1964

Merged
chaojhou merged 57 commits into
mainfrom
feature/chenyi/cc-grid-runner
Oct 22, 2026
Merged

chaojhou merged 57 commits into
mainfrom
feature/chenyi/cc-grid-runner

Conversation

@chennyiiis

@chennyiiis chennyiiis commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Moves the inline phases of run_grid into named functions driven by _measure_variant. Each stopped or failed variant is represented by _VariantAborted, and run_grid records its result and applies the stop policy in one handler. The handler contains the loop's only break; it stops on _Halt.STOP, or on _Halt.ON_FAILURE when keep_going_on_failure is false.

Nine NamedTuple carriers are added: _RoundProcess, _RoundError, _VariantSlot, _VariantFailure, _GridRun, _VariantRounds, _BenchmarkPass, _MeasuredVariant and _VariantOutcome. _RoundAttempt already exists on main and is moved. No module is added or removed.

The production diff against main is net -26 lines. _grid_runner.py changes by +1225/-1259 and remains 2045 lines. run_grid is 105 lines, down from the recorded 1014-line baseline; its existing size debt stays recorded.

Related launch-composition changes

The grid's YAML composition uses helpers with the same ownership as the other benchmark paths:

  • pin_mlperf_round_concurrency moves from _workload_envs.py to common/agentx_workload.py, beside its backend predicate.
  • remove_moe_runner_backend_arg moves to grid_server_args.py, beside the flag-detection and flag-removal logic. The detection pattern is public as SGLANG_MOE_RUNNER_BACKEND_RE; the pair-removal pattern is _SGLANG_MOE_RUNNER_BACKEND_PAIR_RE.
  • _shell_safe_dedupe becomes shell_safe_dedupe, with its callers updated.
  • grid_server_args.py extracts _removed_token_count and _json_blob_end to simplify flag removal and JSON-blob reserialization while retaining their token and quoting rules.

These helper moves keep the grid's launch phases using the modules that own concurrency and server arguments, without adding a grid-only implementation. Their callers in _benchmark_runtime.py, _workload_envs.py, baseline.py and conc_sweep.py are updated. The redundant separate _build_variant_yaml import in explore.py is removed, restoring that file's import block to main.

Observable effect

Session-budget exhaustion now passes through _abandon_rest_of_grid, preserving the stopped and unstarted result rows while also emitting a finished-variant progress note. Benchmark results, argument precedence, warmup/measurement order, stop attribution and cleanup policy are otherwise intended to remain unchanged.

Validation and size

The branch contains main at bb2da63332dcbc46bdf10702c516ef43eb80a016. Validation covers grid execution, branch and kill paths, runtime overrides, behavior locks, server-argument parsing and MoE backend injection. The complexity baseline retires four entries: two for run_grid, and two for the server-argument parser. Four existing entries are tightened; no entry is added or raised.

size-exception: the in-place grid-loop rewrite changes more than 1000 production Python lines. Its admission, warmup, launch, measurement, cleanup and result paths are reviewed together to preserve their ordering. The large existing modules and the remaining function-size entries stay recorded in the baseline.

chennyiiis and others added 2 commits October 15, 2026 10:27
Sixteen places end a variant, and three different mechanisms decided what that
meant for the grid: the abort handler, `_record_round_stop`, and two inline
tails after a benchmark that returned nothing usable. Each read
`keep_going_on_failure` or returned a flag of its own, and two of them filled
the rest of the grid with not-run rows.

All sixteen now raise `_VariantAborted`, and the halt is data on it: CONTINUE
for an outcome the grid always walks past, ON_FAILURE for one the policy
decides, STOP for one that ends the batch and carries the rows for the
variants it will not reach. The rc asymmetry that was buried in two
`if rc != 0 and not keep_going_on_failure` tails is now the difference between
two enum members.

`_record_round_stop` becomes `_abort_on_round_stop` and only builds the
result: it no longer appends, no longer extends the grid, and no longer
answers a question its caller then re-asked.

Cyclomatic complexity of run_grid drops from 61 to 55, cognitive from 196 to
177, and its length from 1040 lines to 1026.

Refactor with nothing observable: the same rows in the same order, the same
abort markers, and the same decision at every one of the sixteen endings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_grid classified a round that produced no workspace, and a round that
produced one with no valid measurement, inline in a 937-line loop body. Give
each a name, and the round's subprocess output a record rather than three
locals both of them reached for.

run_grid: 1140 -> 1117 lines, cognitive 196 -> 174, cyclomatic 61 -> 50

Observable effect: none -- pure refactor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis
chennyiiis requested a review from a team as a code owner October 17, 2026 08:33
@chennyiiis chennyiiis added the baseline-raise Waives new, worse and baseline-growth code-metrics findings; reason required in the PR description label Oct 17, 2026
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 11 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 14 -1
Cognitive complexity fails > 30 0 0 0 95 -3
Function length (lines) fails > 80 0 0 0 234 +0
Nested block depth fails > 5 0 0 0 8 +0
Parameter count fails > 5 0 0 0 87 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 47 +0
Duplicated lines per file fails > 0 0 0 0 95 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Check Findings Effect
Comments 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1

Module Lines Limit
src/hyperloom/inference_optimizer/grid_server_args.py 997 warns > 800

Already over at the base (backlog, not this change's): 3

Unit Dimension Base Now
src/hyperloom/orchestrator/kernel/conc_sweep.py:717 _sweep_one_arm_single_server Function length (lines) 335 335
src/hyperloom/orchestrator/kernel/conc_sweep.py:1186 run_conc_sweep Function length (lines) 249 249
src/hyperloom/orchestrator/kernel/conc_sweep.py:1 Module length (lines) 1449 1449
63 findings in files this change does not touch

Outside this change (informational): 63

Unit Dimension Baseline Now
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison Cognitive complexity absent 34
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/kernelforge/cli/forge_loop_command.py forge_loop Cognitive complexity absent 31
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment Function length (lines) absent 107
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup Function length (lines) absent 89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology Function length (lines) absent 92
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser Function length (lines) absent 84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors Function length (lines) absent 84
src/hyperloom_kb/sync.py GlobalSync.pull Function length (lines) absent 84
src/kernelforge/loop/runner.py IterationLoop._close_iteration Function length (lines) absent 117
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Function length (lines) absent 185
src/kernelforge/loop/runner.py IterationLoop._implementer_history Function length (lines) absent 112
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Function length (lines) absent 102
src/kernelforge/loop/runner.py IterationLoop._source_attempt Function length (lines) absent 175
src/hyperloom_kb/http_service.py Module length (lines) absent 1525
src/kernelforge/cli/forge_loop_command.py Module length (lines) absent 2003
src/hyperloom/orchestrator/actions/executors/_kv_metrics.py port_from_server_log Nested block depth absent 6
src/hyperloom/orchestrator/specialists/dispatch.py SpecialistDispatchCollaborator.warm_specialist_params Cognitive complexity 93 94
src/hyperloom/agents/quantization/driver/retry.py quantize_via_prompt Function length (lines) 102 103
src/hyperloom/inference_optimizer/breakdown/exporter.py build Function length (lines) 132 136
src/hyperloom/inference_optimizer/cli/__init__.py _run_optimize Function length (lines) 273 279
src/hyperloom/inference_optimizer/cli/preflight.py _preflight Function length (lines) 203 208
src/hyperloom/inference_optimizer/multi_node/scripts/kernel_patch_multinode.py _do_apply Function length (lines) 98 99

...and 38 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

chennyiiis and others added 9 commits October 19, 2026 11:14
…bort-path

# Conflicts:
#	scripts/code_metrics_baseline.txt
main added a parameter-count limit after this branch was written;
_abort_on_round_stop took nine. The seven describing the variant's run are one
thing -- which variant, in which slot, what its process returned, when it
started, and the servers it left behind -- so they travel as _RoundAttempt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…into feature/chenyi/cc-grid-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/actions/executors/_grid_runner.py
_build_variant_yaml inlined the MoE-runner-backend drop, the overlay PYTHONPATH
containment check and the xdit PATH pin, so the function that materializes one
variant's config carried three unrelated rules in the middle of it.

Observable effect: none -- pure refactor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_grid held one 1100-line loop body: the capability gate, config
materialization, the discarded warmup round, the multi-node restart and its
client warmup, the measured round, the watchdog returncodes and the row the
report becomes were all inline, closing over thirty-one names.

Each is now a function named for the job it does, over four records that carry
what a phase needs: the recipe a variant's YAML is derived from, how a round
launches, the session budget it must fit, and the round's own result. The two
liveness watchdogs end on one path keyed by the returncode they came back with,
and every abort writes its marker and raises through one helper.

Observable effect: none -- pure refactor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis chennyiiis removed the baseline-raise Waives new, worse and baseline-growth code-metrics findings; reason required in the PR description label Oct 19, 2026
Test and others added 2 commits October 19, 2026 16:54
…tity

The grid runner materialized each variant's Magpie YAML inline: _build_variant_yaml
and the three env rules only it uses sat in the middle of the module that drives the
grid. They move to _grid_variant_yaml.py, which _grid_runner re-exports so every
caller and the argv-seal census keep their names.

The fourteen abort lines each repeated the same three arguments to say which variant
they were about; one tag names a variant once.

Observable effect: none -- pure refactor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis chennyiiis changed the title refactor(grid): name the two ways a grid round can measure nothing refactor(grid): drive the grid over named variant phases Oct 19, 2026
Comment thread src/hyperloom/orchestrator/actions/executors/_grid_variant_yaml.py Fixed
Comment thread src/hyperloom/orchestrator/actions/executors/_grid_variant_yaml.py Fixed
Test and others added 3 commits October 19, 2026 18:27
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The variant-YAML module reached into hyperloom.inference_optimizer for two
private names; it now owns that reach, and _grid_runner takes both from it
rather than around it. The warmup round's sub-directory is named once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis chennyiiis added the size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description label Oct 19, 2026
Test and others added 5 commits October 19, 2026 18:46
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…public

``shell_safe_dedupe`` and ``SGLANG_MOE_RUNNER_BACKEND_RE`` are used from
``hyperloom.orchestrator``, so they are part of what ``grid_server_args``
offers, not names private to it. Their owner names them that way now and the
callers import them plainly instead of ``as`` themselves across packages.

Observable effect: none -- pure refactor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
Test and others added 6 commits October 20, 2026 19:54
…ir owners

Materializing a variant's YAML reached sideways into the workload-env
materializer for a flag-string edit and an AgentX env pin. The flag removal
now sits beside the regex that detects the same flag in grid_server_args, and
the pin beside the backend predicate it reads in common.agentx_workload, so
the grid's variant YAML no longer depends on the module that materializes the
run's own envs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
…bort-path

Resolve scripts/code_metrics_baseline.txt with resolve_baseline.py: two hunks
retired by main's over-gate deletions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d-runner

Main's #1782 made gpu_runner_type public and routes the benchmark runner
through the GPU product family in both apply_runtime_benchmark_overrides and
_apply_runner_type; conc_sweep now imports the public name. The merge keeps
main's resolution in all three call sites and unions it with this branch's
pin_mlperf_round_concurrency import.

scripts/code_metrics_baseline.txt was resolved with the baseline resolver:
the port_loop entries main's #2026 retired stay retired, and run_grid's stay
retired by this branch's split.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Test and others added 5 commits October 21, 2026 13:55
…bort-path

Conflict: scripts/code_metrics_baseline.txt only, resolved with
resolve_baseline.py (1 hunk: try_flydsl_kb_warmstart retired by main,
run_grid kept at this branch's lower 55).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d-runner

Conflict: scripts/code_metrics_baseline.txt only, resolved with
resolve_baseline.py (try_flydsl_kb_warmstart retired by main, run_grid
retired by this branch).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…k in its module

Delete-first pass over this branch's own structure:

- _GridRecipe, _GridLaunch and _GridBudget fold into one _GridRun. The nesting
  carried two second copies of state the record already held -- cwd was
  str(output_root) and _GridBudget.deadline_sec was _GridLaunch's
  session_deadline_sec -- and both are gone.
- _RoundBudget and _variant_rounds restated the round counts _GridRun already
  carries; only the multi-node warmup count has to be worked out.
- _WatchdogStop restated _VariantFailure's fields for one caller, and
  _RoundHarvest was a second record of the round _MeasuredVariant names.
- _preflight_abort_error and _missing_workspace_error had one caller each and
  go back into it.
- _grid_variant_yaml.py is gone: _build_variant_yaml, _apply_variant_env and
  variant_conc return to _grid_runner.py, which owns them on main, and
  explore.py and conc_sweep.py import them from there again.

No observable change: internal structure only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…grid-runner

#1896 lands first and this branch rebuilt the same abort path over module-level
phases, so _grid_runner.py keeps this branch's shape. Both of #1896's later
commits survive in it: the rows a halted grid never reaches still carry that
stop's own ``never_started`` and ``error_class`` (through the whole
``StoppedByTheRun`` on ``_VariantAborted.abandoned`` rather than a copy of the
two fields), and the round attempt still carries no copy of the loop's state
the slot already names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ated

Twenty-one of this branch's private helpers carried a one-line docstring that
said what the name already says. The ones that state a return contract, a
control-flow exit or a constraint from outside the file stay.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chennyiiis

Copy link
Copy Markdown
Contributor Author

Reworked delete-first. Production net lines are now −36 (+1350 / −1386 against merge-base e25cb2db2), down from the +128 you measured. No new module, five fewer record types, and five baseline entries retire with no new one added.

before now
net production lines +128 −36
new modules 1 0
new record types 17 11
_grid_runner.py module lines 2089 → 1951 + a 208-line sibling 2089 → 2045, no sibling

Cut next — done

  • _GridRecipe. Gone, with _GridLaunch and _GridBudget: the four-level nest is one flat _GridRun. The nesting was itself carrying two second copies of state the record already held — _GridLaunch.cwd was str(output_root) and _GridBudget.deadline_sec was _GridLaunch.session_deadline_sec — and both are gone rather than rebound. (I did not reach for functools.partial: _build_variant_yaml's twelve arguments are not fixed across the two calls — the warmup call writes into a different directory against a different lifecycle — so a partial would have to be rebuilt per variant and would hide which of the twelve the second call changes.)
  • The duplicate _mn_warmup_rounds = 0. Gone with _RoundBudget and _variant_rounds: _GridRun already carries rounds_per_variant and mn_warmup_rounds, so only the multi-node warmup count has to be worked out, by _multi_node_warmup_rounds.
  • _RoundHarvest. Gone; the workspace and its salvage are fields of _MeasuredVariant, which is the record of that same round.
  • Helpers with one caller that only thread arguments through. _preflight_abort_error and _missing_workspace_error are back inside _abort_without_workspace; _watchdog_stop and _WatchdogStop are back inside _abort_on_watchdog (_WatchdogStop was also a second statement of _VariantFailure's own fields).
  • Docstrings that restate a private helper's name. Twenty-one dropped. The ones that state a return contract, a control-flow exit, or a constraint from outside the file stay.
  • _grid_variant_yaml.py. Deleted rather than imported from. _build_variant_yaml, _apply_variant_env and variant_conc are back in _grid_runner.py, and explore.py and conc_sweep.py import them from there again, exactly as on main. That closes CodeQL 4008 and 4085 by construction, and it is why the baseline adds no _grid_variant_yaml.py::_build_variant_yaml entry: the _grid_runner.py entry is lowered 135 → 98 instead.
  • The as X re-exports and __all__. The as X lines at the top of _grid_runner.py are main's, unchanged by this branch. __all__ now has two entries fewer than merge-base: _SGLANG_MOE_RUNNER_BACKEND_RE and moe_runner_requires_aiter are gone.

Not cut, and why

_MeasuredVariant, _VariantOutcome and _RoundAttempt stay. None is single-hop any more: _RoundAttempt is built at three call sites and read at one — it is what merges the warmup, multi-node-warmup and measured reap paths into one abort; _MeasuredVariant is read by _variant_row and _round_warnings; _VariantOutcome is read by four row builders. Replacing them with a per-variant runner object is the redesign you sketched and it is larger than this change; I would rather land the deletions and keep the three records than half-do it.

Blocking item 1 was fixed in eaaeba943: the server_init_dead excerpt is read inside _abort_on_watchdog after harvest_leaked_artifacts, test_server_dead_cause_read_from_the_harvested_log pins it against INFERENCE_OPTIMIZER_LEAK_ROOTS, and the c0 marker assertion in test_grid_runner_behavior_lock.py is back to asserting both markers.

#1896 is merged into this branch, so the two land cleanly in either order; its _grid_runner.py is superseded here, and both of its later commits survive — the rows a halted grid never reaches carry that stop's own never_started and error_class through the whole StoppedByTheRun on _VariantAborted.abandoned, and the round attempt carries no copy of state its slot already names.

Comment thread src/hyperloom/orchestrator/actions/executors/explore.py Fixed
…d-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/actions/executors/_grid_runner.py

@ZhengGong-amd ZhengGong-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1964 -- refactor(grid): drive the grid over named variant phases

What it does: Splits the ~1000-line loop body of run_grid in _grid_runner.py into module-level phase functions driven by _measure_variant, so every variant ends on one abort path. It also rewrites the argument-removal and JSON-blob handling in grid_server_args.py and moves two helpers out of _workload_envs.py. The only intended runtime difference is one extra finished-variant progress note when the session budget runs out.

Blocking issues: 3

  1. [X2] The description omits the second concern in the diff: the grid_server_args.py rewrite and the helper moves [verified]
    Problem: "Functions moved" lists only _build_variant_yaml, _apply_variant_env and variant_conc. The diff also contains:

    • _workload_envs._remove_moe_runner_backend_arg (base _workload_envs.py:839) is now the public grid_server_args.remove_moe_runner_backend_arg (src/hyperloom/inference_optimizer/grid_server_args.py:790), and the regex is now _SGLANG_MOE_RUNNER_BACKEND_PAIR_RE (grid_server_args.py:787).
    • pin_mlperf_round_concurrency moved from base _workload_envs.py:449 to src/hyperloom/common/agentx_workload.py:59.
    • _shell_safe_dedupe (base grid_server_args.py:475) is now the public shell_safe_dedupe (grid_server_args.py:482).
    • grid_server_args.py was rewritten (git diff --numstat: +86/-70). It adds _removed_token_count (:91) and _json_blob_end (:288) and lowers the complexity of remove_server_args and _reserialize_json_blobs. Two of the retired baseline entries come from this rewrite.
    • The callers changed in baseline.py, _benchmark_runtime.py and _workload_envs.py.
      The body does not say why these changes ship with the grid-runner refactor.
      Impact: The release notes would present this as a grid-runner-only refactor. An operator chasing a changed SGLang server command line (flag dedupe, --moe-runner-backend stripping, JSON-blob reserialization) would not learn that this PR reimplemented that logic.
      Action: Add a section that lists the four changes above and gives the one-concern-per-change reason for bundling them, or split them into their own PR. Write the body against git diff origin/main...HEAD only; every name and number must be reproducible from that diff.
  2. [X2] "Mechanisms removed" and the type and control-flow claims describe branch history, not the diff [verified]
    Problem: Checked against merge-base ba808c3, these statements are false:

    • "_GridRecipe, _GridLaunch and _GridBudget are now one flat _GridRun. _RoundBudget, _variant_rounds, _WatchdogStop, _watchdog_stop and _RoundHarvest are deleted outright." git grep -w <name> -- src finds 0 hits for all 8 names at both the merge-base and the head. They existed only in earlier commits on this branch. In the diff, _GridRun is new (_grid_runner.py:687).
    • "_GridBudget.deadline_sec was session_deadline_sec": _GridBudget does not exist on main. The cwd half of that bullet is accurate.
    • "21 docstrings that only restated a private helper's name": the docstrings on functions and classes in _grid_runner.py went from 41 to 60.
    • "no new type beyond the three that carry a phase's inputs": the head adds 9 NamedTuples, at src/hyperloom/orchestrator/actions/executors/_grid_runner.py:594, 602, 651, 659, 687, 718, 727, 740, 750 (_RoundProcess, _RoundError, _VariantSlot, _VariantFailure, _GridRun, _VariantRounds, _BenchmarkPass, _MeasuredVariant, _VariantOutcome). _RoundAttempt only moved.
    • "the loop body contains ... no return, break or continue": _grid_runner.py:1658 is a break inside except _VariantAborted. It runs when abort.halt is STOP, or is ON_FAILURE while keep_going_on_failure is false.
      Impact: The release notes would say that 8 types and 21 docstrings were deleted from main, though main never had them. They would also say the refactor added no types beyond three, when it added 9.
      Action: Delete the bullets about the 8 branch-only names, _GridBudget.deadline_sec and the 21 docstrings. State that 9 NamedTuples were added, each carrying one phase's inputs or outputs. Reword the invariant: the only break is in the abort handler, and all three raise sites funnel into it. Write the body against git diff origin/main...HEAD only; every name and number must be reproducible from that diff.
  3. [X2] The complexity numbers come from before #1896, not from the merge-base [verified]
    Problem: "Complexity" quotes values from 39c5ac1c9^. The figures at merge-base ba808c3, measured for run_grid (_grid_runner.py:1565) and scripts/code_metrics_baseline.txt, are:

    • "ruff C901 61 -> 9": actually 56 -> 9 (ruff check --select C901).
    • "cognitive 196 -> 21": actually 180 -> 21 (complexipy). The baseline entry was 177.
    • "_grid_runner.py 2089 -> 2045 lines": actually 2079 -> 2045.
    • "Five baseline entries retire": the baseline diff deletes 4 entries: _reserialize_json_blobs 45, remove_server_args 37, run_grid cognitive 177 and run_grid cyclomatic 55. scripts/code_metrics.py --base-ref origin/main reports cyclomatic -1 and cognitive -3. "No new entry is added" is accurate.
      Impact: The release notes would publish complexity reductions and a baseline-retirement count that cannot be reproduced against main.
      Action: Replace the figures with C901 56 -> 9, cognitive 180 -> 21, file 2079 -> 2045, and 4 baseline entries retired (2 of them from grid_server_args.py). Write the body against git diff origin/main...HEAD only; every name and number must be reproducible from that diff.

Checked: live PR body; _grid_runner.py (run_grid, NamedTuple carriers, abort handler); grid_server_args.py; _workload_envs.py; common/agentx_workload.py; baseline.py; _benchmark_runtime.py; scripts/code_metrics_baseline.txt | Ran: gh pr view 1964; git grep -w for the 8 names at base and head; ruff check --select C901 and complexipy on base and head _grid_runner.py; git diff --stat/--numstat/--shortstat; trial merge into origin/main b772e94 (clean); scripts/code_metrics.py --base-ref origin/main (PASSED, New 0, Worse 0); ruff check/ruff format --check on the changed files; pytest on the affected test files (the merged tree fails the same 2 tests as the merge-base and adds no new failures) | Base: ba808c3 | Head: f9adc67

Test added 2 commits October 21, 2026 21:57
…d-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/actions/executors/_grid_runner.py

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1964 -- refactor(grid): drive the grid over named variant phases

What it does: run_grid used to be one ~1100-line loop. It now drives each variant through named phase functions under _measure_variant. Every early end of a variant goes through one _abort_variant / _VariantAborted path and one handler in run_grid. The PR also gives helpers to the modules that own them: pin_mlperf_round_concurrency moves to common/agentx_workload.py, remove_moe_runner_backend_arg moves to grid_server_args.py, and shell_safe_dedupe and SGLANG_MOE_RUNNER_BACKEND_RE become public. It also restructures remove_server_args and _reserialize_json_blobs without changing what they do. The only runtime difference is one extra finished-variant progress note when the session budget ends the grid, and the description states it.

Blocking issues: 1

  1. [X3] A moved regex left its explanatory comment behind, above an unrelated block [verified]
    Problem: This PR moves _MOE_RUNNER_BACKEND_RE out of _workload_envs.py to grid_server_args.py:787 (_SGLANG_MOE_RUNNER_BACKEND_PAIR_RE), but leaves its comment in place. The comment explains why the value is optional: "Value is optional so a bare, value-less flag (an operator typo, or a flag left dangling at end-of-string) is stripped too...". That comment now sits directly above the profile-phase capture cap (_DEFAULT_PROFILE_MAX_STEPS = 128), which has no optional value. The comment at the regex's new location (grid_server_args.py:786) says "with the value that follows it" and leaves out the reason the value is optional. src/hyperloom/orchestrator/actions/executors/_workload_envs.py:100
    Impact: Runtime behaviour does not change; test_moe_runner_backend_injection.py:331-332 still pins the bare-flag removal. The effect is on readers. A reader of _workload_envs.py is told the profile cap has an optional value. A reader of the regex is no longer told why the value is optional, so a later edit that makes the value mandatory looks safe. A retry meant to launch without --moe-runner-backend would then keep a bare flag. This breaks AGENTS.md Finish the replacement (the superseded code's leftovers go with it) and Comment below the local average (a comment says what the code cannot). The concern belongs to grid_server_args.py.
    Action: Delete the three comment lines at _workload_envs.py:100-102. Change the comment at grid_server_args.py:786 to say the value is optional, so a bare or dangling --moe-runner-backend is removed too.

SKIPPED: V4 -- "# PARTIAL: #1869 changes 554 files, only its first 100 were compared", overlap there unchecked
SKIPPED: V4 -- "# PARTIAL: #1276 changes 1311 files, only its first 100 were compared", overlap there unchecked
SKIPPED: V4 -- "# PARTIAL: #1154 changes 1232 files, only its first 100 were compared", overlap there unchecked
SKIPPED: V4 -- "# PARTIAL: #1032 changes 1075 files, only its first 100 were compared", overlap there unchecked

Checked: _grid_runner.py base run_grid (:768-1873) against head run_grid + _measure_variant and every phase helper (:594-1669), site by site; _build_variant_yaml/_apply_variant_env/variant_conc and the three extracted env helpers; _resolve_mn_effective_server_args, _not_run_skip_result, _measurement_server_log_path, _RoundAttempt, _VariantAborted; grid_server_args.{remove_server_args,_removed_token_count,_reserialize_json_blobs,_json_blob_end,shell_safe_dedupe,remove_moe_runner_backend_arg}; common/agentx_workload.pin_mlperf_round_concurrency; _workload_envs.py, _benchmark_runtime.py, baseline.py and conc_sweep.py call sites; benchmark_result.extract_benchmark_measurement (use of config_path); a tree-wide git grep at head for every renamed or removed name; the previous card's F1 (fixed: _abort_on_watchdog harvests at :1258 before reading the excerpt at :1264) | Ran: a base-vs-head run_grid differential harness with only _run_magpie and report_progress stubbed, 14 scenarios (session-time-exhausted, cancelled, server_init_dead with a leak-root log, detokenizer_stall, no workspace at rc 1 with keep_going off and at rc 0, AgentX preflight, eval_probe_unpatchable, report missing, lifecycle_boot_only, magpie timeout, nonzero invalid measurement, top-of-loop and post-restart budget skips), each with warmup on and off: result rows identical in all 28 runs, and the only extra progress note is the stated one on the budget skips; a base-vs-head remove_server_args/_reserialize_json_blobs differential over 186,430 cases with 0 mismatches; CI at head 5fcd850 has every check green (Windows host cannot run the async test files: no fcntl or pytest-asyncio) | Base: bb2da63 | Head: 5fcd850

Test and others added 4 commits October 22, 2026 15:10
…d-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
#	src/hyperloom/orchestrator/actions/executors/_workload_envs.py
#	src/hyperloom/orchestrator/actions/executors/baseline.py
The MoE runner backend pair regex moved to grid_server_args.py but its
explanation stayed behind in _workload_envs.py, where it came to sit above
the profile-phase capture cap, which has no optional value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e size

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d-runner

# Conflicts:
#	scripts/code_metrics_baseline.txt
@chennyiiis

Copy link
Copy Markdown
Contributor Author

@chaojhou — [X3] fixed, and origin/main is merged in (branch is MERGEABLE again).

Where the comment now lives. The explanation moved to the regex it describes, in src/hyperloom/inference_optimizer/grid_server_args.py:786-790, directly above _SGLANG_MOE_RUNNER_BACKEND_PAIR_RE:

# Same flag with the value that follows it, so a removal takes the pair and not just the name.
# The value is optional so a bare, value-less flag (an operator typo, or a flag left dangling at
# end-of-string) is stripped too rather than surviving into a retry that is supposed to launch
# without it.
_SGLANG_MOE_RUNNER_BACKEND_PAIR_RE = re.compile(r"(?:^|\s)--moe-runner-backend(?:(?:[=\s]+)(?!--)\S+)?")

The three orphaned lines at _workload_envs.py:100-102 are gone, so _DEFAULT_PROFILE_MAX_STEPS now carries only its own capture-cap comment and nothing about an optional value. Comment placement only — no code moved or changed for this.

On the merge. Three files conflicted against main: _workload_envs.py and baseline.py (both where main extracted inline blocks into helpers — _refuse_framework_script_mismatch / _pin_workload_shape_envs, and _settle_attempt_server_args), plus scripts/code_metrics_baseline.txt. Resolved toward main for behaviour, with this branch's structure re-applied on top. The merge did silently leave two call sites on the deleted private name _remove_moe_runner_backend_arg — inside main's new _settle_attempt_server_args (baseline.py) and at _workload_envs.py:1640 — both repointed at grid_server_args.remove_moe_runner_backend_arg, with a tree-wide grep confirming no stale references to the old names remain.

The three moved helpers were checked explicitly against main at their old locations: main's _workload_envs.py hunks all begin at line 1075, below pin_mlperf_round_concurrency (:449), _MOE_RUNNER_BACKEND_RE (:101) and _remove_moe_runner_backend_arg (:841), and main made no change at all to grid_server_args.py; caller counts per file are identical at base and at main for all of them, so nothing of main's was dropped by the move.

Your previous card's F1 still holds after the merge: _abort_on_watchdog harvests at _grid_runner.py:1258 and reads the death excerpt at :1264. #1896's single abort path (_abort_variant / _abort_on_round_stop / _abandon_rest_of_grid / _VariantAborted) is intact.

Production net lines are unchanged at -26. CI is 33 pass / 0 fail at c426c57.

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1964 -- refactor(grid): drive the grid over named variant phases

What it does: run_grid used to be one ~1100-line loop. It now drives each variant through named phase functions under _measure_variant. Every early end of a variant goes through one _abort_variant / _VariantAborted path and one handler in run_grid. The PR also gives helpers to the modules that own them: pin_mlperf_round_concurrency moves to common/agentx_workload.py, remove_moe_runner_backend_arg moves to grid_server_args.py, and shell_safe_dedupe and SGLANG_MOE_RUNNER_BACKEND_RE become public. It also restructures remove_server_args and _reserialize_json_blobs without changing what they do. The only runtime difference is one extra finished-variant progress note when the session budget ends the grid, and the description states it.

Blocking issues: none

Checked: _grid_runner.py base run_grid (:768-1873) against head run_grid + _measure_variant and every phase helper (:594-1669), site by site; _build_variant_yaml/_apply_variant_env/variant_conc and the three extracted env helpers; _resolve_mn_effective_server_args, _not_run_skip_result, _measurement_server_log_path, _RoundAttempt, _VariantAborted; grid_server_args.{remove_server_args,_removed_token_count,_reserialize_json_blobs,_json_blob_end,shell_safe_dedupe,remove_moe_runner_backend_arg}; common/agentx_workload.pin_mlperf_round_concurrency; _workload_envs.py, _benchmark_runtime.py, baseline.py and conc_sweep.py call sites; benchmark_result.extract_benchmark_measurement (use of config_path); a tree-wide git grep at head for every renamed or removed name; the previous card's F1 (fixed: _abort_on_watchdog harvests at :1258 before reading the excerpt at :1264) | Ran: a base-vs-head run_grid differential harness with only _run_magpie and report_progress stubbed, 14 scenarios (session-time-exhausted, cancelled, server_init_dead with a leak-root log, detokenizer_stall, no workspace at rc 1 with keep_going off and at rc 0, AgentX preflight, eval_probe_unpatchable, report missing, lifecycle_boot_only, magpie timeout, nonzero invalid measurement, top-of-loop and post-restart budget skips), each with warmup on and off: result rows identical in all 28 runs, and the only extra progress note is the stated one on the budget skips; a base-vs-head remove_server_args/_reserialize_json_blobs differential over 186,430 cases with 0 mismatches; CI at head 5fcd850 has every check green (Windows host cannot run the async test files: no fcntl or pytest-asyncio) | Re-review at c426c57: b315dff moves the optional-value note to grid_server_args.py:787-789 and removes it from _workload_envs.py; the hand-resolved merges of main (7d8a139, c426c57) keep #1963's phase split and #1961's baseline.py rewrite intact, with the PR's renames carried to _merge_caller_overrides (_workload_envs.py:1637) and baseline.py:2363; per-file PR diff against main matches the reviewed diff; git merge-tree with origin/main is clean; CI green | Base: bb2da63 | Head: c426c57

LGTM

@chaojhou
chaojhou merged commit b392e8d into main Oct 22, 2026
40 checks passed
@chaojhou
chaojhou deleted the feature/chenyi/cc-grid-runner branch October 22, 2026 07:36
siliangchen-amd added a commit that referenced this pull request Oct 23, 2026
…the merged tree's other readers do

main's agentx_workload.py (#1922, #1964) reads envs.get("CONC"), a name the rule pack owns on
multibranch-dev; the outside-facts sentinel refuses a reader that repeats an owned value. Same
resolution as provenance.py in the sync merge.

Shared-File: src/hyperloom/common/agentx_workload.py @yihan.wang-agent
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants