[zoroyihan7]

• Description: what and why

Two things for release/v1.1.4:

1. Resume fixes. Cherry-picks the minimum set of fixes that a resume on release/v1.1.4 needs (hyperloom recover followed by optimize --resume-from on the original session). Each one is required: without it that resume fails or stalls. No refactors are picked. One docs commit documents the new recover step for operators (below). Resume a crashed integrate: discharge a tree lost with its execution environment, replay an accepted env-only candidate #1879 (merged on main as d77c808bb) is picked on top: once the steps above let a session resume, a crashed integrate_patch window is the next thing that stops it.
2. Sweep and baseline server teardown (partial backport of fix(geak): make an AgentX hand-off to GEAK end in a result the session can keep. #1606). Two commits that stop the server an AgentX client booted when its round ends, so a round killed at its cap or cancelled no longer leaves that server holding the port and nearly all GPU memory for the rounds after it. This changes round teardown in every sweep, explore and baseline round (see its section below); it is not part of the resume path.

hyperloom recover --confirm-scope-gone (#1859). When the container that ran a session is gone, the leases it held name an execution scope that no longer exists. Until now the only way out was --confirm-stopped one task at a time, which refuses tasks with a recorded owner scope, so the resume guard kept refusing the session. recover --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT now cancels the unfinished tasks of that vanished scope and releases their execution and GPU leases, under the session lock and in one transaction. It refuses the current scope, an unobservable current scope, any unexpired lease of another scope, and tasks that span two scopes. Why required: release/v1.1.4 does not have the command (git merge-base --is-ancestor 680b3742 origin/release/v1.1.4 fails), and without it a session whose container is gone cannot be resumed at all; #1870's regression test and #1871 also build on it.

Re-dispatch the conc_sweep when a resume finds SWEEP without one (#1870). recover cancels a vanished scope's tasks directly in the task ledger, so a conc_sweep cancelled that way never records its outcome. A resume does not re-enter the current phase, so a session resumed in SWEEP had no sweep in flight and nothing to exit on: it idled until the grant the lost sweep held ran out, and the sweep was never measured. Before a resumed process's first phase decision, SWEEP with no recorded conc_sweep outcome now re-fires its entry hook, once. Why required: without it a session interrupted in SWEEP resumes and then sits idle instead of finishing.

Owner scope unique per container (#1871). The owner scope recorded on leases was <boot_id>:<pid-namespace inode>. A container started on a node after another one exited can repeat that inode, so recover --confirm-scope-gone refused the lost container's scope as "the current execution environment" and the resume guard kept refusing. The scope now also includes the start time of the namespace's PID 1. Two-part scopes from this boot and namespace are still treated as this environment by the resume guard and reaper, and confirming one checks that no holder process that started before its lease is still running here. Why required: without it, a resume on the same node after a container restart is refused every time.

Resume a crashed integrate (#1879, two commits). Once a vanished container's scope is confirmed gone, a session whose container was lost during an integrate_patch still cannot resume. state.json keeps pending_integrate (phase ready, the operator's stash, the source tree's HEAD) for a tree that lived in the lost container, and the task's last outcome is the operator's stop confirmation with an empty result. Resume held that sentinel as an unknown outcome, and a rollback would have refused anyway because the recorded stash is absent from the new checkout: every resume stopped with patch_recovery_incomplete, with no way forward short of editing state.json.

• First commit: an integrate attempt records an identifier in the per-worktree git directory of the tree it mutates (recovery.tree_instance). On resume, a different identifier at the same path means another checkout: the sentinel is marked restored with tree_replaced, artifact targets outside the tree are restored from their preimages, and the stash that was not restored is reported (unrestored_stash_oid), never popped into the new tree. A task whose last outcome is an operator stop confirmation, with no KEEP on the bus and no stack row, now rolls back instead of being held; an accepted window is still held. A sentinel written before identifiers existed (every sentinel a v1.1.4 session wrote before this pick) counts as a replaced tree only when the task was confirmed stopped, the recorded stash object is not in the tree's object store at all, every patch of the candidate still applies forward cleanly and none reverse-applies, and every artifact target inside the tree holds its recorded preimage; otherwise it refuses as before. That inference can misread the same tree with its stash dropped and pruned in two cases, both recorded in the tree_replaced docstring: the candidate already taken out by hand (nothing is lost), and an in-tree artifact whose content is back at its preimage but whose file mode the candidate changed (the mode stays as the candidate set it).
• Second commit: resume read a crashed window's task row only when the window carried a patch or an artifact. A candidate that changed only environment variables or server arguments has neither, so if its KEEP reached the task row but not the bus, resume cleared the sentinel as stale and the accepted candidate never reached the optimization stack. Every window's task row is now read; a confirmed KEEP in the accepted phase is replayed into the stack (replayed_pending_integrate), anything else with nothing in the tree is cleared as before, never held.

Why required: a release user whose container is lost mid-integrate cannot resume at all after the fixes above (the first commit), and a resume can silently drop a candidate it had already accepted (the second). Both defects are in v1.1.4: the reproductions below fail on this branch before the pick and pass after it.

Sweep and baseline: stop the server a round's AgentX client booted (partial backport of #1606). Two commits on top of the resume picks: d09c9e1df (sweep / explore rounds in run_grid) and 384eb89d1 (the baseline executor's single-round cleanup).

What. In a concurrency sweep whose rounds each boot their own server through an AgentX client (aiperf_client.sh / mlperf_agentic_client.sh, per-round restart), a round that hits the benchmark cap is killed as magpie_timeout. The client dies, but the server it booted leads its own session and is re-parented away from it, so reaping the round's process tree never reaches it. On release/v1.1.4, run_grid only tears down a server it started itself (auto-warm-up lifecycle), so that leftover server keeps the port and nearly all GPU memory for the rest of the sweep: the next rungs fail to bind the port and measure against the leftover, and once it goes idle the later rungs fail at engine start for lack of memory (magpie_nonzero_invalid_measurement). The stop marker still records "tree reaped".

Why it is a bug, not a feature. Every rung after a timed-out one reports a number that is not its own (measured against another rung's server) or no number at all. Nothing in the sweep is meant to keep a timed-out round's server alive; the client scripts already record its pid in agentx_server.pid for exactly this teardown, and the release simply never reads it.

Partial backport scope. #1606 (c9818d50e) bundles this fix with unrelated behaviour changes. Only its server-teardown part is taken, three production files, with -x-style reference in the commit message:

• common/agentx_workload.py: AGENTX_SERVER_PIDFILE.

• _grid_runner.py: the variant teardown runs after every round (measured, failed, timed out, cancelled, warm-up stop, multi-node warm-up stop), not only auto-warm-up rounds, and also stops the server agentx_server.pid names; a stopped round's marker and nonfatal_warnings name any server that outlived SIGKILL.

• _server_lifecycle.py: stop_agentx_servers / reap_recorded_servers; the reaper treats a zombie leader as gone while still reaching live ranks in its group, refuses to signal its own process group, confirms the SIGKILL and keeps a survivor's pidfile.

• baseline.py (384eb89d1): the single-round cleanup calls _lifecycle.stop_agentx_servers(output_dir), the same line fix(geak): make an AgentX hand-off to GEAK end in a result the session can keep. #1606 adds on main (baseline.py:2740 at c9818d50e). On a single node the baseline executor runs the same AgentX client (MAGPIE_RUN_PHASE stays full, so the client boots the server and records it in agentx_server.pid), and without this a killed or cancelled baseline round left that server up until the next session-wide reap.

Not taken: the GEAK metric axis, the GEAK hand-off GPU wait (free_gpus_for_handoff and the rocm_smi helper it needs; it has no caller without the KERNEL-phase change), the extend-budget cap, the GEAK budget reservation and #1606's code-metrics baseline update.

Tests. #1606's TestRoundServerTeardown and orphan-reaper tests, plus the timed-out-round regression test from #1894 (TestRoundServerTeardown::test_timed_out_round_stops_its_server_before_the_next_round, run with a sglang and a vllm server cmdline). Python 3.10.22:

• On the PR head before the commit (5b9f21250, production files unchanged): the new test fails for both cmdlines with assert [True] == [False] (the first round's server is still alive when the second round starts); an independent reproduction written outside the PR fails the same way.
• With the commit: both pass, the reproduction passes; test_grid_runner_behavior_lock.py + test_orphan_server_reaper.py: 38 passed.
• Every test file that mentions the grid runner, orphan reaper, conc sweep or resume, or imports _server_lifecycle / run_grid (43 files): 2251 passed, 4 skipped, 2 failed; the same 2 (test_coverage_gap_units.py::test_credentials_validate_and_reset_claude_config, test_integrate_patch_executor.py::test_enablement_zero_patch_round_does_not_erase_prior_accepted_work) fail on 5b9f21250 on this host as well, so neither comes from the commit.
• ruff check / ruff format --check (0.16.2) on the five touched files: clean.
• 384eb89d1: test_baseline_warmup_double_run.py::test_baseline_single_round_stops_the_server_its_agentx_client_booted (a completed and a cancelled round, each leaving a setsid'd server with agentx_server.pid) fails on d09c9e1df (the server is still alive) and passes with the commit. test_baseline*.py + test_grid_runner_behavior_lock.py + test_orphan_server_reaper.py + test_source_tree_identity_unit.py: 358 passed. The same commit adds an explanatory comment to the empty except FileExistsError in source_tree_identity.ensure_tree_instance (CodeQL "empty except"); no behaviour change.
• Code-metrics gate, --base-ref 5b9f21250: the only findings are run_grid (cognitive complexity 222 -> 228, length 1073 -> 1076) and _grid_runner.py module length (2258 -> 2274), from the teardown now running on every exit path. fix(geak): make an AgentX hand-off to GEAK end in a result the session can keep. #1606 carried the same growth on main under baseline-raise; this PR takes that label for the same reason rather than reshaping run_grid on a release branch away from main.

Pick list, in order (git cherry-pick -x -m 1 for the merged PRs, plain git cherry-pick -x for #1879's two commits; each commit keeps its (cherry picked from commit ...) trailer):

#	main merge commit	release commit
#1859	680b374289328ca36a189798e70b9006861b46c2	21695dc29
#1870	c831742d37b18310fe3ba8cb387fa116e43d9e31	330d9bc5d
#1871	25ea91feacbd20769ce252249f8994876dd4ebfa	123ad1849
#1879 (1/2)	fd2ab7a546992447ec2b592fb2ab39b31d706ce7 (PR commit; merged on main by d77c808bb)	f25d8a41f
#1879 (2/2)	a436bd4e675cfe4cda06fcf1c124270698b128ed (PR commit; merged on main by d77c808bb)	5b9f21250
#1606 (partial)	c9818d50e909f322a81b5c7259f1b1e0efa62a7b	d09c9e1df, 384eb89d1

Base: release/v1.1.4 at 60b57706e.

Conflicts and how they were resolved:

• Let an operator confirm a vanished execution scope so a lost session can resume #1859: applied cleanly.
• fix(sweep): re-dispatch the conc_sweep when a resume finds SWEEP without one #1870, src/hyperloom/orchestrator/phases/machine.py: on main, the context next to the new _release_holds_no_task_will_settle helper includes _drain_for_transition, which landed on main after the release was cut. Only the new helper is taken; _drain_for_transition is not brought to the release.
• fix(sweep): re-dispatch the conc_sweep when a resume finds SWEEP without one #1870, scripts/code_metrics_baseline.txt: the release's baseline for MachinePhase.advance_phase_if_needed is 192 (main's was 156). The pick shortens the function by 4 lines here too, so the entry is 188.
• Tell apart two containers that repeat one boot and PID-namespace inode in the owner scope #1871, src/hyperloom/inference_optimizer/session/resume_guard.py: on main, refactor(optimizer): bring the optimizer and scripts under the complexity ceiling #1840 split _inspect into _scan_leases / _scan_gpu_leases / _scan_tasks / _scan_open_rounds before Tell apart two containers that repeat one boot and PID-namespace inode in the owner scope #1871 changed one line in _scan_leases. The release keeps the unsplit _inspect, and the same one-line change (match the lease's scope with same_environment) is applied there. The refactor is not brought to the release. Every other file applied cleanly.
• Resume a crashed integrate: discharge a tree lost with its execution environment, replay an accepted env-only candidate #1879 (1/2), src/hyperloom/inference_optimizer/session/resume_guard.py: on main the new confirmed_stopped_scope sits next to _confirmation_entry / _write_confirmation, a split of _confirm_task_stopped_in_transaction that is not on the release. The release keeps the unsplit function, and confirmed_stopped_scope is added unchanged after it; only its docstring names _confirm_task_stopped_in_transaction as the writer. The evidence it reads (reason, cleanup_confirmed, outcome, confirmed_scope_gone) is what the release writes.
• Resume a crashed integrate: discharge a tree lost with its execution environment, replay an accepted env-only candidate #1879 (1/2), scripts/code_metrics_baseline.txt: the release's entries for IntegratePatchExecutor._stage_apply (521) and integrate_patch.py (4886) differ from main's. The pick shortens them here too, so the entries are 515 and 4881, measured on this branch. The new import line in integrate_patch.py applied cleanly.
• Resume a crashed integrate: discharge a tree lost with its execution environment, replay an accepted env-only candidate #1879 (2/2): applied cleanly.

Docs for the recover step (review request). One commit on top of the picks, docs and help text only, no behaviour change:

• docs/reference/operations.md, Disaster recovery / Scenario A: a new step 4 for a resume that exits 2 with resume blocked: ...: the hyperloom recover --session-dir S --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT command, what it cancels and releases, what it refuses (current scope, unobservable current scope, an unexpired lease of another scope, a task holding execution leases in two scopes, a two-part scope whose holder still runs here, a ledger without owner scopes, a held session lock), what happens if the runtime/pod_history.jsonl append fails, and when --confirm-stopped applies instead (legacy empty-scope ownership, one task at a time).
• docs/reference/operations.md: the crash-recovery lifecycle row, the hyperloom recover exit codes (2 also on a refused confirmation; 0 also when there was nothing to confirm) and the --resume-from / recover summary line.
• docs/reference/operator-scripts.md: the recover entry.
• recover subcommand help in parser.py: names the cleanup confirmations (same two lines, so _build_parser does not grow).

The same docs change for main is #1880.

No resume pick brought in unrelated changes. The #1859 edits to parser.py and resource_lock.py move the recover flags next to their handler and move local_owner_scope unchanged into hyperloom.common.owner_scope, which #1871 then extends.

• Linked issue(s): close/fix refs

Cherry-picks of #1859, #1870, #1871 and #1879 (all merged to main), and a partial backport of #1606 (server teardown only).

• Tests: added/updated? commands run? (what to test; if a test was replaced, where its assertions live now):

The picked tests come from the PRs: test_resume_scope_gone.py (#1859, extended by #1871), four TestASweepCancelledByRecoverIsRedispatchedOnResume tests in test_resume.py (#1870), and common/tests/test_owner_scope.py (#1871). Everything below was run on this branch with Python 3.10.22:

# py3.10 import of every touched module (no datetime.UTC anywhere in the diff)
python3.10 -c "import hyperloom.common.owner_scope, hyperloom.inference_optimizer.cli.recover, hyperloom.inference_optimizer.cli.parser, hyperloom.inference_optimizer.session.resume_guard, hyperloom.orchestrator.bus.resource_lock, hyperloom.orchestrator.phases.machine, hyperloom.orchestrator.phases.sweep"
# -> OK

T=src/hyperloom/inference_optimizer/tests
python3.10 -m pytest -q src/hyperloom/common/tests/test_owner_scope.py $T/test_resume.py $T/test_resume_scope_gone.py \
  $T/test_resume_guard.py $T/test_sweep_phase_auto.py $T/test_conc_sweep.py $T/test_conc_sweep_ceiling.py \
  $T/test_conc_sweep_session_deadline.py $T/test_cli_resume_launch_shape.py $T/test_resume_budget_objective.py \
  $T/test_preflight_resume_framework.py $T/test_geak_resume_recovery.py $T/test_cli_conc_sweep_default.py
# -> 577 passed, 2 skipped

Per file (collected): test_owner_scope 14, test_resume 40, test_resume_scope_gone 30, test_resume_guard 133, test_sweep_phase_auto 101, test_conc_sweep 121, test_conc_sweep_ceiling 7, test_conc_sweep_session_deadline 38, test_cli_resume_launch_shape 22, test_resume_budget_objective 7, test_preflight_resume_framework 36, test_geak_resume_recovery 7, test_cli_conc_sweep_default 23 (579 total = 577 passed + 2 skipped).

The two hand-resolved hunks were checked to be load-bearing on this branch: reverting the same_environment line in the release's _inspect fails test_resume_scope_gone.py::test_resume_after_an_in_place_upgrade_still_sees_its_own_two_part_lease, and removing the resume_unsettled_sweep() call from the new machine helper fails two of the #1870 tests in test_resume.py.

#1879 picks (Python 3.10.22): test_coordinator_async_batch2_unit.py (166, including the six new resume scenarios) and test_source_tree_identity_unit.py (19) pass, as do test_resume_guard.py and test_resume_scope_gone.py (348 passed across the four files). The wider run over every test file that mentions pending_integrate, the resume guard, tree identity or writeback (69 files): 3452 passed, 7 skipped, 17 failed; the same 17 fail on the release base 6c3c2cd8d on this host (no httpx module, no patch binary), so none comes from the picks. Two independent reproductions written outside the PR, run on this branch and on 6c3c2cd8d: a lost container's tree re-cloned at the same path (with an identifier and with a legacy sentinel) now discharges and restores the artifact outside the tree, where the base refuses; the same tree with an intact stash now rolls back normally, where the base holds it; the same tree with its stash pruned, or with the candidate edited by hand, still refuses on both; an environment-only and a server-args-only KEEP now reach the stack, where the base clears them (13 passed here, 6 failed on the base).

Docs commit: python -m hyperloom recover --help renders the new subcommand help; recover --confirm-scope-gone <this environment's scope> prints ERROR: the named owner scope is the current execution environment and exits 2, and combining it with --force exits 2, as the docs say; test_resume_scope_gone.py 30 passed; ruff check / ruff format --check on parser.py clean.

Code-metrics gate, run with the gate scripts taken from the release base: python code_metrics.py --root . --base-ref 60b57706e -> PASSED (11 files judged, 0 new, 0 worse, 0 stale baseline entries). For the #1879 picks, --base-ref 6c3c2cd8d (this branch before them) -> PASSED (8 files judged, 0 new, 0 worse, 0 stale entries). ruff check and ruff format --check (0.16.2) on the touched Python files: clean.

Re-qualification: resume and SWEEP paths, resume of a session interrupted during an integrate, a concurrency sweep with a timed-out rung, and a single-node AgentX baseline round that is cancelled or hits its cap.

A real resume E2E on this branch (head 5b9f21250, two pod evictions, recover --confirm-scope-gone then optimize --resume-from) is posted as a PR comment; the teardown commits are not on the resume path.

• Size/complexity triggers crossed: split, or why the shape is right:

None crossed by the resume picks; the code-metrics gate passes against the release base for them. The sweep teardown commit grows run_grid as #1606 did on main and carries baseline-raise (see its section). With it the PR changes 1146 production Python lines, over the 1000-line diff budget (the resume picks alone were under it); the PR carries size-exception because the sweep fix was asked to ship in this release PR rather than a separate one, and it is a separate commit that can be reviewed on its own (git show d09c9e1df). The production diff is about 600 lines for #1859-#1871 because it carries #1859 whole, which the other two fixes depend on. #1879 puts its new logic in a new module (actions/executors/source_tree_identity.py) and TaskRegistry.find, so writeback.py and integrate_patch.py do not grow.

• If this simplifies or refactors: which mechanism was removed, which contract was preserved, and the tests that show it:

Not a refactor. Main-only refactors (#1840's _inspect split, _drain_for_transition) were deliberately left out.

• Observable effect: what an operator will now see, and what the old behaviour cost them (if none, why the change is unobservable):

On v1.1.4 an operator can confirm a vanished container's scope with hyperloom recover --confirm-scope-gone and resume the session, including on the same node after a container restart. A session resumed in SWEEP measures its sweep instead of idling until the grant runs out. A session whose container was lost during an integrate resumes once its tasks are confirmed stopped; the resume report carries rolled_back_pending_integrate with tree_replaced: true and, when there was one, unrestored_stash_oid (the operator's uncommitted changes in the lost checkout are not recovered, and the report says so). An environment-only candidate whose KEEP reached the task row is promoted on resume instead of dropped. Before, such sessions could not be resumed, or resumed and then stalled or lost an accepted candidate.

In every sweep, explore and baseline round, the server an AgentX client booted is now stopped when the round ends, however it ended (measured, failed, timed out at the cap, cancelled, warm-up stop). The teardown reads agentx_server.pid under the round's own workspace, where $RESULT_DIR points by default; a round whose params.result_dir override points outside its workspace is not covered, as on main; the session-wide orphan reap still finds it when that directory lies under the session's runs/, as the failure-recovery reference recommends. Before, a round killed at its cap or cancelled left that server holding the port and nearly all GPU memory: later rungs measured against the leftover server or failed at port bind or engine start, and the stop marker still said "tree reaped". Operators may now see a server_survived_teardown:<pid> warning in a round's nonfatal_warnings and the stop marker when a server outlives SIGKILL; the orphan reaper waits up to 30 s to confirm a SIGKILL and keeps a survivor's pidfile for the next reap.

• Breaking changes: yes/no (details if yes)

no. New leases record a three-part owner scope; two-part scopes written by earlier versions are still recognised. New integrate sentinels carry one extra field (recovery.tree_instance), which older code ignores.

• PR addresses single concern: yes/no (details if no):

no: two concerns, both approved for this release. Resume on release/v1.1.4, and the round server teardown (partial backport of #1606, its own commits d09c9e1df and 384eb89d1, reviewable on their own).

• Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed:

No, the root cause is in Hyperloom.

🤖 Generated with Claude Code

[github-actions]

Code metrics gate: PASSED with waived findings (baseline-raise)

Judged: the 26 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Worse	Baseline out of date	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	0	0	122	+0
Cognitive complexity	fails > 30	0	1	0	323	+0
Function length (lines)	fails > 80	0	2	0	477	+0
Nested block depth	fails > 5	0	0	0	29	+0
Module length (lines)	warns > 800, fails > 1200	0	2	0	55	+0
Duplicated lines per file	fails > 0	0	0	0	120	+0
Dead code (vulture)	fails > 0	0	0	0	2	+0

Check	Findings	Effect
Comments	0
English only	0
Production code imports test code	0
Repeated literals	0
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	0

Worse than the baseline (waived by the baseline-raise label): 5

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/actions/executors/_grid_runner.py:863 run_grid	Cognitive complexity	222	228
src/hyperloom/orchestrator/actions/executors/_grid_runner.py:863 run_grid	Function length (lines)	1073	1076
src/hyperloom/orchestrator/actions/executors/baseline.py:2309 BenchmarkRunExecutor._run_once	Function length (lines)	682	683
src/hyperloom/orchestrator/actions/executors/_grid_runner.py:1	Module length (lines)	2258	2274
src/hyperloom/orchestrator/actions/executors/baseline.py:1	Module length (lines)	4253	4292

How to fix

• New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
• Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
• Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
• Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).

Already over at the base (backlog, not this change's): 4

Unit	Dimension	Base	Now
src/hyperloom/inference_optimizer/cli/parser.py:1	Module length (lines)	1214	1202
src/hyperloom/inference_optimizer/cli/parser.py:178 _build_parser	Function length (lines)	1038	1025
src/hyperloom/orchestrator/actions/executors/baseline.py:3371 BenchmarkRunExecutor._run_single_benchmark	Function length (lines)	580	580
src/hyperloom/orchestrator/loop/writeback.py:1	Module length (lines)	4809	4807

33 findings in files this change does not touch

Outside this change (informational): 33

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/enablement/recipe/projections.py project_launch_evidence	Cognitive complexity	absent	36
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison	Cognitive complexity	absent	35
src/hyperloom/common/launch_log_evidence.py _server_args_from_log	Cognitive complexity	48	49
src/hyperloom/orchestrator/actions/executors/explore.py ExploreExecutor.__call__	Cognitive complexity	369	372
src/hyperloom/orchestrator/loop/dispatcher.py DispatcherCollaborator._spawn_fitting_queued	Cognitive complexity	110	111
src/hyperloom/orchestrator/state/_shared_state/render.py _RenderMixin._format_variant_line	Cognitive complexity	36	40
src/hyperloom/common/launch_log_evidence.py _server_args_from_log	Cyclomatic complexity	22	23
src/hyperloom/orchestrator/actions/executors/_workload_envs.py materialize_config_with_envs	Cyclomatic complexity	110	111
src/hyperloom/inference_optimizer/cli/__init__.py _run_optimize	Function length (lines)	940	946
src/hyperloom/inference_optimizer/cli/bootstrap.py _seed_shared_state	Function length (lines)	155	157
src/hyperloom/orchestrator/actions/executors/_workload_envs.py materialize_config_with_envs	Function length (lines)	922	928
src/hyperloom/orchestrator/actions/executors/explore.py ExploreExecutor.__call__	Function length (lines)	1124	1133
src/hyperloom/orchestrator/loop/dispatcher.py DispatcherCollaborator._spawn_fitting_queued	Function length (lines)	242	243
src/hyperloom/orchestrator/loop/dispatcher.py DispatcherCollaborator.run_task_registered	Function length (lines)	109	111
src/hyperloom/orchestrator/phases/prelude.py PreludePhase._settle_warm_replay	Function length (lines)	388	390
src/hyperloom/orchestrator/policy/gate.py PolicyGate._validate_specialist_dispatch	Function length (lines)	91	93
src/hyperloom/orchestrator/prompts/prompt_builder.py build_orchestration_prompt	Function length (lines)	139	142
src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py _section_identity	Function length (lines)	94	95
src/hyperloom/orchestrator/specialists/runner.py SpecialistRunner._prepare	Function length (lines)	192	200
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison	Function length (lines)	100	101
src/hyperloom/inference_optimizer/cli/__init__.py	Module length (lines)	2569	2740
src/hyperloom/orchestrator/actions/executors/explore.py	Module length (lines)	1905	1914
src/hyperloom/orchestrator/loop/coordinator.py	Module length (lines)	1429	1432
src/hyperloom/orchestrator/loop/dispatcher.py	Module length (lines)	1503	1551
src/hyperloom/orchestrator/loop/intent_router.py	Module length (lines)	1548	1556

...and 8 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

[chaojhou]

PR #1874 -- release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871)

What it does: On release/v1.1.4 a session whose container is gone cannot be resumed: its leases name an owner scope that no longer exists, and the only release path, recover --confirm-stopped, refuses tasks with a recorded scope. This PR cherry-picks the three main fixes that close this -- hyperloom recover --confirm-scope-gone (#1859), re-dispatching a lost conc_sweep when a resume lands in SWEEP (#1870), and an owner scope that tells two containers on one node apart (#1871) -- applying #1871's one-line scope check to the release's unsplit _inspect and leaving main's refactors behind.

Blocking issues: 1

1. [X5] The new --confirm-scope-gone recovery step is missing from the operator docs this branch ships [verified]
   Problem: the flag is new on this base (absent from release/v1.1.4 at 60b5770) and no doc names it. docs/reference/operations.md "Disaster recovery / Scenario A: pod was OOM-killed mid-session" -- the scenario this PR exists for -- still tells the operator to relaunch with --resume-from, and says recover only rebuilds the breakdown artifact (operations.md:278); by this PR's own description that resume is refused until the vanished scope is confirmed. src/hyperloom/inference_optimizer/cli/recover.py:29
   Impact: an operator following the v1.1.4 disaster-recovery page after a lost container hits the resume guard's refusal, and the step that resolves it is discoverable only from that refusal message (resume_guard.py:196-197) or --help. The same page on main has the same gap.
   Action: add the hyperloom recover --session-dir S --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT step to Scenario A -- when to use it instead of --confirm-stopped, and what it refuses -- and update the other places that still describe recover only as a breakdown rebuild (operations.md:144 crash-recovery row, operations.md:214 exit codes, the recover subcommand help at parser.py:1186-1187), in this PR and the same change on main.

Checked: per-file interdiff of each pick against its main merge commit's first-parent diff (#1859 680b374 -> 21695dc, #1870 c831742 -> 330d9bc, #1871 25ea91f -> 123ad18): identical except the baseline numbers (192 -> 188) and the indentation of the one same_environment line in _inspect, as the description states; the owner-scope comparisons in resume_guard.py / resource_lock.py on the release head against main (same set, same kinds); every local_owner_scope importer on the release (the old import path is kept by resource_lock.py:16); the new machine helper's dependencies (release_finished_roofline_gate and resume_unsettled_sweep only, nothing from _drain_for_transition); open PRs on release/v1.1.4 (none besides this one); the docs for hyperloom recover on the head and on main; an independent reader pass over this card, the diff and the merge base | Ran: the interdiff script and git grep sweeps on both branches; no local tests (CI green at the head: 12 test shards, both coverage jobs, code-metrics, pr-hygiene, import-linter) | SKIPPED: V4 -- openprs.txt "# PARTIAL:" lines for #1032, #1154, #1276 and #1675 (only their first 100 files compared); none of them targets this base -- gh pr list --base release/v1.1.4 lists only this PR | Base: 60b5770 | Head: 123ad18

[zoroyihan7]

Thanks. Addressed in 6c3c2cd (docs and help text only, no behaviour change):

• docs/reference/operations.md:263-302, Scenario A: new step 4 for a resume that exits 2 with resume blocked: ...: the hyperloom recover --session-dir S --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT command, what it cancels and releases, what it refuses (current scope, unobservable current scope, an unexpired lease of another scope, a task with execution leases in two scopes, a two-part scope whose holder still runs here, a ledger without owner scopes, a held session lock; exit 2, ownership DB unchanged), the ledger-append warning case, and when --confirm-stopped applies instead (legacy empty-scope ownership, one task at a time).
• docs/reference/operations.md:144: crash-recovery row points at the confirmation step.
• docs/reference/operations.md:214: recover exit codes (0 also when there is nothing to confirm; 2 also on a refused confirmation).
• docs/reference/operations.md:320: the --resume-from / recover summary line.
• docs/reference/operator-scripts.md:194: the recover entry.
• src/hyperloom/inference_optimizer/cli/parser.py:1186-1187: recover subcommand help names the cleanup confirmations.

Same change on main: #1880.

[zoroyihan7]

Pushed d09c9e1df on top of 5b9f21250: the sweep teardown fix, a partial backport of #1606 (server-teardown part only; see the new section in the description). It touches only sweep teardown (agentx_workload.py, _grid_runner.py, _server_lifecycle.py and their tests), nothing on the resume path, so an E2E result already posted for 5b9f21250 still describes the resume picks.

Test result (Python 3.10.22):

• TestRoundServerTeardown::test_timed_out_round_stops_its_server_before_the_next_round (from test(grid-runner): a timed-out round stops the server its client booted #1894, sglang and vllm server cmdlines): fails on 5b9f21250 (assert [True] == [False], the timed-out round's server is still alive when the next round starts), passes on d09c9e1df.
• test_grid_runner_behavior_lock.py + test_orphan_server_reaper.py: 38 passed.
• 43 grid-runner / orphan-reaper / conc-sweep / resume test files: 2251 passed, 4 skipped, 2 failed; the same 2 fail on 5b9f21250 on the same host (environment-only).
• ruff clean. Code-metrics: run_grid grows as it did with fix(geak): make an AgentX hand-off to GEAK end in a result the session can keep. #1606 on main, so this PR now carries baseline-raise.

@chaojhou could you take another look?

[zoroyihan7]

E2E: resume after sandbox loss, on head 5b9f212

Re-ran the real resume end-to-end test on 5b9f21250813f94d3b8d9d4e7ab239db4ca23f1c (the PR head when the run started), the same way as the earlier run on 123ad1849. It used one GLM-5.2 MXFP4 / SGLang / MI355X session with a 3 h budget, driven through the production dispatch path. The session checkout and the recovery checkout were both at the PR head.

Interruptions (Kubernetes Eviction API on the session's own sandbox pod):

1. In PRELUDE after the baseline was measured (current_best = baseline 386.71 tok/s, ~7170 s charged, no lease held).
2. In FRAMEWORK_AGENT 2.7 min after the first resume, while a research-lane specialist held a lease. The replacement pod had landed on the same node as the first one. Its owner scope had the same boot id and PID-namespace inode, and only the third component differed (the Tell apart two containers that repeat one boot and PID-namespace inode in the owner scope #1871 case).

Verified

• After eviction 2 the vanished scope's lease stayed in the coordinator DB until recovery. Then recover --confirm-scope-gone matched the evicted pod's 3-part scope, cancelled the task as operator_confirmed_stopped and released the research-lane lease.
• The guard accepted each resume: the new process took optimizer.lock under the new sandbox's hostname.
• optimize --resume-from continued the same session directory both times. It kept current_best, the stack (0), the validated gain (0.0), elapsed_charged (7167 s, then 7299 s) and the phase.
• The session settled normally: FRAMEWORK_AGENT -> SWEEP -> CLOSE, sweep_done.
• Side observation: before eviction 1 the session was stuck in PRELUDE (details below). The resume's warm-replay recovery rolled it back and cleared warm_replay_pending, and PRELUDE -> FRAMEWORK_AGENT followed 5 s after resume.

Not covered

• The pending-integrate resume path added by the last two commits (Resume a crashed integrate: discharge a tree lost with its execution environment, replay an accepted env-only candidate #1879). No candidate reached integrate within the budget, so no interruption landed during an in-flight integrate. That path is covered by its unit tests only.
• The later commit d09c9e1df (sweep server teardown after a timed-out round) is not in the tested tree. It does not touch the resume code.
• SWEEP re-dispatch. The concurrency sweep was skipped by its time budget, as in the earlier run.

Pre-existing issue seen (not introduced by this PR; same code on main)
The warm recipe matched after the baseline, and the replay_warm_recipe task was cancelled at dispatch by the time-budget admission check ("needs ~97 min but only 79 min left"). That path does not settle warm_replay_pending, so it stayed in_flight and PRELUDE never reached prelude_done (22+ min with no progress until the eviction). This deserves a separate fix: settle or clear the warm-replay pending record when its task is cancelled before dispatch.

🤖 Generated with Claude Code

[chaojhou]

PR #1874 -- release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879)

What it does: On release/v1.1.4 a session whose container is lost cannot be resumed: its leases name an owner scope that no longer exists (and a restarted container on the same node can repeat that scope), a session resumed in SWEEP idles with no sweep in flight, and the sentinel an interrupted integrate left for the lost tree blocks every resume. This PR cherry-picks the main fixes (#1859 hyperloom recover --confirm-scope-gone, #1871 a three-part owner scope, #1870 SWEEP re-dispatch on resume, and #1879, still open on main, which adds a source-tree identity and replays env-only KEEPs) with their runbook docs, and partially backports #1606 so that run_grid stops the server an AgentX client booted after every round.

Blocking issues: 2

1. [C2] The partial #1606 backport stops a killed round's AgentX server only in run_grid; #1606's same teardown in the baseline executor is left out for a reason that does not hold [verified]
   Problem: stop_agentx_servers has a single production caller on this head, _teardown_variant_server (_grid_runner.py:2054). #1606 also calls it from the baseline executor's round cleanup (c9818d5 baseline.py:2740; its commit message: "so does the baseline executor's cleanup"). On this head the baseline executor runs the same AgentX client (baseline.py:2470-2476) through run_with_session_kill (baseline.py:3613); on a single node MAGPIE_RUN_PHASE stays full (it is set to client only for multi-node, baseline.py:3428), so the client boots the server itself and records it in agentx_server.pid (aiperf_client.sh:132). Neither the round cleanup (baseline.py:2735-2741) nor _pre_start_cleanup reads that file; only the session-wide reap_orphaned_servers does, from the roofline preflight and at coordinator boot and shutdown. The description lists "the baseline executor's cleanup call" as not taken under "unrelated behaviour changes ... Only its server-teardown part is taken", but that call is the same server teardown. src/hyperloom/orchestrator/actions/executors/_server_lifecycle.py:325
   Impact: in a single-node AgentX session, a baseline round killed at its cap or cancelled leaves its setsid'd server holding the port and nearly all GPU memory until the next roofline action or the end of the session, so the rounds after it hit the same port-bind failures and engine-start OOMs this commit fixes for sweep rungs.
   Action: add #1606's _lifecycle.stop_agentx_servers(output_dir) to the baseline executor's round cleanup (baseline.py:2735-2741), or keep it out and state in the description that a killed AgentX baseline round still leaves its server running on v1.1.4, and why that path is out of scope.
2. [X2] The title and the description's summary describe a resume-only PR, but the head also ships the partial #1606 sweep-teardown backport [verified]
   Problem: the title names only #1859, #1870, #1871 and #1879, but the head carries d09c9e1 ("fix(sweep): stop the server a timed-out round's client booted (partial backport of #1606)"), which changes round teardown in _grid_runner.py, _server_lifecycle.py and agentx_workload.py. The opening paragraph still says "Nothing else is picked: no refactors or unrelated fixes", and the observable-effect section lists only resume effects. src/hyperloom/orchestrator/actions/executors/_grid_runner.py:1594
   Impact: the body has its own section on the sweep fix, but the parts that summarise the PR -- the title, the opening paragraph and the observable-effect section -- leave it out or contradict it. The release cut aggregates PR descriptions into the GitHub release (AGENTS.md, "Every change states its observable effect in its own PR description"), so the v1.1.4 notes would present this PR as resume fixes only and omit the behaviour it changes in every sweep and explore round: teardown after every round, server_survived_teardown:<pid> warnings, and a 30 s SIGKILL confirmation in the orphan reaper.
   Action: retitle to name the sweep teardown (for example "release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879) and sweep server teardown (partial #1606)"), and correct the opening paragraph, the observable-effect section and the linked references to include it.

SKIPPED: V4 -- PARTIAL overlap for #1032, #1154, #1276, #1675, #1869 (each cut at its first 100 files), overlap there unchecked; all target main, and gh pr list --base release/v1.1.4 lists only this PR

Checked: per-file interdiff of every pick against its source (first-parent diffs of 680b374, c831742 and 25ea91f; #1879's head commits fd2ab7a and a436bd4; #1606 c9818d5): identical except the resolutions the description states and, for #1606, the parts it leaves out; #1879's head is still a436bd4 (OPEN) and its resume functions are byte-identical on this head; every symbol the picks and the backport call exists at the base; stop_agentx_servers callers on this head vs c9818d5; the baseline executor's AgentX path (prepare_agentx_runtime, run_with_session_kill, MAGPIE_RUN_PHASE) and the client scripts' agentx_server.pid; the earlier card's [X5], fixed by 6c3c2cd (operations.md:144, :214-215, :263-302, :320-322; operator-scripts.md:194-198; parser.py:1186-1187) | Ran: interdiff and function-compare scripts; git grep at 60b5770, d09c9e1 and c9818d5; python ..\pr_tools\c901.py 60b57706e d09c9e1df <14 src files> (ruff C901 counting; ruff is not installed here): no added or raised function over 20; gh run list --commit d09c9e1d (latest run of all 9 workflows green); gh pr list --base release/v1.1.4; findings traced statically, no local tests | Base: 60b5770 | Head: d09c9e1

[zoroyihan7]

Thanks. Both addressed in 384eb89d1 plus the PR title and description.

1. [C2] baseline executor teardown. Taken. The single-round cleanup in baseline.py now calls _lifecycle.stop_agentx_servers(output_dir), the same line #1606 adds on main (c9818d50e baseline.py:2740), in the same finally. New test test_baseline_warmup_double_run.py::test_baseline_single_round_stops_the_server_its_agentx_client_booted runs one baseline round through the executor whose fake client leaves a setsid'd server with agentx_server.pid in the round's workspace, for a completed and a cancelled round: both fail on d09c9e1df (server still alive) and pass on 384eb89d1. test_baseline*.py + test_grid_runner_behavior_lock.py + test_orphan_server_reaper.py + test_source_tree_identity_unit.py: 358 passed (Python 3.10.22); ruff clean. Scope note, now in the description: like main, the teardown scans the round's workspace, which is where $RESULT_DIR points by default; a params.result_dir override outside the workspace is not covered by the per-round teardown.

2. [X2] title and summary. Retitled to "release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879) and round server teardown (partial #1606)". The opening paragraph now lists the two concerns (resume fixes; sweep and baseline server teardown) and drops "Nothing else is picked"; the teardown section covers both commits and no longer lists the baseline cleanup call as not taken; the pick table has a #1606 row (d09c9e1df, 384eb89d1); linked references name the #1606 partial backport and #1879 as merged on main (d77c808bb); the observable-effect section adds the teardown after every sweep, explore and baseline round, the server_survived_teardown:<pid> warning and the 30 s SIGKILL confirmation in the reaper; "single concern" now says no, two concerns, with the teardown in its own commits.

CodeQL "empty except" (source_tree_identity.py:65). The except FileExistsError: pass in ensure_tree_instance now carries a comment saying why: another creator linked its identifier first and that one is read back. No behaviour change; main (d77c808bb) has the same code.

[xiaofei-zheng]

PR #1874 -- release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879) and round server teardown (partial #1606)

What it does. A session on release/v1.1.4 whose pod is lost cannot be resumed: the branch has no recover --confirm-scope-gone, its owner scope cannot tell two containers that repeat one boot and PID namespace apart, a resume landing in SWEEP never re-dispatches the sweep recover cancelled, and a pending_integrate whose source tree went with the container always stops at patch_recovery_incomplete. Separately, a sweep, explore or baseline round that boots its own server through an AgentX client leaves that server holding the port and the GPUs when the round is killed at its cap or cancelled, because the server is re-parented out of the round's process tree and run_grid only tore down a server it started itself. The PR cherry-picks #1859, #1870, #1871 and #1879's two commits onto the release branch, adds one docs commit for the new recover step, and hand-assembles the server-teardown part of #1606 (AGENTX_SERVER_PIDFILE, _teardown_variant_server on every round exit, stop_agentx_servers / reap_recorded_servers, and the one baseline.py call), leaving out #1606's GEAK metric-axis and hand-off work.

Blocking issues: none.

Checked, since the value of a backport review is whether the picks are faithful and the partial one is self-contained:

• Each pick against its upstream original. git show of 21695dc29, 330d9bc5d, 123ad1849, f25d8a41f and 5b9f21250 against 680b374289, c831742d3, 25ea91fea, fd2ab7a54 and a436bd4e6: the only differences are the three code_metrics_baseline.txt numbers measured on this branch (advance_phase_if_needed 192→188, _stage_apply 521→515, integrate_patch.py 4886→4881) and the one docstring cross-reference that names the release's unsplit _confirm_task_stopped_in_transaction. Both hand-resolved hunks (_inspect's same_environment line, the _release_holds_no_task_will_settle helper without _drain_for_transition) land in the release's unsplit functions and match main's logic.
• The partial fix(geak): make an AgentX hand-off to GEAK end in a result the session can keep. #1606 is complete. _server_lifecycle.py, _grid_runner.py, common/agentx_workload.py and baseline.py at head are byte-identical to c9818d50e's versions apart from the free_gpus_for_handoff / _serving_process_groups / _program_of block and the three imports and two constants only it uses — none of which has a caller on this branch. The not-taken hunks (_workload_envs.py, _gpu_pin.py, writeback.py, _geak_sweep.py, kernel.py) are all GEAK metric-axis work and touch neither $RESULT_DIR nor the pidfile, so nothing the taken code depends on was left behind.
• The teardown has a writer. assets/agentx/aiperf_client.sh:132 and mlperf_agentic_client.sh:72 already write agentx_server.pid on release/v1.1.4, so the backported reader is not inert. stop_agentx_servers reaches both AgentX round paths: _grid_runner.py:2054 (sweep / explore, now on every exit — measured, failed, timed out, cancelled, warm-up and multi-node warm-up) and baseline.py:2740. nonfatal_warnings exists at _grid_base.py:137.
• Release-base wiring for the resume picks. TaskRegistry.find, TERMINAL_STATES, Coordinator.phase_sweep (coordinator.py:439) and writeback.resumed_from (writeback.py:3195) all resolve; resume_unsettled_sweep is gated on _resume_checked and is_resume, so it fires once per process before the first phase decision. No dangling reference to anything not picked.
• The docs commit against the code it describes. The new Scenario A step 4 and the exit-code line check out: the 20-record cap (resume_guard.py:83), the five-scope cap (resume_guard.py:159, LIMIT 5), the refusal set in _confirm_scope_gone_in_transaction (current scope, unobservable current scope, an unexpired lease of another scope, a two-scope task, a two-part scope whose holder still runs here, a ledger without owner_scope, a held session lock), and the pod_history.jsonl append failure printing WARNING: on stderr while still exiting 0 (recover.py _run_cleanup_confirmation).
• Safety and silent failure. The only new empty except (source_tree_identity.ensure_tree_instance) now carries the comment saying the first linked identifier stands and is read back. A server outliving SIGKILL is reported as server_survived_teardown:<pid> in nonfatal_warnings and in the stop marker instead of the old unconditional "tree reaped". The reaper refuses its own process group and requires a live pid to still look like a serving process.
• Scope and labels. Title, body and the eight commits at head 384eb89d1 agree, including the #1894 regression test carried with d09c9e1df; size-exception and baseline-raise are on the PR as the body states.

Ran: git show / diff comparisons of all five picks and the four #1606 production files against their main originals; head-tree greps for AGENTX_SERVER_PIDFILE, agentx_server.pid, stop_agentx_servers, reap_recorded_servers, free_gpus_for_handoff, nonfatal_warnings, phase_sweep, resumed_from. Local pytest was not used as evidence: this host is Windows and the owner-scope and reaper tests need POSIX symlinks and process groups. CI at head 384eb89d1 is green on all 12 test shards, both coverage jobs, code-metrics, pr-hygiene, ruff, pylint, bandit, CodeQL, import-linter, mypy, REUSE and gitleaks (the two cancelled entries are superseded duplicate runs of jobs that also report success). Base: 60b57706ed2c12b367873ba90f421f0b4c1f924b | Head: 384eb89d15d03e711ef7b3754bc9689bd78cb489

LGTM

[xiaofei-zheng]

Approving on the review posted above: no blocking issues. All five picks verified byte-identical to their main originals apart from the documented baseline numbers and the one docstring cross-reference; the partial #1606 backport is self-contained and its pidfile writer already exists on this branch; the new docs match the code they describe.