docs: document recover --confirm-scope-gone for a refused resume

[zoroyihan7]

• Description: Documents hyperloom recover --confirm-scope-gone in the operator docs and in the recover help text. docs/reference/operations.md Scenario A told an operator whose pod was OOM-killed to relaunch with --resume-from and described recover only as a breakdown rebuild, but a resume in a new pod is refused until the lost pod's owner scope is confirmed gone. Scenario A now covers the refused resume: what it prints, when to run hyperloom recover --session-dir S --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT (a whole execution environment that no longer exists) and, when the relaunch is still refused after that, --confirm-stopped TASK_ID for each listed task that names no owner scope (such as a lane-less CLOSE-phase step), what the confirmation changes, and each case in which it refuses. The crash-recovery lifecycle row, the recover exit codes, operator-scripts.md, the top-level hyperloom command summary and the recover subcommand help/description say the same. The recover help and description strings live as constants in cli/recover.py, beside the other recover options, so _build_parser does not grow (the code-metrics gate fails any growth of it). The refusal message ensure_resume_safe prints (session/resume_guard.py) now gives the same order: the scope confirmation first, then --confirm-stopped for a listed task that names no owner scope. This is the main-branch counterpart of the documentation change requested on release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879) and round server teardown (partial #1606) #1874 (release/v1.1.4).
• Linked issue(s): follow-up to review on release/v1.1.4: cherry-pick resume fixes (#1859, #1870, #1871, #1879) and round server teardown (partial #1606) #1874
• Tests: docs, help text and the resume-refusal message. Added test_lane_less_close_task_needs_confirm_stopped_after_the_scope_is_confirmed in test_resume_scope_gone.py, which follows the runbook path (confirm the scope, relaunch still refused by a lane-less CLOSE task, --confirm-stopped it, relaunch admitted). Ran test_hyperloom_cli.py, test_resume_scope_gone.py, test_setup_cli.py and the recover/resume tests (282 passed), scripts/check_cli_references.py (clean), ruff check / ruff format --check at the pinned v0.16.2, and checked python -m hyperloom recover --help.
• Size/complexity triggers crossed: none; docs and help text plus the refusal message and one test; _build_parser keeps its length.
• If this simplifies or refactors: n/a
• Observable effect: an operator following the disaster-recovery runbook after a lost pod now finds the step that unblocks the refused resume, hyperloom recover --help explains the confirmation mode and exit codes, and the refusal message a refused --resume-from prints now names the scope confirmation first and sends a listed task that names no owner scope to --confirm-stopped. Before, the runbook ended at the refusal and the message pointed only at --confirm-stopped for legacy empty-scope ownership.
• Breaking changes: no
• PR addresses single concern: yes
• Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: n/a

🤖 Generated with Claude Code

[github-actions]

Code metrics gate: PASSED

Judged: the 9 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Worse	Baseline out of date	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	0	0	72	+0
Cognitive complexity	fails > 30	0	0	0	284	+0
Function length (lines)	fails > 80	0	0	0	449	+0
Nested block depth	fails > 5	0	0	0	27	+0
Module length (lines)	warns > 800, fails > 1200	0	0	0	53	+0
Duplicated lines per file	fails > 0	0	0	0	118	+0
Dead code (vulture)	fails > 0	0	0	0	2	+0

Check	Findings	Effect
Comments	0
English only	0
Production code imports test code	0
Repeated literals	0
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	0

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1

Module	Lines	Limit
src/hyperloom/inference_optimizer/cli/parser.py	1192	warns > 800

Already over at the base (backlog, not this change's): 1

Unit	Dimension	Base	Now
src/hyperloom/inference_optimizer/cli/parser.py:176 _build_parser	Function length (lines)	1017	1017

96 findings in files this change does not touch

Outside this change (informational): 96

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/enablement/recipe/projections.py project_launch_evidence	Cognitive complexity	absent	36
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison	Cognitive complexity	absent	34
src/hyperloom_kb/collect/mapping.py _experience	Cognitive complexity	absent	31
src/hyperloom_kb/schema.py FieldDeclaration.__post_init__	Cognitive complexity	absent	32
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver	Cognitive complexity	absent	36
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start	Cognitive complexity	absent	40
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate	Cognitive complexity	absent	44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt	Cognitive complexity	absent	38
src/hyperloom_kb/tests/test_cli.py	Duplicated lines per file	absent	18
src/hyperloom_kb/tests/test_global_identity_and_pull.py	Duplicated lines per file	absent	13
src/hyperloom_kb/tests/test_local_state.py	Duplicated lines per file	absent	18
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment	Function length (lines)	absent	112
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup	Function length (lines)	absent	89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology	Function length (lines)	absent	92
src/hyperloom/inference_optimizer/cli/__init__.py _resume_session	Function length (lines)	absent	171
src/hyperloom/inference_optimizer/cli/__init__.py _seed_fresh_session	Function length (lines)	absent	183
src/hyperloom/inference_optimizer/multi_node/cli.py _restart_multinode	Function length (lines)	absent	107
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser	Function length (lines)	absent	84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors	Function length (lines)	absent	90
src/hyperloom_kb/http_service.py RequestHandler._dispatch	Function length (lines)	absent	91
src/hyperloom_kb/sync.py GlobalSync.pull	Function length (lines)	absent	86
src/kernelforge/cli/forge_loop_command.py _RunReporter.build_result	Function length (lines)	absent	91
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver	Function length (lines)	absent	140
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start	Function length (lines)	absent	154
src/kernelforge/loop/knowledge_integration.py _measure_warmstart_field	Function length (lines)	absent	83

...and 71 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

[chaojhou]

PR #1880 -- docs: document recover --confirm-scope-gone for a refused resume

What it does: Since #1859, a --resume-from in a new pod is refused (exit 2) until the lost pod's owner scope is confirmed gone with hyperloom recover --confirm-scope-gone, yet the disaster-recovery runbook ended at that refusal and the docs and help described recover only as a breakdown rebuild. This PR adds the step to operations.md Scenario A (what the refusal prints, the command, what the confirmation changes, when it refuses, and when --confirm-stopped applies instead), updates the crash-recovery lifecycle row, the recover exit codes and operator-scripts.md, and extends the top-level and recover subcommand help, keeping the subcommand help and description as constants in cli/recover.py so _build_parser does not grow.

Blocking issues: 1

1. [X3] The optimization-loop doc still says recover only reconstructs artifacts [verified]
   Problem: this PR corrects the recover contract in the Scenario A summary ("recover = rebuild the breakdown artifact, or, with --confirm-scope-gone / --confirm-stopped, record a cleanup confirmation that lets a refused resume proceed"), in operator-scripts.md:190-196 and in the help, but leaves its duplicate at docs/conceptual/optimization-loop.md:498-499 -- "recover only reconstructs artifacts offline" -- unchanged. At this head, recover with either confirmation option cancels the unfinished tasks and releases their execution and GPU leases instead of rebuilding anything (cli/recover.py:146-150 -> session/resume_guard.py:380-405). docs/reference/operations.md:330
   Impact: after merge the docs disagree about what recover does. A reader of the optimization-loop page, which AGENTS.md lists as the doc for the loop, is told recover cannot help with the refused resume this PR documents.
   Action: reword optimization-loop.md:498-499 to say recover reconstructs artifacts offline or, with --confirm-scope-gone / --confirm-stopped, records the cleanup confirmation a refused resume asks for, and link Scenario A.

SKIPPED: V4 -- PARTIAL overlap for #1032, #1154, #1276, #1675, #1869 (each cut at its first 100 files), overlap there unchecked

Checked: every operational claim in the added docs and help against the head code -- cli/recover.py (_run_cleanup_confirmation, _run_recover_session), session/resume_guard.py (ensure_resume_safe, _other_scopes, _label, confirm_scope_gone, _confirm_scope_gone_in_transaction, _confirm_task_stopped_in_transaction, _task_leases, _holder_runs_here, _in_locked_ledger), session/lock.py (SessionLock.acquire), common/owner_scope.py, cli/__init__.py:2060-2070 and main, cli.py, parser.py:152-163 and :1166-1183; the four places the #1874 card named (operations.md:144, :214-215, Scenario A, the recover subcommand help), now all correct; commit 3 as a pure move of the commit-1 help strings; tree-wide sweeps at head for RECOVER_HELP / RECOVER_DESCRIPTION, the replaced help literals, the three confirmation flags and every recover mention in docs/**, **/SKILL.md and prompts; #1874's docs commit against this head | Ran: git show / git grep sweeps at base and head; git diff of the #1874 and #1880 docs commits; gh run list --commit 1b03a28a (latest run per workflow: 10 success, 2 label-gated E2E skipped); gh pr view 1880 --json mergeable (MERGEABLE); claims traced statically (no direct call or c901.py run on this host; the diff adds no def or branch and the code-metrics job reports cyclomatic +0) | Base: d755931 | Head: 1b03a28

[zoroyihan7]

Thanks for the review. Addressed in 9f4ffa4.

1. [X3] optimization-loop.md still says recover only reconstructs artifacts -- fixed.

• docs/conceptual/optimization-loop.md:498-501 now reads: "recover reconstructs artifacts offline or, with --confirm-scope-gone / --confirm-stopped, records the cleanup confirmation a refused resume asks for (see Scenario A)."
• The same stale claim was also in src/hyperloom/inference_optimizer/SKILL.md:871-872 ("recover remains an offline artifact-reconstruction tool"), so it is corrected there too. That text names --confirm-scope-gone for a lost owner scope and reserves --confirm-stopped for ownership recorded without a scope, matching ensure_resume_safe / _confirm_task_stopped_in_transaction, and points at Scenario A.
• A tree-wide sweep of docs/**, **/SKILL.md and prompts finds no other place that says recover only rebuilds artifacts. scripts/check_cli_references.py passes, and the Sphinx and Read the Docs builds resolve the anchor.

CI at 9f4ffa4: every lane passes (label-gated E2E skipped).

[chaojhou]

PR #1880 -- docs: document recover --confirm-scope-gone for a refused resume

What it does: A --resume-from in a new pod is refused (exit 2) until the lost pod's ownership is confirmed. This PR documents that step. In operations.md Scenario A it covers the refusal output, hyperloom recover --confirm-scope-gone, what the confirmation changes, when it refuses, and when --confirm-stopped applies. It also updates the lifecycle row, the recover exit codes, operator-scripts.md, optimization-loop.md, SKILL.md, and the top-level and recover help. The help strings are kept as constants in cli/recover.py. The previous card's item (optimization-loop.md still saying recover only reconstructs artifacts) is fixed at this head, along with the same stale line in SKILL.md.

Blocking issues: 1

1. [free:confirm-stopped-scope] The runbook says scope-less ownership comes only from earlier versions, but the current code creates running tasks with no lease [verified]
   Problem: Scenario A says to confirm "once for the whole scope, then relaunch" (docs/reference/operations.md:279). It limits --confirm-stopped to ownership recorded without an owner scope "by earlier versions" (:313). At head, CLOSE-phase tasks are created with requires_lanes=[] (phases/close.py:594, :923), and the dispatcher takes a lease only when task.requires_lanes is non-empty (loop/dispatcher.py:847), yet the task is still moved to running. A pod lost during CLOSE therefore leaves a running task that _scan_tasks blocks with holder unknown and no scope. --confirm-scope-gone cannot reach it, because _confirm_scope_gone_in_transaction selects targets only from leases under the named scope. --confirm-stopped does work for it. docs/reference/operations.md:313
   Impact: an operator whose pod was lost during CLOSE follows the runbook: confirms the scope (exit 0), relaunches, and is refused again. The runbook then tells them the remaining tool is only for records from earlier versions.
   Action: describe --confirm-stopped by what the operator sees rather than by version: use it for each blocked task the refusal lists without an owner scope. Drop "by earlier versions". Add one sentence to Scenario A: if the relaunch is still refused after the scope confirmation for tasks that name no scope, confirm each with --confirm-stopped.

SKIPPED: V4 -- open PRs #1032, #1154, #1276, #1675, #1869 were compared on their first 100 files only

Checked: every added sentence against cli/recover.py (_run_cleanup_confirmation, _run_recover_session, RECOVER_HELP, RECOVER_DESCRIPTION), session/resume_guard.py (ensure_resume_safe message, _scan_tasks, _other_scopes, _label, _confirm_scope_gone_in_transaction, _confirm_task_stopped_in_transaction, _task_leases, _in_locked_ledger), session/lock.py, common/owner_scope.py, loop/dispatcher.py, phases/close.py, sub_agent_runner.py, resource_lock.py, cli/init.py exit handling; tree-wide sweep of docs/**, **/SKILL.md and prompts for recover mentions | Ran: git diff 1b03a28a8 9f4ffa47e; git show/git grep at base and head; two independent refutation passes per finding; confirm_scope_gone traced statically (it needs fcntl, unavailable on this host) | Base: d755931 | Head: 9f4ffa4

[zoroyihan7]

Thanks for the review. Addressed in cbca632.

1. [free:confirm-stopped-scope] the runbook says scope-less ownership comes only from earlier versions -- fixed.

• Confirmed the shape: CLOSE-phase tasks are created with requires_lanes=[] (phases/close.py:594, :923), so a pod lost during CLOSE leaves a running task with no lease; _scan_tasks blocks it as lane='unknown' holder='unknown', _confirm_scope_gone_in_transaction never selects it, and --confirm-stopped releases it.
• docs/reference/operations.md Scenario A: dropped "by earlier versions" and added the follow-up step: "If the relaunch is still refused after the scope confirmation for tasks that name no owner scope, confirm each of them with --confirm-stopped TASK_ID, then relaunch. Such a task holds no ownership record under any owner scope, so --confirm-scope-gone releases nothing for it; a CLOSE-phase step, for example, runs without a lane and so is recorded without one."
• One thing I added beyond the card: the refusal prints owner scopes only on its other owner scopes: line, not per task, so an operator cannot read a task's scope off its line (and the absence of that line does not prove the rest are scope-less: the current scope is excluded from it). The paragraph therefore says to let --confirm-stopped check: it refuses, with exit code 2 and nothing changed, a task recorded under a nonempty owner scope (_task_leases with an empty target scope).
• Same wording swept into docs/reference/operator-scripts.md, src/hyperloom/inference_optimizer/SKILL.md and RECOVER_DESCRIPTION in cli/recover.py ("one blocked task that names no owner scope"). The --confirm-stopped option help already described it by scope, not version, and is unchanged. The "ownership database predates owner scopes" refusal bullet stays: that case really is version-based.
• The refusal message itself (ensure_resume_safe) said "For legacy empty-scope ownership only". It now gives the scope step first, then "For a listed task that names no owner scope (such as a lane-less CLOSE-phase step) ... --confirm-stopped ...; it refuses, changing nothing, a task recorded under an owner scope." The confirm_task_stopped docstring no longer calls it a legacy task.
• New test test_lane_less_close_task_needs_confirm_stopped_after_the_scope_is_confirmed (tests/test_resume_scope_gone.py) runs the runbook path: a lost scope with an explore task plus a lane-less running report task; confirm_scope_gone cancels only the explore task; the resume is still refused listing task='report-1' lane='unknown' holder='unknown' and pointing at --confirm-stopped; confirm_task_stopped then lets the resume through. It fails at 9f4ffa4 (the message still says "legacy"), and fails under a mutant where _scan_tasks stops blocking lane-less running tasks.

Ran: the recover/resume test files (282 passed), ruff check / ruff format --check, scripts/check_cli_references.py.

CI at cbca632: every lane passes (33 pass; label-gated E2E and the other conditional lanes skipped).

[chaojhou]

PR #1880 -- docs: document recover --confirm-scope-gone for a refused resume

What it does: A --resume-from in a new pod is refused (exit 2) until the lost pod's ownership is confirmed. This PR documents that step in operations.md Scenario A, the lifecycle row, the recover exit codes, operator-scripts.md, optimization-loop.md, SKILL.md and the recover help, and -- since cbca632 -- rewrites the refusal message ensure_resume_safe prints so it gives the scope step first and then --confirm-stopped for a listed task that names no owner scope (such as a lane-less CLOSE-phase step), with a test that runs that runbook path. The previous card's item is fixed at this head: --confirm-stopped is no longer described as being for "earlier versions", and Scenario A now says what to do when the relaunch is still refused after the scope confirmation.

Blocking issues: 1

1. [X2] The description still says the PR is docs and help text only, with no behaviour change, but the head changes the runtime refusal message [verified]
   Problem: since cbca632 the PR changes ensure_resume_safe's refusal text (src/hyperloom/inference_optimizer/session/resume_guard.py:228-234), which every operator whose resume is refused reads, and adds test_lane_less_close_task_needs_confirm_stopped_after_the_scope_is_confirmed to test_resume_scope_gone.py. The description was not updated: its Tests line says "docs and help text only, no behaviour change", the size line says "about 60 lines of docs and help text", the Description still calls --confirm-stopped TASK_ID "one task recorded without an owner scope" -- the framing this commit replaced -- and the Observable effect line mentions only the runbook and recover --help. AGENTS.md: "Every change states its observable effect in its own PR description." src/hyperloom/inference_optimizer/session/resume_guard.py:229
   Impact: the release notes are built from PR descriptions, so the changed refusal message -- the text an operator sees at the moment the runbook exists for -- would ship unannounced, and a reader of the description is told nothing at runtime changed.
   Action: update the description: state that the refusal message now names the scope step first and routes a listed task that names no owner scope to --confirm-stopped, describe --confirm-stopped as the head does, drop "no behaviour change" and "docs and help text only", and name the new test.

Checked: the cbca632 delta in operations.md, operator-scripts.md, SKILL.md, cli/recover.py RECOVER_DESCRIPTION and session/resume_guard.py (ensure_resume_safe message, _scan_tasks, _scan_leases, _other_scopes, _label, _task_leases, confirm_task_stopped, _confirm_task_stopped_in_transaction, _confirm_scope_gone_in_transaction); phases/close.py and dispatcher.py for the lane-less CLOSE steps; the new test; the description and the author's reply; tree-wide grep at head for the replaced "legacy empty-scope" text | Ran: git diff 9f4ffa47e cbca632dc; CI at cbca632 (33 pass, 7 label-gated skipped); confirm_scope_gone traced statically (it needs fcntl, unavailable on this host) | Base: d755931 | Head: cbca632

[zoroyihan7]

@chaojhou [X2] fixed — the description now matches the head: it states that the refusal message ensure_resume_safe prints gives the scope confirmation first and sends a listed task that names no owner scope to --confirm-stopped, describes --confirm-stopped that way, drops "docs and help text only, no behaviour change", names test_lane_less_close_task_needs_confirm_stopped_after_the_scope_is_confirmed, and the Observable effect line covers the changed message. No code change at cbca632.

[chaojhou]

PR #1880 -- docs: document recover --confirm-scope-gone for a refused resume

What it does: documents hyperloom recover --confirm-scope-gone in the disaster-recovery runbook, the lifecycle row, the recover exit codes and help, operator-scripts.md, optimization-loop.md and SKILL.md, and changes the refusal message ensure_resume_safe prints so it names the scope confirmation first and sends a listed task that names no owner scope to --confirm-stopped. The previous card's item is fixed: the description now states the changed refusal message, describes --confirm-stopped as the head does, drops "docs and help text only, no behaviour change" and names the new test.

Blocking issues: none

Checked: the description as edited at 2026-10-16T14:35Z against the head diff (session/resume_guard.py message and docstring, test_resume_scope_gone.py, the cli/recover.py help constants); head unchanged since the previous card | Ran: gh api for the description, checks and reviews; diff of resume_guard.py | Base: d755931 | Head: cbca632