Repository navigation
chore(release): merge release/v1.1.4 into main - #1955
Merged
Merged
Conversation
… and round server teardown (partial #1606) (#1874) * feat(recover): let an operator confirm a vanished execution scope (#1859) Cherry-pick of #1859 onto release/v1.1.4, as the prerequisite of #1871 (which changes the owner-scope code this adds) and of #1870's regression test (which cancels the sweep through confirm_scope_gone). Applied cleanly. Adds `hyperloom recover --confirm-scope-gone OWNER_SCOPE --confirmation-reason TEXT`: under the session lock and in one transaction it cancels the unfinished tasks holding a lease in that scope and releases their execution and GPU leases, and refuses the current scope, an unobservable current scope, any unexpired lease of another scope, and tasks spanning two scopes. local_owner_scope moves unchanged to hyperloom.common.owner_scope. (cherry picked from commit 680b374) Co-Authored-By: Claude <noreply@anthropic.com> * fix(sweep): re-dispatch the conc_sweep when a resume finds SWEEP without one (#1870) Cherry-pick of #1870 onto release/v1.1.4. `hyperloom recover` cancels the tasks of a vanished execution scope directly in the task ledger, so a conc_sweep cancelled that way never records last_conc_sweep. A resume does not re-enter the current phase, so a session resumed in SWEEP had no sweep in flight and no outcome to exit on: it idled until the grant the lost sweep held ran out, and the sweep was never measured. Before a resumed process's first phase decision, SWEEP with no recorded conc_sweep outcome now re-fires its entry hook, once. Conflicts resolved: - src/hyperloom/orchestrator/phases/machine.py: main's context next to the new _release_holds_no_task_will_settle helper includes _drain_for_transition, which landed on main after the release cut. Only the new helper is taken; _drain_for_transition is not brought to the release. - scripts/code_metrics_baseline.txt: the release's baseline for MachinePhase.advance_phase_if_needed is 192 (main's was 156). The pick shortens the function by 4 lines on this branch too, so the entry is 188. (cherry picked from commit c831742) Co-Authored-By: Claude <noreply@anthropic.com> * fix(owner-scope): tell apart two containers that repeat one boot and PID namespace (#1871) Cherry-pick of #1871 onto release/v1.1.4. The owner scope recorded on leases was `<boot_id>:<pid-namespace inode>`; a container started on a node after another one exited can repeat that inode, so `recover --confirm-scope-gone` refused the lost container's scope as the current environment. The scope now adds the start time of the namespace's PID 1; two-part scopes of this boot and namespace are still treated as this environment by the resume guard and reaper, and confirming one checks that no holder that started before its lease still runs here. Conflict resolved: - src/hyperloom/inference_optimizer/session/resume_guard.py: on main, #1840 split _inspect into _scan_leases/_scan_gpu_leases/_scan_tasks/ _scan_open_rounds before #1871 changed one line in _scan_leases. The release keeps the unsplit _inspect; the same one-line change (match the lease's scope with same_environment) is applied there. The refactor is not brought to the release. Every other file applied cleanly. (cherry picked from commit 25ea91f) Co-Authored-By: Claude <noreply@anthropic.com> * docs(recover): document --confirm-scope-gone in the resume runbook Scenario A of the disaster-recovery runbook still told the operator only to relaunch with --resume-from and described recover only as a breakdown rebuild, while a resume after a lost pod is refused until the vanished owner scope is confirmed (#1859). - operations.md Scenario A: a step for a blocked resume, with the recover --confirm-scope-gone command, what it does and does not do, what it refuses, and when --confirm-stopped applies instead. - operations.md: the crash-recovery lifecycle row, the recover exit codes and the --resume-from / recover summary line. - operator-scripts.md: the recover entry. - recover subcommand help: names the cleanup confirmations. Docs and help text only; no behaviour change. Co-Authored-By: Claude <noreply@anthropic.com> * Discharge a pending integrate whose source tree went with its execution environment A node loss during an integrate left pending_integrate at phase "ready" with the operator's stash recorded in a tree that lived in the dead container. On resume in a new container the task carried the operator's stopped confirmation with an empty result, so recovery held the sentinel as an unknown outcome, and a rollback would have refused anyway because the stash is absent from the new checkout: every resume stopped with patch_recovery_incomplete. - An integrate attempt now records an identifier in the per-worktree git directory of the tree it mutates (tree_instance). On resume, a different identifier at the same path means another checkout: the tree-side obligations (reset the candidate out, hand the stash back) went with the old tree, in every phase short of restored or accepted. The sentinel is marked restored with tree_replaced, artifact targets outside the tree are still restored from their preimages, and the stash that was not restored is reported. A matching identifier keeps today's behaviour, so a stash dropped or pruned from the same tree still refuses. - A task whose last outcome is an operator confirmation that it stopped, with no kept result on the bus and no stack row, now proceeds to rollback instead of being held as an unknown outcome. An accepted window is still held. - A sentinel written before identifiers existed is judged replaced only when the task was confirmed stopped, the recorded stash object is not in the tree's object store at all, and nothing of the candidate is in the tree: every patch still applies forward cleanly (an applied candidate, or one edited over by hand, keeps the refusal) and every artifact target inside the tree holds its recorded preimage. (cherry picked from commit fd2ab7a) Co-Authored-By: Claude <noreply@anthropic.com> * Replay an accepted integrate that changed only its environment on resume Resume settled a crashed integrate_patch window from its task row only when the window carried a patch or an artifact. A candidate that changed only environment variables or server arguments has neither, so a resume after its KEEP was recorded on the task row, but before it reached the bus, cleared the sentinel as stale and the accepted candidate never reached the stack. Every window now has its task row read. A confirmed KEEP in the accepted phase is replayed into the stack, the same as for a patch candidate; a window with nothing in the tree is otherwise cleared as before, since there is nothing to roll back or hold. (cherry picked from commit a436bd4) Co-Authored-By: Claude <noreply@anthropic.com> * fix(sweep): stop the server a timed-out round's client booted (partial backport of #1606) In a concurrency sweep whose rounds each boot their own server through an AgentX client (aiperf_client.sh / mlperf_agentic_client.sh), a round that hits the benchmark cap is killed as magpie_timeout. The client dies, but the server it booted leads its own session and is re-parented away from the client, so reaping the round's process tree never reaches it. The leftover server keeps the port and nearly all GPU memory: the next rounds either measure against it (their own server fails to bind) or fail at engine start for lack of memory, until the sweep ends. The stop marker still said "tree reaped". This is a partial backport of #1606 (cherry picked from commit c9818d5, server-teardown part only): - agentx_workload: AGENTX_SERVER_PIDFILE, the pidfile both client scripts already write. - _grid_runner: the variant teardown runs after every round (not only auto-warmup rounds), including a multi-node warm-up stop, and also stops the server agentx_server.pid names; a stopped round's record names any server that outlived SIGKILL. - _server_lifecycle: stop_agentx_servers / reap_recorded_servers; the reaper treats a zombie leader as gone while still reaching live ranks in its group, refuses to signal its own process group, confirms the SIGKILL and keeps a survivor's pidfile. Not backported from #1606: the GEAK metric axis, the GEAK hand-off GPU wait (free_gpus_for_handoff and its rocm_smi helper), the extend-budget cap, the GEAK budget reservation, the baseline executor's cleanup call and the code-metrics baseline update. Those are behaviour changes or features, not this bug. Tests: #1606's TestRoundServerTeardown and orphan-reaper tests, plus the timed-out-round regression test from #1894 (test_timed_out_round_stops_its_server_before_the_next_round, run for a sglang and a vllm server cmdline). The new test fails on the parent of this commit (the server is still alive when the next round starts) and passes with it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(baseline): stop the server a baseline round's AgentX client booted (partial backport of #1606) An AgentX client boots its server setsid'd, outside the round's process tree, and records it in agentx_server.pid. On a single node the baseline executor runs that client too (MAGPIE_RUN_PHASE stays full), and its single-round cleanup never read the pidfile, so a baseline round killed at its cap or cancelled left its server holding the port and the GPUs until the next session-wide reap. The single-round cleanup now calls _lifecycle.stop_agentx_servers on the round's output directory, the same call #1606 adds on main. A test runs a single baseline round whose fake client leaves a setsid'd server behind and checks it is stopped after both a completed and a cancelled round; it fails without the fix. Also explain the empty FileExistsError branch in ensure_tree_instance (CodeQL "empty except"): the identifier another creator linked first stands and is read back. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: chaojhou <chaojhou@amd.com> Co-authored-by: Claude <noreply@anthropic.com>
* chore(release): cut 1.1.4 Every site that tracks the packaged version moves together: pyproject's [project].version, the docs version_number kept in sync with it, the README badge, the compatibility matrix row, and the three copy-paste `pip install hyperloom-inference-optimizer==` pins, which would otherwise keep installing 1.1.3 after 1.1.4 ships. HYPERLOOM_WHEEL_TAG moves with it, which requires the v1.1.4 GitHub release to carry the rocm-profiler-hotfix-libs.tar.gz asset the bare-metal installer downloads from that tag. The Unreleased section of the release notes becomes the 1.1.4 section, and an empty Unreleased heading is kept for the next cycle. Co-Authored-By: Claude <noreply@anthropic.com> * docs(release-notes): complete the 1.1.4 section The section carried over the Unreleased entries verbatim, and those missed changes that ship in 1.1.4 and that an operator upgrading from 1.1.3 will hit: - the single `hyperloom <command>` entry point and the old entry points, flags and critic subcommands it removed (#1736); - the removal of in-process specialist dispatch and its two flags, and the installer owning the one claude CLI (#1786); - kernelforge no longer reading its historical workspace formats, and the `--op-name` alias and old entry-point group it dropped (#1768); - four of the variables #1739 made fail on a malformed value were missing from the strict-parsing list: INFERENCE_OPTIMIZER_CYCLE_RELOOP_MIN_REMAINING_SEC, HYPERLOOM_GEMM_PAIRED_PAIRS, MAX_MODEL_LEN and GPU_MEMORY_UTILIZATION. Co-Authored-By: Claude <noreply@anthropic.com> * docs(release-notes): cover every operator-visible change in 1.1.4 The 1.1.4 section listed six of the 98 operator-visible changes merged between v1.1.3 and the release branch. Add a "Before upgrading from 1.1.3" section for the removed options, module paths and environment variables, the moved defaults, the installation prerequisites and the persisted-state changes, and add Added, Changed, Removed and Fixed entries for the rest. 1.1.4 is no longer described as a patch release. The fixes for the AgentX agentic-recipe path (#1696, #1749) are left out: the path itself is removed in the same release (#1784). Allow the release notes to name the retired kernel_agents.agent_providers entry-point group and KERNEL_AGENTS_MAX_TURNS in the rename-completeness test, as they already may name the retired kernel-agents console script. Co-Authored-By: Claude <noreply@anthropic.com> * docs(release-notes): correct six 1.1.4 entries against the code - GLM-5.3 corpus: the old corpus stays usable through AGENTX_CANONICAL_DATASET; pinning it with AGENTX_DATASET stamps every result submission_valid=false. - Workload knobs: state that --extra-env pins are now exported under their own names, that a shell PRECISION still overrides a resumed session, and that 1.1.3 already read MAX_MODEL_LEN. - Resume still type-checks crash_count / crash_timestamps; drop that claim. - optimization_journal.json outcomes: name the literals it writes (KEEP, REVERT, no_promote, recorded). - Roofline: the profile runs once and logs one "roofline profile failed" warning. Also quote the real UPDATE_STATE refusal, note that the MLPerf backend measures Kimi-K3 only, scope the warm-replay flag removal to the flag, and write placeholders as <name> like the rest of the page. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Zeng <lzeng@amd.com> Co-authored-by: Claude <noreply@anthropic.com>
Code metrics gate: PASSEDJudged: the 10 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
93 findings in files this change does not touchOutside this change (informational): 93
...and 68 more; the job log lists every row.
Thresholds and sources: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges
release/v1.1.4back intomainafter the v1.1.4 release, somaincarries the 1.1.4 version bump and release notes and thev1.1.4tag becomes an ancestor ofmain.The release branch has two commits
mainlacks:mainhas not toucheddocs/release-notes.mdsince the cut (60b57706e), so the file equals the released version.main's version. All of those PRs, plus docs: document recover --confirm-scope-gone for a refused resume #1880 (its docs counterpart) and test(grid-runner): a timed-out round stops the server its client booted #1894 (its regression test), are already onmain, andmainhas since reworked the same code (refactor(optimizer): bring the optimizer and scripts under the complexity ceiling #1840), so taking the cherry-picks would re-apply hunksmainalready has.Net diff against
main: the eight version sites move 1.1.3 -> 1.1.4,docs/release-notes.mdgains the 1.1.4 section, andtest_rename_completeness.pygains the allowlist entry those notes need.Please merge with "Create a merge commit", not squash: squashing drops the release branch as a parent, and
v1.1.4stops being reachable frommain.main; CI runs the release and doc version tests. Checked thatgit diff origin/mainis limited to the files above, that no1.1.3version pin remains outside the release notes, and thatv1.1.4is an ancestor of the merge commit.mainreports version 1.1.4 and its install pins andHYPERLOOM_WHEEL_TAGdefault point at the published v1.1.4 release.main.