Skip to content

fix(conc-sweep): boot the single-server sweep with the reuse rounds' envs - #1895

Closed
stojanovic-ana wants to merge 1 commit into
AMD-AGI:mainfrom
stojanovic-ana:fix/conc-sweep-boot-envs
Closed

stojanovic-ana wants to merge 1 commit into
AMD-AGI:mainfrom
stojanovic-ana:fix/conc-sweep-boot-envs

Conversation

@stojanovic-ana

Copy link
Copy Markdown
  • Description: The single-server concurrency sweep boots one shared server on its top rung and measures every lower rung as a reuse round against that server. The boot rung passed MAGPIE_RUN_PHASE=server as an extra env; the reuse rounds do not. Magpie past the current pin (e6833b81) keeps every non-client config env as the running server's identity and refuses a reuse round whose envs differ. So once MAGPIE_REF moves, every MI300X/MI355X single-server sweep fails below the boot rung with "Reuse metadata mismatch". Magpie already sets MAGPIE_RUN_PHASE for each subprocess it spawns, so the boot rung never needed to pass it. This drops it, and the now-unused base_extra_envs parameter of _SweepRun.run_rung.
  • Linked issue(s): none. This is a prerequisite for moving MAGPIE_REF past e6833b81, which feat(gpu): support the Radeon AI PRO R9700 (gfx1201) with vLLM #1782 is waiting on.
  • Tests: updated test_single_server_boot_only_is_measured_by_reuse_round to assert that the boot round's envs equal the measured reuse round's envs, instead of pinning {"MAGPIE_RUN_PHASE": "server"}. It fails on main (assert {'MAGPIE_RUN_PHASE': 'server'} == {}) and passes with the fix.
    • pytest src/hyperloom/inference_optimizer/tests/test_conc_sweep.py: 119 passed, 2 skipped (plot tests, matplotlib not installed)
    • pytest on the sweep test files (test_conc_sweep*.py, test_cli_conc_sweep_default.py, test_sweep_phase_auto.py, test_geak_sweep_units.py, test_sbd_v6_conc_sweep_*.py): 385 passed, 2 skipped
    • ruff check and ruff format --check (0.16.2) on both changed files: clean
  • Size/complexity triggers crossed: none. Three lines removed, one assertion changed.
  • If this simplifies or refactors: removes the base_extra_envs parameter of _SweepRun.run_rung; only the boot rung ever passed it. The contract that a single-server sweep boots one server and measures every rung against it is unchanged, and the test above pins it.
  • Observable effect: none on the current pin, because Magpie sets MAGPIE_RUN_PHASE for each subprocess itself. After the Magpie bump, MI300X/MI355X single-server sweeps measure every rung instead of failing everything below the boot rung with "Reuse metadata mismatch".
  • Breaking changes: no. _SweepRun.run_rung is private to conc_sweep.py.
  • PR addresses single concern: yes.
  • Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: no. Magpie's reuse identity check is intended; the stray env came from Hyperloom.

…envs

The single-server sweep booted its shared server with
MAGPIE_RUN_PHASE=server and then measured every lower rung as a reuse
round without it. Magpie past the current pin (e6833b81) keeps every
non-client config env as the server's identity and refuses a reuse
round whose envs differ, so once MAGPIE_REF moves, every MI300X/MI355X
single-server sweep fails below the boot rung with "Reuse metadata
mismatch".

Magpie already sets MAGPIE_RUN_PHASE for each subprocess it spawns, so
the boot rung has no reason to pass it. Drop it, and the now-unused
base_extra_envs parameter of _SweepRun.run_rung.
@stojanovic-ana
stojanovic-ana requested a review from a team as a code owner October 14, 2026 15:13
@mehdi-saeedi
mehdi-saeedi requested a review from chaojhou October 15, 2026 01:04

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1895 -- fix(conc-sweep): boot the single-server sweep with the reuse rounds' envs

What it does: The single-server concurrency sweep booted its shared server with an extra MAGPIE_RUN_PHASE=server env that the reuse rounds did not carry; Magpie past e6833b81 treats non-client envs as the server's identity, so every reuse rung would fail with "Reuse metadata mismatch". The PR drops that env from the boot call and the now-unused base_extra_envs parameter of _SweepRun.run_rung (src/hyperloom/orchestrator/kernel/conc_sweep.py).

Blocking issues: 1

  1. [V5] CI has not run on this head [verified]
    Problem: At head 657c7fe only ReadTheDocs has reported. Tests with Coverage, Lint, Code metrics, CodeQL, PR Hygiene, Packaging, REUSE, Secret Scan, Docs and both E2E workflows are action_required (fork workflow approval pending), while the diff changes src/hyperloom/orchestrator/kernel/conc_sweep.py:580.
    Impact: Nothing has executed the changed sweep path or the edited test; the required checks cannot pass until the runs are approved.
    Action: A maintainer approves the workflow runs for this fork; merge only once the matrix is green.

SKIPPED: V4 -- openprs.txt has # PARTIAL: lines for #1869, #1675, #1276, #1154 and #1032; overlap past their first 100 files was not checked.
SKIPPED: pytest -- not run locally; relies on the CI matrix once approved.

Checked: conc_sweep.py run_rung and the single-server boot/reuse calls; _grid_runner.py _build_variant_yaml base_extra_envs merge; _server_lifecycle.py resolve_lifecycle_params; Magpie benchmarker.py at e6833b81 and main (reuse metadata, client-only env keys); tree-wide grep at head for every writer and reader of MAGPIE_RUN_PHASE and base_extra_envs and every caller of run_rung | Ran: git grep at head; gh api AMD-AGI/Magpie contents; gh api actions/runs for the head | Base: a56028d | Head: 657c7fe

@github-actions

Copy link
Copy Markdown

Gate job conclusion: failure (run) on 657c7fe3c34cca46bad9071326749b9ae64744c4, as reported by GitHub.

The report below was uploaded by that run; the conclusion above is the result.


Code metrics gate: FAILED

Judged: the 2 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 73 +0
Cognitive complexity fails > 30 0 0 0 289 +0
Function length (lines) fails > 80 0 0 1 453 +0
Nested block depth fails > 5 0 0 0 27 +0
Module length (lines) warns > 800, fails > 1200 0 0 1 53 +0
Duplicated lines per file fails > 0 0 0 0 120 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Check Findings Effect
Comments 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0

Baseline out of date (run python scripts/code_metrics.py --update-baseline): 2

Baseline entry Dimension Baseline Now
src/hyperloom/orchestrator/kernel/conc_sweep.py _sweep_one_arm_single_server Function length (lines) 335 334
src/hyperloom/orchestrator/kernel/conc_sweep.py Module length (lines) 1429 1426

How to fix

  • New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
  • Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
  • Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
  • Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).
89 findings in files this change does not touch

Outside this change (informational): 89

Unit Dimension Baseline Now
src/hyperloom/orchestrator/enablement/recipe/projections.py project_launch_evidence Cognitive complexity absent 36
src/hyperloom/orchestrator/state/shared_state.py resolve_graded_comparison Cognitive complexity absent 34
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/hyperloom_kb/schema.py FieldDeclaration.__post_init__ Cognitive complexity absent 32
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver Cognitive complexity absent 36
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start Cognitive complexity absent 40
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom_kb/tests/test_cli.py Duplicated lines per file absent 18
src/hyperloom_kb/tests/test_global_identity_and_pull.py Duplicated lines per file absent 13
src/hyperloom_kb/tests/test_local_state.py Duplicated lines per file absent 18
src/hyperloom/inference_optimizer/cli/__init__.py _reexport_session_environment Function length (lines) absent 112
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_agent_lineup Function length (lines) absent 89
src/hyperloom/inference_optimizer/cli/__init__.py _resolve_run_topology Function length (lines) absent 92
src/hyperloom/inference_optimizer/cli/__init__.py _resume_session Function length (lines) absent 171
src/hyperloom/inference_optimizer/cli/__init__.py _seed_fresh_session Function length (lines) absent 183
src/hyperloom/inference_optimizer/multi_node/cli.py _restart_multinode Function length (lines) absent 107
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser Function length (lines) absent 84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors Function length (lines) absent 90
src/hyperloom_kb/http_service.py RequestHandler._dispatch Function length (lines) absent 91
src/hyperloom_kb/sync.py GlobalSync.pull Function length (lines) absent 86
src/kernelforge/cli/forge_loop_command.py _RunReporter.build_result Function length (lines) absent 91
src/kernelforge/cli/forge_loop_command.py _prepare_measurement_driver Function length (lines) absent 140
src/kernelforge/cli/forge_loop_command.py _read_kb_warm_start Function length (lines) absent 154
src/kernelforge/loop/knowledge_integration.py _measure_warmstart_field Function length (lines) absent 83

...and 64 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

@chaojhou

Copy link
Copy Markdown
Collaborator

Closing as superseded by #1892, which removes the single-server boot-and-reuse path this PR fixes: every rung boots its own server there, so the MAGPIE_RUN_PHASE=server boot env and the reuse-metadata mismatch no longer arise. Thanks for tracking down the Magpie reuse-identity change.

@chaojhou chaojhou closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants