[rpoornac]

Follow-up to #1730, which landed the measured-phase GPU power sampler and the assert-only --gpu-power-cap-w / --gpu-perf-level. This PR adds the power budgets split out of #1730 in review. They are opt-in: with neither flag set, KEEP behaviour is unchanged (GEAK replays are now sampled, which only adds a gpu_power.json). The opt-in --apply-gpu-power-settings that was bundled here has been split into its own branch and will come as a separate PR.

--max-power-w and --max-per-gpu-power-w: a power budget is a constrained maximum

The deployment it serves. Data-centre racks and rows are provisioned for average power, with each card left at its full cap as headroom for peaks. The question that deployment asks is "the most throughput whose average draw fits the budget". A per-card cap doesn't answer it: capping every card at the budget throws away the boost headroom the rack was provisioned to keep. Perf-per-watt doesn't answer it either.

How it differs from Perf/Watt. Take the example from the #1730 review, with a 1000 W budget:

• 1200 tok/s at 900 W is the right deployment config. It's the most throughput that fits.
• 800 tok/s at 500 W wins on tok/s/W, but leaves 400 tok/s unused under the same budget.

They answer different questions, so this PR doesn't claim to be a Perf/Watt objective. That belongs in resolve_graded_comparison as an objective, and can follow on the same measurement.

Why the numbers transfer. The session runs at the deployed cap, which it asserts with --gpu-power-cap-w. So candidates are measured under the same DVFS and boost conditions the deployment sees; only the average-power budget is added. Grading below the deployed cap would not transfer, and that is not what this does. A site that caps the cards themselves at 1000 W should amd-smi set the cap and assert it, with no ceiling, exactly as the #1730 review suggested. The docs say so.

Cost. There's no new mechanism. constraint_veto_reason folds latency and power into the veto channel reviewed for #1297, and it applies only to would-be KEEPs. Every KEEP decision point honours it, so a lane reverts its own over-budget change:

• explore, integrate_patch (including the correctness-fix route, which KEEPs a verified fix after a REVERT and so applies the budgets itself), assess_integrate_performance (including the stack_positive_keep window), warm replay, and the lift;
• GEAK replays, which are now sampled from the logs of the servers they boot, so GEAK is not structurally refused.

What is budgeted. Two flags, either or both:

• --max-power-w N is the total for the serving GPUs: the sum of each serving card's measured-phase mean, for example 2800 W for four cards. A rack or row budget is a total, so the flag is one too.
• --max-per-gpu-power-w GPU_ID W, repeated once per card, limits one card by its physical amd-smi index, for a card with less headroom (a hotter slot, a weaker supply). A serving card with no entry is bound only by the total, and a listed card that didn't serve a round isn't judged on it.

Both are enforced twice:

• At launch the session refuses (exit 2) if a listed GPU isn't one it can use (its visible-device mask, or every card amd-smi reports when there is no mask), or if the per-GPU limits add up to more than the total. Either budget is refused, at launch and on resume, when no round could be measured: on multi-node (the sampler reads only its own host), on a scriptable framework (xdit, custom, which run no server for the sampler to follow), with HYPERLOOM_GPU_POWER_SAMPLING=0, or without amd-smi.
• On every round the veto reads each serving card's reading, now carried as gpu_power_by_gpu_w next to gpu_power_avg_w: gpu_power_budget_exceeded when a card is over its own limit, power_budget_exceeded when the total is over, and power_unmeasured when there is no per-card reading. Only the feat(power): measure per-GPU power over the measured phase; assert and record the GPU power settings #1730 sampler reads each card, so the benchmark report's host-wide mean counts as unmeasured (fail closed). A round is also unmeasured when a card that may be serving couldn't be read (a card holding a model with no power reading, where a 0 W or negative socket power counts as none, or a card in scope whose VRAM was never read); such cards are listed as unread_gpus in gpu_power.json, so a missing card can't drop out of the total or escape its own limit.

A baseline that breaks either budget, or has no reading, stops the run with baseline_over_power_budget. The stop reason is in both contract versions, with the digests regenerated. A resume with a different value for either flag is refused, and the prompt gets one line naming the total and each card's limit.

Validation (MI355X, cards 4–7)

• Ceiling (run when the budget was a per-card mean): a 300 W budget stopped a 702 W MiniMax M3 AgentX baseline with baseline_over_power_budget, measured by the feat(power): measure per-GPU power over the measured phase; assert and record the GPU power settings #1730 sampler, before any candidate ran. In an 8-hour session at a 725 W budget, two candidates measured 708 W and 686 W. Both lost on throughput, so they carried no power veto.
• Per-GPU launch checks (MiniMax M3 container, mask 4–7): --max-per-gpu-power-w 0 700 refused with exit 2, naming GPU 0 and the available 4–7; limits of 3 × 700 W against --max-power-w 2000 refused with exit 2 ("add up to 2100 W, more than --max-power-w 2000 W"). Both stopped before any model loaded.
• Per-GPU stop: with --max-power-w 4000 --max-per-gpu-power-w 4 300, the MiniMax M3 AgentX baseline (306 tok/s) stopped with baseline_over_power_budget (gpu_power_budget_exceeded): 2877 W total, inside the 4000 W total, but GPU 4 at 710 W against its 300 W limit (GPU 5 735 W, GPU 6 724 W, GPU 7 708 W, each read by the sampler).

Operator-visible changes

• With neither flag set: nothing changes.
• With --max-power-w N: KEEPs whose serving GPUs draw more than N W in total, or with no per-card reading, are refused, and a baseline over N stops the run. An explore variant refused this way is recorded with the decision gate power_budget (the session breakdown's blocked_by), not latency_budget. The launch is refused where power can't be sampled (multi-node, a scriptable framework, sampling off, no amd-smi).
• With --max-per-gpu-power-w GPU_ID W: the same for that one card; the launch is refused if the limits add up to more than --max-power-w, or name a card the session can't use.

Breaking changes: none.

Code layout and the code-metrics gate

Main's code-metrics ratchet landed while this PR was open, so the constraint code now has its own homes instead of growing over-limit units:

• perf_metric.constraint_fields() copies e2el_mean_ms, gpu_power_avg_w and gpu_power_by_gpu_w in every measurement carrier (explore, integrate_patch, warm replay, rebench, the grid runner, writeback), so no lane can drop one.
• orchestrator/measurement/baseline_budget.py holds the baseline stop (latency first, then power) and its report text.
• cli/constraint_args.py holds --max-latency-ms, --max-power-w and --max-per-gpu-power-w; cli/bootstrap.py holds the launch and resume budget checks.
• gpu_power_sampling_unavailable() lives in common/gpu_power_settings.py, so the CLI does not import the orchestrator for it.

Behaviour is unchanged by the move. Thirteen touched units are smaller than on main, and the baseline is lowered for them.

baseline-raise: three units stay over main. kernel.py and KernelPhase.promote_geak_from_candidate are two lines over: gpu_power_avg_w and gpu_power_by_gpu_w in the tuple of fields a GEAK promotion copies. _grid_runner.py is one line over: the import of constraint_fields. Those lines carry the power reading on these paths. Without them, a GEAK promotion or grid-runner KEEP would have no reading and fail closed under any power budget. The only other way to pass the gate would be deleting unrelated lines.

Tests

• Predicate and verdict: a card over its own limit, the total over, an unlisted card bound only by the total, a listed card that didn't serve, unmeasured rounds (including a 0 W or negative card), and the stack-positive window.
• The per-card reading from a real recorder round, the report fallback leaving it empty, and unread_gpus (a serving card with no power reading or a 0 W one, a masked card missing from amd-smi, unreadable VRAM; an idle card with no reading does not block).
• The unmeasurable-budget refusal: multi-node, a scriptable framework, sampling off, no amd-smi.
• Each lane's real result through its decision point under a budget: explore, integrate_patch, the correctness-fix route (total and per-GPU budget), assess_integrate_performance, warm replay, the GEAK lift.
• GEAK replay phase driving, a replay that raises recording no power, an unreadable replay log warned once, and a driver bug raising with the recorder still closed.
• CLI flags (including bad, negative and repeated GPU ids), the launch check (missing GPU, sum over the total), and both resume conflicts.
• The contract digest.

[chaojhou]

Thanks @rpoornac. The budget argument is well made: an average-power budget is a different question from a per-card cap or from Perf/Watt. With none of the new flags set, nothing changes except that GEAK replays are now sampled.

Requesting changes for one blocker and two must-fix items, all on the opt-in paths, plus a scope point. Line numbers are at 16abb36b5; paths are under src/hyperloom/.

Blocker

1. --apply-gpu-power-settings can leave card-wide settings changed after the session, and doesn't stop two sessions from stacking on the same cards.

• Restore runs only from atexit (inference_optimizer/cli/__init__.py:1367).
  • SIGKILL and OOM kills skip it.
  • SIGTERM is only drained inside the tick loop. The drain is closed before the terminal close sequence (orchestrator/loop/coordinator.py:1072-1089), and nothing handles SIGTERM during bootstrap. A SIGTERM in either window ends the process without running atexit.
  • The usual Slurm time limit sends SIGTERM, which starts a graceful CLOSE (stack revalidation, report), then sends SIGKILL after KillWait. If CLOSE outlasts KillWait, the cards stay at the applied cap.
  • The record is recovered only by a later launch on the same host, with the same ledger directory and --apply. Any other launch only warns (cli/__init__.py:1369-1370).
• The ledger isn't host-scoped. It lives under $HYPERLOOM_RUNTIME_DIR, else $USER_DATA_PATH/runtime, else ./runtime (cli/bootstrap.py:290-295).
  • For several runs on one node, the docs recommend a per-job runtime directory such as runtime-$SLURM_JOB_ID (docs/reference/troubleshooting.md:314-315). Two such sessions don't see each other's locks.
  • Scenario:
    • Session A runs without a mask, so it sets all eight cards.
    • Session B then starts on cards 4–7 with its own ledger, and records A's applied cap as their original.
    • A's exit resets cards 4–7 while B is still running on them.
    • B's exit then "restores" them to A's cap, where they stay.
  • The installer default $USER_DATA_PATH/runtime follows the shared mount (docs/reference/operations.md:87-90). A record names only the local card index (common/gpu_power_settings.py:300), so hosts sharing that mount contend for each other's card indices.
• Without a parseable integer mask, it sets every card on the host (gpu_power_settings.py:57-68, bootstrap.py:342-344).
  • That covers both no mask at all and a UUID ROCR_VISIBLE_DEVICES, and it includes cards the session doesn't serve on.
  • The resident-model refusal is checked once, at launch (bootstrap.py:376), and a card whose VRAM can't be read counts as free (gpu_power_settings.py:186-190).
• Suggested: split the apply path out, as agreed on #1730 (see Scope below). If it stays, at minimum:
  • refuse without a parseable integer mask, and set only the cards in it;
  • keep the ledger at a fixed host-local path that doesn't depend on HYPERLOOM_RUNTIME_DIR or USER_DATA_PATH, and stamp each record with the host (for example, its boot id) so that records from another host are ignored;
  • handle SIGTERM from apply until restore, and restore in a finally rather than only from atexit;
  • document that after a SIGKILL the cards stay at the applied settings until a launch with the flag recovers them, and update inference_optimizer/SKILL.md:748 ("Hyperloom never changes them").

Must-fix

2. --max-power-w is accepted when power can't be measured at all.

• With only the total budget set, the launch check returns early (bootstrap.py:202-203).
• So the flag is accepted:
  • on multi-node sessions;
  • with HYPERLOOM_GPU_POWER_SAMPLING=0;
  • on hosts without amd-smi.
• In the last two cases no recorder is built (orchestrator/actions/executors/_gpu_power.py:304-309).
• Every round is then power_unmeasured. The run spends a full baseline and stops with baseline_over_power_budget (orchestrator/loop/writeback.py:3925), which reads as "over budget" rather than "can't be measured".
• Please refuse both budget flags with exit 2 in these cases, at launch and on resume.

3. A serving card without a reading drops out of the budget, so a partial reading fails open.

• A card counts as serving only when its peak VRAM is at least 10% and it has a power reading (_gpu_power.py:185). Only serving cards are returned (:356-372).
• power_veto_reason sums whatever readings are present and skips the limits of cards that are absent. Only an empty map is power_unmeasured (common/perf_metric.py:361-371).
• Example: TP4 on cards 4–7 with --max-power-w 2000. If card 7 reports socket_power as N/A, or stays under 10% VRAM:
  • the total covers only three cards (about 1500 W), so an over-budget candidate can KEEP;
  • a --max-per-gpu-power-w 7 … limit is never checked.
• Please treat a round as unmeasured when a card in the session's mask was active but has no reading, and record such cards in gpu_power.json. That matches the PR's own fail-closed rule.

Scope

• On #1730, @ZhengGong-amd asked to drop the mutating apply path and --max-power-w from that PR and to take each up separately, and I agreed.
• This PR's description answers the earlier objections: an average-power budget is a different question from a cap, and a power cap is reversible and evicts nothing. But it bundles both pieces again, and no agreement is recorded here.
• Blocker 1 is a concrete case of the "cards are shared" concern behind that request.

I suggest splitting the apply path into its own PR, and landing the budgets first once 2 and 3 are fixed. @ZhengGong-amd, can you confirm the direction?

[rpoornac]

Thanks @chaojhou. Addressed in 60b7d998b:

Scope / blocker 1. Split as you suggested. --apply-gpu-power-settings is out of this PR; it's parked on feat/apply-gpu-power-settings and will come as its own PR with your points addressed there (integer mask required and only those cards set, a host-local ledger stamped with the boot id, SIGTERM handled from apply until restore with restore in a finally, the SIGKILL caveat documented and the SKILL.md row updated). This PR is now the budgets only; with neither flag set, nothing changes except that GEAK replays are sampled.

2. Unmeasurable budgets are refused. New power_budget_unmeasurable_error (inference_optimizer/cli/bootstrap.py) refuses either budget with exit 2 on multi-node, with HYPERLOOM_GPU_POWER_SAMPLING=0, or without amd-smi, both at launch and on resume (from the archived budgets). The sampling test is one shared helper, gpu_power_sampling_unavailable in _gpu_power.py, which build_gpu_power_recorder now uses too, so the launch check and the recorder can't disagree.

3. Partial readings fail closed. The sampler now records unread_gpus in gpu_power.json: a card holding a model (at least 10% VRAM) with no power reading, or a card in scope (the mask, or every card amd-smi reported) whose VRAM was never read or that was missing from amd-smi, so it can't be ruled out as serving. Any unread card makes the round unmeasured (avg_power_w is null and the per-card map is empty, giving power_unmeasured). For your example, TP4 on 4–7 with card 7 at N/A, the round is now unmeasured instead of summing three cards. On "stays under 10% VRAM": a TP shard's weights plus the KV cache the framework reserves (vLLM's gpu_memory_utilization, SGLang's mem_fraction_static) put every serving card well above 10%; on the MiniMax M3 run each held about 260 of 288 GB. A card in the mask under 10% therefore holds no model and doesn't serve. An idle card with no power reading on an unpinned host still doesn't block, so a dead idle sensor can't veto every round.

Tests cover each case, plus the refusal paths. The description is updated to match.

[rpoornac]

Rebased onto main (dc46f8d97); the branch is now d0e3b8871, with no change to what the PR does. Three conflicts were resolved:

• integrate_patch.py: kept main's grade_accuracy comment rename beside the branch's gpu_power_avg_w / gpu_power_by_gpu_w fields.
• The two workflow-evaluation schemas: took main's versions, kept the baseline_over_power_budget stop reason, and regenerated both contract digests from the merged contracts in a separate commit (d0e3b8871).

Checked after the rebase: ruff check and format (0.16.2) clean; the power and contract tests (132) pass; and the broad suites show no failures beyond those on a clean main worktree in this environment.

[rpoornac]

Rebased onto main (ac6ae792a); the branch is now d97ef353c. Two fixes came out of reviewing the rebased branch before pushing:

1. Main's correctness-fix KEEP route applied only the latency budget (06c940fe5). feat(runtime-findings): scan server logs, surface findings to every role, allow verified correctness-fix KEEP #1793 added _correctness_fix_refusals in integrate_patch.py, which can KEEP a verified fix after the graded verdict reverted it, so it reads the budgets itself; it called latency_veto_reason directly. A correctness fix over --max-power-w or a card's limit would have been kept. It now calls constraint_veto_reason, like every other KEEP point; new tests (total and per-GPU) fail on the old call.
2. Power budgets are refused on scriptable frameworks (d97ef353c). An independent pre-push review found that xdit and custom rounds run no server, so the measured-phase sampler is never built and every round is power_unmeasured: the launch accepted the budget, then spent a baseline and stopped as if over budget, which is the case your must-fix dfs-actions, knowledge base as RAG, hooks for adding to KB, baselines #2 asked to refuse up front. power_budget_unmeasurable_error now refuses either budget there at launch and on resume, using framework_registry.is_scriptable (as the latency-budget scope check does). The docs, the env-var reference and the SKILL.md row no longer say the budgets apply to every framework.

Rebase conflicts: resolve_graded_comparison keeps main's AgentX refused gates and incomparability_reason, then applies constraint_veto_reason only when nothing refused (where main applies the latency veto); the CLI resume chain keeps main's _custom_specialist_conflict alongside the power checks; both new integrate-patch tests are kept. Contract digests are unchanged and still match.

Checked before pushing: ruff 0.16.2 check and format clean; pylint --errors-only on the CI package set reports the same errors as main (environment-only, none new); the power, contract and correctness-fix tests pass; the broad suites show no branch-only failure that reproduces on its own; a session saved without the new state fields resumes with no budget; Gitleaks finds no leaks. The description lists the correctness-fix route and the scriptable refusal.

[rpoornac]

/review

[github-actions]

Automated review started for head d97ef35. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38288907446

[github-actions]

Code metrics gate: PASSED with waived findings (baseline-raise)

Judged: the 42 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Worse	Baseline out of date	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	0	0	120	+0
Cognitive complexity	fails > 30	0	0	0	320	+0
Function length (lines)	fails > 80	0	1	0	472	+0
Nested block depth	fails > 5	0	0	0	29	+0
Module length (lines)	warns > 800, fails > 1200	0	2	0	54	+0
Duplicated lines per file	fails > 0	0	0	0	120	+0
Dead code (vulture)	fails > 0	0	0	0	2	+0

Check	Findings	Effect
Comments	0
English only	0
Production code imports test code	0
Repeated literals	0
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	0

Worse than the baseline (waived by the baseline-raise label): 3

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/phases/kernel.py:2057 KernelPhase.promote_geak_from_candidate	Function length (lines)	165	167
src/hyperloom/orchestrator/actions/executors/_grid_runner.py:1	Module length (lines)	2258	2259
src/hyperloom/orchestrator/phases/kernel.py:1	Module length (lines)	4227	4229

How to fix

• New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
• Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
• Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
• Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 5

Module	Lines	Limit
src/hyperloom/inference_optimizer/cli/bootstrap.py	816	warns > 800
src/hyperloom/inference_optimizer/cli/parser.py	1163	warns > 800
src/hyperloom/orchestrator/actions/executors/benchmark_result.py	986	warns > 800
src/hyperloom/orchestrator/loop/conversation.py	942	warns > 800
src/hyperloom/orchestrator/state/_shared_state/render.py	923	warns > 800

Already over at the base (backlog, not this change's): 11

Unit	Dimension	Base	Now
src/hyperloom/orchestrator/state/shared_state.py:101 resolve_graded_comparison	Cognitive complexity	35	34
src/hyperloom/orchestrator/actions/executors/explore.py:732 ExploreExecutor.__call__	Cognitive complexity	372	372
src/hyperloom/orchestrator/state/_shared_state/render.py:705 _RenderMixin._format_variant_line	Cognitive complexity	40	40
src/hyperloom/inference_optimizer/cli/bootstrap.py:424 _seed_shared_state	Function length (lines)	157	157
src/hyperloom/orchestrator/actions/executors/explore.py:732 ExploreExecutor.__call__	Function length (lines)	1133	1132
src/hyperloom/orchestrator/phases/prelude.py:2080 PreludePhase._settle_warm_replay	Function length (lines)	390	390
src/hyperloom/inference_optimizer/cli/__init__.py:1	Module length (lines)	2740	2734
src/hyperloom/orchestrator/actions/executors/explore.py:1	Module length (lines)	1914	1914
src/hyperloom/orchestrator/loop/writeback.py:1	Module length (lines)	4809	4790
src/hyperloom/orchestrator/phases/prelude.py:1	Module length (lines)	2642	2642
src/hyperloom/orchestrator/state/shared_state.py:1	Module length (lines)	2697	2697

20 findings in files this change does not touch

Outside this change (informational): 20

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/enablement/recipe/projections.py project_launch_evidence	Cognitive complexity	absent	36
src/kernelforge/cli/forge_loop_command.py	Module length (lines)	absent	1725
src/hyperloom/common/launch_log_evidence.py _server_args_from_log	Cognitive complexity	48	49
src/hyperloom/orchestrator/loop/dispatcher.py DispatcherCollaborator._spawn_fitting_queued	Cognitive complexity	110	111
src/hyperloom/common/launch_log_evidence.py _server_args_from_log	Cyclomatic complexity	22	23
src/hyperloom/orchestrator/actions/executors/_workload_envs.py materialize_config_with_envs	Cyclomatic complexity	110	111
src/hyperloom/orchestrator/actions/executors/_workload_envs.py materialize_config_with_envs	Function length (lines)	922	928
src/hyperloom/orchestrator/actions/executors/baseline.py BenchmarkRunExecutor._run_single_benchmark	Function length (lines)	578	580
src/hyperloom/orchestrator/loop/dispatcher.py DispatcherCollaborator._spawn_fitting_queued	Function length (lines)	242	243
src/hyperloom/orchestrator/loop/dispatcher.py DispatcherCollaborator.run_task_registered	Function length (lines)	109	111
src/hyperloom/orchestrator/policy/gate.py PolicyGate._validate_specialist_dispatch	Function length (lines)	91	93
src/hyperloom/orchestrator/prompts/prompt_builder.py build_orchestration_prompt	Function length (lines)	139	142
src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py _section_identity	Function length (lines)	94	95
src/hyperloom/orchestrator/specialists/runner.py SpecialistRunner._prepare	Function length (lines)	192	200
src/hyperloom/orchestrator/actions/executors/baseline.py	Module length (lines)	4253	4291
src/hyperloom/orchestrator/loop/dispatcher.py	Module length (lines)	1503	1551
src/hyperloom/orchestrator/loop/intent_router.py	Module length (lines)	1548	1556
src/hyperloom/orchestrator/policy/gate.py	Module length (lines)	1626	1647
src/hyperloom/orchestrator/prompts/prompt_builder.py	Module length (lines)	1205	1223
src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py	Module length (lines)	2573	2597

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

[github-actions]

PR #1770 -- feat(power): --max-power-w total and --max-per-gpu-power-w budgets

What it does: Adds opt-in GPU power budgets for serving frameworks: --max-power-w caps the summed mean power of the serving cards over each round's measured phase, and repeatable --max-per-gpu-power-w GPU_ID W caps one card by its physical amd-smi index. Both ride the existing KEEP veto channel (constraint_veto_reason in common/perf_metric.py), so every KEEP decision point (explore, integrate_patch including the correctness-fix route, the stacked-integrate window, warm replay, the lift, and GEAK replays, which now get a sampled measured phase via a log-driven phase driver) reverts its own over-budget candidate; unmeasurable budgets are refused with exit 2 at launch and on resume, and an over-budget or unread baseline stops the run with baseline_over_power_budget. With neither flag set, decision behaviour is unchanged; gpu_power.json now also records unread possibly-serving cards (leaving that round's mean null), and GEAK replays are sampled.

Blocking issues: 2

1. [V5] import-linter (architecture contracts) fails at head: the launch check imports a private orchestrator module across the layer boundary [verified]
   Problem: src/hyperloom/inference_optimizer/cli/bootstrap.py:232 does from hyperloom.orchestrator.actions.executors._gpu_power import gpu_power_sampling_unavailable, reaching an _-prefixed module of another package from the inference_optimizer layer (the A1 private-name reach the code-metrics gate also reports). The .importlinter hl-top-layers contract carries a closed baseline of exactly six such cli -> executors entries ("the list may only shrink"), so this seventh cannot be waived by a baseline line; I ran PYTHONPATH=src lint-imports --no-cache on the head tree and it fails on precisely this import (both the layers and the acyclic contract), while all contracts are kept at the merge base.
   Impact: the import-linter (architecture contracts) CI job is red at head d97ef35, so the PR cannot merge; the owner of _gpu_power also gains a caller it cannot see the next time it refactors the module.
   Action: Route the sampler-availability check through a surface the CLI layer may import (re-home gpu_power_sampling_unavailable where the CLI can reach it legally, or have the launch refusal read it through a module the contract already allows), so the contract holds without adding a baseline entry.
2. [D12] The code-metrics gate fails at head: this diff raises three functions further past the complexity ceiling and adds a module-length violation and repeated literals, with no baseline-raise reason stated [verified]
   Problem: Measured with ruff --select C901 (max-complexity 20) against the merge base: _run_optimize 97->98 (src/hyperloom/inference_optimizer/cli/__init__.py:1743, the new launch-check branch), ConversationCollaborator.compose_prompt 29->30 (src/hyperloom/orchestrator/loop/conversation.py:446, the power block), WritebackCollaborator._promote_baseline 29->30 (src/hyperloom/orchestrator/loop/writeback.py:1763, the power stop block) — AGENTS.md's ceiling forbids raising a function past 20 without a stated reason, and the description gives none. The same gate also fails on src/hyperloom/inference_optimizer/cli/parser.py reaching 1269 lines (over the 1200 failure limit) and on three repeated string literals the baseline-raise label cannot waive: 'max_power_w' and 'max_per_gpu_power_w' each written 3 times in cli/__init__.py:1962-1963 and 2401-2407, and 'boot' now written 3 times in src/hyperloom/orchestrator/actions/executors/_gpu_power.py:65 (the phase driver's boot note).
   Impact: the code-metrics CI job is red at head d97ef35, so the PR cannot merge as it stands.
   Action: Name each repeated literal once as a module-level constant, split parser.py under the module-length limit, and extract the added branches (the power launch check out of _run_optimize, the baseline stop out of _promote_baseline's current-best block, the prompt section out of compose_prompt) so the functions are no worse than their recorded values — and where the gate allows a waiver, apply the baseline-raise label with the reason in the PR description.

deferred: A1 -- the private-name reach at bootstrap.py:232 is the defect reported inside finding 1, kept under the V5 key where the red check names it

Checked: diff.txt in full; perf_metric.py (power_veto_reason/constraint_veto_reason/_ceiling_veto), _gpu_power.py (unread_gpus, ServerLogPhaseDriver, read_measured_gpu_power_by_gpu), bootstrap.py (budget conflict/unmeasurable/per-GPU checks), cli/init.py:1940-2000 and 2250-2420 (resume chain, framework resolution before the launch check), writeback.py:1763-2030 (_promote_baseline stops), shared_state.py from_dict, baseline.py:3801, _geak_sweep.py replay loop, integrate_patch.py:1817, integrate_performance.py, prelude.py _settle_warm_replay, render.py to_power_budget_summary; tree-wide greps for baseline_over_latency_budget consumers, latency_veto readers, and gpu_power_by_gpu_w writers/readers; .importlinter contract; ruff C901 at head and base | Ran: PYTHONPATH=src lint-imports --no-cache (fails on bootstrap.py:232); ruff C901 head vs base (three functions raised); pytest test_power_budget.py + test_latency_budget.py (123 passed), test_gpu_power_sampling.py (22), test_integrate_correctness_fix.py (14), warm-replay/GEAK/explore/integrate power tests (9), contract tests (145) | Base: ac6ae79 | Head: d97ef35

[rpoornac]

Rebased onto main 1a444f6b8 (61 commits, including #1842's fix for the serving_slot Ray test that failed the last run's py3.11 shard 6). The branch is now 4e13f7724.

• Conflicts: scripts/code_metrics_baseline.txt (kept main's entries and re-ran --update-baseline at the refactor commit: the same 13 units lowered) and the tail of test_gpu_power_sampling.py (Delete four duplicated admission-gate rules and the complexity they carried #1853 rewrote the last assertion of test_round_recorders_combine_kv_and_power; kept main's version, then this PR's tests). git range-diff shows nothing else changed in the existing commits.
• One new commit: fix(enablement): the setup/boot/grade path of an enablement round #1841/fix(enablement): keep enablement rounds, the kept stack, and the framework tree intact #1858 made bind_integrate_to_lane read state.phase and state.enablement and added both to the latency-budget twin of test_a_source_patch_over_the_power_budget_is_reverted; the power test gets the same two fields.
• Locally at this head: ruff, lint-imports (7 kept), code-metrics (only the three units baseline-raise covers), pylint errors-only identical to main (227/227), and the full src suite has no failure main lacks except the timing-flaky test_aiperf_client_sh / test_inferencex_patcher concurrency tests, which pass on rerun.

[rpoornac]

/review

[github-actions]

Automated review started for head 4e13f77. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38418986039

[github-actions]

Automated review: no verdict applied -- the review job ended with result 'failure'. Comment /review to retry. Run: https://github.com/AMD-AGI/Hyperloom/actions/runs/38418986039

[rpoornac]

/review

[github-actions]

Automated review started for head 4e13f77. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38426070608

[github-actions]

Automated review: no verdict applied -- the review job ended with result 'failure'. Comment /review to retry. Run: https://github.com/AMD-AGI/Hyperloom/actions/runs/38426070608

[rpoornac]

/review

[github-actions]

Automated review started for head 4e13f77. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38442590460

[github-actions]

PR #1770 -- feat(power): --max-power-w total and --max-per-gpu-power-w budgets

What it does: Adds two opt-in GPU power budgets for the serving frameworks: --max-power-w caps the summed mean power of the serving cards over each round's measured phase, and repeatable --max-per-gpu-power-w GPU_ID W caps one card by its physical amd-smi index. Both ride the existing KEEP veto channel (constraint_veto_reason in common/perf_metric.py), so every KEEP decision point (explore, integrate_patch including the correctness-fix route and the stacked window, warm replay, the lift, and GEAK replays, which now get a sampled measured phase via a log-driven phase driver) reverts its own over-budget candidate; unmeasurable budgets are refused with exit 2 at launch and on resume, an unread or over-budget baseline stops the run with baseline_over_power_budget, and a round with an unread possibly-serving card (listed in gpu_power.json as unread_gpus, including a 0 W or negative reading) fails closed as power_unmeasured. With neither flag set, decision behaviour is unchanged; GEAK replays are newly sampled.

Blocking issues: 1

1. [X3] A power veto is recorded and logged as "latency_budget" [verified]
   Problem: The KEEP veto channel was widened to carry power_budget_exceeded / gpu_power_budget_exceeded / power_unmeasured, but the two texts that name the refusing constraint after the old latency-only contract were not updated: the explore decision gate is labelled "latency_budget" whenever graded.veto_reason is set (src/hyperloom/orchestrator/actions/executors/explore.py:1421), and the lift logs "winner refused by the latency budget" (src/hyperloom/orchestrator/loop/writeback.py:1452) — both fire with the power reasons. breakdown/recorder/framework_event.py:987 (_blocking_gate) persists that gate name into the session breakdown's blocked_by field, so a variant refused by the power budget is recorded as blocked by the latency budget. At the merge base the label was always correct, because latency_veto_reason was the only veto producer.
   Impact: The explore ledger row and the persisted session_breakdown blocked_by misname the constraint that refused a variant; the orchestration prompt renders this ledger and its power-budget section tells the model to look for throughput-per-watt levers, so a power refusal mislabelled as latency steers the model toward latency levers; the lift log line names the wrong budget to the operator reading the session log.
   Action: Name the gate for the constraint that fired — derive the gate label from graded.veto_reason (or a "power_budget" arm beside "latency_budget") at explore.py:1421, and reword the lift log at writeback.py:1452 to name the session constraints rather than the latency budget; extend the over-budget leg of test_explore_grades_the_rounds_own_gpu_power_against_the_budget to pin the gate label.

Checked: diff.txt in full; perf_metric.py (power_veto_reason/constraint_veto_reason/_ceiling_veto/CONSTRAINT_FIELDS/constraint_fields), _gpu_power.py (parse_power_sample's non-positive rule, summary/unread_gpus, ServerLogPhaseDriver, _read_round_artifact, read_measured_gpu_power_by_gpu, build_gpu_power_recorder), _geak_sweep.py (_run_with_power_sampling/_run_sampled/_replay_server_logs, the replay loop at :311-345 and _geak_replay_server_log still called at :371), bootstrap.py (all four budget checks, resolve_launch_gpu_power, constraint_budgets_from_args, _budget_resume_conflict), cli/init.py resume chain and launch path, writeback.py (lift_to_current_best veto and log, _promote_baseline stop, _replay_keep_from_result), baseline_budget.py, baseline.py:3821 result assembly, shared_state.py (resolve_graded_comparison veto placement, from_dict tolerance), integrate_performance.py, integrate_patch.py:1754, prelude.py _settle_warm_replay, kernel.py promote_geak_from_candidate, render.py, conversation.py, explore.py decision gates, stop_reasons.py and the four contract JSONs, constraint_args.py, report.py explanation merge, rebench.py; tree-wide sweeps (head and base) for gpu_power_by_gpu_w, power_budget_w, power_budget_per_gpu_w, baseline_over_power_budget, constraint_veto/latency_veto, latency_veto, GPU_POWER_ENV, _geak_replay_server_log, constraint_fields and the warm-replay outcome key; SharedState loader read and a pre-PR state.json load run; both workflow contract digests recomputed (b25e226a.../ece338b24...) and matching; ruff C901 (max 20) on all 24 changed production files at head and base; per-file comment/code ratio from the diff; comments.txt (all five review rounds' findings re-verified fixed at head); openprs.txt (#1798/#1845 file overlaps, no premise conflict); an independent reader pass over card.md, diff.txt and base.txt | Ran: pytest test_power_budget.py (65 passed), test_gpu_power_sampling.py (28), test_latency_budget.py (61), warm-replay/GEAK-promotion/explore/integrate/correctness-fix power tests (13), all at head 4e13f77; power_veto_reason/constraint_veto_reason called directly on the fail-closed shapes (None/{}/scalar/0.0/-5.0/NaN/inf/bool/string, per-GPU over, total over, at-limit, listed-not-serving, unlisted-inside-total); SharedState.from_dict on a pre-PR raw dict; workflow_contract_digest for both versions | Base: 1a444f6 | Head: 4e13f77

[rpoornac]

Re the review at 4e13f77, [X3] a power veto recorded and logged as "latency_budget": verified and fixed in the latest commit.

• GradedComparison.vetoed_by (common/perf_metric.py) names the constraint behind veto_reason: power_budget for power_budget_exceeded / gpu_power_budget_exceeded / power_unmeasured, latency_budget for the latency reasons, "" when nothing vetoed. The power reason strings are now named once there.
• Explore's decision gate uses it, so a power refusal is recorded as power_budget in the explore ledger and the session breakdown's blocked_by. The lift log now reads "winner refused by power_budget (power_budget_exceeded)".
• Class sweep: these were the only two places naming the vetoing constraint. _blocking_gate passes the gate name through unchanged, and no doc, prompt or contract lists gate names.
• Tests: the over-budget leg of test_explore_grades_the_rounds_own_gpu_power_against_the_budget now pins power_budget (and no latency_budget gate), and vetoed_by is pinned for every reason. Both fail on 4e13f77.

The description's operator-visible changes now mention the power_budget gate.

[rpoornac]

/review

[github-actions]

Automated review started for head c599b3f. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38452258614

Superseded by a clean automated review at c599b3f.

[github-actions]

PR #1770 -- feat(power): --max-power-w total and --max-per-gpu-power-w budgets

What it does: Adds two opt-in GPU power budgets for the serving frameworks: --max-power-w caps the summed mean power of the serving cards over each round's measured phase, and repeatable --max-per-gpu-power-w GPU_ID W caps one card by its physical amd-smi index. Both ride the existing KEEP veto channel (constraint_veto_reason in common/perf_metric.py), so every KEEP decision point (explore, integrate_patch including the correctness-fix route and the stacked window, warm replay, the lift, and GEAK replays, which now get a sampled measured phase via a log-driven phase driver) reverts its own over-budget candidate; unmeasurable budgets are refused with exit 2 at launch and on resume, an unread or over-budget baseline stops the run with baseline_over_power_budget, and a round with an unread possibly-serving card (listed in gpu_power.json as unread_gpus, including a 0 W or negative reading) fails closed as power_unmeasured. With neither flag set, decision behaviour is unchanged; GEAK replays are newly sampled.

Blocking issues: none

Checked: diff.txt in full; perf_metric.py (power_veto_reason/constraint_veto_reason/_ceiling_veto/CONSTRAINT_FIELDS/constraint_fields/vetoed_by vs the latency twin's behaviour), _gpu_power.py (parse_power_sample's non-positive rule, summary/unread_gpus, ServerLogPhaseDriver vs the watchdog's own marker→phase mapping in _subprocess_kill.py:1095-1106, _read_round_artifact, read_measured_gpu_power_by_gpu, build_gpu_power_recorder), _geak_sweep.py (_run_with_power_sampling/_run_sampled/_replay_server_logs and the replay entry's path to promotion_measurement), bootstrap.py (all four budget checks, resolve_launch_gpu_power, constraint_budgets_from_args, _budget_resume_conflict), cli/init.py resume chain and launch path, writeback.py (lift_to_current_best veto and log, _promote_baseline stop, _replay_keep_from_result carrier), baseline_budget.py, baseline.py result assembly (extract_benchmark_measurement feeds the baseline stop at :3821), shared_state.py (resolve_graded_comparison veto placement, from_dict tolerance for pre-PR archives), integrate_performance.py, integrate_patch.py:1753 (correctness-fix route), prelude.py _settle_warm_replay, kernel.py promote_geak_from_candidate, render.py to_power_budget_summary/to_constraint_sections, stop_reasons.py and the four contract JSONs, constraint_args.py, report.py explanations merge, explore.py decision gate, _grid_base.py, _grid_runner.py, conversation.py, rebench.py; tree-wide sweeps (head) for every writer/reader of gpu_power_by_gpu_w, power_budget_w, power_budget_per_gpu_w, baseline_over_power_budget, veto_reason/latency_veto, GPU_POWER_ENV, _round_gpu_power_w, _positive_ms_arg/_positive_watts_arg; ruff --select C901 (max-complexity 20) on the 24 changed production files at head and at base; per-file comment/code ratio from the diff; ci.txt and comments.txt (all prior rounds' findings re-verified fixed at head); openprs.txt (#1798/#1316/#1845 overlaps, no premise conflict) | Ran: pytest test_power_budget.py + test_gpu_power_sampling.py (99 passed), test_latency_budget.py (61), test_integrate_correctness_fix.py (14), warm-replay/GEAK-promotion/explore/integrate power tests (9), all at head c599b3f; power_veto_reason/constraint_veto_reason/latency_veto_reason called directly on the fail-closed shapes (None/{}/0.0/-5.0/NaN/inf/bool/string, per-GPU over, total over, at-limit, listed-not-serving, unlisted-inside-total); SharedState.from_dict on a pre-PR raw dict; both workflow contract digests recomputed via workflow_contract_digest (b25e226a…/ece338b24…, matching) | Base: 1a444f6 | Head: c599b3f

LGTM

[chaojhou]

Re-reviewed at c599b3f1a. Everything from my reviews is resolved:

• From 16abb36b5:
  • The apply path is out of this PR.
  • Both budgets are refused with exit 2 when power can't be measured, at launch and on resume.
  • A possibly-serving card without a reading, or reading 0 W or less, makes the round power_unmeasured.
• From da75556dd: the replay phase driver's two broad except Exception are gone. _poll_logs catches only OSError and warns once, anything else raises, and _run_sampled still closes the recorder in a nested finally. The PR adds no broad except anywhere.
• New since then: GradedComparison.vetoed_by names power_budget for the three power reasons, so the explore gate and the lift log no longer call a power veto a latency one. The rebase only changed the test_gpu_power_sampling.py conflict resolution and one lowered code_metrics_baseline.txt entry.

The baseline-raise label covers three units that grow by one or two lines to carry the power reading, and the description gives that reason. CI is green, including code-metrics and pr-hygiene. LGTM.

[chaojhou]

PR #1770 -- feat(power): --max-power-w total and --max-per-gpu-power-w budgets

What it does: Adds two opt-in GPU power budgets for serving frameworks: --max-power-w limits the summed measured-phase mean power of the serving cards, and the repeatable --max-per-gpu-power-w GPU_ID W limits one card by its amd-smi index. Both go through the existing KEEP veto (constraint_veto_reason), so every KEEP point refuses an over-budget or unmeasured candidate, and explore records such a refusal under the power_budget gate. A budget that cannot be measured is refused with exit 2 at launch and on resume, and an over-budget or unread baseline stops the run with baseline_over_power_budget. With neither flag set, decisions are unchanged; GEAK replays now also write gpu_power.json.

Blocking issues: none

Checked: the three commits since my re-review at da75556dd (e5ab20170, 4e13f7724, c599b3f1a), and git range-diff for the rest (the rebase changed only conflict context: the baseline file and one test assertion). My earlier blocker is resolved: ServerLogPhaseDriver._poll_logs (_gpu_power.py:298) catches only OSError; the scanner it calls (_subprocess_kill.py:711-796) already absorbs read and decode errors, so only a bug can raise; the per-variant handler that exists on base (_geak_sweep.py:367) marks that variant failed, and _run_sampled still closes the recorder. GradedComparison.vetoed_by was checked against every writer of veto_reason (shared_state.py:172, :198, both from constraint_veto_reason) and every reader of the gate names (none compares them; no contract, prompt or doc lists them). A tree-wide sweep after the rebase onto 61 main commits found every KEEP point going through constraint_veto_reason and every measurement carrier through constraint_fields; main's new KEEP_INERT keeps switched-off code and lets nothing into current_best. The diff adds no except Exception. | Ran: vetoed_by for every veto reason, power_veto_reason on 0 W / -5 W / NaN / {} / None and a per-GPU overrun, and _poll_logs with an unlistable log directory (warns once, then debug, no raise) and with a scan bug (raises), all loaded from the head commit; c901.py, an AST port of ruff's C901 count (ruff is not installed here), on the 24 changed production modules at base and head: no function above 20 grows (_run_optimize 97 -> 96, compose_prompt 29 -> 28); CI: full matrix green at this head, including code-metrics and import-linter; no test suite run locally | Base: 1a444f6 | Head: c599b3f

LGTM