Skip to content

Reject candidate changes to frozen AgentX replay controls - #1667

Open
iraj465 wants to merge 3 commits into
mainfrom
fix/agentx-frozen-workload
Open

iraj465 wants to merge 3 commits into
mainfrom
fix/agentx-frozen-workload

Conversation

@iraj465

@iraj465 iraj465 commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Keep operator AgentX replay controls authoritative when materializing candidate configurations. Candidates cannot replace frozen replay inputs, and inherited AGENTX_WARMUP_GRACE_PERIOD is discarded so the workload switch derives the grace for the candidate concurrency. This prevents valid optimized concurrency-sweep rungs from failing YAML construction or overwriting their resolved warmup grace.

The branch is based on current main 9b8a3e2f3b2920d04fb3f511ca88c90e2d1c8297. The regression covers lower and higher concurrency, inherited base and variant environments, unchanged candidate mappings, and fixed C32/1,800-second materialization. All 34 frozen-workload tests and 477 surrounding tests passed, with 5 skips; repository-wide Ruff lint and formatting passed. Hosted CI must run after publication; no GPU A/B result is claimed.

@iraj465
iraj465 marked this pull request as ready for review September 24, 2026 15:42
@iraj465
iraj465 requested a review from a team as a code owner September 24, 2026 15:42

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must-fix: AGENTX_WARMUP_GRACE_PERIOD is a derived value, so the "identical inherited value is allowed" rule fails for it on every conc-sweep rung whose CONC differs from the session's.

_build_variant_yaml validates against bench["envs"] right after apply_runtime_benchmark_overrides(conc=variant_conc(variant)). That re-runs apply_agentx_switch(conc=...), which rewrites envs["AGENTX_WARMUP_GRACE_PERIOD"] scaled to the rung's CONC. Conc-sweep rungs carry the current best's envs plus their own CONC (conc_sweep.py:451-466), and those envs come from normalize_proposal(state.current_best) unfiltered.

So if the current best carries AGENTX_WARMUP_GRACE_PERIOD at the session value -- the identical-inherited case this PR allows, which passes and can be kept at the session CONC -- every optimized-arm rung at another CONC now fails with yaml_build_error. The AgentX sweep is on by default with the 1,4,8,10,14,20,28 ladder, so that is all rungs, or all but one. Before this change the carried value only overrode the rung's grace (a longer warmup wait); after it, the rung is lost. Every new test runs at the session CONC (32), so this case is not covered.

Whether GEAK's env_map or an LLM proposal actually carries this key is not visible from this repo, but the PR's premise is that GEAK echoes replay controls.

Requested change: don't compare the derived key. Drop AGENTX_WARMUP_GRACE_PERIOD from candidate envs instead -- the switch owns it and re-derives it per rung, and dropping it also stops _build_variant_yaml's later env merge from overriding the rung's value. Add a case where a variant with a CONC different from the session's inherits the session value.

Before merge: CI ran on f3a736a on 09-24, and main has 86 commits since, including changes to _grid_runner.py. The merge is textually clean and the code this PR relies on is unchanged, but please update the branch and re-run CI.

@chaojhou

Copy link
Copy Markdown
Collaborator

@iraj465 this PR conflicts with current main. Could you rebase it?

Files that conflict: src/hyperloom/orchestrator/actions/executors/_grid_runner.py, src/hyperloom/orchestrator/actions/executors/_workload_envs.py.

Since 2026-09-26, #1922 moved AgentX onto Magpie's native contract (apply_agentx_switch, the new magpie_agentx.py), and #1963 / #1964 split materialize_config_with_envs and _build_variant_yaml. The override check probably belongs on the native launch path now.

Once it merges cleanly I'll review it.

@chaojhou

Copy link
Copy Markdown
Collaborator

Closing this for now, as noted in the 2026-10-22 rebase request. There has been no reply or push since then, and the branch still conflicts with main: a trial merge against ba689ef conflicts in 6 files. Main has moved under it. #1922 put AgentX on Magpie's native contract (apply_agentx_switch, magpie_agentx.py), and #1963 / #1964 split materialize_config_with_envs and _build_variant_yaml.

Thank you for this. If candidate changes to the frozen AgentX replay controls are still possible on current main, please re-port the check onto the native Magpie launch path as a fresh PR, or reopen this one after rebasing. The regression tests here are a good starting point.

@chaojhou chaojhou closed this Oct 25, 2026
@chaojhou chaojhou reopened this Oct 25, 2026
@chaojhou

Copy link
Copy Markdown
Collaborator

Reopening: closing three days after the 2026-10-22 rebase request was too soon. This stays open for a month from that request, until 2026-11-22. If there is still no rebase or reply by then, it will be closed. The notes in the comment above still apply.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants