feat(agentx) Use Magpie native AgentX for Hyperloom optimization

[haofrank]

Fresh HYPERLOOM_AGENTX=1 sessions use Magpie-managed AgentX through the existing Hyperloom CLI. Hyperloom keeps optimization, candidate grading and selection. Magpie manages the server lifecycle and optional trace capture; InferenceX/AIPerf replays and measures the workload. No Slurm integration is required.

Fixes #1601.

What this enables

• Single-node SGLang/vLLM sessions automatically use the native backend. Hyperloom translates model, image, topology and concurrency settings into Magpie configuration.
• Server arguments, environment tuning and verified source/kernel changes reach the actual process. Hyperloom checks the launch receipt while keeping workload identity fixed.
• KEEP/resume and GEAK remeasurement remain in Hyperloom. GEAK proposals require review and canonical remeasurement; proxy scores cannot promote a candidate.
• Historical sessions retain their recorded backend, measurement contract and accepted records. New measurements still face current validity checks. Synthetic and explicit MLPerf paths remain available.

Compatibility and trace handling

• Successful standalone diagnostic profiles now reach the Coordinator's trace-analysis path. Failed or incomplete captures are rejected, and diagnostic results cannot change baseline/KEEP evidence.
• The installer and CLI share one Magpie health check: generic sessions require an importable package; native AgentX additionally requires the pinned capabilities, commit and audited execution tree. Compatible custom/offline installs and saved legacy sessions retain their existing behavior.
• Native CLI preflight rejects non-empty legacy AIPERF_*, AGENTIC_*, AGENTX_DATASET, AGENTX_WARMUP_REQUESTS_PER_LANE and WEKA_LOADER_OVERRIDE controls in the shell or benchmark.envs, before dependency installation or benchmarking. Errors identify the variable names without printing their values. Use supported benchmark.agentx settings and the resolved InferenceX recipe. Legacy and MLPerf controls remain available.

Profiling

Hyperloom profiles an accepted configuration using Magpie's AgentX replay. The corresponding Magpie settings can request repeated captures:

benchmark:
  envs:
    CONC: 32
  profiler:
    torch_profiler:
      enabled: true
      num_steps: 20
      num_profiles: 3
      start_seconds: 30
      interval_seconds: 200
      detailed_annotations: false

Use the accepted workload's concurrency. The initial delay starts after warmup, and later intervals start after the previous trace finishes flushing. These are server execution steps, not request counts. Profile results are diagnostic; KEEP requires separate unprofiled canonical evidence. Rank-file inventory and selected-rank quality certificates remain distinct.

The example uses basic tracing. Enhanced shape/roofline analysis requires a compatible instrumented runtime and passing trace-quality checks.

Dependencies and validation

Hyperloom d9bb74c pins merged Magpie d80eb4d from #117, with its updated audited package fingerprint and InferenceX 408c015. SGLang shape discovery is optional; per-batch graph trace export is preserved. Both InferenceX layouts are supported. The generic packaged throughput client has an isolated Python 3.12 / Transformers 5 runtime; native AgentX/AIPerf remains unchanged. The validation harness protects GSM8K's existing environment with framework constraints and version checks. Generic compatibility preparation leaves the audited Magpie package unchanged.

At 40a143aa, main 53ed5072 is merged and the PR has no conflicts. All 12 Linux Python 3.10/3.11 test shards and both coverage checks pass. Lint, including all seven architecture contracts, also passes. The current-head single-GPU smoke E2E is still running.

Remaining merge gates: PR hygiene counts 10,027 changed production Python lines against the 1,000-line limit; code-metrics retains complexity/size and two flag-argument findings. Repeated literals, cross-package private imports and stale baseline entries are cleared. No thresholds were relaxed; six improved baseline entries were lowered, and no exception labels have been added. These remaining gates require restructuring or a documented exception under the repository's review rules.

GPU baseline/candidate and trace capture previously passed on earlier revisions; the slower --disable-cuda-graph candidate was not promoted. Selected-rank trace quality reported blocked/warn, while other ranks received file-inventory checks. Full TraceLens/Roofline analysis has not run. Current-head native GPU, ordinary benchmark/GSM8K, and the full Coordinator/KEEP/GEAK GPU loop remain unverified. See detailed results and limitations.

[Copilot]

Copilot review overview

🟡 Changes recommended

Compatibility checkout preparation blocks the async event loop, and the native GEAK skip leaves superseded handoff code reachable only through direct tests.

Review effort: Balanced
Findings: 2

Open (2)

• Run synchronous AgentX checkout in a worker thread · New
• Remove unreachable native AgentX handoff and proxy path · New

What changed in this PR

Routes AgentX measurements through pinned Magpie/InferenceX native launchers while preserving recipe topology, cache behavior, and measurement provenance.

Changes:

• Adds native AgentX configuration, runtime validation, topology-aware metrics, and resume state.
• Rejects unsupported optimization, GEAK, mutation, and sweep paths.
• Updates installation, profiling compatibility, tests, examples, and documentation.

File	Description
src/​hyperloom/​orchestrator/​state/​shared_state.py	Persists AgentX pins and source config.
src/​hyperloom/​orchestrator/​phases/​prelude.py	Preserves pinned InferenceX checkout.
src/​hyperloom/​orchestrator/​phases/​kernel.py	Skips unsupported native GEAK execution.
src/​hyperloom/​orchestrator/​loop/​writeback.py	Persists AgentX GPU topology.
src/​hyperloom/​orchestrator/​kernel/​request_handlers.py	Rejects native mutation integration.
src/​hyperloom/​orchestrator/​kernel/​conc_sweep.py	Records AgentX GPU count.
src/​hyperloom/​orchestrator/​kernel/​conc_sweep_plot.py	Normalizes plots by recipe GPUs.
src/​hyperloom/​orchestrator/​actions/​executors/​tests/​test_integrate_patch_executor.py	Tests native integration rejection.
src/​hyperloom/​orchestrator/​actions/​executors/​profile.py	Adds isolated compatibility profiling.
src/​hyperloom/​orchestrator/​actions/​executors/​integrate_patch.py	Blocks native source mutations.
src/​hyperloom/​orchestrator/​actions/​executors/​baseline.py	Adds native launch validation hooks.
src/​hyperloom/​orchestrator/​actions/​executors/​_recipe_script.py	Resolves native server proxies.
src/​hyperloom/​orchestrator/​actions/​executors/​_ray_backend.py	Forces AgentX local execution.
src/​hyperloom/​orchestrator/​actions/​executors/​_inferencex_patcher.py	Limits patching to explicit checkouts.
src/​hyperloom/​orchestrator/​actions/​executors/​_grid_runner.py	Validates native grid topology.
src/​hyperloom/​orchestrator/​actions/​executors/​_grid_base.py	Adds GPU-count result metadata.
src/​hyperloom/​orchestrator/​actions/​executors/​_benchmark_runtime.py	Preserves native launcher selection.
src/​hyperloom/​orchestrator/​actions/​executors/​_accuracy_gate.py	Parses native request-error gates.
src/​hyperloom/​inference_optimizer/​tests/​test_workload_envs.py	Tests diagnostic profile materialization.
src/​hyperloom/​inference_optimizer/​tests/​test_workload_envs_branches_unit.py	Tests native configuration safeguards.
src/​hyperloom/​inference_optimizer/​tests/​test_warm_replay.py	Tests pinned checkout preservation.
src/​hyperloom/​inference_optimizer/​tests/​test_shared_state_units.py	Tests disabled AgentX warm replay.
src/​hyperloom/​inference_optimizer/​tests/​test_run_optimize_exit_code_lock.py	Strengthens CLI error assertions.
src/​hyperloom/​inference_optimizer/​tests/​test_ray_backend_unit.py	Tests local execution and GPU counts.
src/​hyperloom/​inference_optimizer/​tests/​test_profile_and_kernel_handlers.py	Tests isolated compatibility profiling.
src/​hyperloom/​inference_optimizer/​tests/​test_map_aiperf.py	Tests native accounting mapping.
src/​hyperloom/​inference_optimizer/​tests/​test_kernel_integrate_and_report.py	Tests measurement-only integration behavior.
src/​hyperloom/​inference_optimizer/​tests/​test_install_magpie_idempotent.py	Tests Magpie capability repair.
src/​hyperloom/​inference_optimizer/​tests/​test_install_inferencex.py	Tests checkout ownership preservation.
src/​hyperloom/​inference_optimizer/​tests/​test_inferencex_preflight_clone.py	Tests pinned checkout identity.
src/​hyperloom/​inference_optimizer/​tests/​test_inferencex_patcher.py	Tests isolated patch scope.
src/​hyperloom/​inference_optimizer/​tests/​test_grid_runner.py	Updates legacy timeout coverage.
src/​hyperloom/​inference_optimizer/​tests/​test_graded_comparison.py	Tests native grading guards.
src/​hyperloom/​inference_optimizer/​tests/​test_geak_revalidation_dispatch.py	Tests GEAK refusal and recovery.
src/​hyperloom/​inference_optimizer/​tests/​test_geak_handoff_workload_contract.py	Tests pre-handoff GEAK skipping.
src/​hyperloom/​inference_optimizer/​tests/​test_geak_handoff_launch_server_script.py	Tests native proxy resolution.
src/​hyperloom/​inference_optimizer/​tests/​test_eval_result_dir_wiring.py	Isolates native accuracy mapping.
src/​hyperloom/​inference_optimizer/​tests/​test_env_safety.py	Tests blocked AgentX identity overrides.
src/​hyperloom/​inference_optimizer/​tests/​test_coordinator_gemm_promote_units.py	Separates generic grading tests.
src/​hyperloom/​inference_optimizer/​tests/​test_cli_conc_sweep_default.py	Tests mode-dependent sweep defaults.
src/​hyperloom/​inference_optimizer/​tests/​test_cli_bootstrap.py	Tests source and pin snapshots.
src/​hyperloom/​inference_optimizer/​tests/​test_baseline_param_overrides.py	Tests unsupported native overrides.
src/​hyperloom/​inference_optimizer/​tests/​test_backend_gating.py	Tests immutable install behavior.
src/​hyperloom/​inference_optimizer/​tests/​test_agentx_runtime.py	Tests native launch-boundary validation.
src/​hyperloom/​inference_optimizer/​tests/​test_agentx_repair.py	Updates installer extraction coverage.
src/​hyperloom/​inference_optimizer/​tests/​test_agentx_baseline_timeout.py	Tests profile compatibility timeout.
src/​hyperloom/​inference_optimizer/​tests/​test_accuracy_gate_units.py	Tests native error thresholds.
src/​hyperloom/​inference_optimizer/​cli/​parser.py	Adds benchmark-config CLI semantics.
src/​hyperloom/​inference_optimizer/​cli/​bootstrap.py	Seeds native state and runtime pins.
src/​hyperloom/​inference_optimizer/​baseline_comparison/​research_hints.py	Uses physical recipe GPU count.
src/​hyperloom/​inference_optimizer/​assets/​install.sh	Pins and validates Magpie 0.3.0.
src/​hyperloom/​inference_optimizer/​assets/​agentx/​aiperf_client.sh	Uses canonical AgentX model identity.
src/​hyperloom/​inference_optimizer/​agentx/​mapping.py	Maps native accounting fields.
src/​hyperloom/​common/​tests/​test_perf_metric.py	Tests per-chip throughput guards.
src/​hyperloom/​common/​perf_metric.py	Adds topology-aware throughput helpers.
src/​hyperloom/​common/​gain_math.py	Normalizes sweep guards per chip.
src/​hyperloom/​common/​env_safety.py	Blocks recipe-identity candidate overrides.
examples/​hyperloom-agentx-glm-5.2-mi355x/​README.md	Documents the runnable GLM example.
examples/​hyperloom-agentx-glm-5.2-mi355x/​benchmark.yaml	Adds the native GLM recipe config.
docs/​release-notes.md	Records native AgentX behavior.
docs/​reference/​environment-variables.md	Documents native controls and limits.
docs/​reference/​authentication.md	Documents dependency path ownership.
docs/​how-to/​optimize.md	Adds native AgentX instructions.
docs/​conceptual/​optimization-loop.md	Documents measurement-only phase behavior.
docs/​components/​magpie.md	Updates Magpie integration details.
docs/​compatibility.rst	Records validated dependency pins.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chaojhou]

Blocker: several native-only refusals are gated on AgentX in general, so they also switch off optimization for legacy AgentX sessions.

Native mode is opt-in (benchmark.agentx: enable), and SKILL.md keeps HYPERLOOM_AGENTX as the supported legacy switch. Most of the PR scopes its limits correctly with native_agentx_enabled(bench["agentx"]) (preflight, grid runner, benchmark runtime, CLI). These do not:

• kernel.py:1050 is if agentx:, where agentx = is_agentx_mode(benchmark_mode) if benchmark_mode else agentx_enabled() (:1011). The GEAK phase is skipped for every AgentX session, and the legacy AgentX GEAK branches (proposal proxies with canonical revalidation in Hyperloom) are deleted.
• integrate_patch.py:2350, request_handlers.py:4778, baseline.py:2366 / :2445 and writeback.py:6190 use agentx_active(...), which is HYPERLOOM_AGENTX or benchmark_mode == "agentx", unchanged from main.

Their messages all say "Native AgentX cannot ...", but a legacy HYPERLOOM_AGENTX=1 session -- today's production AgentX path -- would lose GEAK, integrate_patch source/kernel mutations, kernel mutation benchmarks and runtime_override, leaving measurement plus explore. The description does not mention this.

Requested change: gate these on native mode (the same native_agentx_enabled check on the session's source/accepted config), or, if legacy AgentX is meant to lose these levers too, say so in the description and get that signed off.

Must-fix: no GPU run. This bumps Magpie for every session, including synthetic ones, drops the install-time compat patch step (the runtime equivalents in preflight and baseline remain), and adds the native path, but per the description no GPU workload was executed, and ci-e2e has not run on this branch. Before merge, please run at least one synthetic GPU E2E and the shipped GLM-5.2 native example, and post the results.

[haofrank]

Review follow-up: legacy HYPERLOOM_AGENTX=1 optimization is preserved; the native-only refusals now follow explicit native session identity. Server tuning, GEAK handoff/revalidation, source/kernel integration, concurrency sweeps, and legacy resume retain their existing paths.

Both real BaselineExecutor GPU integration validations below passed on a newly allocated exclusive Crusoe 8×MI355X node. The full LLM optimizer E2E has not run: the operator's LLM credential configuration is unavailable. These baseline runs do not close that remaining E2E request, and no optimization gain is claimed.

Workload	Measured revisions	Result
Shipped GLM-5.2 native example, TP4/EP4, CONC8, HiCache + MTP	Hyperloom 2080dadd4; Magpie release 5d6aaa29	286.78509 output tokens/s across 4 GPUs, or 71.69627 tokens/s/GPU; P50 E2E-normalized interactivity 72.13054 tokens/s/user.
Synthetic Qwen3-0.6B BF16, TP1/EP1, CONC64, ISL/OSL 1024/1024	Current Hyperloom 0ef71137f; Magpie hotfix a3339dc	320/320 throughput requests, 15,637.91443 output tokens/s on one GPU; real GSM8K strict-match 43/64 = 67.1875%.

The synthetic retry completed at 2026-09-29 04:03 UTC, with RUN_EVAL=true, Magpie exit code 0, and unchanged Torch/SGLang packages. Its 64 unique sample IDs and per-sample scores reproduce the reported accuracy. This is a 64-example smoke slice, not the full 1,319-example GSM8K evaluation. KV telemetry was unavailable (/metrics returned 404); throughput and accuracy artifacts were present.

The first synthetic attempt completed throughput but failed GSM8K because Magpie changed the working directory after sourcing InferenceX by a relative path. Fixed upstream in Magpie #113, currently draft and unmerged, and pinned the immutable fix in this PR. The v0.3.0 tag remains unchanged. Only ten generic script source blocks changed; native AgentX and shared runner Python code are byte-identical. The new source/wheel execution manifest was independently audited, and the old native manifest remains accepted for resume.

Native accounting caveats: the 3600-second sending window issued 990 requests; 989 completed, one tail request was cancelled, and there were zero recorded request errors. After the 30-second drain and a further 10-second cancellation-credit wait, AIPerf logged an ERROR and forced phase completion. This is not 990/990 completion. The 87 warmup records are excluded. All native schema/protocol, measurement, publishability, and submission-validity checks nevertheless passed. P50 TTFT/TPOT were 539.43/11.79 ms. accuracy=1.0 is the recorded-request error-rate gate, not model-answer accuracy.

The native table uses Magpie/InferenceX's 3606.24392-second completed-record span. AIPerf uses a 3640.00082-second phase window including drain, yielding 284.12549 output tokens/s for the same run. Server metrics emitted counter-reset warnings, so cache/MTP counter statistics are diagnostic. Both parsers report the nonfatal raw_inferencex_result_used provenance marker; neither successful validation relies on salvaging a nonzero process exit.

At current head 0ef71137f32e5775f133426d752f4e0555ec9c0f, the newly installed Magpie hotfix passed native identity/materialization checks, and the current parser reproduced the original native measurement exactly. That is a new preflight/reparse, not a repeated one-hour GPU measurement. Native logs/reports now display per-GPU values using the recorded physical GPU count, while keeping raw scoring values unchanged.

Reproduction and evidence:

• Shipped config: examples/hyperloom-agentx-glm-5.2-mi355x/benchmark.yaml; mounted models: /shared_nfs/models/GLM-5.2-MXFP4 and /shared_nfs/models/Qwen3-0.6B.
• Image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728, digest sha256:a95431225d5079bb398a3f49e7049630f62a2489197fbfd3c5ff84b02f026145.
• InferenceX: 3d5581562f643f9bdeb8410cd924e2c70906c966; AIPerf: 754356e9a39acc6cc6afb242d123bb57c3fb6f75. Magpie release/hotfix: 5d6aaa29f2ee78e53ec35c8bc2e082d8a10c47f0 / a3339dc2776ee0c977fb3313fe89f56da7a91555.
• NFS evidence: /shared_nfs/haoli103/hyperloom-pr1675-validation-20260929, sessions native/executor-session-02 and synthetic/executor-session-02; drivers in native/executor-validation.py and synthetic/executor-validation.py; current native check in native/hotfix-head-preflight-01 and native/hotfix-head-reparse.json.
• Both task containers were removed and the new lease released after verifying no GPU processes remained. NFS evidence is retained.
• Current-head full CI passed: Python 3.10/3.11 each total 24,391 passed / 81 skipped across six shards, with 91.27% coverage. Magpie hotfix CI also passed: 629 passed / 1 skipped on Python 3.10 and 3.12, with 90.16% coverage.

[chaojhou]

@haofrank #1800 and #1802, filed by @mohbasit from MiniMax-M3 runs on this branch, are gaps in the native AgentX profiling path this PR introduces. I checked both at 40a143aa2:

• TraceLens patches dont apply in AgentX profiling session #1800, TraceLens vLLM patch: _managed_profile and prepare_managed_profile never call ensure_vllm_patched_for_tracelens, and the compat leg skips source patches on purpose (_workload_envs.py:2141-2150). vLLM 0.26+ traces then have no graph-capture sidecars, more than 10% of kernel time sits under hipGraphLaunch, and TraceLens falls back to its deterministic analysis.
• kernel_shape_tool doesnt invoke in SGLang during AgentX profiling #1802, SGLang kernel_shape_tool: nothing on the native path puts kernel_shape_tool_dir() on the profile server's PYTHONPATH or sets TRACELENS_SHAPE_DISCOVERY=1, so SGLang 0.5.18+ traces carry no sglang_profiler:: ops and custom kernels have no operand shapes.

Fresh HYPERLOOM_AGENTX=1 sessions default to this path, so both regress AgentX profiling against today's main. Please cover them here, or land them before the default switch:

• kernel_shape_tool doesnt invoke in SGLang during AgentX profiling #1802 changes no framework source, so passing the overlay and the env var into Magpie's profile server spec should be enough.
• TraceLens patches dont apply in AgentX profiling session #1800 patches framework source, which the native path keeps untouched by design, so it needs a decision: for example, patch an isolated copy for diagnostic profiles only, or get graph-capture profiling into vLLM's V2 runner upstream or into Magpie.

[chaojhou]

Closing as superseded by #1922 (merged), which moved AgentX onto Magpie's native contract. Remaining native-path gaps are tracked in #1948 (TraceLens collection) and #1919 (AIPerf reinstall).