[kyle-hoffmeyer]

Summary

flowchart LR
    subgraph BEFORE["Before — requires Hyperloom-patched image"]
        direction TB
        B_1["Coordinator: PRELUDE → roofline task"] --> B_2
        B_2["RooflineExecutor: profile + analyze"] --> B_3
        B_3["ProfileExecutor._after_materialize_config()"] --> B_4{"is_scriptable?"}
        B_4 -->|Yes| B_5["_xdit_patcher.verify_xdit_profiler_baked()"]
        B_5 --> B_6{"Sentinels in\nbase_model.py?"}
        B_6 -->|No — stock image| B_7["⚠ Log warning"]
        B_6 -->|Yes — baked image| B_8["Return True"]
        B_7 --> B_9["return None — skip patching"]
        B_8 --> B_9
        B_9 --> B_10["Magpie xdit_mi300x.sh\n--profile --profile_wait 0"]
        B_10 --> B_11["xDiT base_model.profile()\n✓ trace generated"]
        B_11 --> B_12["RooflineExecutor:\ntrace_analyze_handler()"]
        B_12 --> B_13["Route A: TraceLens (default)\ntracelens_analysis.py\nfind_steady_state_generic()"]
        B_13 --> B_14["TraceLens CLI:\nper-chunk analysis\nkernel roofline"]
        B_14 --> B_15["analysis.md\nkernel_candidates.json\ndiffusion_roofline.json"]
    end

    subgraph AFTER["After — works on any xDiT image"]
        direction TB
        A_1["Coordinator: PRELUDE → roofline task"] --> A_2
        A_2["RooflineExecutor: profile + analyze"] --> A_3
        A_3["_workload_envs.py:\nprofile config materialization"] --> A_4{"framework == xdit?"}
        A_4 -->|Yes| A_5["_server_patcher:\nensure_xdit_patched_for_tracelens()"]
        A_5 --> A_6{"SHA in known\nimage table?"}
        A_6 -->|Yes| A_7["git apply TraceLens patch\nEXTRA_XDIT_ARGS (appended):\n--profile_wait 1\n--profile_capture_phase\n--profile_with_stack"]
        A_6 -->|No| A_8["EXTRA_XDIT_ARGS:\n--profile_wait 1"]
        A_7 --> A_9["Magpie xdit_mi300x.sh\n--profile --profile_wait 1"]
        A_8 --> A_9
        A_9 --> A_10["xDiT base_model.profile()\n✓ trace generated"]
        A_10 --> A_11["RooflineExecutor:\ntrace_analyze_handler()"]
        A_11 --> A_12["Route A: TraceLens (default)\ntracelens_analysis.py\nfind_steady_state_generic()"]
        A_12 --> A_13["TraceLens CLI:\nper-chunk analysis\nkernel roofline"]
        A_13 --> A_14["analysis.md\nkernel_candidates.json\nkernel_roofline.json\ndiffusion_roofline.json"]
    end

    BEFORE ~~~ AFTER

    style B_7 fill:#fee,stroke:#c00
    style A_7 fill:#efe,stroke:#0a0
    style A_8 fill:#efe,stroke:#0a0

Loading

This PR adds support for profiling and analysis for xDiT workloads.

Changes

1. Empty traces on stock xDiT images. Every profiling run on a non-hyperloom-patched xdit docker image was failing. The root cause is an interaction between Magpie's xdit_bench_common.sh (which hardcodes --profile_wait 0) and xDiT's profile() method. The method loops for exactly wait + warmup + active iterations — one full torch.profiler.schedule cycle. With wait=0, the profiler schedule action at each step and the state after the final step() call look like this:

   Broken: wait=0, warmup=2, active=1 (3 iterations)

   iter 0:  WARMUP          ─── step() ──→  iter 1 action: WARMUP
   iter 1:  WARMUP          ─── step() ──→  iter 2 action: RECORD_AND_SAVE
   iter 2:  RECORD_AND_SAVE ─── step() ──→  next action: WARMUP  ← cycle 2 starts
                                             transition (R&S → WARMUP) calls:
                                               stop_trace, _trace_ready, prepare_trace
                                             ⚠ prepare_trace creates a NEW profiler
                                               object (self.profiler = prof.profile(...)),
                                               wiping all recorded events
                                             __exit__() → export_chrome_trace() exports
                                               from the empty new profiler → 0 events
   

   The mechanism: the profiler defines transition callbacks for each (prev_action, current_action) pair. The (RECORD_AND_SAVE → WARMUP) transition includes prepare_trace, which replaces self.profiler with a brand new prof.profile(...) object — clearing all accumulated events. The (RECORD_AND_SAVE → NONE) transition does NOT call prepare_trace, so the profiler object retains its data. When xDiT later calls export_chrome_trace(), it exports from whichever profiler object is current — empty (broken) or full (fixed).
   There are two configurations that fix this issue:

   Fix 1: wait=1, warmup=2, active=1 (4 iterations)

   iter 0:  NONE            ─── step() ──→  iter 1 action: WARMUP
   iter 1:  WARMUP          ─── step() ──→  iter 2 action: WARMUP
   iter 2:  WARMUP          ─── step() ──→  iter 3 action: RECORD_AND_SAVE
   iter 3:  RECORD_AND_SAVE ─── step() ──→  next action: NONE   ← idle
                                             transition (R&S → NONE) calls:
                                               stop_trace, _trace_ready
                                             ✓ NO prepare_trace — profiler object
                                               and its recorded events are preserved
                                             __exit__() → export_chrome_trace() exports
                                               from the original profiler → 143,748 events
   

   Fix 2: wait=0, warmup=2, active=1, repeat=1 (3 iterations)

   The previous Hyperloom bake patched repeat=1 into xDiT's torch.profiler.schedule() call. With repeat=1, the schedule returns NONE for all steps after the first cycle completes, so the post-active transition is (RECORD_AND_SAVE → NONE) regardless of wait:

   iter 0:  WARMUP          ─── step() ──→  iter 1 action: WARMUP
   iter 1:  WARMUP          ─── step() ──→  iter 2 action: RECORD_AND_SAVE
   iter 2:  RECORD_AND_SAVE ─── step() ──→  next action: NONE   ← repeat=1, cycle done
                                             transition (R&S → NONE) calls:
                                               stop_trace, _trace_ready
                                             ✓ NO prepare_trace — data preserved
                                             __exit__() → export_chrome_trace() → 143,748 events
   

   Both fixes work by ensuring the post-RECORD_AND_SAVE transition lands on NONE instead of WARMUP. Fix 1 is preferred because --profile_wait is a CLI argument passed via the EXTRA_XDIT_ARGS environment variable. On the other hand, --profile_repeat is not a CLI argument and thus Fix 2 requires patching repeat=1 into xDiT's base_model.py, which is the dependency this PR eliminates.

2. Unnecessary dependency on a Hyperloom-specific Docker image overlay. A verifier module (_xdit_patcher.py) required the pytorch-xdit:v26.6-hyperloom15 image tag, which baked two patches into xDiT's base_model.py at build time. The first patch was the repeat=1 patch explained above and the second patch added per-step annotations. Neither patch was needed: repeat=1 is unnecessary when --profile_wait 1 is set, and the denoise_step_N per-step annotations were never consumed by any production code. Hyperloom assumed the hyperloom-patched docker image was being used. _xdit_patcher.py existed to verify that the patched image was being used. Since we're no longer using this patched image, _xdit_patcher.py and test_xdit_patcher.py have been deleted.

3. TraceLens xDiT patches now applied at runtime. TraceLens maintains version-specific xDiT patches that add capture-phase profiling support for models that support graph-mode execution.
   These patches are now applied at runtime to rocm xDiT images instead of requiring pre-patched images. In _server_patcher.py, xDiT patching support has been added following the convention of vllm/sglang.

• SHA detection reads git rev-parse HEAD from the xDiT checkout (via _git_head_sha from _git.py) to determine the patch to apply. xDiT is different from vllm and sglang in that it is not a library. There is no reliable way of getting the xDiT version and correlating it back to the version of the rocm docker image release. Furthermore, there is no reliable way of getting the docker image information from within a container. However, each xDiT image release is tied to a specific commit in xDiT:

Image	SHA	git describe
rocm/pytorch-xdit:v26.7	c30266378c815c2f31f95b9d21d2818276050547	0.4.5-118-gc302663
rocm/pytorch-xdit:v26.8	2f74acb12830163d1fe0dd1abb5422f0bff7acf0	0.4.5-134-g2f74acb

• _discover_xdit_plan(tracelens_root) reads the HEAD SHA, matches it with the image via this table and applies the correct patch.
• Similar to vllm/sglang, helper functions ensure_xdit_patched_for_tracelens and _resolve_xdit_patch_file have been created. These function similarly to vllm/sglang
• When the xDiT SHA is not in the known-image table the patch is skipped, the profiler set degrades to --profile_wait 1 only, and this is surfaced to the Coordinator via HYPERLOOM_TRACELENS_PATCH_STATUS (ok / unavailable / not_attempted) and HYPERLOOM_PROFILE_DEGRADED_REASON.

4. xDiT profiler flags now set through _workload_envs.py. vLLM and SGLang profiler flags (EXTRA_VLLM_ARGS, EXTRA_SGLANG_ARGS) are set dynamically in _workload_envs.py during profile config materialization. xDiT was excluded from this path — profiler flags were supplied by Magpie's xdit_mi300x.sh launcher and Hyperloom never set them in-repo. This meant there was no place to conditionally set flags based on runtime state (whether a patch was successfully applied).

5. infer_analysis_mode now includes xDiT. tracelens_skill_runner.py:infer_analysis_mode() resolves the TraceLens analysis mode from the framework name. It defaulted vLLM, SGLang, and Atom to "inference" mode, but xDiT fell through to "default", causing xDiT to use the non-inference TraceLens analysis script. Only the inference variant accepts --capture_folder for graph-capture merging. xDiT has been added to the list of inference frameworks so that the inference analysis script is invoked.

6. A model name is now required for xDiT runs. xDiT uses a model registry to dispatch model names to pipeline classes. Unlike vLLM/SGLang, xDiT has multiple pipelines for different models and requires the model name in order to choose the correct pipeline. Previously, the benchmark script extracted the model name via basename of the model path, which is very error prone as this requires the model path to be in a specific format. Unfortunately, there is no reliable way to extract the model name from the HuggingFace config. A new --xdit-model-name CLI flag is thus required to pass the xDiT registry name explicitly. The value is exported as XDIT_MODEL_NAME and flows through _workload_envs.py into the benchmark YAML's benchmark.envs, where Magpie's shell script ${XDIT_MODEL_NAME:-$(basename ...)} picks it up. The flag is persisted in state.xdit_model_name for resume. XDIT_MODEL_ARG (which controlled the basename vs path derivation strategy) has been removed from _workload_envs.py as it is no longer needed.
   The requirement is satisfiable by either the --xdit-model-name flag or an already-exported XDIT_MODEL_NAME (and is restored from state.xdit_model_name on resume); only the absence of both fails launch. Consistent with the other operator-supplied overrides in the same CLI path (FRAMEWORK_REPO_PATH, HYPERLOOM_BYPASS_SCRIPTS_DIR), an exported XDIT_MODEL_NAME outranks the flag — a deliberate shell export is authoritative and a stale flag in a launcher cannot silently override it.

Baseline raises

All raises are from adding xDiT as a supported framework — the same fields, flags, and code paths that vLLM/SGLang/ATOM already have.

• _server_patcher.py module lines (1303, ceiling 1200): xDiT is the third framework _server_patcher.py supports. The 103-line increase is the xDiT discovery and patching functions (_xdit_default_source_root, ensure_xdit_patched_for_tracelens, _resolve_xdit_patch_file, _discover_xdit_plan), following the same structure as the existing vLLM and SGLang paths. Splitting xDiT out alone would be inconsistent.
• _workload_envs.py module lines (2192) and materialize_config_with_envs function lines (924) / cyclomatic (+2) / cognitive (+2): the xDiT profiler-flag block and the _record_tracelens_patch_status helper extraction.
• cli/__init__.py module lines (2805): --xdit-model-name flag handling, _apply_operator_supplied_paths xDiT guard, and _restore/_persist paths — same pattern as the sibling FRAMEWORK_REPO_PATH / HYPERLOOM_BYPASS_SCRIPTS_DIR overrides.
• parser.py::_build_parser function lines (987): the --xdit-model-name argparse entry.
• shared_state.py module lines (2622): the xdit_model_name field (+2 lines).

[github-actions]

Code metrics gate: FAILED with waived findings (baseline-raise)

Judged: the 17 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Worse	Baseline out of date	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	1	0	45	+0
Cognitive complexity	fails > 30	0	1	0	170	+0
Function length (lines)	fails > 80	0	3	0	315	+0
Nested block depth	fails > 5	0	0	0	19	+0
Parameter count	fails > 5	0	0	0	89	+0
Module length (lines)	warns > 800, fails > 1200	1	4	0	47	+0
Duplicated lines per file	fails > 0	0	0	0	96	+0
Dead code (vulture)	fails > 0	0	0	0	2	+0

Check	Findings	Effect
Comments	0
English only	0
Production code imports test code	0
Repeated literals	1	fails
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	0

New violations (not in the baseline) (waived by the baseline-raise label): 1

Unit	Dimension	Value	Limit
src/hyperloom/orchestrator/actions/executors/_server_patcher.py:1	Module length (lines)	1303	fails > 1200

Worse than the baseline (waived by the baseline-raise label): 9

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/actions/executors/_workload_envs.py:1263 materialize_config_with_envs	Cognitive complexity	285	287
src/hyperloom/orchestrator/actions/executors/_workload_envs.py:1263 materialize_config_with_envs	Cyclomatic complexity	109	111
src/hyperloom/inference_optimizer/cli/bootstrap.py:504 _seed_shared_state	Function length (lines)	155	156
src/hyperloom/inference_optimizer/cli/parser.py:179 _build_parser	Function length (lines)	961	967
src/hyperloom/orchestrator/actions/executors/_workload_envs.py:1263 materialize_config_with_envs	Function length (lines)	915	924
src/hyperloom/inference_optimizer/cli/__init__.py:1	Module length (lines)	2569	2771
src/hyperloom/orchestrator/actions/executors/_workload_envs.py:1	Module length (lines)	2158	2192
src/hyperloom/orchestrator/state/shared_state.py:1	Module length (lines)	2549	2551
src/hyperloom/orchestrator/trace_analysis/tracelens_skill_runner.py:1	Module length (lines)	1322	1326

Repeated literals: 1

Where	Problem
src/hyperloom/orchestrator/actions/executors/_workload_envs.py:1927	'xdit' is written 3 times (was 1); name it as a module-level constant

How to fix

• New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
• Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
• Repeated literals: name the value once as a module-level constant and use the name.
• Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
• Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 2

Module	Lines	Limit
src/hyperloom/inference_optimizer/cli/bootstrap.py	931	warns > 800
src/hyperloom/inference_optimizer/cli/parser.py	1145	warns > 800

Already over at the base (backlog, not this change's): 5

Unit	Dimension	Base	Now
src/hyperloom/orchestrator/state/shared_state.py:96 resolve_graded_comparison	Cognitive complexity	34	34
src/hyperloom/inference_optimizer/cli/__init__.py:1805 _reexport_session_environment	Function length (lines)	107	107
src/hyperloom/inference_optimizer/cli/__init__.py:2331 _resolve_agent_lineup	Function length (lines)	89	89
src/hyperloom/inference_optimizer/cli/__init__.py:2226 _resolve_run_topology	Function length (lines)	92	92
src/hyperloom/inference_optimizer/cli/__init__.py:2438 _run_optimize	Function length (lines)	279	279

84 findings in files this change does not touch

Outside this change (informational): 84

Unit	Dimension	Baseline	Now
src/hyperloom_kb/collect/mapping.py _experience	Cognitive complexity	absent	31
src/kernelforge/cli/forge_loop_command.py forge_loop	Cognitive complexity	absent	31
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate	Cognitive complexity	absent	44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt	Cognitive complexity	absent	38
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _build_parser	Function length (lines)	absent	84
src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py _spawn_rank_actors	Function length (lines)	absent	84
src/hyperloom_kb/sync.py GlobalSync.pull	Function length (lines)	absent	84
src/kernelforge/loop/runner.py IterationLoop._close_iteration	Function length (lines)	absent	117
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate	Function length (lines)	absent	185
src/kernelforge/loop/runner.py IterationLoop._implementer_history	Function length (lines)	absent	112
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt	Function length (lines)	absent	102
src/kernelforge/loop/runner.py IterationLoop._source_attempt	Function length (lines)	absent	175
src/hyperloom_kb/http_service.py	Module length (lines)	absent	1525
src/kernelforge/cli/forge_loop_command.py	Module length (lines)	absent	2003
src/hyperloom/orchestrator/actions/executors/_geak_sweep.py sweep_via_geak	Cognitive complexity	67	70
src/hyperloom/orchestrator/actions/executors/_grid_runner.py run_grid	Cognitive complexity	177	180
src/hyperloom/orchestrator/actions/executors/explore.py ExploreExecutor.__call__	Cognitive complexity	369	383
src/hyperloom/orchestrator/kernel/request_handlers.py _run_forge_gemm_tuning	Cognitive complexity	124	125
src/hyperloom/orchestrator/phases/kernel.py KernelPhase._run_geak_kernel_phase	Cognitive complexity	122	130
src/hyperloom/orchestrator/phases/prelude.py PreludePhase.maybe_enqueue_warm_replay	Cognitive complexity	190	191
src/hyperloom/orchestrator/actions/executors/_grid_runner.py run_grid	Cyclomatic complexity	55	56
src/hyperloom/orchestrator/actions/executors/explore.py ExploreExecutor.__call__	Cyclomatic complexity	76	83
src/hyperloom/orchestrator/kernel/request_handlers.py _run_forge_gemm_tuning	Cyclomatic complexity	56	57
src/hyperloom/orchestrator/phases/kernel.py KernelPhase._run_geak_kernel_phase	Cyclomatic complexity	49	52
src/hyperloom/orchestrator/phases/prelude.py PreludePhase.maybe_enqueue_warm_replay	Cyclomatic complexity	48	49

...and 59 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

[kyle-hoffmeyer]

/review

[github-actions]

Automated review started for head 7a37a9f. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38467019785

[github-actions]

PR #1762 -- xDiT profiling and TraceLens analysis support

What it does: xDiT profiling runs on stock rocm/pytorch-xdit images produced empty traces because Magpie's launcher hardcodes --profile_wait 0, whose torch-profiler schedule transition wipes the recorded events; the PR passes --profile_wait 1 through EXTRA_XDIT_ARGS set at config materialization, replaces the deleted baked-image verifier with SHA-gated runtime patching in _server_patcher (following the vllm/sglang convention), routes xDiT to TraceLens's inference analysis mode, and makes --xdit-model-name a required CLI flag persisted for resume.

Blocking issues: 5

1. [C1] --xdit-model-name resume path is dead: the apply gate rejects the value the restore path just exported [verified]
   Problem: _restore_operator_supplied_paths_from_state (src/hyperloom/inference_optimizer/cli/init.py:229) exports os.environ["XDIT_MODEL_NAME"] from the persisted state.xdit_model_name, then _apply_operator_supplied_paths (src/hyperloom/inference_optimizer/cli/init.py:196) checks only args.xdit_model_name, finds it empty and exits 2 — unlike the custom-framework path in the same function, which accepts the exported env (or export FRAMEWORK_REPO_PATH). Reproduced: restore-then-apply with state.xdit_model_name="FLUX.2-dev" prints the re-exported env and then exits 2; an operator-exported XDIT_MODEL_NAME without the flag fails the same way.
   Impact: resuming any xDiT session without re-passing --xdit-model-name aborts at launch with "required for xDiT runs", so the persisted-for-resume field the PR adds can never be used, and the env-var configuration route is refused.
   Action: have the xDiT branch of _apply_operator_supplied_paths consult os.environ.get("XDIT_MODEL_NAME") when the flag is absent, as the custom path does for its flags, and add a resume-without-flag test.

2. [C3] the roofline_in_flight PRELUDE hold duplicates a hold the file already owns, and its test passes on the merge base [verified]
   Problem: the branch at src/hyperloom/phases/machine_state.py:1830 re-implements _holds_for_initial_analysis (machine_state.py:1722): both read the same state field auto_roofline_pending_task_id (projected as initial_analysis_pending at :1253 and again as roofline_in_flight at :1260) with the same affordable is None or affordable > 0.0 gate, and the pre-existing check at :1817 returns None first, so the new branch is unreachable. Reproduced on the merge base 1a444f6: with auto_roofline_pending_task_id set, compute_next_phase already returns None, and the new test test_prelude_blocked_while_roofline_in_flight passes unchanged. The change also adds an roofline_in_flight key to persisted predicate_inputs in both session_breakdown schema copies and a comment block narrating it — none of which the PR description states.
   Impact: a second copy of a decision the module already owns (the merge from main brought the original in), a schema field that changes no behaviour, a regression test that pins nothing, and the cyclomatic raise of replay_next_phase (30→31) that is part of the failing code-metrics run.
   Action: delete the machine_state.py:1828-1831 block, the roofline_in_flight key at :1260, both schema additions and the new test — or, if a distinct signal is wanted, say what distinguishes it from initial_analysis_pending and state the change in the description.

3. [S3] a skipped xDiT patch degrades profiling with no operator-visible signal, diverging from the vllm/sglang convention [verified]
   Problem: when _discover_xdit_plan skips (SHA not in the two-entry _XDIT_SHA_TO_VERSION table, source root or TRACELENS_ROOT missing — src/hyperloom/orchestrator/actions/executors/_server_patcher.py:446/451/457/481), _workload_envs.py:1556 falls back to --profile_wait 1 only, dropping --profile_capture_phase --profile_with_stack; no HYPERLOOM_TRACELENS_PATCH_STATUS or HYPERLOOM_PROFILE_DEGRADED_REASON is written — the vllm/sglang block in the same function writes the status in every branch for exactly this reason (_workload_envs.py:1659-1672) — and the skip paths log at info. The instrumentation preflight (profile.py:145-175), the only check that carries this signal, therefore reports "passed"; the deleted patchers["xdit_profiler_baked"] record was the last xDiT entry it could fail on.
   Impact: the first run on an rocm/pytorch-xdit image newer than v26.8 (the table's two entries) silently loses capture-phase profiling: the trace and analysis still run, so the degradation surfaces later as missing capture/annotation data with a preflight that said instrumentation was fine. The new _server_patcher xDiT logic also has no test file, where vllm and sglang each have one.
   Action: write HYPERLOOM_TRACELENS_PATCH_STATUS (and the degraded reason on failure) from the xDiT branch as the sibling branch does, raise the unknown-SHA skip to the same warning level the sibling uses for a failed patch, and add a _server_patcher xDiT plan-discovery test alongside the vllm/sglang ones.

4. [X2] the description's change 4 claims the profiler flags "were set statically in profile_xdit.yaml" — they never were [verified]
   Problem: body.txt change 4 states xDiT profiler flags were set statically in profile_xdit.yaml; at the merge base that file sets no EXTRA_XDIT_ARGS (it is byte-identical base vs head and untouched by this PR) and nothing in the repo set the variable — the PR's own BEFORE diagram correctly attributes the flags to Magpie's xdit_mi300x.sh.
   Impact: a config-value claim in the description that fails verification against the merge base distorts the change's motivation for every later reader (there was no static in-repo setting to move; the flags were only ever assembled out of tree).
   Action: correct change 4 to say the flags were supplied by Magpie's script and Hyperloom never set them, so the dynamic _workload_envs path is where they first become controllable in-repo.

5. [X3] infer_analysis_mode docstring still lists the inference frameworks without xDiT [verified]
   Problem: the docstring at src/hyperloom/orchestrator/trace_analysis/tracelens_skill_runner.py:313 says "inference frameworks (vllm/sglang/atom) default to inference", one line above the set at :327 that this diff changed to {"vllm", "sglang", "atom", "xdit"}.
   Impact: the next reader of the function the PR changed takes the docstring's membership as the contract and misses that xDiT traces now group as inference.
   Action: add xdit to the docstring's framework list in the same edit that adds it to the set.

Checked: machine_state.py replay_next_phase/_holds_for_initial_analysis/_prelude_predicate_inputs (base and head), cli/init.py apply/restore/persist paths, _workload_envs.py xdit block and vllm/sglang patch block nesting, _server_patcher.py xdit discovery + _ensure_patched/_apply_atomic, profile.py instrumentation preflight, tracelens_skill_runner.py infer_analysis_mode, SharedState.from_dict tolerance, profile_xdit.yaml base-vs-head; tree-wide greps for roofline_in_flight, XDIT_MODEL_NAME, XDIT_MODEL_ARG, EXTRA_XDIT_ARGS, verify_xdit_profiler_baked, HYPERLOOM_TRACELENS_PATCH_STATUS, HYPERLOOM_PROFILE_DEGRADED_REASON | Ran: restore-then-apply reproduction (SystemExit 2, both flag-absent and env-exported variants); new phase test's scenario executed against base-tree src via PYTHONPATH (returns None / prelude_done — passes on base); no test suite run | Base: 1a444f6 | Head: 7a37a9f

[kyle-hoffmeyer]

/review

[github-actions]

Automated review started for head 2d30c11. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38489157888

[github-actions]

PR #1762 -- xDiT profiling and TraceLens analysis support

What it does: xDiT profiling runs on stock rocm/pytorch-xdit images produced empty traces because Magpie's launcher hardcodes --profile_wait 0, whose torch-profiler schedule transition wipes the recorded events; this PR passes --profile_wait 1 (plus capture-phase flags when the TraceLens runtime patch applies) through EXTRA_XDIT_ARGS set at config materialization, replaces the deleted baked-image verifier with SHA-gated runtime patching in _server_patcher following the vllm/sglang convention, routes xDiT to TraceLens's inference analysis mode, and makes --xdit-model-name a required CLI flag persisted for resume.

Blocking issues: 5

1. [free:xdit-patch-kill-switch] The xDiT patch branch ignores the HYPERLOOM_ENABLE_PATCH kill switch [verified]
   Problem: the new branch at src/hyperloom/orchestrator/actions/executors/_workload_envs.py:1552 calls ensure_xdit_patched_for_tracelens() unconditionally, while the sibling vllm/sglang branch in the same function gates on patch_attempted = _tracelens_patch_enabled() and ... (:1664); reproduced with HYPERLOOM_ENABLE_PATCH=0 — the patcher is still invoked, and on a real image it git applys into the /app/xDiT checkout and injects --profile_capture_phase --profile_with_stack, exactly what profile_vllm.yaml:35 documents the switch disables ("disable runtime patching entirely").
   Impact: an operator who disables runtime patching still gets the xDiT checkout mutated at profile time, with no log line or status distinguishing it from the switch being on — the sibling writes not_attempted, the xDiT branch writes ok.
   Action: gate the xDiT branch on _tracelens_patch_enabled() as the sibling does, writing the not_attempted status (and safe flags only) when it is off, and name the switch's widened scope in the description.
2. [V5] The code-metrics gate fails at the current head with new violations this diff introduces [verified]
   Problem: the code-metrics check failed at head 2d30c11 (ci.txt); the new violations are all in files this PR touches — _server_patcher.py crosses the 1200-line module ceiling (1320, +137 lines from this diff), parser.py crosses it (1209), materialize_config_with_envs rises to cyclomatic 114 / 941 lines, and 'xdit_model_name' is written three times as a literal in cli/init.py:197/233/234 — and the description claims no baseline-raise reason for any of them.
   Impact: the PR cannot merge with a red gating check; the repeated literal is a design-check failure independent of the label.
   Action: split the new xDiT patcher discovery out of _server_patcher.py (or bring the module back under 1200), bring parser.py back under the ceiling, and name the xdit_model_name constant once — or add the baseline-raise label with the reason stated in the PR description.
3. [T4] The patch gate's crossing cases have no test [verified]
   Problem: the xDiT branch adds a gate (ensure_xdit_patched_for_tracelens() outcome) together with a runtime fallback (EXTRA_XDIT_ARGS="--profile_wait 1" only plus the degraded reason), and no test drives either case: nothing in the test tree asserts any EXTRA_XDIT_ARGS value or the status envs for a patched or unpatched xDiT profile, and no test touches _discover_xdit_plan/_resolve_xdit_patch_file, where each sibling patcher has dedicated test files (test_server_patcher_vllm_version_fallback.py, test_server_patcher_sglang_version_fallback.py); a test asserting the fallback flags fails on the merge base.
   Impact: a regression to the pre-PR shape — the empty trace on a stock image, or patch-only flags on an unpatched build — fails no test, and the SHA-table/nearest-lower selection this PR adds is unpinned.
   Action: add a _server_patcher xDiT test (known SHA → patch applied; unknown SHA → skip; missing patch file → nearest-lower) and a materialize test asserting EXTRA_XDIT_ARGS and the status envs for both patcher outcomes.
4. [X2] The description's change 4 claims the profiler flags "were set statically in profile_xdit.yaml" — they never were [verified]
   Problem: body.txt change 4 states xDiT profiler flags were set statically in profile_xdit.yaml; at the merge base that file sets no EXTRA_XDIT_ARGS (it is byte-identical base vs head and untouched by this PR) and nothing in the base tree set the variable — the PR's own BEFORE diagram correctly attributes the flags to Magpie's script.
   Impact: a config-value claim in the description that fails verification against the merge base distorts the change's motivation for every later reader (there was no static in-repo setting to move; the flags were only ever assembled out of tree).
   Action: correct change 4 to say the flags were supplied by Magpie's launcher and Hyperloom never set them, so the _workload_envs path is where they first become controllable in-repo.
5. [X3] ensure_xdit_patched_for_tracelens docstring says unknown SHAs get the newest patch; the code skips them [verified]
   Problem: the docstring added at src/hyperloom/orchestrator/actions/executors/_server_patcher.py:383 says "Unknown SHAs try the newest available patch via nearest-lower fallback", but _discover_xdit_plan (:455-462) returns None for any SHA not in the two-entry _XDIT_SHA_TO_VERSION table, as its own docstring states ("Unknown SHAs are skipped"); the nearest-lower fallback runs only for a known version whose exact patch file is missing.
   Impact: the next maintainer debugging a degraded run on a new image reads the public entry point's docstring and expects a fallback apply that never happens.
   Action: correct the docstring to say unknown SHAs are skipped and the nearest-lower fallback applies only to a known version with no exact patch file.

deferred: D10 -- _server_patcher.py:54 hardcodes _XDIT_DEFAULT_SOURCE_ROOT = Path("/app/xDiT"), re-stating the per-framework source root framework_registry.py:66 already owns and exports (source_root("xdit"), consumed by framework_paths.py:49/314); two homes for the same path diverge silently (registry updated, patcher skips with an info log and loses capture-phase profiling). Reportable under D10 (AGENTS.md Clean design), but latent rather than current wrong behaviour, and five findings rank ahead of it.

Checked: cli/init.py apply/restore/persist paths and call order (:2081-2087), parser.py --xdit-model-name, shared_state.py from_dict tolerance, _workload_envs.py xdit profile branch and vllm/sglang patch block, _server_patcher.py xDiT discovery + shared _PatchPlan/_ensure_patched/_apply_atomic core, profile.py instrumentation preflight, tracelens_skill_runner.infer_analysis_mode, profile_xdit.yaml base-vs-head, SKILL.md flag table; tree-wide greps for XDIT_MODEL_NAME, xdit_model_name, EXTRA_XDIT_ARGS, HYPERLOOM_TRACELENS_PATCH_STATUS, HYPERLOOM_PROFILE_DEGRADED_REASON, HYPERLOOM_ENABLE_PATCH, XDIT_MODEL_ARG, verify_xdit_profiler_baked, source_root | Ran: kill-switch repro via materialize_config_with_envs with the patcher stubbed (HYPERLOOM_ENABLE_PATCH=0/1, patcher invoked both times); infer_analysis_mode driven directly for 6 shapes; pytest test_custom_framework.py -k xdit (2 passed), test_workload_envs.py -k xdit (2 passed); ruff C901 head-vs-base on the changed non-test files (materialize_config_with_envs 111/111); test_tracelens_csv.py blocked by the pre-existing out-of-tree TraceLens import, function driven directly instead | Base: 1a444f6 | Head: 2d30c11

[kyle-hoffmeyer]

/review

[github-actions]

Automated review started for head 395fcff. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/38654169220

[github-actions]

PR #1762 -- xDiT profiling and TraceLens analysis support

What it does: xDiT profiling on stock rocm/pytorch-xdit images produced empty traces because Magpie's launcher hardcodes --profile_wait 0, whose torch-profiler schedule transition wipes the recorded events; the PR injects --profile_wait 1 (plus --profile_capture_phase --profile_with_stack when a TraceLens runtime patch applies) through EXTRA_XDIT_ARGS set at profile config materialization, replaces the deleted baked-image verifier with SHA-gated runtime patching in _server_patcher following the vllm/sglang convention, routes xDiT to TraceLens's inference analysis mode, and makes --xdit-model-name (or an exported XDIT_MODEL_NAME) a launch requirement persisted for resume.

Blocking issues: 3

1. [V5] the PR conflicts with main and the code-metrics gate fails at the current head [verified]
   Problem: meta.txt reports mergeable: CONFLICTING (re-confirmed via gh pr view: DIRTY at head 395fcff), and the code-metrics gate that failed at 2d30c11 fails again at this head: src/hyperloom/inference_optimizer/cli/parser.py is 1209 lines and src/hyperloom/orchestrator/actions/executors/_server_patcher.py is 1321, both over the 1200-line module ceiling with no baseline entry (scripts/code_metrics_baseline.txt moves only profile.py entries), beside worse-than-baseline rows this diff adds (materialize_config_with_envs cyclomatic 111→113, cli/__init__.py module 2569→2760) — with no baseline-raise label on the PR.
   Impact: the PR cannot merge: it needs a rebase, and the gating check is red on violations the diff itself introduces.
   Action: rebase on main, then split the xDiT plan discovery out of _server_patcher.py (or bring both modules back under 1200 lines) — or add the baseline-raise label with the reason stated in the PR description.

2. [X2] the description's change 4 claims xDiT profiler flags "were set statically in profile_xdit.yaml" — they never were [verified]
   Problem: body.txt change 4 states the flags were set statically in profile_xdit.yaml; at the merge base that file sets no EXTRA_XDIT_ARGS (it is byte-identical base vs head and untouched by this PR) and nothing in the base tree set the variable — the PR's own BEFORE diagram correctly attributes the flags to Magpie's xdit_mi300x.sh.
   Impact: a config-value claim that fails verification against the merge base distorts the change's motivation for every later reader: there was no static in-repo setting to move, and the _workload_envs path is where the flags first become controllable in-repo.
   Action: correct change 4 to say the flags were supplied by Magpie's launcher and Hyperloom never set them.

3. [D10] _XDIT_DEFAULT_SOURCE_ROOT duplicates the xDiT source root framework_registry already owns [verified]
   Problem: src/hyperloom/orchestrator/actions/executors/_server_patcher.py:54 hardcodes Path("/app/xDiT"), re-stating the root src/hyperloom/inference_optimizer/framework_registry.py:66 carries and exports as source_root("xdit") (framework_registry.py:131); nothing prevents the import (the same package's _workload_envs.py already imports framework_registry), and if the registry's root is ever updated the patcher silently skips patching on the stale path with only an info log (AGENTS.md Clean design: "duplicated state or a duplicated decision is the same problem: name the owner it derives from").
   Impact: two homes for the xDiT source root diverge silently; the degradation surfaces later as HYPERLOOM_TRACELENS_PATCH_STATUS=unavailable with no hint that a stale path copy is the cause, and capture-phase profiling is lost.
   Action: derive the default from the owner — Path(framework_registry.source_root("xdit") or "/app/xDiT") — and drop the literal.

SKIPPED: V4 -- openprs.txt lists 5 # PARTIAL: lines (#1276, #1675, #1032, #1869, #1154; 1311/161/1075/269/1232 changed files, only the first 100 compared each) — overlap with those PRs is unchecked, not absent.

Checked: cli/init.py apply/restore/persist and call order (2085-2086, 2265), parser.py --xdit-model-name, _workload_envs.py xdit profile branch, _record_tracelens_patch_status and the vllm/sglang block nesting (not _is_scriptable_profile gate, tracelens_patch_ok init at 1689), _server_patcher.py xDiT discovery + shared _PatchPlan/_ensure_patched/_apply_atomic (md5-compared to base), profile.py instrumentation preflight, tracelens_skill_runner.infer_analysis_mode and its exec_mode consumer, shared_state.from_dict tolerance, profile_xdit.yaml base-vs-head, SKILL.md flag row, install_kernel_tools.sh + tracelens_analysis.py pin pair; tree-wide sweeps (src, docs, scripts) for XDIT_MODEL_NAME, xdit_model_name, XDIT_MODEL_ARG, EXTRA_XDIT_ARGS, HYPERLOOM_TRACELENS_PATCH_STATUS, HYPERLOOM_PROFILE_DEGRADED_REASON, verify_xdit_profiler_baked, _xdit_patcher, pytorch-xdit, /app/xDiT, 6489fbc/ae16eacb | Ran: pytest test_workload_envs_branches_unit.py -k xdit (5 passed), test_custom_framework.py -k "xdit or model_name" (3 passed), test_workload_envs.py (66 passed), test_workload_envs_golden_lock.py (10 passed), branches_unit + custom_framework full (104 passed); the five new xdit tests against base-tree/src from a neutral directory (all fail: AttributeError ensure_xdit_patched_for_tracelens); ruff C901 max-complexity=20 head vs base on the changed files (materialize_config_with_envs 111→113, no new function over 20); wc -l vs the 1200 ceiling; infer_analysis_mode driven directly for 8 shapes; gh api to TraceLens xdit_patches at the pinned SHA; gh pr view mergeable/labels; no full-suite run | Base: 1a444f6 | Head: 395fcff

[kyle-hoffmeyer]

/review

[github-actions]

Automated review started for head 3964130. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/39324945236

[github-actions]

PR #1762 -- xDiT profiling and TraceLens analysis support

What it does: xDiT profiling on stock rocm/pytorch-xdit images produced empty traces because Magpie's launcher hardcodes --profile_wait 0, whose torch-profiler schedule transition wipes the recorded events; the PR passes --profile_wait 1 (plus capture-phase flags when a TraceLens runtime patch applies) through EXTRA_XDIT_ARGS at profile config materialization, replaces the deleted baked-image verifier with SHA-gated runtime patching in _server_patcher following the vllm/sglang convention, routes xDiT to TraceLens's inference analysis mode, and makes --xdit-model-name (or an exported XDIT_MODEL_NAME) a launch requirement persisted for resume.

Blocking issues: 3

1. [C1] --xdit-model-name is never persisted on a fresh session, so the resume path the PR adds cannot work [verified]
   Problem: the only writer of state.xdit_model_name is _persist_operator_supplied_paths (src/hyperloom/inference_optimizer/cli/init.py:306), and its sole call site (:1923) is _reexport_session_environment — the resume path. The fresh-session seeder bootstrap.py::_seed_shared_state persists the sibling fields (bypass_scripts_dir, framework_repo_path, benchmark_backend, src/hyperloom/inference_optimizer/cli/bootstrap.py:538-540) but never seeds xdit_model_name, so a fresh xDiT session persists "". _restore_operator_supplied_paths_from_state (cli/init.py:251-257) then has nothing to export, and _apply_operator_supplied_paths exits 2 with "--xdit-model-name is required for xDiT runs". Reproduced: fresh launch with the flag exports the env and seeds state.xdit_model_name=""; a fresh-shell resume without the flag raises SystemExit 2.
   Impact: every resume of a fresh xDiT session in a new shell aborts at launch unless the operator re-passes the flag or re-exports the env — the documented "restored from state.xdit_model_name on resume" never happens, so the field the PR adds is dead weight and the resume requirement is silently stricter than the description states.
   Action: seed xdit_model_name=os.environ.get("XDIT_MODEL_NAME", "").strip() in bootstrap.py::_seed_shared_state beside the sibling fields at :538-540 (after _apply_operator_supplied_paths has exported it at launch), and add a resume-without-flag test that fails until the seeder writes it.

2. [X2] the description's change 7 claims a TraceLens pin bump that the diff does not contain [verified]
   Problem: body.txt change 7 states "TraceLens pin bumped to ae16eacb3d31857486a99439ad72824d5274428d"; at head ae16eacb occurs nowhere in the tree and both pin sites still carry the base value 647cbf2702687a656f2480e07517824369898aa9 (src/hyperloom/inference_optimizer/assets/install_kernel_tools.sh:96 and src/hyperloom/orchestrator/trace_analysis/tracelens_analysis.py:3193, whose comment says "Kept in sync with install.sh"). Checked against the TraceLens repo: at the still-pinned SHA, xdit_patches/ holds only config_xdit_v26.6.patch and config_xdit_v26.7.patch; the v26.8 patch exists only at ae16eacb.
   Impact: a fresh install self-heals TraceLens to a SHA without config_xdit_v26.8.patch, so on rocm/pytorch-xdit:v26.8 — the newest image in _XDIT_SHA_TO_VERSION (2f74acb... → "26.8") — _resolve_xdit_patch_file falls back to the v26.7 patch for a v26.8 checkout, and capture-phase profiling either applies a mismatched patch or degrades to --profile_wait 1 only; the description also misstates the diff for every later reader.
   Action: commit the pin bump in both places at once (install_kernel_tools.sh:96 and tracelens_analysis.py:3193), or delete change 7 from the description — one of the two must hold at the head that merges.

3. [V5] the code-metrics gate is red at head on violations this diff introduces, and the waiver route it documents is not exercised [verified]
   Problem: code-metrics failed three times at head 3964130 (runs 39324879407, 39324497261, 39322312279) on: worse-than-baseline rows this diff adds — materialize_config_with_envs cyclomatic 109→111 and cognitive 285→287, _build_parser function length 981→987, cli/__init__.py module lines 2569→2814, shared_state.py 2639→2641 — baseline entries added or raised relative to the base branch (including _server_patcher.py absent→1314), and the repeated literal 'xdit' written 3 times at _workload_envs.py:1927. The PR carries no baseline-raise label, and the description's "Baseline raise" section states a reason only for the _server_patcher.py module-lines raise, not for the other gated-limit raises.
   Impact: the PR cannot merge with a red gating check; the gated limits AGENTS.md's Size is a design signal defines are exceeded by this diff's additions with the documented waiver (label plus reason in the description) only partially satisfied.
   Action: apply the baseline-raise label and extend the description's reason to cover each raise this diff introduces (the materialize_config_with_envs branch, the cli/__init__.py/shared_state.py/_build_parser growth, the baseline entries, the repeated 'xdit' literal), or bring the units back under their recorded values — name the 'xdit' literal once either way.

SKIPPED: V4 -- openprs.txt carries 4 # PARTIAL: lines (#2004: 104 files, #1907: 188 files, #1869: 462 files, only the first 100 compared each; and 103 open PRs with only the newest 100 compared) — overlap with those PRs is unchecked, not absent. Compared rows do conflict in flight: #1963 splits materialize_config_with_envs (the function this diff grows), #1993 splits _server_patcher.py (the module this diff grows past the 1200-line ceiling), #2025 moves profiler setup and patching in _workload_envs.py — settle a landing order before merging.

Checked: cli/init.py apply (:211-221)/restore (:251-257)/persist (:297-306) and their call order (:1917-1923, :2128); bootstrap.py::_seed_shared_state field seeding (:525-555); shared_state.py xdit_model_name field and from_dict (:1090-1095); _workload_envs.py xdit profile branch (:1484-1506), _record_tracelens_patch_status (:997), tracelens_patch_ok init (:1598), XDIT_MODEL_NAME wiring (:1928-1930); _server_patcher.py xdit discovery (:56-69, :385-483) against the vllm/sglang twins (_resolve_vllm_patch_file, _discover_sglang_plan, _ensure_patched/_apply_atomic); profile.py preflight status readers (:157/:170) and the removed verifier branch; tracelens_skill_runner.infer_analysis_mode; install_kernel_tools.sh + tracelens_analysis.py pin pair base-vs-head; profile_xdit.yaml base-vs-head (byte-identical); SKILL.md flag row; tree-wide sweeps (src, docs, scripts, assets) for XDIT_MODEL_NAME, xdit_model_name, XDIT_MODEL_ARG, verify_xdit_profiler_baked, ensure_xdit_patched_for_tracelens, HYPERLOOM_TRACELENS_PATCH_STATUS, HYPERLOOM_PROFILE_DEGRADED_REASON, ae16eacb, TRACELENS_REF | Ran: fresh-session persistence reproduction (apply→seed→restore→apply, SystemExit 2); infer_analysis_mode driven directly for 9 shapes; pytest test_server_patcher_xdit_plan.py (11 passed), test_workload_envs_branches_unit.py -k xdit (5 passed), test_custom_framework.py -k "xdit or model_name" (3 passed); ruff C901 and PLR0913 head vs base on the 7 changed production files; gh api to TraceLens xdit_patches at both SHAs; gh pr view labels (none); wc -l on the gated modules; no full-suite run | Base: b7004d1 | Head: 3964130