[zoroyihan7]

Summary

The shipped profile configurations (profile_sglang.yaml, profile_vllm.yaml, profile_atom.yaml) now set the HIP runtime's switch that replays a captured graph node by node instead of from pre-recorded AQL packets while a profile runs. The switch differs per ROCm line, and a runtime ignores the ones it does not define, so all three are set:

HIP runtime (ROCm/rocm-systems projects/clr)	Switch	What it selects
rocm-7.1.0 .. rocm-7.2.3 (and release/therock-7.9, 7.10)	DEBUG_CLR_GRAPH_PACKET_CAPTURE=0 (flags.hpp@rocm-7.2.3)	no packet capture at instantiate (hip_graph_internal.cpp)
rocm-7.2.4 (and release/therock-7.11 .. 7.14)	DEBUG_HIP_GRAPH_SEGMENT_SCHEDULING=0 (flags.hpp@rocm-7.2.4)	segmented executor off; packets are captured only when it is on (Init), and Run takes the classic RunNodes path (Run)
release/therock-10.0 .. 10.2, develop	DEBUG_HIP_GRAPH_CLASSIC_PATH=1 (flags.hpp@therock-10.0)	GraphExecClassic instead of GraphExecSegmented (hip_graph.cpp): "classic topological enqueue, no AQL capture" (GraphExecClassic)

On 10.x DEBUG_HIP_GRAPH_SEGMENT_SCHEDULING=0 is that runtime's default (hybrid stream assignment), so the 7.2.4 entry changes nothing there. GraphExecClassic refuses multi-device graphs; the frameworks' per-rank decode graphs are single-device.

With the runtime's default replay (from the graph's recorded packets) the ROCm profiler records a replay's kernels only for the replay that a host sync follows: the profiler's stop, or a prefill step. Measured on one Qwen3-14B-FP8 SGLang server (764-kernel decode replays, concurrency 64, ISL/OSL 1k/1k): a 128-step window had the kernels of 1 of 127 replays, a 32-step window 1 of 32, a 4-step by-stage window 1 of 4, and single-step by-stage windows had none in 3 of 3. A profile's op shares were therefore those of the eager ops around the replays and of its prefill steps, not of the decode steps the graded workload runs.

Replayed node by node, on the same server and load: 128 of 128 replays recorded in a 128-step window, 32 of 32 in a 32-step one, 4 of 4 in a 4-step by-stage window and 4 of 4 single-step windows, with the kernel names in capture order and no misreported eager event. Decode throughput at concurrency 64 is unchanged within noise while windows were profiled (median 5277 against 5208 tok/s). The window setting is unchanged: the long window keeps its prefill steps, and every decode replay in it now counts.

The baselines do not carry the variable, so a measured run replays as shipped.

New test_profile_replay_config.py pins, per framework, that the profile carries it and the baseline does not.

Not measured

• Only the 7.2.0 switch is measured (the numbers above, one ROCm 7.2.0 SGLang image). Whether the missing-replay symptom reproduces on 7.2.4 and 10.x, and whether their switches restore every replay, is not measured: no GPU run on those images was made for this change. Each switch selects the path without AQL packet capture, the path the measured 7.2.0 setting selects.
• The variable is a HIP runtime setting, independent of the framework, so vLLM and ATOM carry it as SGLang does, but the measurement above is SGLang only. The SGLang numbers come from the same profile_sglang.yaml rendered by the multibranch measurement backend; this change was not re-run through ProfileExecutor and Magpie on a GPU.

Test plan

• pytest on test_profile_replay_config.py (9: 3 frameworks x 3 switches), test_profile_and_kernel_handlers.py, test_custom_framework.py, test_profile_gpu_preflight.py, test_profile_startup_shim.py: 225 passed; the one failure, test_trace_analyze_handler_dry_run_returns_structured_result, fails the same way on main in my environment (TraceLens is not installed there).
• Reverse proof: deleting DEBUG_HIP_GRAPH_CLASSIC_PATH from profile_vllm.yaml turns that case red (KeyError: 'DEBUG_HIP_GRAPH_CLASSIC_PATH'), the other 8 stay green.
• ruff check, ruff format --check (0.16.2), scripts/code_metrics.py --base-ref origin/main (PASSED).

Review template

• Linked issue(s): none.
• Size/complexity triggers crossed: none; three YAML entries and one small test file.
• Observable effect: on a ROCm 7.1.0-7.2.3 runtime (measured on 7.2.0, SGLang) a profile run records the kernels of every decode graph replay in its trace, where it recorded one replay in many before, so the per-op shares the profile analysis reads are those of the decode steps; throughput while profiling is unchanged within noise. On 7.2.4 and 10.x the runtime's own switch selects the same non-captured path; its effect there is not measured. Baseline and benchmark runs are unchanged.
• Breaking changes: no.
• PR addresses single concern: yes.
• Root cause is upstream: the default replay is the HIP runtime's; each setting is that runtime's own switch for it.

🤖 Generated with Claude Code

[github-actions]

Code metrics gate: PASSED

Judged: the 4 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Worse	Baseline out of date	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	0	0	6	+0
Cognitive complexity	fails > 30	0	0	0	53	+0
Function length (lines)	fails > 50	0	0	0	545	+0
Nested block depth	fails > 4	0	0	0	44	+0
Parameter count	fails > 5	0	0	0	3	+0
Module length (lines)	warns > 800, fails > 1200	0	0	0	43	+0
Duplicated lines per file	fails > 0	0	0	0	95	+0
Dead code (vulture)	fails > 0	0	0	0	2	+0
Directory size (entries)	fails > 50	0	0	0	2	+0

Check	Findings	Effect
Comments	0
Broad catch without a reason	0
English only	0
Production code imports test code	0
Repeated literals	0
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	0

22 findings in files this change does not touch

Outside this change (informational): 22

Unit	Dimension	Baseline	Now
src/hyperloom/orchestrator/actions/executors/_integrate_apply.py ApplyStageMixin._stage_apply	Cognitive complexity	absent	32
src/hyperloom/orchestrator/kernel/serving_log_evidence.py resolve_forge_server_log	Cognitive complexity	absent	49
src/hyperloom_kb/collect/mapping.py _experience	Cognitive complexity	absent	31
src/kernelforge/cli/forge_loop_command.py forge_loop	Cognitive complexity	absent	31
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate	Cognitive complexity	absent	44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt	Cognitive complexity	absent	38
src/hyperloom/orchestrator/kernel/integrate_accuracy.py grade_integrate_accuracy	Function length (lines)	absent	56
src/hyperloom/orchestrator/loop/_resume_recovery.py ResumeRecoveryMixin._resume_recover_pending_warm_replay	Function length (lines)	absent	75
src/hyperloom_kb/http_service.py	Module length (lines)	absent	1525
src/kernelforge/cli/forge_loop_command.py	Module length (lines)	absent	2003
src/hyperloom/inference_optimizer/cli/preflight.py _probe_rocm_build	Function length (lines)	54	57
src/kernelforge/gemm_tune/tuners/_aiter_dense_common.py run_aiter_dense_tuner	Function length (lines)	118	119
src/hyperloom/inference_optimizer/breakdown/schema.py	Module length (lines)	2681	2686
src/hyperloom/orchestrator/prompts/prompt_builder.py	Module length (lines)	1205	1212
src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py	Module length (lines)	2556	2567
src/kernelforge/fusion/command.py	Module length (lines)	1584	1708
src/kernelforge/fusion/validate.py	Module length (lines)	1513	1514
src/kernelforge/gemm_tune/tuners/_aiter_dense_common.py	Module length (lines)	1248	1249
src/kernelforge/loop/insession_gate.py	Module length (lines)	1218	1260
src/kernelforge/loop/knowledge_integration.py	Module length (lines)	1457	1459
src/kernelforge/loop/runner.py	Module length (lines)	6051	6510
src/kernelforge/loop/task_preparer.py	Module length (lines)	1299	1557

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

[chaojhou]

PR #2144 -- fix(profile): replay captured graphs node by node so a trace holds the kernels of every decode replay

What it does: sets DEBUG_CLR_GRAPH_PACKET_CAPTURE: "0" in the shipped profile_sglang.yaml, profile_vllm.yaml and profile_atom.yaml. The goal is for the HIP runtime to replay a captured graph node by node while a profile runs, so the ROCm profiler records the kernels of every decode replay instead of only the replay a host sync follows. On every single-node path that uses these files, the variable reaches the server process: Magpie exports benchmark.envs into the launch shell, or as docker run -e, and the reuse fingerprint keeps a baseline-booted server from being reused. Profile throughput is never graded, so a slower replay mode cannot skew KEEP/REVERT.

Blocking issues: 1

1. [X2] The variable does not exist in the HIP runtime of the repo's validated ROCm stacks, so on those stacks the change does nothing, while the description and comments present it as unconditional [verified]
   Problem: DEBUG_CLR_GRAPH_PACKET_CAPTURE is defined in ROCm CLR's projects/clr/rocclr/utils/flags.hpp in rocm-7.2.0 through rocm-7.2.3. It is absent at rocm-7.2.4, release/therock-10.0 and develop, where the segmented graph executor replaced packet capture. docs/compatibility.rst:125,155-159 names ROCm 10.0 (rocm10 images) as the validated docker stack for SGLang and vLLM, and SGLang bare metal on 7.2.x resolves the 7.2.4 layer (:226). On those runtimes the new entry (src/hyperloom/inference_optimizer/assets/configs/profile_sglang.yaml:65, profile_vllm.yaml:77, profile_atom.yaml:72) is silently ignored. The description's measurement comes from one ROCm 7.2.0 SGLang image, but its observable-effect line and the YAML comments state the effect without a version bound ("A HIP runtime setting, independent of the framework ... Replayed node by node (0) every replay is recorded").
   Impact: on the default stacks a profile records exactly what it recorded before. If the missing-replay symptom also exists under ROCm 10's executor, decode kernels are still missing while the release notes say they are fixed. If it does not exist there, the comments describe a mechanism those runtimes do not have.
   Action: measure the missing-replay symptom on a ROCm 10.0 image and on 7.2.4. If it reproduces, find that runtime's own switch (or report it upstream alongside ROCm/rocm-systems#13882) instead of shipping a variable it ignores. If it does not reproduce, scope the change: the YAML comments and the observable-effect line should say the variable applies to HIP runtimes up to ROCm 7.2.3 and is ignored on 7.2.4 and ROCm 10.

Checked: the three YAML hunks and the new test; ProfileExecutor config selection (profile.py _default_profile_config), materialize_config_with_envs, the native AgentX path (magpie_agentx.enable_profile, which never reads these files) and the multi-node restart path (which forwards only reference-launch envs); Magpie 6a1ee26f config.get_env_vars, benchmarker.py local/docker env export and reuse fingerprint, and the *_mi300x.sh launchers; consumers of profile throughput (writeback.py profile path, audit extras, roofline anchor); docs/compatibility.rst stack versions | Ran: gh api for ROCm/rocm-systems projects/clr/rocclr/utils/flags.hpp at rocm-7.2.0, 7.2.3, 7.2.4, release/therock-10.0 and develop (the flag is present only through 7.2.3); test_profile_replay_config.py at head (3 passed) and with the sglang YAML reverted (fails); gh pr checks 2144 (35 pass, 7 skipped); two independent refutation passes per finding | Base: 5c33315 | Head: 6ea73ec

[zoroyihan7]

@chaojhou Thanks, agreed: on 7.2.4 and 10.x the variable alone did nothing. Each later runtime has its own switch to the non-captured path, so 37db4c5 sets all three (a runtime ignores the ones it does not define) and scopes the comments and the description by version:

• ROCm 7.1.0-7.2.3: DEBUG_CLR_GRAPH_PACKET_CAPTURE=0 (flags.hpp@rocm-7.2.3).
• ROCm 7.2.4 (also release/therock-7.11..7.14): DEBUG_HIP_GRAPH_SEGMENT_SCHEDULING=0 (flags.hpp@rocm-7.2.4, "0 = Disable"). Packets are captured only when the segmented executor is on (Init); off, Run takes the classic RunNodes path (Run).
• ROCm 10.x (release/therock-10.0..10.2, develop): DEBUG_HIP_GRAPH_CLASSIC_PATH=1 (flags.hpp@therock-10.0) instantiates GraphExecClassic instead of GraphExecSegmented (hip_graph.cpp), "classic topological enqueue, no AQL capture" (GraphExecClassic). There SEGMENT_SCHEDULING=0 is the default (hybrid), so the 7.2.4 entry is a no-op. The classic executor refuses multi-device graphs; per-rank decode graphs are single-device.

Not measured, and stated so in the YAML comments and the description: whether the missing-replay symptom reproduces on 7.2.4 / 10.x and whether these switches restore every replay there. Only the 7.2.0 setting has GPU numbers.

Test: test_profile_replay_config.py now pins all three per framework (9 cases); deleting DEBUG_HIP_GRAPH_CLASSIC_PATH from profile_vllm.yaml turns that case red with KeyError. Latest main is merged in.

Fixed in 37db4c5 (each ROCm line's switch set and the effect scoped by version); superseded by the LGTM comment below.

[chaojhou]

PR #2144 -- fix(profile): replay captured graphs node by node so a trace holds the kernels of every decode replay

What it does: sets the HIP runtime's switch for the non-captured, per-node graph replay in the shipped profile_sglang.yaml, profile_vllm.yaml and profile_atom.yaml, so the ROCm profiler records the kernels of every decode replay rather than only the one a host sync follows. The switch differs per ROCm line, and a runtime ignores the ones it does not define, so all three are set: DEBUG_CLR_GRAPH_PACKET_CAPTURE=0 (7.1.0-7.2.3), DEBUG_HIP_GRAPH_SEGMENT_SCHEDULING=0 (7.2.4) and DEBUG_HIP_GRAPH_CLASSIC_PATH=1 (10.x). Baselines are unchanged.

Blocking issues: none

The finding from my review at 6ea73ec is fixed in 37db4c5. The repo's validated stacks (ROCm 10.0 images, 7.2.4 bare metal) now get their own runtime's switch. I checked each flag in projects/clr/rocclr/utils/flags.hpp at rocm-7.2.3, rocm-7.2.4, release/therock-10.0 and develop: each exists where the comment says, SEGMENT_SCHEDULING's default on 10.x is 0, and CLASSIC_PATH is false by default. The YAML comments and the description now scope the effect by version and state that only the 7.2.0 switch was measured. The observable-effect line says the 7.2.4 and 10.x effect is not measured.

Checked: the three YAML hunks and the test at 37db4c5 against a merge of main; ROCm CLR flags.hpp at the four refs above; the description and the author's reply | Ran: gh api for flags.hpp at each ref; gh pr checks 2144 (35 pass, 7 skipped) | Base: merge base with main | Head: 37db4c5

LGTM

[chaojhou]

Approved for merge; see the LGTM comment above.