[kyle-hoffmeyer]

Summary

This PR refactors HyperLoom's trace splitting pipeline to support generic (non-LLM-inference) workloads and removes dead code.

PR restriction exceptions

baseline-raise

_check_chunk_quality consolidates _check_selected_chunk_has_gpu_events (cognitive 36, 103 lines) and _check_selected_chunk_has_gpu_events_quality (cognitive 35, 100 lines) into one function at cognitive 23 / 88 lines. This is a net reduction from both predecessors and shouldn't be treated as a regression. Only the 88-line entry needs the waiver; the cognitive and cyclomatic figures are within their limits.

size-exception

This PR removes ~1000 lines of code added in #1292 as the logic covered by that PR has been migrated into TraceLens. Additionally, this PR is meant to accompany the TraceLens trace splitting refactor and thus contains many changes.

Changes

1. Remove --skip-split scriptable gate (trace_analyze.py)

LLM inference frameworks are defined as serving. Non-LLM-inference workloads are defined as scriptable. The if scriptable: cmd += ["--skip-split"] conditional in _build_trace_analyze_cmd was meant to prevent non-LLM-inference workloads from being split with the TraceLens splitter. This conditional has been removed so that all frameworks now go through the splitting pipeline.
Scriptable frameworks now enter the TraceLens split pipeline with --steady-state-mode generic instead of bypassing splitting with --skip-split. Traces whose splitting fails will fail with trace_split_no_steady_state where they previously produced an unsplit report.

2. Wire --llm-inference for serving frameworks (trace_analyze.py, tracelens_analysis.py)

--split-llm-inference is passed to tracelens_analysis.py, which forwards it as --llm-inference to the TraceLens splitter. This enables shape-based phase classification (decode vs prefill) for traces that lack serving annotations. The LLM-specific splitter flags (--CONC, --OSL, --R) are also gated behind this flag, they are only added when --split-llm-inference is set.

3. Add generic steady-state mode (trace_analyze.py, tracelens_analysis.py)

Previously, --steady-state-mode was only meaningful for LLM inference traces. Scriptable frameworks never reached this code because --skip-split bypassed the entire splitting pipeline.

With the gate removed, scriptable frameworks now need a steady-state mode that matches how the TraceLens splitter handles non-LLM traces. The splitter has two output modes:

• LLM inference: writes three phase-specific chunks — mixed_steady_state_*, decode_only_steady_state_*, prefilldecode_steady_state_* — same as before.
• Generic: writes a single steady_state_* chunk using duration-based steady-state windowing. There are no prefill/decode phases to distinguish.

The mixed/decode_only/prefilldecode modes map to the LLM chunk prefixes. A generic trace produces steady_state_* which doesn't match any of them. So we add generic as a new --steady-state-mode choice. In _build_trace_analyze_cmd, the mode is now set based on framework kind:

• LLM inference: no explicit --steady-state-mode is set, so it defaults to mixed — same behavior as before.
• Generic: --steady-state-mode generic is explicitly set. This tells _collect_split_chunks to look for steady_state_* files instead of the phase-specific prefixes.

_collect_split_chunks is updated to collect generic chunks and exclude them from the LLM-inference chunk sets.

4. Consolidate _check_selected_chunk_has_gpu_events() and _check_selected_chunk_has_gpu_events_quality (tracelens_analysis.py)

_check_selected_chunk_has_gpu_events() and _check_selected_chunk_has_gpu_events_quality have been consolidated into and replaced by _check_chunk_quality. If there is little GPU work, it selects a better chunk if it exists. However, if there is no GPU work, the check refuses the chunk unconditionally. _check_chunk_quality surfaces a steady_state_chunk_low_quality warning. The steady_state_chunk_empty warning code is removed; both cases now surface as steady_state_chunk_low_quality.

5. Remove pretrim logic (tracelens_analysis.py)

All pretrim functions and their invocation in main() are deleted:

• pretrim_startup_transient()
• _step_annotation_spans()
• _host_step_starts()
• _step_phase()
• _phase_step_medians()
• _build_pretrim_no_steady_state_warning()
• Constants: _PRETRIM_OUTLIER_FACTOR, _PRETRIM_MIN_STEPS, _PRETRIM_MAX_DROPPED_STEPS, _PRETRIM_MIN_PHASE_STEPS

Pretrim warmup removal is now handled inside the TraceLens splitter itself: AMD-AGI/TraceLens#1080.

6. Extract splitting logic into functions (tracelens_analysis.py)

Much of the inline splitting logic which now depends on the framework type are extracted into three functions:

• _build_split_cmd() — Assembles the TraceLens splitter CLI argv. Gates LLM-specific flags (--llm-inference, --CONC, --OSL, --R) behind args.split_llm_inference.
• _collect_split_chunks() — Globs the split directory for chunk files, maps steady_state_mode to the right chunk family (LLM: mixed/decode_only/prefilldecode, generic: steady_state), returns the selected chunk and metadata.
• _validate_selected_chunk() — Runs the low-quality busy-ratio gate on the selected chunk.

7. Remove splitting forecast from self-certification probe (trace_selfcert.py)

The self-certification probe (trace_selfcert.py) produces a diagnostic report about whether a trace is usable. It contained a "split forecast" subsystem that duplicated the TraceLens splitter's logic to predict what splitting would produce — without ever running the splitter. This forecast had zero runtime impact: the splitter always ran independently afterward regardless of what the forecast predicted. The forecast results (viable_modes, recommended_steady_state_mode, split_forecast, idle_gate) were written to a selfcert.json certificate file and recorded in the roofline event recorder's metadata, but no code path read them to make decisions about splitting, mode selection, or any other runtime behavior.

We verified this by searching all consumers of the certificate data:

• profile.py:254 (_steady_state_forecast) — extracts forecast fields into metadata. Doesn't do anything with the metadata though.
• profile.py:270 (_build_trace_validate) — passes certificate through to the event record, no conditionals
• roofline_event.py:420 — records recommended_steady_state_mode, viable_modes, steady_state_forecast into event metadata, never reads them back

8. Update TraceLens commit SHA

TraceLens pin bumped to 647cbf2, which adds the busy_ratio column to execution_details.csv. _check_chunk_quality reads this column. Against an older splitter the column is absent and the check falls back to gpu_busy_duration / gpu_duration. When neither is available, a chunk carrying GPU events is let through unchecked and a chunk with no GPU events is still refused, so the pin bump restores full gating rather than being required for correctness.

9. Remove step_roots_sufficient gate

The self-cert probe no longer blocks annotation-less traces. The step_roots_sufficient gate in build_verdict assumed the splitter required pattern-matched step annotations, but this is no longer required, making the
gate a false negative. Removed the parameter and its blocking branch; the splitter's own execution_details.csv is the ground truth for chunk quality.

Tests

New test file: test_trace_splitting.py (11 tests)

TestCollectSplitChunks (4 tests) — unit tests for _collect_split_chunks() in tracelens_analysis.py:

• test_llm_mixed_mode: given mixed/decode/prefilldecode chunk files, selects the mixed chunk correctly
• test_generic_mode: given a steady_state_* chunk file, selects it via the generic mode
• test_no_chunks_raises: empty split directory raises trace_split_no_steady_state
• test_missing_mode_raises: requested mode has no chunks raises steady_state_chunk_missing

TestSplitterE2E (2 tests) — end-to-end tests that call _run_trace_split() directly (the same function main() uses) on synthetic traces. This tests the full splitting chain: args → _build_split_cmd → TraceLens splitter subprocess → _collect_split_chunks → _validate_selected_chunk. Verifies the pipeline runs without errors and produces the expected output files:

• test_vllm_trace_produces_three_chunks: synthetic vLLM trace with step annotations + split_llm_inference=True → splitter produces mixed_steady_state_* chunks, _collect_split_chunks selects the mixed chunk
• test_xdit_trace_produces_generic_chunk: synthetic xDiT trace with ProfilerStep markers + steady_state_mode="generic" → splitter produces a steady_state_* chunk, _collect_split_chunks selects it via generic mode

TestBuildTraceAnalyzeCmd (5 tests) — unit tests for _build_trace_analyze_cmd() in trace_analyze.py, verifying the orchestrator wires the correct flags based on framework type:

• test_serving_gets_llm_inference: vLLM (serving) gets --split-llm-inference in the command, no --skip-split
• test_scriptable_gets_generic_mode: xDiT (scriptable) gets --steady-state-mode generic, no LLM flags, no --skip-split
• test_serving_forwards_conc_osl_r: CONC/OSL/R from the payload are forwarded for serving frameworks
• test_scriptable_no_conc_osl_r: CONC/OSL/R are absent for scriptable frameworks even if present in payload
• test_scriptable_payload_override_cannot_shadow_generic: payload-supplied steady_state_mode cannot override the hardcoded generic for scriptable frameworks

[tsrikris]

/review

[github-actions]

Automated review started for head b3cc7a7. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/37815170112

[github-actions]

PR #1612 -- Cleanup trace splitter code

What it does: Replaces the blanket --skip-split gate that kept scriptable (non-LLM-inference) frameworks out of trace splitting with per-kind flags — --steady-state-mode generic for scriptable, --split-llm-inference for serving — extracting the split pipeline into testable functions, adding a generic chunk family, and deleting the selfcert split-forecast subsystem and the Hyperloom-side pretrim.

Blocking issues: 3

1. [S4] Deleting the structural GPU-events gate lets a zero-GPU-event chunk reach analysis [verified]
   Problem: _check_selected_chunk_has_gpu_events was deleted on the claim that the splitter cannot produce an empty chunk, but the only surviving gate, _check_selected_chunk_has_gpu_events_quality, returns None when the busy ratio is unmeasurable — which is exactly the num_gpu_events=0 case its helper _busy_ratio punts with "the caller defers to the N25 structural gate", a gate this PR removed. The splitter's coverage gate grades whole-trace root attribution, not per-chunk content. src/hyperloom/agents/kernel/tools/tracelens_analysis.py:4597
   Impact: A chunk with zero GPU events (TP=1 + CUDA-graph mixed windows where rocprofiler-sdk emits no kernel events for the replay — the case the deleted gate's own tests pinned) is fed to TraceLens analysis, producing the misleading "Compute %=~0, Idle %=~100" Executive Summary the gate existed to refuse.
   Action: Keep the structural zero check — either restore the gate or make _validate_selected_chunk treat num_gpu_events==0 as blocking instead of deferring to a deleted callee.
2. [C1] The chunk-failure warnings that drive the roofline auto-retry are no longer emitted [verified]
   Problem: _collect_split_chunks and _validate_selected_chunk raise RuntimeError without appending the structured warning dicts; base appended steady_state_chunk_missing/steady_state_chunk_low_quality (with available_modes/non_empty_modes) to trace_health_warnings before raising, and _extract_steady_state_retry_mode in the roofline executor reads only those dicts — it never parses the error string. steady_state_chunk_empty also still sits in _AUTO_RETRY_WARNING_CODES with no producer left. src/hyperloom/agents/kernel/tools/tracelens_analysis.py:4583
   Impact: Every missing-mode, empty-chunk and low-quality-chunk failure becomes a hard roofline failure; the N26 self-healing re-issue on the same trace is dead code, and the executor's generic tracelens_analysis_failed warning does not carry a retry mode.
   Action: Emit the structured warnings (appended before the raise, or returned by _run_trace_split and extended into trace_health_warnings by main()) so the existing retry consumer keeps working.
3. [X2] The description states a change the diff does not contain, and an upstream handoff the pin does not carry [verified]
   Problem: Item 6 claims _PHASE_SUFFIX_RE and its call site were removed; the symbol, its comment and the quoted example exist nowhere in the diff or the base tree. Item 5 claims pretrim warmup removal is "now handled inside the TraceLens splitter itself" (TraceLens#1080), but the pinned TRACELENS_REF b79f906 (2026-09-29) predates that merge (2026-10-07), and this PR does not move the pin. src/hyperloom/agents/kernel/scripts/install.sh:160
   Impact: At the current pin nobody trims the profiler-start rank-barrier transient — the reference capture's 15.78 s barrier step lands in the steady-state window and every downstream percentage is computed against that inflated denominator, the exact regression the deleted pretrim documented.
   Action: Move TRACELENS_REF to a commit containing TraceLens#1080 in this PR (with the compatibility doc, per the repo's four-places rule), remove the fabricated item 6, and state the pretrim behavioural change as a dependency on the new pin.

Checked: tracelens_analysis.py split pipeline (_build_split_cmd/_collect_split_chunks/_validate_selected_chunk/_run_trace_split), trace_analyze.py _build_trace_analyze_cmd, roofline.py auto-retry (_AUTO_RETRY_WARNING_CODES/_extract_steady_state_retry_mode), trace_selfcert.py verdict/certify_trace_dir, profile.py _steady_state_forecast, roofline_event.py, framework_registry.py, pinned TraceLens splitter (root_detection.py coverage gate, steady_state_window.py, trace_extraction.py, startup_transient.py absence) | Ran: pytest test_trace_splitting.py (10 pass), test_n25_steady_state_mode.py + test_trace_selfcert_observability.py + test_kernel_request_handlers_units.py (272 pass), test_profile_and_kernel_handlers.py (176 pass + 1 failure reproduced on base = environment), test_tracelens_csv.py (220 pass + 1 failure reproduced on base = environment), test_n36_chunk_quality_gate.py (7 pass), test_roofline_executor.py + test_roofline_executor_units.py (42 pass), test_sbd_v6_roofline_timeline.py (35 pass); direct reproduction of the empty-chunk acceptance on head vs steady_state_chunk_empty on base; full tla.main() drives of the missing-chunk and low-quality cases on head and base; _build_trace_analyze_cmd duplicate-flag and env-override runs; ruff on changed files | Base: f2b32cd | Head: b3cc7a7

[kyle-hoffmeyer]

S4 has been addressed. Created a "busy_ratio" metric in TraceLens. check_chunk_quality looks at this metric to handle both the no gpu work check and the little gpu work check.
C1 has also been addressed.
Regarding X2, the PHASE_SUFFIX_RE removal description in the PR description is now outdated since it seems to be gone. This PR will need to be merged after the next TraceLens release as the code in this PR is dependent upon the latest head of TraceLens.

[kyle-hoffmeyer]

@chaojhou addressed the comments

[xiaofei-zheng]

PR #1612 -- Cleanup trace splitter code

What it does: trace splitting was gated off for non-LLM-inference workloads (--skip-split) and the repo
carried its own pretrim and split-forecast reimplementations of logic TraceLens now owns. This PR routes
every framework through the TraceLens splitter -- serving frameworks get --split-llm-inference plus the
CONC/OSL/R hints, scriptable frameworks get a new --steady-state-mode generic that selects the splitter's
single steady_state_* chunk -- extracts the splitting block of main() into four testable functions,
merges the two chunk gates into _check_chunk_quality reading the splitter's new busy_ratio column, and
deletes pretrim_startup_transient, forecast_split and idle_gate along with the certificate fields they
fed.

Blocking issues: 4

1. [S4] The chunk-quality gate cannot tell a missing busy_ratio column from a 0% busy chunk, so against
   any TraceLens older than the new pin it stops blocking entirely. [verified]
   Problem: _read_busy_ratio returns 0.0 both when the column is absent and when the chunk really is 0%
   busy. With the column absent, every row reads 0.0, so no alternate can clear
   oth_ratio >= threshold and alternates stays empty; if not alternates and sel_events > 0: return None
   then passes every chunk that has any GPU events. gpu_busy_duration, which the two deleted gates read and
   which older splitters do emit, is no longer read by any production line (only the docstring at
   tracelens_analysis.py:375). src/hyperloom/orchestrator/trace_analysis/tracelens_analysis.py:429
   Impact: _TRACELENS_REF_DEFAULT is a default and TRACELENS_ROOT / --tracelens-root let an operator
   point at another checkout, which the install script has explicit machinery for. On such a pod a
   structurally empty or 1%-busy chunk is now fed to TraceLens and produces the misleading high-idle
   Executive Summary the gate exists to prevent, with no warning and no log line. The description (section 8)
   says the check "degrades to busy_ratio=0.0", which reads as degrading strict; it degrades to off.
   Action: detect the absent column explicitly -- fall back to gpu_busy_duration / gpu_duration when
   busy_ratio is not in the row, or emit a warning that the gate could not run -- and correct section 8.

2. [C3] StepPass.gpu_intervals loses its only reader in this PR and is left accumulating one tuple per GPU
   event on every self-certification scan. [verified]
   Problem: the attribute is initialised at :255 and appended at :304 (cat == "kernel") and :308
   (gpu_memcpy / gpu_memset). Its only reader was _idle_in_span, at merge base 788ab26
   trace_selfcert.py:715, which this diff deletes; a grep of the head tree including tests returns three
   hits, all writes. src/hyperloom/orchestrator/trace_analysis/trace_selfcert.py:255
   Impact: certify_trace_dir holds a list proportional to the GPU event count of the capture -- production
   rank traces reach millions of events -- for data nothing consumes. The PR is titled a cleanup and removes
   the forecast subsystem; this is the piece of it left behind.
   Action: delete gpu_intervals and its two append sites.

3. [X3] A #: doc-comment survives the constant it documented and now asserts a certificate property this PR
   removed. [verified]
   Problem: "The splitter's internal mode names differ from the value a caller sets in
   INFERENCE_OPTIMIZER_STEADY_STATE_MODE. Recommendations are reported in the caller's vocabulary so the
   certificate is directly actionable." documented _CONSUMER_MODE, which this PR deletes; the certificate no
   longer carries recommended_steady_state_mode or recommended_splitter_mode either. The block now sits
   directly above def _iso_now(), where #: means nothing.
   src/hyperloom/orchestrator/trace_analysis/trace_selfcert.py:65
   Impact: the next reader of this module is told the certificate reports an actionable mode recommendation
   that no longer exists anywhere in the tree.
   Action: delete the block; while there, :48 still justifies the root patterns by what they "would
   forecast", and the forecast is gone.

4. [X2] The description does not match the head tree on two checkable numbers. [verified]
   Problem: the body says test_trace_splitting.py has "10 tests" and lists four under
   TestBuildTraceAnalyzeCmd; the file has 11, with five in that class --
   test_scriptable_payload_override_cannot_shadow_generic at :381 is unlisted. The baseline-raise
   exception says _check_chunk_quality is "cognitive 33 / 101 lines"; scripts/code_metrics_baseline.txt
   committed in this same PR records function-lines ... _check_chunk_quality 96.
   src/hyperloom/orchestrator/trace_analysis/tests/test_trace_splitting.py:381
   Impact: the baseline-raise exception is the text a reviewer reads instead of re-deriving the metric, and
   it quotes a number its own artifact contradicts.
   Action: update both counts in the description.

Checked: tracelens_analysis.py (_build_split_cmd / _collect_split_chunks / _validate_selected_chunk /
_run_trace_split / _check_chunk_quality / _is_derived_trace / argparse), trace_analyze.py
(build_trace_analyze_cmd, all four scriptable x bypass branches), trace_selfcert.py (StepPass.scan,
build_verdict, certify_trace_dir), roofline.py (AUTO_RETRY_WARNING_CODES,
extract_steady_state_retry_mode), profile.py, roofline_event.py, bypass_trace_analysis.py (argv surface),
test_trace_splitting.py / test_n25 / test_n36 / test_tracelens_csv, and TraceLens at 647cbf2
(split_trace/main.py, steady_state_window.py, trace_extraction.py) | Ran: head-tree and merge-base greps for
every removed identifier (steady_state_chunk_empty, pretrim*, forecast_split, viable_modes, recommended*,
gpu_intervals, 6489fbc), glob-exclusivity check for steady_state* vs mixed_steady_state*, gh api reads of
the pinned TraceLens tree; local pytest blocked by the module-level TraceLens import, CI green at head |
Base: 788ab26 | Head: c70e45b

[kyle-hoffmeyer]

@xiaofei-zheng addressed comments

[xiaofei-zheng]

Re-reviewed at head a76f356. Two of the four are fixed (gpu_intervals removed, the orphaned #: block removed), and the description numbers now match. Three remain.

1. [S4] Still open: _read_busy_ratio cannot distinguish an absent busy_ratio column from a 0% busy chunk

tracelens_analysis.py is unchanged at this head and section 8 of the description is unchanged, so I think this one was read as already-answered. It is a different point from the S4 in the automated review of 2026-10-08, which asked for the metric to exist. Adding busy_ratio to TraceLens is exactly right; the issue is how this side reads it.

_read_busy_ratio (src/hyperloom/orchestrator/trace_analysis/tracelens_analysis.py:429) returns 0.0 both when the column is missing and when the chunk genuinely has no GPU work. With the column missing every row reads 0.0, so no alternate can clear oth_ratio >= threshold at :472, alternates stays empty, and if not alternates and sel_events > 0: return None at :474 passes every chunk that has any GPU events. gpu_busy_duration — which both deleted gates read, and which older splitters do emit — is no longer read by any production line (the only remaining occurrence is the docstring at :375).

_TRACELENS_REF_DEFAULT is a default, and TRACELENS_ROOT / --tracelens-root let an operator point at another checkout; install_kernel_tools.sh has explicit machinery for that case. On such a pod a structurally empty or 1%-busy chunk is now analysed and produces the misleading high-idle Executive Summary the gate exists to prevent, with no warning and no log line. Section 8 says the check "degrades to busy_ratio=0.0", which reads as degrading strict; it degrades to off.

Please detect the absent column explicitly — fall back to gpu_busy_duration / gpu_duration when busy_ratio is not in the row, or emit a warning that the gate could not run — and correct section 8.

2. [X1] Removing pretrim is not the equivalent migration section 5 describes

Section 5 says "Pretrim warmup removal is now handled inside the TraceLens splitter itself: TraceLens#1080". I read TraceLens/TraceUtils/split_trace/startup_transient.py::trim_startup_transient at the pinned 647cbf2. It is deliberately narrower than the pretrim_startup_transient this PR deletes, in three ways that change what Hyperloom reports:

• Collective-only. It returns unchanged when trace_index.collective_kernels is empty, and otherwise requires the worst collective-op inflation to clear 5.0x its later-iteration median and to beat the worst non-collective inflation by 5.0x. Its own docstring states the intent: "legitimate variance (a slow first iteration from cache warmup, allocator setup, etc.) touches compute and collective kernels roughly alike" and is therefore not trimmed. The deleted pretrim detected on step duration, not kernel name, and its docstring named exactly the cases now dropped: "the same guard catches other head-of-window transients (first-replay JIT, KV-pool growth) without knowing what they are". Single-rank captures and xDiT-style workloads with no collectives at all are now outside the trim entirely.
• One iteration, not up to four. Upstream does new_roots = roots[1:]. The deleted pretrim looped over leading outliers up to _PRETRIM_MAX_DROPPED_STEPS = 4.
• The pretrim_no_steady_state warning is gone. When more than four leading steps were outliers, the old path refused to trim and raised a trace-health warning saying the capture never reached steady state and that every percentage below includes the warm-up. Upstream records startup_transient_trimmed: false in its own diagnostics, which does not surface as a Hyperloom trace-health warning. An operator reading the report no longer has any signal that the window contains warm-up.

I am not arguing the upstream behaviour is wrong — the selective-inflation test is a better discriminator for the collective case. The problem is that section 5 presents this as the same capability moving, which it is not. Please state the narrowed scope, and say whether the lost pretrim_no_steady_state signal is intended to go away or needs an equivalent on the splitter side.

3. [X2] The baseline-raise reason does not cover two of the three findings it waives

The gate report at this head waives three findings. The description's baseline-raise section explains only _check_chunk_quality (a merge of two deleted gates, a net reduction against both predecessors — that reason is sound). The other two have no reason given:

• _collect_split_chunks — new violation, function length 88 > 80. This is a newly written function, not a merge of existing ones.
• _build_trace_analyze_cmd — worse than baseline, function length 112 -> 116, grown by the --split-llm-inference and generic-mode branches added here.

The gate's own instruction is "Use it only with the reason in the PR description". Please add both, or bring the two units back under their limits.

[github-actions]

CI E2E report — ❌ Failed

item	value
result	❌ Failed
model	Qwen/Qwen3-0.6B (dense)
resources	1× GPU, TP=1
PR branch	fix/khoffmey/split_trace_cleanup
commit	bc7958b38b05ebd4aab779f3514d14bbb96b7888
session_id	hyperloom:oneoff-fb24a2e031b24911
backend run	30f152cb-2089-48ec-845b-717890045945 on -
queue → dispatch	0s
run time	40m 50s
total	40m 50s
reason	safe-workload: the platform ended the run (Terminated, Pod was terminated because the allocation this Pod ran on ended: allocation 54c907a4-98dc-4639-b5be-75d574b63147 generation 1 ended: its scheduler job 217472 reached PREEMPTED by job 217519 under QoS amd-aifw-dev-qos (mode Cancel, recorded CANCELLED); the node's pods were stopped (the node stopped this workload at 2026-10-15T07:08:39.824Z because the allocation it ran on ended: the allocation's payload asked for it); DisruptionTarget: PreemptionByScheduler (2026-10-15T07:08:42): Scheduler: the allocation this Pod ran on was preempted to make room for other work (allocation 54c907a4-98dc-4639-b5be-75d574b63147 generation 1 ended: its scheduler job 217472 reached PREEMPTED by job 217519 under QoS amd-aifw-dev-qos (mode Cancel, recorded CANCELLED)); a node whose allocation the site's scheduler may preempt is labelled autopilot.amd.com/preemptible=true)

platform error

safe-workload: the platform ended the run (Terminated, Pod was terminated because the allocation this Pod ran on ended: allocation 54c907a4-98dc-4639-b5be-75d574b63147 generation 1 ended: its scheduler job 217472 reached PREEMPTED by job 217519 under QoS amd-aifw-dev-qos (mode Cancel, recorded CANCELLED); the node's pods were stopped (the node stopped this workload at 2026-10-15T07:08:39.824Z because the allocation it ran on ended: the allocation's payload asked for it); DisruptionTarget: PreemptionByScheduler (2026-10-15T07:08:42): Scheduler: the allocation this Pod ran on was preempted to make room for other work (allocation 54c907a4-98dc-4639-b5be-75d574b63147 generation 1 ended: its scheduler job 217472 reached PREEMPTED by job 217519 under QoS amd-aifw-dev-qos (mode Cancel, recorded CANCELLED)); a node whose allocation the site's scheduler may preempt is labelled autopilot.amd.com/preemptible=true)

details

[kyle-hoffmeyer]

@xiaofei-zheng
Regarding S4, added a fallback if users are using an old TraceLens commit which doesn't have busy_ratio populated.
Regarding X1, the pretrim removal in the trace splitter is deliberately simpler than the one implemented in HyperLoom.

• It only focuses on collective operations since that was the initial focus of the feature. As you mentioned, "The deleted pretrim detected on step duration, not kernel name, and its docstring named exactly the cases now dropped: 'the same guard catches other head-of-window transients (first-replay JIT, KV-pool growth) without knowing what they are'." However, guarding for these cases is unnecessary as HyperLoom already profiles a steady state window. There is already a warmup period before profiling begins so that the mentioned head-of-window transients aren't profiled.
• The profiling delay would only affect the first iteration so it is not necessary to check the first four iterations for outliers.
• The pretrim_no_steady_state warning is not needed since TraceLens only drops or keeps the first iteration. If the first iteration has an unusually long collective operation, it is dropped. If it doesn't it is kept. startup_transient_trimmed is recorded in the trace splitter metadata and no warnings need to be emitted.

Regarding X2, reduced the function lengths to be within limits.

[xiaofei-zheng]

Re-reviewed at head 42779ce against the new merge base a56028d. The three points from the last round are addressed: the busy_ratio fallback is in, the pretrim scope question is answered (the delay_iters argument holds — I checked _workload_envs._profile_capture_window), and the function lengths came down. The merge of main is clean: all four files main touched since the old base kept their changes.

Four blocking issues remain, two of them introduced by the fix in dc184e93.

1. [S1] The busy_ratio fallback lets a zero-GPU-work chunk through on exactly the configuration it was added for

src/hyperloom/orchestrator/trace_analysis/tracelens_analysis.py:424-451 and :470-472

_read_busy_ratio returns None when busy_ratio is absent and gpu_duration is not positive, and the caller then does if sel_ratio is None: return None — accept.

A structurally empty chunk is exactly that shape. In TraceLens trace_extraction.py:156, gpu_dur is only accumulated when a GPU event was seen (max(end_time) - min(start_time) if start_time else 0), so a chunk with num_gpu_events=0 carries gpu_duration=0. I ran the function as it stands at this head against the two rows:

row {num_gpu_events:0, gpu_duration:0, gpu_busy_duration:0, busy_ratio:0}  -> 0.0    (new splitter)
row {num_gpu_events:0, gpu_duration:0, gpu_busy_duration:0}                -> None   (old splitter)
row {num_gpu_events:2790, gpu_duration:4538984, gpu_busy_duration:2723452} -> 0.600  (old splitter, healthy)

With the new splitter the busy_ratio=0 column makes the gate reach sel_events == 0 and refuse, which is right. With an older splitter the same chunk returns None and is accepted, and TraceLens analyses an empty window. Note this is a regression against the previous head too: at a76f356 the missing column read as 0.0, which refused correctly. The fallback works on healthy chunks and fails open on the one case the gate exists for.

Fix: check num_gpu_events (and gpu_busy_duration) before the sel_ratio is None and sel_ratio >= threshold early returns, so the structural refusal is unconditional the way _check_selected_chunk_has_gpu_events was.

2. [S1] The 10 ppt anti-thrash margin is now applied to structurally empty chunks, so a recoverable failure became a hard one

tracelens_analysis.py:479-491 with src/hyperloom/orchestrator/actions/executors/roofline.py:133-163

_CHUNK_QUALITY_ALTERNATE_MARGIN exists so a chunk that works but not well does not thrash against an equally mediocre sibling — its own comment says "Alternate must beat the requested mode by this margin to avoid thrashing". The deleted _check_selected_chunk_has_gpu_events did not use it: it listed as non_empty_modes every alternate with other_events > 0 and other_busy > 0.0. After the merge, the margin governs both cases.

Trigger: requested mixed chunk is structurally empty (num_gpu_events=0, busy_ratio=0.0), and the best alternate sits at busy_ratio=0.07. 0.07 >= 0.05 passes the threshold but 0.07 - 0.0 < 0.10 fails the margin, so alternates is empty, the warning carries non_empty_modes: [], _extract_steady_state_retry_mode finds no candidate, and roofline fails without retrying. On the merge base the same inputs retried into prefilldecode and the session recovered on the same captured trace. A 7% busy chunk is not a corner case here — the default threshold is 5% and the remediation text cites a 1% case.

When the selected chunk has no GPU work at all, every alternate carrying GPU work is strictly better and there is nothing to thrash against. Fix: when sel_events == 0, populate non_empty_modes from every alternate with GPU work and skip the margin.

3. [X2] _check_chunk_quality is now measurably worse than the description's waiver claims, and the baseline file no longer matches the code

The baseline-raise section argues: "into one function at cognitive 33 / 96 lines. This is a net reduction from both predecessors" (36/103 and 35/100).

At this head that is no longer true. I ran complexipy 8.0.1 locally on the file: _check_chunk_quality is cognitive 46, higher than both predecessors, not a reduction. scripts/code_metrics_baseline.txt:219 still records 33 and :789 still records 96; the gate report at 42779ce reads 46 and 100 against them. The gate also lists a new cyclomatic-complexity violation, 25 > 20, and grep cyclomatic-complexity.*_check_chunk_quality scripts/code_metrics_baseline.txt confirms the baseline has no entry for it — the base-branch predecessor it replaces was 21.

So the waiver now covers a new violation and two regressions the description does not mention, on an argument its own numbers contradict. _build_trace_analyze_cmd did come back to its baseline of 104 lines — that half of the last round's point is resolved.

Worth saying plainly: findings 1 and 2 are both consequences of putting a structural refusal and a comparative quality judgement in one function. Splitting them back apart would fix the complexity and the two bugs together.

4. [X2] Two statements in the description are false at this head

• §8: "against an older splitter the column is absent and the check degrades to busy_ratio=0.0". It degrades to gpu_busy_duration / gpu_duration, and to skipping the gate when neither is available. Those are opposite operator-visible outcomes, and an operator reads this line to decide whether the pin bump is mandatory.
• §4: "If there is no GPU work, the check refuses the chunk unconditionally". Falsified by finding 1. This is the one guarantee §4 offers to justify deleting _check_selected_chunk_has_gpu_events.

Checked: _check_chunk_quality / _read_busy_ratio / _collect_split_chunks / _validate_selected_chunk / _run_trace_split, _build_trace_analyze_cmd across all four (scriptable, is_bypass) combinations against both receivers' argparse, roofline.py _AUTO_RETRY_WARNING_CODES / _extract_steady_state_retry_mode / the one-retry bound, trace_selfcert.py after the deletion (no dangling state; build_verdict's keyword-only signature is legal), _workload_envs._profile_capture_window for the warm-up argument, and TraceLens at 647cbf2 (trace_extraction.py, split_trace/main.py, startup_transient.py) | Ran: _read_busy_ratio extracted verbatim from this head against four concrete execution_details.csv rows; complexipy 8.0.1 on tracelens_analysis.py; greps of scripts/code_metrics_baseline.txt; a merge-audit of the four files main changed since the old base | Base: a56028d | Head: 42779ce

[xiaofei-zheng]

Thanks Kyle — I pulled 490161a and re-read it. All three code findings are fixed, and fixed the way I'd have asked for. Confirming what I checked, then one correction of mine and two lines of description left.

Fixed, verified:

• Zero-GPU-work chunks: if sel_ratio is None: sel_ratio = 0.0 after the sel_events > 0 guard means an unreadable ratio on a chunk with no GPU events now falls through to the refusal instead of the early accept. The _read_busy_ratio body is unchanged from 42779ce (moved to module scope verbatim), and I ran it there against the old-splitter empty row returning None, so the new guard is exactly what was catching nothing before.
• Retry target: if sel_events == 0: if oth_ratio > 0.0 skips the anti-thrash margin when the selected chunk has no GPU work, so a 7% alternate reaches non_empty_modes again and the roofline retry has a target.
• Complexity: pulling _row_for_chunk / _read_busy_ratio / _num_gpu_events out to module scope brings _check_chunk_quality to cognitive 23 (complexipy 8.0.1, run locally). The gate at this head is New 0 / Worse 0 across every dimension, with one baseline-growth entry left.

Withdrawing one of mine: my §4 finding ("refuses unconditionally" is false) no longer holds. With the sel_ratio = 0.0 fallback in place, every path where the CSV row is readable and num_gpu_events == 0 now reaches the refusal. §4 is accurate as written — please disregard that point.

One clarification on scope, so we're not talking past each other: X2 is not an edge case that needs a pinned old TraceLens, or any runtime condition. It is the description disagreeing with the code at head, which is true for every reader on every configuration. Separately, the margin finding was not old-splitter-dependent either — an empty mixed chunk with a 7% alternate reproduces on the new splitter, since busy_ratio=0 is exactly what trace_extraction.py writes for a chunk with no GPU events. Both are moot now that the code is fixed; flagging it only so a description mismatch doesn't get sorted into the edge-case bucket next time.

What's left is two lines of text, and I'd rather not defer them:

1. §8 still says "against an older splitter the column is absent and the check degrades to busy_ratio=0.0". At head it degrades to gpu_busy_duration / gpu_duration, and skips the gate when neither is available and the chunk has GPU work. An operator reads this line to decide whether the pin bump is mandatory, and it currently gives the opposite answer.
2. The baseline-raise section still says "cognitive 33 / 96 lines". It is now cognitive 23 / 88 lines, and the baseline file carries only the function-lines 88 entry. Worth updating because it makes your own argument stronger, not weaker: 23 is genuinely below both predecessors (36 and 35), which is what the waiver claims. As written the numbers don't support the sentence.

Neither needs code or CI. With those two edits I have no blocking issues left and no objection to this landing — I agree the remaining hardening (the nan/inf clamp in _read_busy_ratio, and the num_gpu_events > 0 with gpu_busy_duration == 0 row) is follow-up material; neither is reachable from what the splitter can emit.

[xiaofei-zheng]

LGTM. All findings from the earlier rounds are resolved at bc7958b.

I edited two factual lines in the description directly rather than send them back: §8 now describes the actual busy_ratio fallback (gpu_busy_duration / gpu_duration, gate skipped only when neither is available and the chunk has GPU events), and the baseline-raise waiver now carries the real figures, cognitive 23 / 88 lines, which is what makes the net-reduction argument hold against the 36 and 35 of the two predecessors. No other text was touched.

Checked: _check_chunk_quality and the three helpers lifted to module scope in 490161a — the sel_ratio = 0.0 fallback after the sel_events > 0 guard closes the accept path for an unreadable ratio on a chunk with no GPU events, and the sel_events == 0 branch in the alternate loop skips the anti-thrash margin so the roofline retry gets a target again; _build_trace_analyze_cmd across all four (scriptable, is_bypass) combinations against both receivers' argparse; _collect_split_chunks generic/LLM chunk separation; roofline.py _AUTO_RETRY_WARNING_CODES and _extract_steady_state_retry_mode with the one-retry bound; trace_selfcert.py after the deletion, with every removed certificate key's reader removed in the same diff; the merge of main, where all four files main had touched since the previous base kept their changes.

Ran: complexipy 8.0.1 on tracelens_analysis.py (_check_chunk_quality cognitive 23, down from 46); _read_busy_ratio extracted verbatim from head against old- and new-splitter execution_details.csv rows; greps of scripts/code_metrics_baseline.txt; TraceLens at 647cbf2 read for busy_ratio (trace_extraction.py), the generic/inference split paths (split_trace/main.py) and the startup-transient trim (startup_transient.py, confirmed an ancestor of the pin).

The ci-e2e/run failure at this head is infrastructure — "Pod was terminated because the allocation this Pod ran on ended" — and the same job succeeded on the two preceding heads. Worth a rerun before merge, not a code issue.

Two items I'd leave for follow-up, neither reachable from what the splitter can emit: _read_busy_ratio clamps a non-finite busy_ratio to 1.0 rather than rejecting it, and a row with num_gpu_events > 0 and gpu_busy_duration == 0 is accepted when no alternate clears the margin.