Skip to content

Upgrade the MI355X default runtime to ROCm 10 - #123

Merged
irvineoy merged 7 commits into
mainfrom
codex/mi355x-rocm10-runtime
Oct 20, 2026
Merged

irvineoy merged 7 commits into
mainfrom
codex/mi355x-rocm10-runtime

Conversation

@irvineoy

@irvineoy irvineoy commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

The MI355X default uses ROCm 7.2, while the ROCm 10 image changes Python, Triton, FlyDSL and SDK library layout. This change selects the digest-pinned SGLang 0.5.20 / ROCm 10 image and integrates the runtime changes with current main.

Changes

  • Bind scoring and resumed workspaces to their runtime image. Separate SDK caches by Python ABI, select one ROCm SDK library tree and use it for rocprofv3.
  • Update optional evaluation-tool runtimes and FlyDSL API adapters. Preserve capability checks, required controls and task-local attribution. Retain main's stronger rotating-input checks and serving-runtime setup.
  • Restore the gfx950 MLA two-stage pipeline with four waves per block. Add reversed-page and partial-length correctness checks and reset the output on every measured invocation, equally for both roles. Keep the independent reference, six original workloads, 1% numerical gate, ten external warmups and 100 graph samples. Bind source assertions to the matching finalized GPU report. Keep the original protected harness hash by comparing additive checks separately, with their own positive and negative coverage. Reject ambiguous constexpr positions and out-of-range Triton AOT argument indices. Stabilize the process-exit regression after asynchronous SIGKILL delivery without changing action deadlines. Construct signed-zero regression inputs with explicit bytes so fast-math cannot remove the intended mutation; byte-exact checks remain unchanged.
  • Revise the FP8 fused-MoE normalized maximum-error limit from 1% to 1.3%, equally for baseline and candidate correctness, measured outputs and replay. This is a task-contract revision, not an accuracy improvement.
  • Retain the original hidden/weight replay and add a separate routing replay. Change expert IDs and routing weights, rebuild the untimed reference plan when IDs change, and restore every input after each draw. Both draws must pass; cases, reference math, warmups, samples and measured work remain unchanged.

Validation

Hardware: AMD Instinct MI355X on Linux. Scoring image: lmsysorg/sglang@sha256:e20849665c105d389ef91d23c0dc73931aaa6f02056dd10e7b43e4f16c79df69.

  • Full Linux CI on the final revision: 15,049 passed, 144 skipped, with 18 passing subtests. The benchmark helper audit and Python compilation passed. The MLA follow-up also passed 126 focused CPU regressions, 345 Linux framework tests and a fresh source-bound MI355X task-validator run. Runtime control sources match the qualified snapshot.
  • Focused fused-MoE regressions: 110 passed, including rejection of cached routing, input mutation and errors above 1.3%. Original reference, input-generation and timing fingerprints still pass. Eleven native MLA regressions also pass with each of GCC 15 and Clang 17 after the signed-zero fixture correction.
  • Fresh framework-finalized task_validator reports are PASS for five task packages: batched INT8 GEMM, INT8 preshuffle GEMM, HGEMM, FP8 fused-MoE and MLA. The whole runs and post-processing completed successfully. Captured actions, report markers, 85 raw task files and 90 materialized files were checked against the final source.
  • The final fused-MoE GPU job passed its own arithmetic/event-timing check and 345 Linux framework tests. All three cases passed correctness, measured-output validation and both independent replay draws, retaining 100 timing samples per case. Batched INT8 GEMM and fused-MoE intentionally start with unimplemented candidates; those initial reports qualify the task and baseline, not generated candidates or speedups.
  • Five earlier finalized PASS reports match the other changed task packages and the same image. Their raw sources match. An audit against the previous PR revision found the old canonical timing implementation unchanged, and added opt-in/serving helpers are unused by these tasks; saved historical materialized-helper copies were not available locally. These historical reports are reused as task-source evidence, rather than presented as fresh integrated-framework runs.
  • MLA: 120/120 repeated GPU comparisons passed for the repaired configuration. A fresh-process, matched ABBA measurement includes the same per-invocation output reset for both configurations. The 128-head/rank-512 workload measured about 21.46 us with the single-stage fallback and 13.67 us with the repair, or 36% lower latency; the other five cases ranged from 2.9% faster to 2.1% slower. Timings from the earlier no-reset harness are not directly comparable.
  • A bounded check of SGLang 0.5.22 / ROCm 10 (lmsysorg/sglang@sha256:1438dd8b9000adaefb5828edb1cc5713992815d4296f4ad1a1e98dee4e9c7e5f) reproduced the original two-wave error on 104/120 checks, with the same frozen inputs as 0.5.20. Both images contain the same Triton compiler binary. This is an MLA diagnostic, not full-catalog qualification of the newer image; the default image remains unchanged.
  • Historical coverage remains 448 passing initial-action or independent-validator results among 459 selections. That catalog differs from current main. Ten selections require their dedicated vLLM runtime; the old fused-MoE numerical failures retain their original outcomes. This is not a fresh full-catalog PASS claim.

Limits

  • The 1.3% fused-MoE limit does not fix the native operator's numerical variation. Earlier diagnostics include an approximately 1.24% outlier, which falls within the revised 1.3% limit. Its original failure under the earlier limit is retained; the fresh finite validation does not establish that every invocation stays below 1.3%.
  • GPU ASan remains unqualified on the tested hosts because ordinary host-allocation GPU access fails its prerequisites. ASan qualification is excluded from this runtime update. Enabling it still requires the original safe and known-bug controls, with no failed control relabeled as a pass.
  • The earlier failed MLA attempts and five non-ASan tool startup results retain their source/runtime boundaries. The MLA change avoids the failing compiler schedule; it does not establish an upstream compiler root-cause fix. Waitcheck and ConSan remain advisory. No overall kernel performance improvement is claimed.

Original failures, complete validation reports, source bindings and diagnostic operands are retained with the experiment records.

Pin the new scoring image, adapt SDK loading and task dependencies, and
update all six evaluation-tool runtimes with matching documentation and
regression coverage. Keep qualification pending for GPU ASan and the
existing fused-MoE numerical failures.
Resolve runner and task integration conflicts, bind the MLA regression to its validated source, reject ambiguous Triton ABI metadata, and stabilize process-exit observation in the materialization test.

Set the declared FP8 fused-MoE normalized-error limit to 1.2% and check changed expert IDs and routing weights during timed replay.
Keep the original hidden/weight perturbation and add a separate routing draw from restored inputs. Validate each draw, restore all inputs on failure, and retain stale-routing and input-mutation controls.
Use memcpy for both signed-zero representations so the native buffer-integrity regression remains meaningful under fast-math optimizations. Keep the byte-exact rejection, restoration assertions and compiler flags unchanged.
Apply the same limit to baseline and candidate correctness, measured outputs
and both replay draws. Keep reference math, inputs, samples and timing intact.
Update the documented contract and boundary regressions.

Validation: 110 focused CPU regressions, 345 Linux framework tests, and a fresh
MI355X task-validator PASS with current-source/report binding.
Keep two pipeline stages, add page-routing correctness checks, and reset
measured outputs on every invocation for both roles. Preserve the original
reference, six workloads, 1% numerical gate and graph sample counts.
Exclude only the two newly covered MLA check helpers from the historical
comparison. Retain the original protected hash and all core assertions.
@irvineoy
irvineoy marked this pull request as ready for review October 20, 2026 21:23
@irvineoy
irvineoy merged commit 894daa0 into main Oct 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant