Skip to content

Compile CP softmax LSE corrections with dynamic shapes - #695

Open
JessicaJiang-123 wants to merge 1 commit into
ROCm:devfrom
JessicaJiang-123:fix/cp-lse-dynamic-compile-rocm-dev
Open

Compile CP softmax LSE corrections with dynamic shapes#695
JessicaJiang-123 wants to merge 1 commit into
ROCm:devfrom
JessicaJiang-123:fix/cp-lse-dynamic-compile-rocm-dev

Conversation

@JessicaJiang-123

Copy link
Copy Markdown

Description

Fixes the flaky ROCm nightly test test_qwen3_30B_A3B/test_baseline.py in miles, where rollout-0 log_probs and ref_log_probs intermittently fail the abs_tol=1e-8 check.

The two CP softmax LSE correction functions use @jit_fuser, which wraps torch.compile:

  • flash_attn_fwd_softmax_lse_correction
  • flash_attn_fwd_second_half_softmax_lse_correction

Dynamo may compile multiple variants of these functions after seeing different input shapes. On gfx950, those variants can select different launch configurations, and libdevice.log1p is not bit-exact across them.

In failing runs, the reference and actor forwards receive bit-identical operands but use different log1p paths. The resulting approximately 1-ULP difference is repeatedly introduced across the model's attention layers and produces a final 1e-5 to 2e-4 log-probability mismatch.

This PR compiles only these two functions with dynamic=True from their first invocation. This avoids the initial static specialization and keeps both forwards on the same numerical path.

CP=1 is unaffected because the LSE merge is only used when CP > 1.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add jit_fuser_dynamic to jit.py, mirroring jit_fuser but using torch.compile(..., dynamic=True).
  • Apply jit_fuser_dynamic only to the two CP softmax LSE correction functions.

Verification

Tested test_qwen3_30B_A3B/test_baseline.py on 4× MI355X with: CP=2/PP=2/TP=1/EP=2

Comparing rollout-0 reference and actor log-probabilities:

Configuration Passes Bit-identical
Before ~2/9 No
This PR 5/5 Yes

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@sudhu2k sudhu2k added ci-level 3 CI test level 3 ci-level 1 CI test level 1 and removed ci-level 3 CI test level 3 labels Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-level 1 CI test level 1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants