Skip to content

[TPU] Qwen3.5 EP on the regular trainer with per-layer torch.compile - #206

Draft
jialei777 wants to merge 1 commit into
qwen3_5-35b-a3b-tpu-moe-ep-fastfrom
fix/tpu-qwen35-ep-regular-trainer-compile
Draft

jialei777 wants to merge 1 commit into
qwen3_5-35b-a3b-tpu-moe-ep-fastfrom
fix/tpu-qwen35-ep-regular-trainer-compile

Conversation

@jialei777

Copy link
Copy Markdown
Contributor

Stacked on #142 (qwen3_5-35b-a3b-tpu-moe-ep-fast). #201 (CP on EP) is restacked on top of this PR.

Summary

With Expert Parallelism, the regular TPUTrainer's per-layer compile (--compile.enable --compile.backend=tpu, i.e. torch.compile(block, fullgraph=True) per transformer block) failed at the first MoE block. Only TpuGraphTrainer worked. This PR:

  • makes the TPU-generation and Tokamax KDA lookups that Ring-of-Experts and the Tokamax GatedDeltaNet call from traced code safe under Dynamo;
  • adds qwen35_moe_35b_ep: qwen35_moe_35b on the regular trainer with EP=4 and the Tokamax GatedDeltaNet. It runs eagerly by default, and --compile.enable --compile.backend=tpu now works.

verl's TPU engine uses the same per-block torch.compile(fullgraph=True) path, with EP and the Tokamax GatedDeltaNet, for example in the Qwen3.5-397B GRPO runs. verl-hardware-plugin #115 works around this bug by monkeypatching these helpers at runtime (patch_torchtitan_tpu_dynamo_caches). Once this lands, the plugin can drop that patch. Its patch stays compatible with this PR in the meantime.

Root cause

A12 (tpu7x-2x2x1, debug MoE, EP2, --compile.enable --compile.backend=tpu; the same failure with and without CP):

TpuQwen35RoutedExperts.forward -> TpuQwen35TokenDispatcher.dispatch
-> _dispatch_ring_of_experts -> ragged_gather_tc.safe_reduce_block_size
-> _tpu_vmem_limit_bytes -> _tpu_generation()   # @assume_constant_result over @lru_cache
-> torch.tpu.get_device_properties() -> hardware.get_tpu_spec()
   (torch_tpu's host device scan, which Dynamo cannot trace)
  • Why the lookups fail: Dynamo never consults an lru_cache; it traces the wrapped function.
    • With assume_constant_result stacked on top, the decorated object is still the lru_cache wrapper, so Dynamo traced into torch_tpu's device query.
    • tokamax_gated_delta._tpu_generation and _import_kda (both lru_cache only) fail the same way when the Tokamax GatedDeltaNet runs inside a compiled block.
  • Why _import_kda needs a different fix: it returns a callable, so it cannot go through assume_constant_result either. Dynamo gives the callable a ConstantSource, whose CLOSURE_MATCH guard is not implemented inside activation checkpointing.

Fix

  • ragged_gather_tc._tpu_generation and tokamax_gated_delta._tpu_generation now cache their result in a module-level list under assume_constant_result, the same pattern as causal_conv1d._tpu_generation. Dynamo calls them once at trace time and bakes the result into the graph.
  • tokamax_gated_delta._import_kda now caches in module globals, which Dynamo guards through a regular GlobalSource. On TPUs that run the kernels, TokamaxGatedDeltaKernel.__init__ fills both caches eagerly, before any tracing.
  • New recipe qwen35_moe_35b_ep = qwen35_moe_35b + _use_tokamax_gated_delta (document resets stay on) + _use_expert_parallel(ep_degree=4), which configures:
    • Ring-of-Experts dispatch with token chunks;
    • SparseCore offload flags;
    • the eager DEFER_AND_FUSE gradient norm (compile_grad_norm must stay off with EP).
  • Why the recipe uses Tokamax: the default XLA GatedDeltaNet cannot be compiled per block at 8K context; see Known issues.

Results: Qwen3.5-35B-A3B on the regular trainer

Setup: tpu7x-2x2x2 (16 ranks), ctx 8K, 32K tokens per device, 20 steps, warmup clamped to 20 steps, per-block compile, Tokamax GatedDeltaNet, image torchtitan-container:2026-11-13 + this branch. Throughput is the mean over steps 10-20.

run loss @1 / @10 / @20 s/step tok/s/device MFU
no EP (qwen35_moe_35b, compiled) 12.88448 / 3.96318 / 2.64803 6.25 5,245 10.00%
EP2 compiled (qwen35_moe_35b_ep) 12.88440 / 3.88662 / 2.65545 7.82 4,189 7.97%
EP2 eager (qwen35_moe_35b_ep, 10 steps) 12.88442 / - / - 8.69 3,773 7.19%
  • EP2 compiled is deterministic: two runs, one with the recipe default and one passing the same Tokamax override on the command line, are bit-identical at steps 1/5/10/15/20.
  • Compiled and eager agree at step 1 (12.88440 vs 12.88442; the eager run's 10-step schedule differs after that).
  • Per-block compile is about 10% faster than eager for EP2.
  • EP2 costs +25% step time over no EP at 32K tokens/device, which is roughly in line with TpuGraphTrainer (+40% at 64K tokens/device in [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142).
  • The memory column from torchtitan's monitor (about 6.6 GiB) does not capture the compiled programs on this trainer, so it is not reported. All runs fit in the 96 GiB HBM.

TPU tests (tpu7x-2x2x1, 8 devices, this branch)

  • Unit tests: test_ragged_gather_tc 7 OK, test_tokamax_gated_delta 22 OK, test_causal_conv1d 11 OK. This includes the two new compile tests.
  • test_train_distributed passes 4/4. The cases are EP2 SimpleFSDP and EP2 with the TPU MoE kernels, each eager and compiled; the compiled cases use the Tokamax GatedDeltaNet.
  • test_qwen3_5_tpu passes 60/60.
  • Compiled debug MoE runs (16 ranks, Tokamax GatedDeltaNet) train with no EP and with EP2, under both selective and full activation checkpointing.

Tests

  • test_ragged_gather_tc: new test_vmem_helpers_compile_fullgraph_without_tracing_device_query. It compiles the VMEM helpers with fullgraph=True against a device query that Dynamo cannot trace, and fails without the fix.
  • test_tokamax_gated_delta: new test_lookups_compile_fullgraph_under_selective_ac. It compiles the generation + KDA lookups with fullgraph=True inside selective activation checkpointing, and fails without the fix.
  • test_train_distributed: new TPU-only compiled cases qwen3_5_moe_ep2_simplefsdp_compile and qwen3_5_moe_ep2_tpu_moe_kernels_compile.
    • Both use the Tokamax GatedDeltaNet (see Known issues).
    • The second also exercises the Pallas grouped matmuls and the weights-on-activation combine.
  • test_qwen3_5_tpu: new test_35b_ep_recipe_on_regular_trainer, covering the recipe, its Tokamax and fused-loss overrides, and its CLI with --compile.enable --compile.backend=tpu.
  • CPU (docker torchtitan-container:2026-11-13):
    • Unit tests: ragged 7 OK (3 skipped), tokamax 22 OK (17 skipped), causal_conv1d 11 OK (9 skipped).
    • The two new tests pass both with and without torch_tpu's torch.tpu module loaded.
    • The Qwen3.5 recipe tests pass.
    • The whole regular trainer with --compile.enable --compile.backend=aot_eager and EP2 (4 gloo ranks) gives exactly the eager losses: 12.92785 / 12.78477 / 12.55228.

Known issues (pre-existing, not EP)

Per-block compile of the default XLA GatedDeltaNet fails on TPU, with or without EP:

  • 35B, 8K context: CompileTimeHbmOom in a block's backward. The compiler needs 202-205 GB of HLO temporaries against 94.7 GiB of HBM (no EP, EP2, and EP2 + CP2 alike).
  • Debug MoE model, selective or no activation checkpointing: RuntimeProgramInputMismatch at the first backward. A torch_tpu split-compiled backward partition receives a buffer of the wrong size, e.g. expected f32[2088960] ... got f32[2097024].
    • The same happens with no EP.
    • Full AC or the Tokamax GatedDeltaNet avoid it; the dense debug model is fine.

Hence the recipe and the compiled test cases use the Tokamax GatedDeltaNet, which is also what the RL stack runs.

Known gaps

  • More EP buffer memory than TpuGraphTrainer: the regular trainer does not run TpuGraphTrainer's EP chunk-barrier FX pass (insert_ep_chunk_barriers). As a result, the per-chunk annotate(ep_chunk_idx) has no effect there, and XLA can keep more chunks' EP buffers live at once. verl-hardware-plugin [pallas:mosaic] Use pltpu.TensorCoreMesh #115 has an autograd-level barrier (_EpChunkWindowBarrier) that could be upstreamed as a follow-up.

…rch.compile

The regular TPUTrainer's `--compile.enable --compile.backend=tpu` compiles
every transformer block with `torch.compile(fullgraph=True)`. With Expert
Parallelism this failed at the first MoE block:

  TpuQwen35TokenDispatcher._dispatch_ring_of_experts
  -> ragged_gather_tc.safe_reduce_block_size -> _tpu_vmem_limit_bytes
  -> _tpu_generation()  (@assume_constant_result over @lru_cache)
  -> torch.tpu.get_device_properties() -> hardware.get_tpu_spec()
     (torch_tpu's host device scan, which Dynamo cannot trace)

Dynamo does not consult an lru_cache; it traces the wrapped function. With
`assume_constant_result` stacked on top, the decorated object is still the
lru_cache wrapper, so Dynamo traced into the device query. `tokamax_gated_delta._tpu_generation` and `_import_kda` (lru_cache only)
fail the same way when the Tokamax GatedDeltaNet runs inside a compiled block.
`_import_kda` returns a callable, which cannot go through
`assume_constant_result` either: Dynamo gives it a ConstantSource whose
CLOSURE_MATCH guard is not implemented inside activation checkpointing.

- `ragged_gather_tc._tpu_generation`, `tokamax_gated_delta._tpu_generation`:
  cache in a module-level list under `assume_constant_result`, as
  `causal_conv1d._tpu_generation` does, so Dynamo evaluates them once at trace
  time and bakes the result into the graph.
- `tokamax_gated_delta._import_kda`: cache in module globals, guarded through a
  regular GlobalSource. `TokamaxGatedDeltaKernel.__init__` fills both caches
  eagerly on TPUs that run the kernels.
- New `qwen35_moe_35b_ep` recipe: `qwen35_moe_35b` on the regular trainer with
  EP=4 (`_use_expert_parallel`: Ring-of-Experts, token chunks, SparseCore
  offload flags, eager DEFER_AND_FUSE gradient norm) and Tokamax's Mosaic KDA
  GatedDeltaNet (the per-block compile of the default XLA recurrence needs
  ~204 GB of HLO temporaries for one block's backward at 8K context). Eager by
  default; `--compile.enable --compile.backend=tpu` now works.

Tests:
- `test_ragged_gather_tc`: VMEM helpers under `torch.compile(fullgraph=True)`
  with a device query that calls `os.listdir` (fails without the fix).
- `test_tokamax_gated_delta`: generation + KDA lookups under fullgraph compile
  inside selective activation checkpointing (fails without the fix).
- `test_train_distributed`: TPU-only compiled EP2 cases
  (`qwen3_5_moe_ep2_simplefsdp_compile`,
  `qwen3_5_moe_ep2_tpu_moe_kernels_compile`).
- `test_qwen3_5_tpu`: `qwen35_moe_35b_ep` recipe and CLI with
  `--compile.enable --compile.backend=tpu`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant