Repository navigation
Conversation
…rch.compile
The regular TPUTrainer's `--compile.enable --compile.backend=tpu` compiles
every transformer block with `torch.compile(fullgraph=True)`. With Expert
Parallelism this failed at the first MoE block:
TpuQwen35TokenDispatcher._dispatch_ring_of_experts
-> ragged_gather_tc.safe_reduce_block_size -> _tpu_vmem_limit_bytes
-> _tpu_generation() (@assume_constant_result over @lru_cache)
-> torch.tpu.get_device_properties() -> hardware.get_tpu_spec()
(torch_tpu's host device scan, which Dynamo cannot trace)
Dynamo does not consult an lru_cache; it traces the wrapped function. With
`assume_constant_result` stacked on top, the decorated object is still the
lru_cache wrapper, so Dynamo traced into the device query. `tokamax_gated_delta._tpu_generation` and `_import_kda` (lru_cache only)
fail the same way when the Tokamax GatedDeltaNet runs inside a compiled block.
`_import_kda` returns a callable, which cannot go through
`assume_constant_result` either: Dynamo gives it a ConstantSource whose
CLOSURE_MATCH guard is not implemented inside activation checkpointing.
- `ragged_gather_tc._tpu_generation`, `tokamax_gated_delta._tpu_generation`:
cache in a module-level list under `assume_constant_result`, as
`causal_conv1d._tpu_generation` does, so Dynamo evaluates them once at trace
time and bakes the result into the graph.
- `tokamax_gated_delta._import_kda`: cache in module globals, guarded through a
regular GlobalSource. `TokamaxGatedDeltaKernel.__init__` fills both caches
eagerly on TPUs that run the kernels.
- New `qwen35_moe_35b_ep` recipe: `qwen35_moe_35b` on the regular trainer with
EP=4 (`_use_expert_parallel`: Ring-of-Experts, token chunks, SparseCore
offload flags, eager DEFER_AND_FUSE gradient norm) and Tokamax's Mosaic KDA
GatedDeltaNet (the per-block compile of the default XLA recurrence needs
~204 GB of HLO temporaries for one block's backward at 8K context). Eager by
default; `--compile.enable --compile.backend=tpu` now works.
Tests:
- `test_ragged_gather_tc`: VMEM helpers under `torch.compile(fullgraph=True)`
with a device query that calls `os.listdir` (fails without the fix).
- `test_tokamax_gated_delta`: generation + KDA lookups under fullgraph compile
inside selective activation checkpointing (fails without the fix).
- `test_train_distributed`: TPU-only compiled EP2 cases
(`qwen3_5_moe_ep2_simplefsdp_compile`,
`qwen3_5_moe_ep2_tpu_moe_kernels_compile`).
- `test_qwen3_5_tpu`: `qwen35_moe_35b_ep` recipe and CLI with
`--compile.enable --compile.backend=tpu`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #142 (
qwen3_5-35b-a3b-tpu-moe-ep-fast). #201 (CP on EP) is restacked on top of this PR.Summary
With Expert Parallelism, the regular
TPUTrainer's per-layer compile (--compile.enable --compile.backend=tpu, i.e.torch.compile(block, fullgraph=True)per transformer block) failed at the first MoE block. OnlyTpuGraphTrainerworked. This PR:qwen35_moe_35b_ep:qwen35_moe_35bon the regular trainer with EP=4 and the Tokamax GatedDeltaNet. It runs eagerly by default, and--compile.enable --compile.backend=tpunow works.verl's TPU engine uses the same per-block
torch.compile(fullgraph=True)path, with EP and the Tokamax GatedDeltaNet, for example in the Qwen3.5-397B GRPO runs. verl-hardware-plugin #115 works around this bug by monkeypatching these helpers at runtime (patch_torchtitan_tpu_dynamo_caches). Once this lands, the plugin can drop that patch. Its patch stays compatible with this PR in the meantime.Root cause
A12 (tpu7x-2x2x1, debug MoE, EP2,
--compile.enable --compile.backend=tpu; the same failure with and without CP):lru_cache; it traces the wrapped function.assume_constant_resultstacked on top, the decorated object is still thelru_cachewrapper, so Dynamo traced into torch_tpu's device query.tokamax_gated_delta._tpu_generationand_import_kda(bothlru_cacheonly) fail the same way when the Tokamax GatedDeltaNet runs inside a compiled block._import_kdaneeds a different fix: it returns a callable, so it cannot go throughassume_constant_resulteither. Dynamo gives the callable aConstantSource, whoseCLOSURE_MATCHguard is not implemented inside activation checkpointing.Fix
ragged_gather_tc._tpu_generationandtokamax_gated_delta._tpu_generationnow cache their result in a module-level list underassume_constant_result, the same pattern ascausal_conv1d._tpu_generation. Dynamo calls them once at trace time and bakes the result into the graph.tokamax_gated_delta._import_kdanow caches in module globals, which Dynamo guards through a regularGlobalSource. On TPUs that run the kernels,TokamaxGatedDeltaKernel.__init__fills both caches eagerly, before any tracing.qwen35_moe_35b_ep=qwen35_moe_35b+_use_tokamax_gated_delta(document resets stay on) +_use_expert_parallel(ep_degree=4), which configures:DEFER_AND_FUSEgradient norm (compile_grad_normmust stay off with EP).Results: Qwen3.5-35B-A3B on the regular trainer
Setup: tpu7x-2x2x2 (16 ranks), ctx 8K, 32K tokens per device, 20 steps, warmup clamped to 20 steps, per-block compile, Tokamax GatedDeltaNet, image
torchtitan-container:2026-11-13+ this branch. Throughput is the mean over steps 10-20.qwen35_moe_35b, compiled)qwen35_moe_35b_ep)qwen35_moe_35b_ep, 10 steps)TpuGraphTrainer(+40% at 64K tokens/device in [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142).TPU tests (tpu7x-2x2x1, 8 devices, this branch)
test_ragged_gather_tc7 OK,test_tokamax_gated_delta22 OK,test_causal_conv1d11 OK. This includes the two new compile tests.test_train_distributedpasses 4/4. The cases are EP2 SimpleFSDP and EP2 with the TPU MoE kernels, each eager and compiled; the compiled cases use the Tokamax GatedDeltaNet.test_qwen3_5_tpupasses 60/60.Tests
test_ragged_gather_tc: newtest_vmem_helpers_compile_fullgraph_without_tracing_device_query. It compiles the VMEM helpers withfullgraph=Trueagainst a device query that Dynamo cannot trace, and fails without the fix.test_tokamax_gated_delta: newtest_lookups_compile_fullgraph_under_selective_ac. It compiles the generation + KDA lookups withfullgraph=Trueinside selective activation checkpointing, and fails without the fix.test_train_distributed: new TPU-only compiled casesqwen3_5_moe_ep2_simplefsdp_compileandqwen3_5_moe_ep2_tpu_moe_kernels_compile.test_qwen3_5_tpu: newtest_35b_ep_recipe_on_regular_trainer, covering the recipe, its Tokamax and fused-loss overrides, and its CLI with--compile.enable --compile.backend=tpu.torchtitan-container:2026-11-13):torch.tpumodule loaded.--compile.enable --compile.backend=aot_eagerand EP2 (4 gloo ranks) gives exactly the eager losses: 12.92785 / 12.78477 / 12.55228.Known issues (pre-existing, not EP)
Per-block compile of the default XLA GatedDeltaNet fails on TPU, with or without EP:
CompileTimeHbmOomin a block's backward. The compiler needs 202-205 GB of HLO temporaries against 94.7 GiB of HBM (no EP, EP2, and EP2 + CP2 alike).RuntimeProgramInputMismatchat the first backward. A torch_tpu split-compiled backward partition receives a buffer of the wrong size, e.g.expected f32[2088960] ... got f32[2097024].Hence the recipe and the compiled test cases use the Tokamax GatedDeltaNet, which is also what the RL stack runs.
Known gaps
TpuGraphTrainer: the regular trainer does not runTpuGraphTrainer's EP chunk-barrier FX pass (insert_ep_chunk_barriers). As a result, the per-chunkannotate(ep_chunk_idx)has no effect there, and XLA can keep more chunks' EP buffers live at once. verl-hardware-plugin [pallas:mosaic] Usepltpu.TensorCoreMesh#115 has an autograd-level barrier (_EpChunkWindowBarrier) that could be upstreamed as a follow-up.