Repository navigation
Conversation
jialei777
force-pushed
the
feat/tpu-qwen35-context-parallel
branch
from
November 19, 2026 22:40
7e53e0f to
d63e308
Compare
…DeltaNet TPU CP (#2) only rewrote attention, so on Qwen3.5 the GatedDeltaNet layers (30 of 40 in 35B-A3B) would run their causal conv and delta-rule recurrence on each rank's token shard as if it were a whole sequence, silently. - qwen3_5/cp_gated_delta.py: CPTpuInnerGatedDeltaNet runs the GatedDeltaNet head-parallel: all-to-alls take q/k/v/a/b from token- to head-sharded (one per input: packing them measured slower), the tokens go back in sequence order (head-tail undone for KV all-gather CP), conv + recurrence run over whole sequences on the rank's heads with its slice of the conv weights / A_log / dt_bias (index_select on the cached CP-rank tensor), and one all-to-all brings the output back. Exact; works with both full-attention strategies. Plus GatedDeltaNetContextParallelTransform and a registered validator that rejects CP configs whose GatedDeltaNet layers are not converted. - muse_glimmer/attention.py: Ulysses / KV all-gather CP versions of SlidingWindowScaledDotProductInnerAttention (Qwen3.5's TPU full attention). - qwen3_5/model.py: TpuQwen35Model shards CP inputs through the CP attention's cp_shard (per-sequence head-tail, CP-rank cache), as core Decoder does; upstream Qwen35Model calls prepare_context_parallel_input. - qwen3_5/infra/parallelize.py: TPUSelectiveAC under CP; upstream's parallelize steps without its CP NotImplementedError. - distributed/cp_attention.py: public cp_all_to_all / cp_all_gather / cp_rank_tensor; register_tpu_cp_inner_attention and register_tpu_cp_validator registries. - qwen3_5/config_registry.py: _qwen35_cp and debug / 35B CP recipes.
- graph_trainer/qwen3_5.py: allow CP (still rejects TP / PP / EP). - graph_trainer/graph_split.py: fsdp_pipeline raised for any backward all-gather / reduce-scatter that is neither a parameter gather nor a gradient reduction, including KV all-gather CP's recomputed K/V gathers and dK/dV reduce-scatters. Raise only when such a collective runs on the process group of a gradient reduction (gathers of saved activations are also fed only by graph inputs, so only reductions name the FSDP groups); on other groups (CP) it stays with its inputs, the existing fallback.
…n tests
- qwen3_5/tests/test_cp_gated_delta.py (CPU, emulated CP ranks, eager and
traced collectives): CP GatedDeltaNet vs single device (cp 2/4, both token
layouts, packed-document resets, outputs and every gradient), the CP debug
model vs single device (loss and all parameter gradients), recipes,
transform and validation.
- test_train_minimal_distributed: qwen3_5_simplefsdp_cp2_{all_gather,ulysses}
and their per-layer torch.compile + Tokamax Splash variants (parity vs the
CPU reference).
- test_train_distributed: Qwen3.5 dense / MoE + Splash CP cases and FSDP2 CP.
…rallel GatedDeltaNet - CPTpuInnerGatedDeltaNet raises without an active CP mesh axis instead of running the single-device recurrence on each rank's token shard. cp == 1 reruns of CP recipes revert it: revert_tpu_context_parallel now runs reverters registered with register_tpu_cp_reverter (revert_gated_delta_context_parallel). - cp_shard (KV all-gather and Ulysses) records the token layout it sharded with (cp_input_layout). The GatedDeltaNet checks the head-tail rows it derives from set_unflatten_seq_len against it, and that Ulysses sharded one contiguous folded stream, so a layout mismatch between sharding and forward raises instead of silently scrambling the token order. - validate_tpu_context_parallel also rejects non-causal KV all-gather windows given as lists (config files / command line). - cp_rank_tensor warns once (instead of a debug log) when it bakes the rank into a traced graph. - TpuQwen35Model.preprocess_inputs raises instead of asserting on missing positions. - Tests: no-CP-group error, revert, cp_shard layout recording, head-tail inverse against torch's _HeadTailLoadBalancer for an unfolded sequence (cp 2/4), layout mismatch errors, list windows.
…r layout and graph-split checks - cp_rank_tensor: a cache miss under Dynamo raises a clear RuntimeError instead of calling the logger, which Dynamo cannot trace (a hard error under per-layer fullgraph compile); non-strict tracing still warns once. - GatedDeltaNet CP _rows raises when the sequence length does not divide the token count (TpuInnerGatedDeltaNet rejects it too), instead of a dead one-sequence fallback. - graph_split: without any recognized gradient reduction, keep the strict check on unclassified backward collectives instead of skipping it. - validate_gated_delta_context_parallel checks TP once, only for models with GatedDeltaNet layers; register_tpu_cp_inner_attention rejects a second, different CP pair for the same attention. - _parallelize_text_qwen3_5_with_cp returns the model and notes to stay in sync with upstream; clearer comments (layout global, reference attention masks, positions all-gather). - Tests: real Dynamo fullgraph compile of the CP GatedDeltaNet (both layouts, one graph) and the uncached-rank error; Ulysses CP over the sliding-window SDPA vs single device (cp 2/4); duplicate registration; reset of the layout / rank-miss globals between tests.
… test coverage, docstrings - _record_layout_of_input clears the recorded CP layout when there is no input to record, so a stale layout is never checked against. - cp_rank_tensor docstring describes the eager and tracing miss paths. - CPQwen35ModelParityTest shards through the recipe attention's real cp_shard (mocked torch chunking), so the recorded layout and the GatedDeltaNet layout checks run in the end-to-end CPU parity test. - Tests: TpuQwen35Model.preprocess_inputs rejects missing positions and mrope_positions; the 35B and split-Tokamax GraphTrainer CP recipes build and convert every layer; one shared _reset_cp_state for the CP globals; the graph-split test restores the mutated node on failure. - Docstring summaries fit in 80 columns.
…ther, docs and test gaps - cp_all_gather raises if its input requires a gradient (the traced collective has none, so eager and compiled runs would differ); _full_sequence_segment_ids uses it instead of repeating the collective. - The CP reverts build configs from init fields only. - _qwen35_cp keeps the config subclass in its return type (TypeVar), as the split-Tokamax GraphTrainer CP recipes return TpuGraphTrainer.Config. - Docstrings and comments match the code after the cp_rank_tensor extraction: _normalize_cp_device callers, cp_rank_tensor Raises, what the qwen3_5 import registers, why CP preprocessing needs positions. - Tests: GatedDeltaNet CP rejects TP; register_tpu_cp_inner_attention rejects a wrong mixin; cp_rank_tensor non-strict tracing miss warns once without caching; layout recording with TpuQwen35Model's real input sharding; the uncached-rank compile error pinned to RuntimeError.
jialei777
force-pushed
the
feat/tpu-qwen35-context-parallel
branch
from
November 21, 2026 04:57
e01195a to
8dfd350
Compare
…ment resets, all-gather/reduce-scatter GDN CP strategy, fused GDN input projections - TpuQwen35Model.forward(cp_full_positions=...) threads the unsharded positions to every layer so CP attention and the head-parallel GatedDeltaNet build document boundaries without per-layer positions all-gathers (both CP strategies); preprocess_inputs sets it automatically. - gated_delta_document_resets per-batch override (single-document rows). - CPTpuGatedDeltaNet + GatedDeltaNetContextParallelTransform(strategy= 'allgather_reducescatter'): AG(x) + head-sliced projections/GDN/norm + RS(out_proj) for any cp dividing the head counts and either load balancer. - TpuGatedDeltaNet.Config.fuse_input_projections. - cp_all_gather_with_grad / cp_reduce_scatter collectives. - Tests: model parity over strategies x eager/traced x GDN strategy x cp_full_positions x document-reset override; fullgraph compile test for the new strategy; transform/projection/preprocess tests. Ported from #216 (stacked on the EP branch) onto main: main's module layout (distributed/cp_attention.py, components/unflatten_shim.py), main's TpuQwen35Model (EP config) and the #200 review fixes (layout checks also on the all-gather / reduce-scatter path; revert_tpu_context_parallel also reverts CPTpuGatedDeltaNet).
jialei777
marked this pull request as ready for review
November 21, 2026 06:46
- test_cp_gated_delta: the all-gather/reduce-scatter compile test class reused the name CompiledCPGatedDeltaNetTest and shadowed the all-to-all one, so its fullgraph-compile and uncached-rank tests never ran; renamed it, and a new test fails on duplicate test class names. - The plain-nn.Linear checks of the fused / head-sliced GatedDeltaNet projections and of out_proj also run while Dynamo compiles (the per-layer compile is the production path); they were skipped there. - CPTpuGatedDeltaNet docstring: correct collective count (five all-to-alls in) and per-rank communication volume (the all-gather/reduce-scatter strategy moves cp * D / C times the all-to-all bytes: even at cp = 2, 2x at cp = 4 on 35B-A3B); recommend benchmarking before choosing it. - Document _CP_INPUT_LAYOUT as a best-effort, trace-time check. - Revert an unrelated formatting change in TpuRMSNormGated.forward.
…docstring numbers, polish - KV all-gather CP: the positions / segment_ids fallbacks use cp_all_gather instead of open-coding the traced / eager all-gather branch. - CPTpuGatedDeltaNet docstring: the "even at cp = 2, 2x at cp = 4" figures assume the default float32 cp_reduce_dtype (0.66x / 1.33x with bfloat16). - TpuQwen35Model.forward: the unsupported-kwargs error lists exactly the rejected kwargs; drop stray parentheses. - Sort cp_attention.__all__. - Tests: the fused / head-sliced projections reject a Linear subclass with its own forward; the graph-split CP-group check restores the node in a finally instead of both a cleanup and an explicit restore.
…ck, uniform position kwargs, docstrings, more tests - TpuGatedDeltaNet passes positions/full_positions uniformly to its inner module; class/forward docstrings describe full_positions, per-call document_resets and fused/head-sliced input projections. - _is_plain_linear shared by input_projections and the AG/RS out_proj check. - Tests: AG/RS rejects a non-CP inner module and a non-plain out_proj; out_proj bias parity at cp=2 (all_to_all and allgather_reducescatter); clearer fused/sliced subTest loop. Drafted with Jetski.
…ers, no dead cu_seqlens op - _gather_positions / _sequence_positions shared by CPTpuInnerGatedDeltaNet and the allgather_reducescatter path (no duplicated gather/reorder logic). - allgather_reducescatter passes cu_seqlens=None instead of building an unused arange; TpuInnerGatedDeltaNet.forward documents its arguments. - Test docstring/config-build polish. Drafted with Jetski.
…CP inner, comment why layer kwargs are conditional Drafted with Jetski.
…rd docstring wording Drafted with Jetski.
…er_reducescatter trade-off TPU7x A/B on Qwen3.5-35B-A3B (TpuGraphTrainer, Ulysses CP2, 8K): fuse_input_projections cost 3 pt MFU (15.55% vs 18.60%), +1.3 s/step and +2.2 GiB, so remove it. allgather_reducescatter at 32K: cp=2 +2% (fp32) / +4% (bf16 reduce, bit-identical at cp=2) throughput for ~4 GiB more HBM; cp=4 fp32 -6%. Documented in CPTpuGatedDeltaNet. Drafted with Jetski.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#2 added TPU context parallelism for attention:
all_gather(load-balanced KV all-gather) andulysses, enabled per recipe through coreContextParallelTransform. On Qwen3.5 that covers only the 10 full-attention layers of 35B-A3B; the other 30 layers are GatedDeltaNet. Core CP and its validation only look atBaseAttention. So withcontext_parallel_degree > 1, the GatedDeltaNet layers would run their causal conv and delta-rule recurrence on each rank's token shard as if it were a whole sequence: no error, wrong results. Upstream (GPU)parallelize_qwen3_5rejects CP for the same reason.This PR makes Qwen3.5 CP work on TPU:
qwen3_5/cp_gated_delta.py, the same idea as Megatron-LM's Mamba CP).q/k/vchannels and thea/bgate inputs from token-sharded (T/cptokens, every head) to head-sharded (Ttokens,1/cpof the heads).A_loganddt_bias, selected withindex_selecton the cached CP-rank tensor, so all ranks run the same compiled graph.n_key_headsandn_value_headsdivisible bycp(35B-A3B: 16 and 32, socpcan be 2, 4, 8 or 16).SlidingWindowScaledDotProductInnerAttentionthat Qwen3.5 uses on TPU, registered sotpu_cp_inner_attentionfinds them. Splash and Tokamax Splash plug in as in feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2.TpuQwen35Model.preprocess_inputsroutes CP sharding through the CP attention'scp_shard, as coreDecoderdoes. UpstreamQwen35Modelcallsprepare_context_parallel_inputitself, which would head-tail the folded[B*S]stream instead of each sequence.cp, and CP + TP.cp_full_positions(TpuQwen35Model.forward): the unsharded[B, S]positions, computed once inpreprocess_inputsbeforecp_shard. With it, no layer all-gatherspositions: neither the attention segment ids nor the GatedDeltaNet document resets. TpuGraphTrainer and the regular trainer both callpreprocess_inputs, so this is automatic.gated_delta_document_resets(TpuQwen35Model.forward, per batch): overridesConfig.document_resetsfor one micro-batch, e.g.Falsewhen every row holds one document.GatedDeltaNetContextParallelTransform(strategy="allgather_reducescatter"), an opt-in GatedDeltaNet CP strategy: all-gatherx, head-sliced input projections, whole-sequence conv and recurrence on the rank's heads, column-slicedout_proj, reduce-scatter (cp_reduce_dtype, fp32 by default). Two collectives instead of six.all_to_all.fuse_input_projectionsoption was removed: it measured 3 pt MFU slower (see below).fsdp_pipelineused to reject backward collectives that have no FSDP role. It now rejects them only on a process group that carries gradient reductions. Elsewhere, for example KV all-gather CP's dK/dV reduce-scatters on the CP group, they stay with their inputs.Everything is under
torchtitan/experiments/tpu/. Core torchtitan is unchanged.Base. Rebased onto
main5a8b0e8, which includes Tanvi's Qwen3.5 EP series (EP 1/9-7/9).TpuQwen35Modelkeeps main's EPConfig.update_from_configand adds the CPpreprocess_inputs.infra/parallelize.pykeeps main'sapply_qwen3_5_simple_fsdpand adds the CP path.CP + EP. The TpuGraphTrainer Qwen3.5 gate now rejects only TP and PP, plus main's "EP requires SimpleFSDP" rule. So CP + EP passes the gate, but it has not been validated with Tanvi's EP (no parity test or TPU run yet). Treat it as experimental until a follow-up adds EP + CP recipes and tests.
Enabling CP (same pattern as #2 / GPU)
New recipes:
qwen35{,_moe}_debugmodel_{allgather,ulysses}_cp2qwen35_moe_35b_{allgather,ulysses}_cp2(regular trainer)qwen35_moe_35b_split_tokamax_benchmark_{allgather,ulysses}_cp2(TpuGraphTrainer)Set
--parallelism.context_parallel_degreeand--training.max_context_lengthon the command line. A CP group shares one DP rank'snum_tokens_per_microbatch_per_dp_rank, so to keep the same tokens per device, multiply it bycp.Results: Qwen3.5-35B-A3B on TPU7x (NAP cluster)
Setup: the
TpuGraphTrainerbenchmark recipeqwen35_moe_35b_split_tokamax_benchmarkand its_ulysses_cp2/_allgather_cp2variants. The recipe uses ZeRO-3 overdp_shard x cp, a per-layer graph split withfsdp_pipeline, Tokamax GatedDeltaNet and Tokamax Splash,tpu_selectiveAC, and bf16. Data is the C4 test set. Image:torchtitan-container:2026-11-13+ this branch.num_tokens_per_microbatch_per_dp_ranktocp x 65536.S x S(6 * H * (d_qk + d_v) * Sper token and layer), with no causal or document discount. The count is the same with and without CP. At long context, where attention dominates, this convention inflates MFU compared with the work causal Splash actually does.Same context with and without CP (the CP cost):
dp 16)dp 8 x cp 2)dp 4 x cp 4)¹dp 32, 1 sequence/device)dp 16 x cp 2)Contexts only CP can train: 64K tokens per device, with each sequence split over
cpdevices.¹
--parallelism.no-enable_tpu_topo_mesh: with the topology-aware mesh,spmd_typesrejects the(dp, cp)layout forcp > 2(see Known issues). The topology-aware mesh makes no difference at CP1/CP2: 26.94% / 25.01% MFU at 32K without it, vs 26.91% / 24.96% with it.² Compiled (about 18 min), but the first step never finished: all ranks were idle for more than 12 min, with or without the topology-aware mesh. Not investigated.
Flags: the no-CP and Ulysses runs from 32K up add
--splash_attention_kernel.no-sa_use_document_aware_block_skipping(KV all-gather never skips), and every run from 64K up adds--splash_attention_kernel.no-sa_use_fused_bwd_kernel(see Notes).Notes
split_dataset_by_node), sodp 16anddp 8see different samples per step. Runs with the same DP degree agree: at step 20 Ulysses gives 2.56204 and KV all-gather 2.56451. Exactness is covered by the step-by-step parity tests.q/k/v/a/bin and the output back for each of the 30 GatedDeltaNet layers, plus Ulysses attention on the 10 full-attention layers. KV all-gather sends the same GatedDeltaNet traffic plus K/V all-gathers and dK/dV reduce-scatters.cpthat fits the context, and prefer Ulysses when the head counts allow it.(S / block_kv_dkv) x H x S x Dpartial-dq buffer: 128 GiB atS = 128K(H = 16,D = 256,block_kv_dkv = 1024). With it, 128K fails at compile time withCompileTimeHbmOomboth without CP (168 GB) and with Ulysses CP2 (104 GB). Even with the unfused backward, no CP OOMs at runtime with a 128K sequence on one device, because activation memory grows with tokens per device.Commands
Each run is
PJRT_DEVICE=TPU torchrun --nnodes=<hosts> --nproc_per_node=8 ... -m torchtitan.experiments.tpu.trainwith:Validation on the current tree
TpuGraphTrainer
qwen35_moe_35b_split_tokamax_benchmark*on TPU7x 2x2x2, 64K tokens/device, 20 steps, same flags as above. Means over steps 10/15/20.fuse_input_projectionswas removed.allgather_reducescatterand fused runs use benchmark-only recipes that are not committed.The current tree reproduces the pre-rebase numbers (losses identical):
GatedDeltaNet
allgather_reducescattervs the defaultall_to_all:all_to_all(B)all_to_all(B)all_to_allall_to_allall_to_all(T, D)input. At CP4 it moves 2x the bytes with fp32 (1.33x with bf16), which matches the slowdown.cp_full_positionsis automatic in images B and C and makes no measurable difference here. GatedDeltaNet resets are off in this recipe, so only the attention segment-id all-gather goes away.fuse_input_projections(removed) measured 15.55% vs 18.60% MFU (Ulysses CP2, 8K): +1.3 s/step and +2.2 GiB.Changes
qwen3_5/cp_gated_delta.py(new)CPTpuInnerGatedDeltaNet: head-parallel GatedDeltaNet. All-to-alls in, head-tail -> sequence order, rank-sliced conv /A_log/dt_bias, all-to-all out. Falls back toTpuInnerGatedDeltaNetwithout a CP axis.GatedDeltaNetContextParallelTransform: the GatedDeltaNet counterpart of coreContextParallelTransform.CPTpuGatedDeltaNet:cp_strategy="allgather_reducescatter"(withcp_reduce_dtype).validate_gated_delta_context_parallel, registered withvalidate_tpu_context_parallel;revert_tpu_context_parallelreverts both classes forcp == 1references.qwen3_5/model.pyTpuQwen35Model: CP input sharding through the CP attention'scp_shard(upstream callsprepare_context_parallel_inputitself). Text-only under CP.cp_full_positionsandgated_delta_document_resetsforward kwargs, passed to every layer.qwen3_5/gated_delta.pyTpuGatedDeltaNet: passespositions/full_positionsto the inner module, takes a per-calldocument_resetsoverride, and addsinput_projections(optionally head-sliced).qwen3_5/__init__.pyuse_tpu_modulesreturnsTpuQwen35Model.Config. Importing the package registers the GatedDeltaNet CP validator.qwen3_5/infra/parallelize.pyTPUSelectiveACfor plain selective AC (as intpu/qwen3), plus upstream's parallelize steps without its CPNotImplementedError(FSDP2 on the(dp_shard, cp)mesh, or SimpleFSDP).qwen3_5/config_registry.py_qwen35_cpand the debug, 35B and 35B GraphTrainer CP recipes.muse_glimmer/attention.pyUlyssesCP...andKVAllGatherCP...SlidingWindowScaledDotProductInnerAttention, registered withregister_tpu_cp_inner_attention. The KV all-gather reference attention masks packed documents, as the single-device SlidingWindow SDPA does.distributed/cp_attention.pycp_all_to_all,cp_all_gatherandcp_rank_tensor, moved out ofKVAllGatherCPInnerAttention.New
register_tpu_cp_inner_attentionandregister_tpu_cp_validatorregistries.Ulysses
cp_shardalso caches the CP-rank tensor.The KV all-gather causal-window check applies only to Varlen
(left, right)windows.graph_trainer/qwen3_5.pygraph_trainer/graph_split.pyfsdp_pipeline: a backward all-gather or reduce-scatter with no FSDP role raises only if it runs on the process group of a gradient reduction. Others, such as KV all-gather CP's dK/dV reduce-scatters on the CP group, stay with their inputs (the existing fallback).qwen3_5/tests/test_cp_gated_delta.py(new).Qwen3.5 CP cases in
test_train_minimal_distributedandtest_train_distributed.test_graph_split: non-FSDP-group reduce-scatter.test_qwen3_5_tpu: CP no longer rejected by the GraphTrainer parallelize.Tests
test_cp_gated_delta,test_cp_attention,test_graph_split,test_workarounds,test_split_optimizers_by_meshandtest_token_dispatchergive 223 passed.test_qwen3_5_tpu --device=cpugives 48 OK (6 skipped) andtest_tokamax_gated_delta --device=cpu21 OK (17 skipped). The new coverage includes:cp_full_positions, with every position all-gather forbidden;gated_delta_document_resetsoverride;allgather_reducescatterparity (both layouts, eager and traced, with and without anout_projbias);qwen3_5/tests/test_cp_gated_delta.py, 31 tests, using the emulated CP ranks fromtests/test_cp_attention.py, eager and traced collectives):CPTpuInnerGatedDeltaNetvs single-deviceTpuInnerGatedDeltaNetforcp2 and 4, both token layouts, with and without packed-document resets, andB=2(documents cross head-tail chunks). Checks the output,dq/dk/dv/da/db, and the summed per-rank conv /A_log/dt_biasgradients, all to 2e-5.cp=2, both strategies) vs a single device: loss to 1e-5 and every parameter gradient to 1e-4.TpuQwen35Model.preprocess_inputssharding throughcp_shard.test_cp_attention,test_workaroundsandtest_graph_split: 124 passed. The pre-existing CPU suites give the same results asmain.dp_shard 4 x cp 2, on the final commit; the eager cases are flaky on this build, see Known issues):test_train_minimal_distributed, step-by-step loss / grad / param parity vs a single-device CPU reference:qwen3_5_simplefsdp_cp2_{all_gather,ulysses}(eager)...._compile(per-layertorch.compile+ Tokamax Splash + traced all-to-alls).test_train_distributed:qwen3_5_cp2_{all_gather,ulysses},qwen3_5_fsdp_cp2_all_gather(FSDP2),qwen3_5_moe_cp2_{all_gather,ulysses}_splash, and the Qwen3 CP cases.test_qwen3_5_tpu: 43/43.Known issues
Eager (uncompiled) Qwen3.5 CP on TPU is unreliable with
dp_shard > 1on this torch_tpu build (imagetorchtitan-container:2026-11-13). The debug recipes ran for 10 steps on 2x2x1 (dp_shard 4 x cp 2):dp_shard 2 x cp 2), the CP runs are exact and reproducible.--compile.enable(per layer), the same 10-step runs were bit-identical 3 of 3 times for each strategy, and the two strategies agree to 3e-5 at step 10 (8.64935 vs 8.64932). The TpuGraphTrainer benchmarks gave the same loss, 2.56204 at step 20, on two images.positionsall-gather), 2x the heads per rank, or--tpu_config.eager_mode=DEFER_NEVER.DEFER_NEVER_AND_LAUNCH_BLOCKINGremoves the blow-ups (0 of 4) but not the run-to-run differences.This looks like an eager-runtime race in torch_tpu, exposed by this op mix: many small CP-subgroup all-to-alls next to Pallas kernels and FSDP-group collectives. It is related to the Ulysses known issue in feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2 (cf. b/540005818, b/567661022, b/574966859). Until it is fixed, train Qwen3.5 with CP compiled on TPU (the TpuGraphTrainer recipes, or
--compile.enable). Effect on the tests:qwen3_5_simplefsdp_cp2_*parity cases passed 11 of 12 times. The failure was one step-1 gradient off by 0.0195 against a 0.01 tolerance.test_train_distributedCP cases can fail on a non-finite step.CP > 2 with the topology-aware mesh. For some shapes,
spmd_typesrejects the(dp, cp)rank layout thattorch_tpu's topology-aware mesh builds, before any CP code runs:Mesh axes mesh_dp and mesh_cp are not orthogonal. This happens fordp 4 x cp 4on 2x2x2 anddp 8 x cp 4on 2x2x4;dp 4 x cp 8on 2x2x4 is accepted. Workaround:--parallelism.no-enable_tpu_topo_mesh, which cost nothing measurable at CP1/CP2.512K context (Ulysses CP8 on 2x2x4) did not finish its first step (see the table).
FSDP2 + compiled CP with the newer torch_tpu stream change (b/588091415). With torch_tpu cl/1026681889,
qwen3_cp_*_compile(FSDP2: side-stream eager collectives next to collectives compiled into the CP blocks) can deadlock. Workaround:TORCH_TPU_INTERNAL_BLOCK_ON_STREAM_EXIT=1, which restores the previous behavior, or use SimpleFSDP.CP + EP passes the GraphTrainer gate but is not validated (see above).
Not supported, rejected by validation: CP + TP for the GatedDeltaNet,
mrope_positions(vision inputs) under CP, head counts not divisible bycp, and pipeline parallelism (as in feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2).Possible follow-ups
allgather_reducescatterwith a bf16 reduce the CP2 default for long context.Review hardening
Five rounds of strict review are folded in:
_is_plain_linearchecks (they also run under Dynamo);nn.Linearsubclasses (e.g. Float8) rejected wherever weights are read directly;allgather_reducescatterpath;