Skip to content

feat(tpu): context parallelism for Qwen3.5 (head-parallel GatedDeltaNet + CP full attention) - #200

Open
jialei777 wants to merge 15 commits into
mainfrom
feat/tpu-qwen35-context-parallel
Open

jialei777 wants to merge 15 commits into
mainfrom
feat/tpu-qwen35-context-parallel

Conversation

@jialei777

@jialei777 jialei777 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

#2 added TPU context parallelism for attention: all_gather (load-balanced KV all-gather) and ulysses, enabled per recipe through core ContextParallelTransform. On Qwen3.5 that covers only the 10 full-attention layers of 35B-A3B; the other 30 layers are GatedDeltaNet. Core CP and its validation only look at BaseAttention. So with context_parallel_degree > 1, the GatedDeltaNet layers would run their causal conv and delta-rule recurrence on each rank's token shard as if it were a whole sequence: no error, wrong results. Upstream (GPU) parallelize_qwen3_5 rejects CP for the same reason.

This PR makes Qwen3.5 CP work on TPU:

  • GatedDeltaNet: head-parallel CP (qwen3_5/cp_gated_delta.py, the same idea as Megatron-LM's Mamba CP).
    • All-to-alls move the projected q/k/v channels and the a/b gate inputs from token-sharded (T/cp tokens, every head) to head-sharded (T tokens, 1/cp of the heads).
    • The tokens go back to sequence order, which undoes KV all-gather's head-tail chunks. The causal conv and the recurrence (XLA chunked or Tokamax KDA) then run over whole sequences on the rank's heads, and one all-to-all brings the output back.
    • Each rank uses its slice of the conv weights, A_log and dt_bias, selected with index_select on the cached CP-rank tensor, so all ranks run the same compiled graph.
    • The result is exactly the single-device layer, with no redundant compute, under both full-attention strategies.
    • Requires n_key_heads and n_value_heads divisible by cp (35B-A3B: 16 and 32, so cp can be 2, 4, 8 or 16).
  • Full attention: CP versions of the SlidingWindowScaledDotProductInnerAttention that Qwen3.5 uses on TPU, registered so tpu_cp_inner_attention finds them. Splash and Tokamax Splash plug in as in feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2.
  • Input sharding: TpuQwen35Model.preprocess_inputs routes CP sharding through the CP attention's cp_shard, as core Decoder does. Upstream Qwen35Model calls prepare_context_parallel_input itself, which would head-tail the folded [B*S] stream instead of each sequence.
  • Validation: these configs now raise: GatedDeltaNet layers that are not CP-aware, a mismatched token layout, head counts not divisible by cp, and CP + TP.
  • CP speedups (opt-in or automatic, folded in from feat(tpu): Qwen3.5 CP speedups: cp_full_positions, per-batch GDN document resets, AG/RS GatedDeltaNet CP, fused GDN projections (stacked on #201) #216):
    • cp_full_positions (TpuQwen35Model.forward): the unsharded [B, S] positions, computed once in preprocess_inputs before cp_shard. With it, no layer all-gathers positions: neither the attention segment ids nor the GatedDeltaNet document resets. TpuGraphTrainer and the regular trainer both call preprocess_inputs, so this is automatic.
    • gated_delta_document_resets (TpuQwen35Model.forward, per batch): overrides Config.document_resets for one micro-batch, e.g. False when every row holds one document.
    • GatedDeltaNetContextParallelTransform(strategy="allgather_reducescatter"), an opt-in GatedDeltaNet CP strategy: all-gather x, head-sliced input projections, whole-sequence conv and recurrence on the rank's heads, column-sliced out_proj, reduce-scatter (cp_reduce_dtype, fp32 by default). Two collectives instead of six.
      • Measured at 32K: +2% (fp32) / +3.7% (bf16 reduce) tokens/s at CP2, for about 4 GiB more HBM. At CP4 with fp32 it is 6% slower.
      • The default stays all_to_all.
    • Both options are exact (parity tests). An earlier fuse_input_projections option was removed: it measured 3 pt MFU slower (see below).
  • GraphTrainer: CP is allowed for Qwen3.5. fsdp_pipeline used to reject backward collectives that have no FSDP role. It now rejects them only on a process group that carries gradient reductions. Elsewhere, for example KV all-gather CP's dK/dV reduce-scatters on the CP group, they stay with their inputs.

Everything is under torchtitan/experiments/tpu/. Core torchtitan is unchanged.

Base. Rebased onto main 5a8b0e8, which includes Tanvi's Qwen3.5 EP series (EP 1/9-7/9). TpuQwen35Model keeps main's EP Config.update_from_config and adds the CP preprocess_inputs. infra/parallelize.py keeps main's apply_qwen3_5_simple_fsdp and adds the CP path.

CP + EP. The TpuGraphTrainer Qwen3.5 gate now rejects only TP and PP, plus main's "EP requires SimpleFSDP" rule. So CP + EP passes the gate, but it has not been validated with Tanvi's EP (no parity test or TPU run yet). Treat it as experimental until a follow-up adds EP + CP recipes and tests.

Enabling CP (same pattern as #2 / GPU)

# qwen3_5/config_registry.py::_qwen35_cp
config.parallelism.context_parallel_degree = cp_degree
config.parallelism.context_parallel_load_balancer = tpu_cp_load_balancer(strategy)  # "headtail" | None
return apply_transforms(config, [
    ContextParallelTransform(inner_attention=tpu_cp_inner_attention(base, strategy)),
    GatedDeltaNetContextParallelTransform(load_balancer=tpu_cp_load_balancer(strategy)),
])

New recipes:

  • qwen35{,_moe}_debugmodel_{allgather,ulysses}_cp2
  • qwen35_moe_35b_{allgather,ulysses}_cp2 (regular trainer)
  • qwen35_moe_35b_split_tokamax_benchmark_{allgather,ulysses}_cp2 (TpuGraphTrainer)

Set --parallelism.context_parallel_degree and --training.max_context_length on the command line. A CP group shares one DP rank's num_tokens_per_microbatch_per_dp_rank, so to keep the same tokens per device, multiply it by cp.

Results: Qwen3.5-35B-A3B on TPU7x (NAP cluster)

All numbers below are TpuGraphTrainer runs (qwen35_moe_35b_split_tokamax_benchmark*). They were measured on the pre-rebase tree (image built from this branch at 4ddedb30f, before the rebase onto Tanvi's EP and before the speedups above). They have not been re-measured on the current tree yet; see "Validation on the current tree".

Setup: the TpuGraphTrainer benchmark recipe qwen35_moe_35b_split_tokamax_benchmark and its _ulysses_cp2 / _allgather_cp2 variants. The recipe uses ZeRO-3 over dp_shard x cp, a per-layer graph split with fsdp_pipeline, Tokamax GatedDeltaNet and Tokamax Splash, tpu_selective AC, and bf16. Data is the C4 test set. Image: torchtitan-container:2026-11-13 + this branch.

  • Every run uses 64K tokens per device. A CP group shares one DP rank's batch, so CP runs set num_tokens_per_microbatch_per_dp_rank to cp x 65536.
  • MFU, step time and tokens/s/device are means over steps 10-20 of 20 at 8K/32K, and over steps 6-10 of 10 (4-8 of 8 at 256K) at longer contexts. Step 1 compiles.
  • MFU uses torchtitan's FLOP count: attention counts the full S x S (6 * H * (d_qk + d_v) * S per token and layer), with no causal or document discount. The count is the same with and without CP. At long context, where attention dominates, this convention inflates MFU compared with the work causal Splash actually does.

Same context with and without CP (the CP cost):

Context Slice Parallelism MFU Step time tokens/s/device HBM
8K 2x2x2 (16 dev) no CP (dp 16) 20.10% 6.22 s 10,543 36.7 GiB
8K 2x2x2 Ulysses CP2 (dp 8 x cp 2) 18.59% 6.72 s 9,749 35.4 GiB
8K 2x2x2 KV all-gather CP2 17.27% 7.23 s 9,062 33.4 GiB
32K 2x2x2 no CP 26.91% 7.19 s 9,112 38.9 GiB
32K 2x2x2 Ulysses CP2 24.96% 7.75 s 8,452 38.4 GiB
32K 2x2x2 KV all-gather CP2 23.28% 8.31 s 7,884 37.6 GiB
32K 2x2x2 Ulysses CP4 (dp 4 x cp 4)¹ 19.85% 9.75 s 6,720 39.4 GiB
32K 2x2x2 KV all-gather CP4¹ 19.21% 10.07 s 6,505 39.4 GiB
64K 2x2x4 (32 dev) no CP (dp 32, 1 sequence/device) 32.41% 8.80 s 7,452 38.8 GiB
64K 2x2x4 Ulysses CP2 (dp 16 x cp 2) 31.05% 9.18 s 7,137 39.5 GiB
64K 2x2x4 KV all-gather CP2 28.18% 10.12 s 6,479 37.9 GiB

Contexts only CP can train: 64K tokens per device, with each sequence split over cp devices.

Context Slice Parallelism MFU Step time tokens/s/device HBM
128K 2x2x4 no CP (128K tokens on one device) OOM - - > 96 GiB
128K 2x2x4 Ulysses CP2 39.02% 12.00 s 5,463 39.4 GiB
128K 2x2x4 KV all-gather CP2 37.54% 12.47 s 5,256 38.4 GiB
256K 2x2x4 Ulysses CP4¹ 41.96% 19.88 s 3,297 39.3 GiB
512K 2x2x4 Ulysses CP8 not run² - - -

¹ --parallelism.no-enable_tpu_topo_mesh: with the topology-aware mesh, spmd_types rejects the (dp, cp) layout for cp > 2 (see Known issues). The topology-aware mesh makes no difference at CP1/CP2: 26.94% / 25.01% MFU at 32K without it, vs 26.91% / 24.96% with it.

² Compiled (about 18 min), but the first step never finished: all ranks were idle for more than 12 min, with or without the topology-aware mesh. Not investigated.

Flags: the no-CP and Ulysses runs from 32K up add --splash_attention_kernel.no-sa_use_document_aware_block_skipping (KV all-gather never skips), and every run from 64K up adds --splash_attention_kernel.no-sa_use_fused_bwd_kernel (see Notes).

Notes

  • Loss. CP runs match no-CP at the start: step 1 is 12.88286 without CP vs 12.88350 for both CP strategies at 8K, and step 10 is 3.95538 vs 3.95540 (Ulysses) and 3.95541 (KV all-gather). Later steps differ by about 0.05 between DP degrees, because the dataloader shards C4 by DP rank (split_dataset_by_node), so dp 16 and dp 8 see different samples per step. Runs with the same DP degree agree: at step 20 Ulysses gives 2.56204 and KV all-gather 2.56451. Exactness is covered by the step-by-step parity tests.
  • CP cost at short context. At 8K/32K, CP2 costs about 0.5 s/step with Ulysses and about 1.0-1.1 s/step with KV all-gather, out of 6-7 s. Ulysses sends q/k/v/a/b in and the output back for each of the 30 GatedDeltaNet layers, plus Ulysses attention on the 10 full-attention layers. KV all-gather sends the same GatedDeltaNet traffic plus K/V all-gathers and dK/dV reduce-scatters.
    • The Ulysses cost stays roughly flat per token (0.52 s at 8K, 0.56 s at 32K), so its share falls as attention grows: -4% MFU at 64K. The KV all-gather cost grows slightly with the context (1.0 s at 8K, 1.32 s at 64K).
    • CP4 is much more expensive than CP2 (+2.6 s at 32K vs +0.6 s): a CP4 group spans two chips and each rank sends 3/4 rather than 1/2 of its activations.
    • Use the smallest cp that fits the context, and prefer Ulysses when the head counts allow it.
  • Packing the GatedDeltaNet all-to-alls into one collective was slower: 18.14% MFU and +2 GiB, vs 18.60% with separate all-to-alls, which XLA overlaps with the earlier inputs' conv. So they stay separate.
  • Long context needs the unfused Splash backward. The fused backward kernel allocates a (S / block_kv_dkv) x H x S x D partial-dq buffer: 128 GiB at S = 128K (H = 16, D = 256, block_kv_dkv = 1024). With it, 128K fails at compile time with CompileTimeHbmOom both without CP (168 GB) and with Ulysses CP2 (104 GB). Even with the unfused backward, no CP OOMs at runtime with a 128K sequence on one device, because activation memory grows with tokens per device.
  • Document-aware block skipping. KV all-gather CP cannot skip document-masked Splash blocks, because its queries are in head-tail order. For the same attention work, the 32K+ Ulysses and no-CP runs disable skipping too. At 8K skipping is worth only about 0.1% MFU (no CP: 20.10% vs 20.00%; Ulysses: 18.59% vs 18.47%).

Commands

Each run is PJRT_DEVICE=TPU torchrun --nnodes=<hosts> --nproc_per_node=8 ... -m torchtitan.experiments.tpu.train with:

--module=torchtitan.experiments.tpu.qwen3_5 --hf_assets_path=./tests/assets/tokenizer \
--dataloader.dataset=c4_test --dataloader.dataset_path=./tests/assets/c4_test \
--dump_folder=/tmp/torchtitan_outputs --lr_scheduler.warmup_steps=200 --profiler.no-enable_profiling \
--training.steps=20 --metrics.log_freq=5 \
--config=qwen35_moe_35b_split_tokamax_benchmark               # no CP
--config=qwen35_moe_35b_split_tokamax_benchmark_ulysses_cp2   # Ulysses CP2 (+ --parallelism.context_parallel_degree=4/8 --parallelism.no-enable_tpu_topo_mesh)
--config=qwen35_moe_35b_split_tokamax_benchmark_allgather_cp2 # KV all-gather CP2
--training.max_context_length=<S> --training.num_tokens_per_microbatch_per_dp_rank=<65536 * cp>

Validation on the current tree

TpuGraphTrainer qwen35_moe_35b_split_tokamax_benchmark* on TPU7x 2x2x2, 64K tokens/device, 20 steps, same flags as above. Means over steps 10/15/20.

  • Image B: 99cb3a5, the speedups before the review rounds.
  • Image C: 6791fd1, this PR after the five review rounds, before fuse_input_projections was removed.
  • The allgather_reducescatter and fused runs use benchmark-only recipes that are not committed.

The current tree reproduces the pre-rebase numbers (losses identical):

Context Run Before Image C
8K no CP 20.10% 20.06% (image B)
8K Ulysses CP2 18.59% 18.56%
8K KV all-gather CP2 17.27% 17.28%
32K no CP 26.91% 26.90%
32K Ulysses CP2 24.96% 24.98%
32K KV all-gather CP2 23.28% 23.30%
32K Ulysses CP4 19.85% 19.82%

GatedDeltaNet allgather_reducescatter vs the default all_to_all:

Context Run MFU Step time tokens/s/device HBM
8K Ulysses CP2, all_to_all (B) 18.60% 6.72 s 9,756 35.4 GiB
8K Ulysses CP2, AG/RS fp32 (B) 18.69% 6.68 s 9,803 35.5 GiB
8K KV all-gather CP2, all_to_all (B) 17.29% 7.23 s 9,068 33.4 GiB
8K KV all-gather CP2, AG/RS fp32 (B) 17.23% 7.25 s 9,036 35.8 GiB
32K Ulysses CP2, all_to_all 24.98% 7.75 s 8,456 38.4 GiB
32K Ulysses CP2, AG/RS fp32 25.48% 7.60 s 8,626 42.5 GiB
32K Ulysses CP2, AG/RS bf16 25.90% 7.47 s 8,769 42.8 GiB
32K KV all-gather CP2, all_to_all 23.30% 8.31 s 7,888 37.6 GiB
32K KV all-gather CP2, AG/RS fp32 23.71% 8.17 s 8,028 44.0 GiB
32K Ulysses CP4, all_to_all 19.82% 9.76 s 6,713 39.4 GiB
32K Ulysses CP4, AG/RS fp32 18.72% 10.34 s 6,337 50.8 GiB
32K Ulysses CP4, AG/RS bf16 20.03% 9.67 s 6,782 44.5 GiB
  • AG/RS helps at CP2 when the context is long, at a memory cost: backward keeps the all-gathered (T, D) input. At CP4 it moves 2x the bytes with fp32 (1.33x with bf16), which matches the slowdown.
  • bf16 reduce at CP2 gives a bit-identical loss to fp32: the sum of two bf16 values is rounded once.
  • cp_full_positions is automatic in images B and C and makes no measurable difference here. GatedDeltaNet resets are off in this recipe, so only the attention segment-id all-gather goes away.
  • fuse_input_projections (removed) measured 15.55% vs 18.60% MFU (Ulysses CP2, 8K): +1.3 s/step and +2.2 GiB.

Changes

File Change
qwen3_5/cp_gated_delta.py (new) CPTpuInnerGatedDeltaNet: head-parallel GatedDeltaNet. All-to-alls in, head-tail -> sequence order, rank-sliced conv / A_log / dt_bias, all-to-all out. Falls back to TpuInnerGatedDeltaNet without a CP axis.
GatedDeltaNetContextParallelTransform: the GatedDeltaNet counterpart of core ContextParallelTransform.
CPTpuGatedDeltaNet: cp_strategy="allgather_reducescatter" (with cp_reduce_dtype).
validate_gated_delta_context_parallel, registered with validate_tpu_context_parallel; revert_tpu_context_parallel reverts both classes for cp == 1 references.
qwen3_5/model.py TpuQwen35Model: CP input sharding through the CP attention's cp_shard (upstream calls prepare_context_parallel_input itself). Text-only under CP. cp_full_positions and gated_delta_document_resets forward kwargs, passed to every layer.
qwen3_5/gated_delta.py TpuGatedDeltaNet: passes positions / full_positions to the inner module, takes a per-call document_resets override, and adds input_projections (optionally head-sliced).
qwen3_5/__init__.py use_tpu_modules returns TpuQwen35Model.Config. Importing the package registers the GatedDeltaNet CP validator.
qwen3_5/infra/parallelize.py With CP: TPUSelectiveAC for plain selective AC (as in tpu/qwen3), plus upstream's parallelize steps without its CP NotImplementedError (FSDP2 on the (dp_shard, cp) mesh, or SimpleFSDP).
qwen3_5/config_registry.py _qwen35_cp and the debug, 35B and 35B GraphTrainer CP recipes.
muse_glimmer/attention.py UlyssesCP... and KVAllGatherCP... SlidingWindowScaledDotProductInnerAttention, registered with register_tpu_cp_inner_attention. The KV all-gather reference attention masks packed documents, as the single-device SlidingWindow SDPA does.
distributed/cp_attention.py Public cp_all_to_all, cp_all_gather and cp_rank_tensor, moved out of KVAllGatherCPInnerAttention.
New register_tpu_cp_inner_attention and register_tpu_cp_validator registries.
Ulysses cp_shard also caches the CP-rank tensor.
The KV all-gather causal-window check applies only to Varlen (left, right) windows.
graph_trainer/qwen3_5.py Allows CP (still rejects TP / PP; EP still requires SimpleFSDP, as on main).
graph_trainer/graph_split.py fsdp_pipeline: a backward all-gather or reduce-scatter with no FSDP role raises only if it runs on the process group of a gradient reduction. Others, such as KV all-gather CP's dK/dV reduce-scatters on the CP group, stay with their inputs (the existing fallback).
tests qwen3_5/tests/test_cp_gated_delta.py (new).
Qwen3.5 CP cases in test_train_minimal_distributed and test_train_distributed.
test_graph_split: non-FSDP-group reduce-scatter.
test_qwen3_5_tpu: CP no longer rejected by the GraphTrainer parallelize.

Tests

  • CPU, current HEAD: test_cp_gated_delta, test_cp_attention, test_graph_split, test_workarounds, test_split_optimizers_by_mesh and test_token_dispatcher give 223 passed. test_qwen3_5_tpu --device=cpu gives 48 OK (6 skipped) and test_tokamax_gated_delta --device=cpu 21 OK (17 skipped). The new coverage includes:
    • cp_full_positions, with every position all-gather forbidden;
    • the per-batch gated_delta_document_resets override;
    • allgather_reducescatter parity (both layouts, eager and traced, with and without an out_proj bias);
    • a compiled graph with one wait per collective;
    • rejection of non-plain projections and of a non-CP inner module.
  • CPU, original version (qwen3_5/tests/test_cp_gated_delta.py, 31 tests, using the emulated CP ranks from tests/test_cp_attention.py, eager and traced collectives):
    • CPTpuInnerGatedDeltaNet vs single-device TpuInnerGatedDeltaNet for cp 2 and 4, both token layouts, with and without packed-document resets, and B=2 (documents cross head-tail chunks). Checks the output, dq/dk/dv/da/db, and the summed per-rank conv / A_log / dt_bias gradients, all to 2e-5.
    • The whole debug model (CP attention + CP GatedDeltaNet, cp=2, both strategies) vs a single device: loss to 1e-5 and every parameter gradient to 1e-4.
    • Recipes, the transform, validation errors, and TpuQwen35Model.preprocess_inputs sharding through cp_shard.
    • Together with test_cp_attention, test_workarounds and test_graph_split: 124 passed. The pre-existing CPU suites give the same results as main.
  • TPU7x 2x2x1 (8 devices, dp_shard 4 x cp 2, on the final commit; the eager cases are flaky on this build, see Known issues):
    • test_train_minimal_distributed, step-by-step loss / grad / param parity vs a single-device CPU reference:
    • test_train_distributed: qwen3_5_cp2_{all_gather,ulysses}, qwen3_5_fsdp_cp2_all_gather (FSDP2), qwen3_5_moe_cp2_{all_gather,ulysses}_splash, and the Qwen3 CP cases.
    • test_qwen3_5_tpu: 43/43.
  • 35B-A3B on TPU7x 2x2x2 / 2x2x4: the table above, both strategies, on TpuGraphTrainer.

Known issues

  1. Eager (uncompiled) Qwen3.5 CP on TPU is unreliable with dp_shard > 1 on this torch_tpu build (image torchtitan-container:2026-11-13). The debug recipes ran for 10 steps on 2x2x1 (dp_shard 4 x cp 2):

    • Ulysses: non-finite or wrong gradients in 18 of 22 runs (4 of 4 at the recipe defaults).
    • KV all-gather: 1 of 11 runs (with fp32 params); every run differs from the others.
    • Without CP, the same model is bit-reproducible on the same build, and so is Qwen3 CP from feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2 (both strategies).
    • On CPU (gloo, dp_shard 2 x cp 2), the CP runs are exact and reproducible.
    • Compiled runs are clean and reproducible. With --compile.enable (per layer), the same 10-step runs were bit-identical 3 of 3 times for each strategy, and the two strategies agree to 3e-5 at step 10 (8.64935 vs 8.64932). The TpuGraphTrainer benchmarks gave the same loss, 2.56204 at step 20, on two images.
    • None of these fixed it: fp32 params, all-to-alls over the head dim instead of the flat channel dim, no packed-document resets (so no positions all-gather), 2x the heads per rank, or --tpu_config.eager_mode=DEFER_NEVER.
    • DEFER_NEVER_AND_LAUNCH_BLOCKING removes the blow-ups (0 of 4) but not the run-to-run differences.

    This looks like an eager-runtime race in torch_tpu, exposed by this op mix: many small CP-subgroup all-to-alls next to Pallas kernels and FSDP-group collectives. It is related to the Ulysses known issue in feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2 (cf. b/540005818, b/567661022, b/574966859). Until it is fixed, train Qwen3.5 with CP compiled on TPU (the TpuGraphTrainer recipes, or --compile.enable). Effect on the tests:

    • The eager qwen3_5_simplefsdp_cp2_* parity cases passed 11 of 12 times. The failure was one step-1 gradient off by 0.0195 against a 0.01 tolerance.
    • The eager test_train_distributed CP cases can fail on a non-finite step.
  2. CP > 2 with the topology-aware mesh. For some shapes, spmd_types rejects the (dp, cp) rank layout that torch_tpu's topology-aware mesh builds, before any CP code runs: Mesh axes mesh_dp and mesh_cp are not orthogonal. This happens for dp 4 x cp 4 on 2x2x2 and dp 8 x cp 4 on 2x2x4; dp 4 x cp 8 on 2x2x4 is accepted. Workaround: --parallelism.no-enable_tpu_topo_mesh, which cost nothing measurable at CP1/CP2.

  3. 512K context (Ulysses CP8 on 2x2x4) did not finish its first step (see the table).

  4. FSDP2 + compiled CP with the newer torch_tpu stream change (b/588091415). With torch_tpu cl/1026681889, qwen3_cp_*_compile (FSDP2: side-stream eager collectives next to collectives compiled into the CP blocks) can deadlock. Workaround: TORCH_TPU_INTERNAL_BLOCK_ON_STREAM_EXIT=1, which restores the previous behavior, or use SimpleFSDP.

  5. CP + EP passes the GraphTrainer gate but is not validated (see above).

  6. Not supported, rejected by validation: CP + TP for the GatedDeltaNet, mrope_positions (vision inputs) under CP, head counts not divisible by cp, and pipeline parallelism (as in feat(tpu): Ulysses and load-balanced KV all-gather context parallelism, enabled via recipes like GPU #2).

Possible follow-ups

  • Root-cause the eager torch_tpu issue.
  • Consider making allgather_reducescatter with a bf16 reduce the CP2 default for long context.
  • EP + CP recipes, parity tests and TPU validation on top of Tanvi's EP.
  • Reduce the CP4 cost (inter-chip all-to-alls) by overlapping it with compute.
  • The fused Splash backward's memory at long context.

Review hardening

Five rounds of strict review are folded in:

  • one all-gather helper and shared CP position helpers;
  • shared _is_plain_linear checks (they also run under Dynamo);
  • nn.Linear subclasses (e.g. Float8) rejected wherever weights are read directly;
  • contiguous-layout check on the allgather_reducescatter path;
  • uniform position kwargs;
  • unique test-class names (a duplicate had hidden the compiled tests), plus a guard;
  • corrected docstring arithmetic.

…DeltaNet

TPU CP (#2) only rewrote attention, so on Qwen3.5 the GatedDeltaNet layers
(30 of 40 in 35B-A3B) would run their causal conv and delta-rule recurrence
on each rank's token shard as if it were a whole sequence, silently.

- qwen3_5/cp_gated_delta.py: CPTpuInnerGatedDeltaNet runs the GatedDeltaNet
  head-parallel: all-to-alls take q/k/v/a/b from token- to head-sharded
  (one per input: packing them measured slower), the tokens go back in
  sequence order (head-tail undone for
  KV all-gather CP), conv + recurrence run over whole sequences on the rank's
  heads with its slice of the conv weights / A_log / dt_bias (index_select on
  the cached CP-rank tensor), and one all-to-all brings the output back.
  Exact; works with both full-attention strategies. Plus
  GatedDeltaNetContextParallelTransform and a registered validator that
  rejects CP configs whose GatedDeltaNet layers are not converted.
- muse_glimmer/attention.py: Ulysses / KV all-gather CP versions of
  SlidingWindowScaledDotProductInnerAttention (Qwen3.5's TPU full attention).
- qwen3_5/model.py: TpuQwen35Model shards CP inputs through the CP
  attention's cp_shard (per-sequence head-tail, CP-rank cache), as core
  Decoder does; upstream Qwen35Model calls prepare_context_parallel_input.
- qwen3_5/infra/parallelize.py: TPUSelectiveAC under CP; upstream's
  parallelize steps without its CP NotImplementedError.
- distributed/cp_attention.py: public cp_all_to_all / cp_all_gather /
  cp_rank_tensor; register_tpu_cp_inner_attention and
  register_tpu_cp_validator registries.
- qwen3_5/config_registry.py: _qwen35_cp and debug / 35B CP recipes.
- graph_trainer/qwen3_5.py: allow CP (still rejects TP / PP / EP).
- graph_trainer/graph_split.py: fsdp_pipeline raised for any backward
  all-gather / reduce-scatter that is neither a parameter gather nor a
  gradient reduction, including KV all-gather CP's recomputed K/V gathers
  and dK/dV reduce-scatters. Raise only when such a collective runs on the
  process group of a gradient reduction (gathers of saved activations are
  also fed only by graph inputs, so only reductions name the FSDP groups);
  on other groups (CP) it stays with its inputs, the existing fallback.
…n tests

- qwen3_5/tests/test_cp_gated_delta.py (CPU, emulated CP ranks, eager and
  traced collectives): CP GatedDeltaNet vs single device (cp 2/4, both token
  layouts, packed-document resets, outputs and every gradient), the CP debug
  model vs single device (loss and all parameter gradients), recipes,
  transform and validation.
- test_train_minimal_distributed: qwen3_5_simplefsdp_cp2_{all_gather,ulysses}
  and their per-layer torch.compile + Tokamax Splash variants (parity vs the
  CPU reference).
- test_train_distributed: Qwen3.5 dense / MoE + Splash CP cases and FSDP2 CP.
…rallel GatedDeltaNet

- CPTpuInnerGatedDeltaNet raises without an active CP mesh axis instead of
  running the single-device recurrence on each rank's token shard. cp == 1
  reruns of CP recipes revert it: revert_tpu_context_parallel now runs
  reverters registered with register_tpu_cp_reverter
  (revert_gated_delta_context_parallel).
- cp_shard (KV all-gather and Ulysses) records the token layout it sharded
  with (cp_input_layout). The GatedDeltaNet checks the head-tail rows it
  derives from set_unflatten_seq_len against it, and that Ulysses sharded one
  contiguous folded stream, so a layout mismatch between sharding and forward
  raises instead of silently scrambling the token order.
- validate_tpu_context_parallel also rejects non-causal KV all-gather windows
  given as lists (config files / command line).
- cp_rank_tensor warns once (instead of a debug log) when it bakes the rank
  into a traced graph.
- TpuQwen35Model.preprocess_inputs raises instead of asserting on missing
  positions.
- Tests: no-CP-group error, revert, cp_shard layout recording, head-tail
  inverse against torch's _HeadTailLoadBalancer for an unfolded sequence
  (cp 2/4), layout mismatch errors, list windows.
…r layout and graph-split checks

- cp_rank_tensor: a cache miss under Dynamo raises a clear RuntimeError
  instead of calling the logger, which Dynamo cannot trace (a hard error under
  per-layer fullgraph compile); non-strict tracing still warns once.
- GatedDeltaNet CP _rows raises when the sequence length does not divide the
  token count (TpuInnerGatedDeltaNet rejects it too), instead of a dead
  one-sequence fallback.
- graph_split: without any recognized gradient reduction, keep the strict
  check on unclassified backward collectives instead of skipping it.
- validate_gated_delta_context_parallel checks TP once, only for models with
  GatedDeltaNet layers; register_tpu_cp_inner_attention rejects a second,
  different CP pair for the same attention.
- _parallelize_text_qwen3_5_with_cp returns the model and notes to stay in
  sync with upstream; clearer comments (layout global, reference attention
  masks, positions all-gather).
- Tests: real Dynamo fullgraph compile of the CP GatedDeltaNet (both layouts,
  one graph) and the uncached-rank error; Ulysses CP over the sliding-window
  SDPA vs single device (cp 2/4); duplicate registration; reset of the
  layout / rank-miss globals between tests.
… test coverage, docstrings

- _record_layout_of_input clears the recorded CP layout when there is no
  input to record, so a stale layout is never checked against.
- cp_rank_tensor docstring describes the eager and tracing miss paths.
- CPQwen35ModelParityTest shards through the recipe attention's real
  cp_shard (mocked torch chunking), so the recorded layout and the
  GatedDeltaNet layout checks run in the end-to-end CPU parity test.
- Tests: TpuQwen35Model.preprocess_inputs rejects missing positions and
  mrope_positions; the 35B and split-Tokamax GraphTrainer CP recipes build and
  convert every layer; one shared _reset_cp_state for the CP globals; the
  graph-split test restores the mutated node on failure.
- Docstring summaries fit in 80 columns.
…ther, docs and test gaps

- cp_all_gather raises if its input requires a gradient (the traced
  collective has none, so eager and compiled runs would differ);
  _full_sequence_segment_ids uses it instead of repeating the collective.
- The CP reverts build configs from init fields only.
- _qwen35_cp keeps the config subclass in its return type (TypeVar), as the
  split-Tokamax GraphTrainer CP recipes return TpuGraphTrainer.Config.
- Docstrings and comments match the code after the cp_rank_tensor
  extraction: _normalize_cp_device callers, cp_rank_tensor Raises, what the
  qwen3_5 import registers, why CP preprocessing needs positions.
- Tests: GatedDeltaNet CP rejects TP; register_tpu_cp_inner_attention rejects
  a wrong mixin; cp_rank_tensor non-strict tracing miss warns once without
  caching; layout recording with TpuQwen35Model's real input sharding; the
  uncached-rank compile error pinned to RuntimeError.
@jialei777
jialei777 force-pushed the feat/tpu-qwen35-context-parallel branch from e01195a to 8dfd350 Compare November 21, 2026 04:57
…ment resets, all-gather/reduce-scatter GDN CP strategy, fused GDN input projections

- TpuQwen35Model.forward(cp_full_positions=...) threads the unsharded positions
  to every layer so CP attention and the head-parallel GatedDeltaNet build
  document boundaries without per-layer positions all-gathers (both CP
  strategies); preprocess_inputs sets it automatically.
- gated_delta_document_resets per-batch override (single-document rows).
- CPTpuGatedDeltaNet + GatedDeltaNetContextParallelTransform(strategy=
  'allgather_reducescatter'): AG(x) + head-sliced projections/GDN/norm +
  RS(out_proj) for any cp dividing the head counts and either load balancer.
- TpuGatedDeltaNet.Config.fuse_input_projections.
- cp_all_gather_with_grad / cp_reduce_scatter collectives.
- Tests: model parity over strategies x eager/traced x GDN strategy x
  cp_full_positions x document-reset override; fullgraph compile test for the
  new strategy; transform/projection/preprocess tests.

Ported from #216 (stacked on the EP branch) onto main: main's module layout
(distributed/cp_attention.py, components/unflatten_shim.py), main's
TpuQwen35Model (EP config) and the #200 review fixes (layout checks also on
the all-gather / reduce-scatter path; revert_tpu_context_parallel also
reverts CPTpuGatedDeltaNet).
@jialei777
jialei777 marked this pull request as ready for review November 21, 2026 06:46
- test_cp_gated_delta: the all-gather/reduce-scatter compile test class reused
  the name CompiledCPGatedDeltaNetTest and shadowed the all-to-all one, so its
  fullgraph-compile and uncached-rank tests never ran; renamed it, and a new
  test fails on duplicate test class names.
- The plain-nn.Linear checks of the fused / head-sliced GatedDeltaNet
  projections and of out_proj also run while Dynamo compiles (the per-layer
  compile is the production path); they were skipped there.
- CPTpuGatedDeltaNet docstring: correct collective count (five all-to-alls in)
  and per-rank communication volume (the all-gather/reduce-scatter strategy
  moves cp * D / C times the all-to-all bytes: even at cp = 2, 2x at cp = 4 on
  35B-A3B); recommend benchmarking before choosing it.
- Document _CP_INPUT_LAYOUT as a best-effort, trace-time check.
- Revert an unrelated formatting change in TpuRMSNormGated.forward.
…docstring numbers, polish

- KV all-gather CP: the positions / segment_ids fallbacks use cp_all_gather
  instead of open-coding the traced / eager all-gather branch.
- CPTpuGatedDeltaNet docstring: the "even at cp = 2, 2x at cp = 4" figures
  assume the default float32 cp_reduce_dtype (0.66x / 1.33x with bfloat16).
- TpuQwen35Model.forward: the unsupported-kwargs error lists exactly the
  rejected kwargs; drop stray parentheses.
- Sort cp_attention.__all__.
- Tests: the fused / head-sliced projections reject a Linear subclass with its
  own forward; the graph-split CP-group check restores the node in a finally
  instead of both a cleanup and an explicit restore.
…ck, uniform position kwargs, docstrings, more tests

- TpuGatedDeltaNet passes positions/full_positions uniformly to its inner
  module; class/forward docstrings describe full_positions, per-call
  document_resets and fused/head-sliced input projections.
- _is_plain_linear shared by input_projections and the AG/RS out_proj check.
- Tests: AG/RS rejects a non-CP inner module and a non-plain out_proj;
  out_proj bias parity at cp=2 (all_to_all and allgather_reducescatter);
  clearer fused/sliced subTest loop.

Drafted with Jetski.
…ers, no dead cu_seqlens op

- _gather_positions / _sequence_positions shared by CPTpuInnerGatedDeltaNet
  and the allgather_reducescatter path (no duplicated gather/reorder logic).
- allgather_reducescatter passes cu_seqlens=None instead of building an
  unused arange; TpuInnerGatedDeltaNet.forward documents its arguments.
- Test docstring/config-build polish.

Drafted with Jetski.
…CP inner, comment why layer kwargs are conditional

Drafted with Jetski.
…er_reducescatter trade-off

TPU7x A/B on Qwen3.5-35B-A3B (TpuGraphTrainer, Ulysses CP2, 8K):
fuse_input_projections cost 3 pt MFU (15.55% vs 18.60%), +1.3 s/step and
+2.2 GiB, so remove it. allgather_reducescatter at 32K: cp=2 +2% (fp32) /
+4% (bf16 reduce, bit-identical at cp=2) throughput for ~4 GiB more HBM;
cp=4 fp32 -6%. Documented in CPTpuGatedDeltaNet.

Drafted with Jetski.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant