Skip to content

feat(tpu): Qwen3.5 context parallelism on top of fast dropless EP (stacked on #142) - #201

Draft
jialei777 wants to merge 5 commits into
fix/tpu-qwen35-ep-regular-trainer-compilefrom
feat/tpu-qwen35-cp-on-ep
Draft

jialei777 wants to merge 5 commits into
fix/tpu-qwen35-ep-regular-trainer-compilefrom
feat/tpu-qwen35-cp-on-ep

Conversation

@jialei777

@jialei777 jialei777 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #206 (regular-trainer per-layer compile with EP), which is stacked on #142 (fast dropless EP) and #6. This is #200 (Qwen3.5 context parallelism on TPU) rebased onto the EP stack, so expert parallelism and context parallelism can be enabled together. See #200 for the CP design, the CP-only results and the known issues; this PR only lists what differs.

What changed vs #200

  • Base. The EP stack branches from main at 2026-11-06, before the TPU experiment root was reorganized into subpackages. The CP code uses the old paths here: experiments/tpu/cp_attention.py, experiments/tpu/unflatten_shim.py and experiments/tpu/activation_checkpoint.py.
  • TpuQwen35Model. [TPU][Qwen3.5] Expert parallelism (EP > 1) for Qwen3.5 MoE #6 already adds a TpuQwen35Model (EP update_from_config and parallelize). The CP preprocess_inputs, which shards the inputs through the CP attention's cp_shard, is added to that class.
  • SPMD backend. On this base parallelism.spmd_backend defaults to partial_dtensor, and CP requires spmd_types. _qwen35_cp sets it, as qwen3/config_registry.py::_qwen3_cp does here. _parallelize_text_qwen3_5_with_cp and preprocess_inputs mirror this base's upstream branches on spmd_backend.
  • GraphTrainer. Qwen3.5 now rejects only TP and PP: [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142 allows EP and this PR allows CP.
  • New recipes: qwen35_moe_35b_split_tokamax_ep_{allgather,ulysses}_cp2. These are qwen35_moe_35b_split_tokamax_ep_benchmark (EP=4; override with --parallelism.expert_parallel_degree) plus CP degree 2. They keep the EP recipe settings: by_layer split, one layer per executable, tpu_selective save regions and SparseCore offload.
  • New tests: test_train_distributed trains the debug MoE with EP 2 + CP 2 under both CP strategies; test_cp_gated_delta checks that the EP + CP recipes keep the EP settings.
  • Regular-trainer EP + CP recipes: qwen35_moe_35b_ep_{allgather,ulysses}_cp2 are qwen35_moe_35b_ep from [TPU] Qwen3.5 EP on the regular trainer with per-layer torch.compile #206 (EP=4, Tokamax GatedDeltaNet) plus CP degree 2, on the regular TPUTrainer. They compile every block (--compile.enable --compile.backend=tpu) by default, because eager CP with dp_shard > 1 is unreliable (feat(tpu): context parallelism for Qwen3.5 (head-parallel GatedDeltaNet + CP full attention) #200). test_cp_gated_delta checks that they keep EP, use Tokamax and spmd_types, compile every block and convert every GatedDeltaNet. See "Regular trainer" below for the results and the known issue.

How EP and CP compose

CP shards each sequence's tokens over the cp mesh axis. The routed experts are token-wise, so EP dispatches each rank's local, CP-sharded tokens over the ep axis. ep must divide dp_shard * cp. ep is the innermost axis of the expert mesh (dp_replicate, efsdp, ep), and cp the innermost data axis of (batch, cp, tp): with cp=2, ep=2 the EP groups are the CP groups, and with cp=2, ep=4 one EP group spans two CP groups. Each token's routed-expert contribution is computed once, on the EP rank that owns the expert, and the expert gradients are summed over efsdp.

Tests

  • CPU (torchtitan-container:2026-11-13):
    • test_cp_gated_delta (incl. the EP + CP recipe checks), test_cp_attention and test_graph_split pass.
    • test_train_distributed EP 2 + CP 2 (all-gather and Ulysses, 4 gloo ranks) pass.
    • test_qwen3_5_tpu, test_token_dispatcher, test_workarounds, test_cp_attention and test_graph_split give the same CPU results as the EP base c56f441fc. The only difference is the renamed graph-trainer rejection test; the failures are the TPU-only Qwen35Test cases, in both.
  • TPU (tpu7x-2x2x1, 8 devices):
    • Unit tests (test_cp_gated_delta, test_cp_attention, test_graph_split, test_token_dispatcher, test_workarounds): 137 passed.
    • Qwen3.5 CP parity against CPU (test_train_minimal_distributed, SimpleFSDP, CP2 all-gather and Ulysses, eager and compiled): 5/5.
    • test_train_distributed: 9/9. Covers CP2 (both strategies), FSDP2 + CP, MoE + CP with Splash, EP2, EP2 with the TPU MoE kernels, and EP2 + CP2 (both strategies).
    • test_qwen3_5_tpu, including [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142's EP tests: 59/59. test_qwen3_5_graph_trainer_matches_eager passes.
    • Qwen3 CP regression: parity 5/5, test_train_distributed 3/3.
    • The regular trainer's per-layer torch.compile with EP used to fail in Dynamo (_tpu_generation() in kernels/ragged_gather_tc.py). [TPU] Qwen3.5 EP on the regular trainer with per-layer torch.compile #206 fixes that: compiled EP2 + CP2 now trains on the regular trainer; see below.

Results: Qwen3.5-35B-A3B, EP and EP + CP

Setup: the TpuGraphTrainer 35B benchmark recipes of this branch on TPU7x (NAP cluster).

Context Slice Parallelism MFU Step time tokens/s/device HBM
8K 2x2x2 (16 dev) EP=1, no CP (dp 16) 20.07% 6.22 s 10,529 36.7 GiB
8K 2x2x2 EP=1 + Ulysses CP2 (dp 8 x cp 2) 18.58% 6.72 s 9,749 35.4 GiB
8K 2x2x2 EP=2, no CP 14.34% 8.71 s 7,520 23.3 GiB
8K 2x2x2 EP=2 + Ulysses CP2 13.41% 9.31 s 7,036 22.3 GiB
8K 2x2x2 EP=2 + KV all-gather CP2 12.70% 9.84 s 6,663 22.3 GiB
64K 2x2x4 (32 dev) EP=2, no CP (1 sequence/device) 25.24% 11.29 s 5,804 26.1 GiB
64K 2x2x4 EP=2 + Ulysses CP2 24.16% 11.80 s 5,554 25.6 GiB
128K 2x2x4 EP=2 + Ulysses CP2 31.82% 14.71 s 4,455 26.1 GiB

Other notes:

Regular trainer (per-layer compile) with EP + CP

Setup: Qwen3.5-35B-A3B on the regular TPUTrainer on tpu7x-2x2x2 (16 ranks).

  • Per-block torch.compile(fullgraph=True), the Tokamax GatedDeltaNet with document resets, full activation checkpointing, and FP32 master weights.
  • ctx 8K, 32K tokens per device, 20 steps, warmup clamped to 20 steps.
  • Image: torchtitan-container:2026-11-13 + this branch (4534b94cf).
  • Throughput is the mean over steps 10-20.
run loss @1 / @10 / @20 s/step tok/s/device MFU
no EP, no CP (qwen35_moe_35b + Tokamax, compiled) 12.88448 / 3.96318 / 2.64803 6.25 5,245 10.00%
EP2 (qwen35_moe_35b_ep, #206) 12.88440 / 3.88662 / 2.65545 7.82 4,189 7.97%
Ulysses CP2 only (qwen35_moe_35b_ulysses_cp2 + Tokamax, compiled) 12.88997 / 3.97498 / 2.65472 7.17 4,571 8.71%
EP2 + Ulysses CP2 (qwen35_moe_35b_ep_ulysses_cp2) 12.88996 / 3.97575 / 2.66672 8.81 3,719 7.09%
EP2 + KV all-gather CP2 (qwen35_moe_35b_ep_allgather_cp2) 12.88996 / 3.97546 / 2.66364 9.12 3,594 6.85%

--parallelism.expert_parallel_degree=2 throughout. CP runs set --training.num_tokens_per_microbatch_per_dp_rank=65536.

Known issue: per-block-compiled CP is intermittently non-finite on this torch_tpu build.

  • 35B: both EP2 + CP2 runs above trained cleanly. Of three identical-config CP-only runs, one crashed in step 1 with a libtpu SLICE_FAILURE_SW_INJECT_ERROR abort, one stopped at step 1 with a non-finite loss / grad norm, and one completed (the row above).
  • Debug MoE model (16 ranks, CP2, 4 steps): runs stop with "Loss or gradient norm is not finite" at step 1 or 2.
    • This happens with and without EP, with the Tokamax or the XLA GatedDeltaNet, under selective or full activation checkpointing, and with Ulysses or KV all-gather.
    • Identical configs pass on reruns (2 of 4 Tokamax CP-only runs passed).
    • --tpu_config.eager_mode=DEFER_NEVER_AND_LAUNCH_BLOCKING does not remove it (2/2 failed).
  • Not the CP code: CPU is exact, and TpuGraphTrainer CP runs are deterministic. The failures hit configs without EP and without the code changed here. My guess is torch_tpu's per-block split compilation, which runs 59-117 partitions per block, but this is not confirmed.
  • Recommendation: use the TpuGraphTrainer recipes for CP. The regular-trainer EP + CP recipes work in the 35B runs above but should not be relied on for long runs until this is fixed.
  • Tests: test_train_distributed therefore has no compiled EP + CP debug cases.

EP + CP loss at step 20: rounding sensitivity, not an EP + CP bug

At 8K, EP=2 + CP2 ends at 2.652 (Ulysses) and 2.644 (KV all-gather) at step 20, against 2.562 for CP2 alone on the same data. Per-step logs show that this is one excursion of a trajectory that already wanders around the CP-only one. Deltas are against CP2 alone:

Step 1 6 13 15 16 17 18 19 20
CP2 alone (loss) 12.88350 4.89614 3.30947 3.01450 2.91143 2.77905 2.73036 2.63779 2.56204
EP recipe at EP=1 + CP2 +0.00001 -0.00002 +0.0004 +0.0009 +0.0007 +0.0001 -0.0003 +0.0001 -0.0007
EP=2 + CP2 +0.0006 -0.0192 +0.0155 +0.0007 +0.0123 +0.0101 +0.0548 +0.0367 +0.0896
EP=2 + CP2, 10x smaller LR (vs CP2 at that LR) +0.0006 -0.0003 +0.0013 -0.0019 -0.0035 -0.0012 -0.0004 -0.0008 -0.0001

Why it happens:

  • Peak LR at the last step. The 35B recipes set warmup_steps=200, which lr_scheduler.build clamps to training.steps. A 20-step benchmark therefore ramps a randomly initialized model to the peak LR (3e-4) at step 20. Grad norms reach 10-17 in the first 10 steps.
  • MoE amplifies rounding. AdamW's first updates are close to lr * sign(g), so rounding-level gradient differences flip the sign of near-zero elements. Top-k routing then turns small weight differences into discrete expert changes for near-tie tokens.
  • EP=2 rounds differently, by design. Ring-of-Experts is mathematically equivalent to EP=1, but it rounds at different points. Its step-1 loss, which is forward-only, differs by 5.5e-4. That is about 50 times the next-largest variant here, and it grows into the excursions above.
    • Most likely the main source: the gather-reduce casts each rank's FP32 partial output to BF16 (ragged_gather_tc.py, total.astype(x.dtype)) before _EpReduceScatter sums the two EP ranks. EP=1 sums all K routes in FP32 and rounds once.
    • Not the source: applying the routing weights on the hidden activation, and the token-chunk count. Turning either off gives the same step-1 loss (12.88404 vs 12.88405).
    • Precedent: the _LocalReorder docstring in token_dispatcher.py already records that reordering equal-expert routes, a pure accumulation-order change, "produced an unacceptable training-trajectory shift".

Evidence:

  1. Exact on CPU. CPU fp32 run with the parity harness, 4 gloo ranks against a single device. EP2 + CP2's step-1 loss and gradients match the single-device run as closely as plain data parallelism does. Both leave the same residual: shared-expert and lm_head gradients within 5e-5. After one AdamW step, every MoE configuration differs from the single-device run by 8e-4 to 9e-3 in loss, plain DP included. The dense model with CP2 matches to < 1e-5.

  2. Deterministic, and not the SPMD backend. Rerunning EP=2 + Ulysses CP2 reproduces every logged step exactly. EP=2 without CP under spmd_types, the backend CP forces, is bit-identical to EP=2 under partial_dtensor.

  3. No accumulating error. With --lr_scheduler.total_steps=200 the warmup is not clamped and LR stays <= 3e-5, while the loss still falls from 12.9 to 3.8. Under that schedule EP=2 + CP2 stays within 3.5e-3 of CP2 alone for 20 steps and ends 1e-4 apart (last row of the table).

  4. Growth matches the perturbation size. Over 20 steps, each perturbation grows by roughly the same factor:

    Perturbation Step 1 Step 20
    CP strategy < 5e-6 2.5e-3
    EP recipe at EP=1 1e-5 7e-4
    EP=2 5.5e-4 0.09
  5. No consistent direction. At 64K and 128K, EP=2 + CP2 ends below CP2 alone at step 10 (-0.065 and -0.037). EP=2 alone deviates from EP=1 by -0.022 at step 10 at 8K, then closes to +0.005 by step 20.

  6. Equivalent variants cluster by family. Five EP=2 + CP2 rounding variants all end at 2.644-2.660 at step 20, with the same stall at step 18: Ulysses or KV all-gather, 4 or 2 token chunks, routing weights on the activation or in the combine. CP2 alone, the EP recipe at EP=1 and the kernel variants all end at 2.561-2.565. Within a family, runs differ by at most 2e-5 at step 1 and end within 0.016. Between families they differ by 5.5e-4 and end 0.09 apart. The separation grows with the initial perturbation.

  7. The gap closes after the LR peak. A 40-step pair with --lr_scheduler.warmup_steps=20 has the same LR as the 20-step runs for steps 1-20 and reproduces them exactly. As the LR then decays, EP=2 + CP2 minus CP2 alone goes from +0.090 at step 20 to +0.019 at step 24, +0.005 at step 30 and +0.0065 at step 40 (1.90070 vs 1.89416).

Code checked for EP + CP interactions:

  • The Ring-of-Experts buffer is worst-case (cap = ep * T * K), so nothing is dropped.
  • The TPU recipes have no load-balancing state: no expert bias, no aux loss.
  • Each token's expert contribution is computed once and summed over efsdp.
  • global_valid_tokens matches CP alone.
  • The saved routing keeps backward on the forward's top-k.
  • The grad norm and clipping split dense and expert parameters correctly.
  • The meshes stay consistent with the topology-aware rank order.

Nothing EP + CP-specific was wrong. To compare EP + CP convergence against CP alone, use runs longer than the warmup, or set --lr_scheduler.total_steps, rather than 20-step benchmarks.

@jialei777
jialei777 force-pushed the feat/tpu-qwen35-cp-on-ep branch from abb6079 to 2f61d1e Compare November 17, 2026 18:14
…DeltaNet

Stacked on the fast dropless EP branch (#142). TPU CP (#2) only rewrote
attention, so on Qwen3.5 the GatedDeltaNet layers (30 of 40 in 35B-A3B)
would run their causal conv and delta-rule recurrence on each rank's token
shard as if it were a whole sequence, silently.

- qwen3_5/cp_gated_delta.py: CPTpuInnerGatedDeltaNet runs the GatedDeltaNet
  head-parallel: all-to-alls take q/k/v/a/b from token- to head-sharded
  (one per input: packing them measured slower), the tokens go back in
  sequence order (head-tail undone for KV all-gather CP), conv + recurrence run over whole sequences on the rank's
  heads with its slice of the conv weights / A_log / dt_bias (index_select on
  the cached CP-rank tensor), and one all-to-all brings the output back.
  Exact; works with both full-attention strategies. Plus
  GatedDeltaNetContextParallelTransform and a registered validator that
  rejects CP configs whose GatedDeltaNet layers are not converted.
- muse_glimmer/attention.py: Ulysses / KV all-gather CP versions of
  SlidingWindowScaledDotProductInnerAttention (Qwen3.5's TPU full attention).
- qwen3_5/model.py: the EP branch's TpuQwen35Model also shards CP inputs
  through the CP attention's cp_shard (per-sequence head-tail, CP-rank cache), as core
  Decoder does; upstream Qwen35Model calls prepare_context_parallel_input.
- qwen3_5/infra/parallelize.py: TPUSelectiveAC under CP; upstream's
  parallelize steps without its CP NotImplementedError.
- cp_attention.py: public cp_all_to_all / cp_all_gather /
  cp_rank_tensor; register_tpu_cp_inner_attention and
  register_tpu_cp_validator registries.
- qwen3_5/config_registry.py: _qwen35_cp (also selects the spmd_types SPMD
  backend, which CP requires on this base) and debug / 35B CP recipes.
- graph_trainer/qwen3_5.py: allow CP (still rejects TP / PP / EP).
- graph_trainer/graph_split.py: fsdp_pipeline raised for any backward
  all-gather / reduce-scatter that is neither a parameter gather nor a
  gradient reduction, including KV all-gather CP's recomputed K/V gathers
  and dK/dV reduce-scatters. Raise only when such a collective runs on the
  process group of a gradient reduction (gathers of saved activations are
  also fed only by graph inputs, so only reductions name the FSDP groups);
  on other groups (CP) it stays with its inputs, the existing fallback.
…n tests

- qwen3_5/tests/test_cp_gated_delta.py (CPU, emulated CP ranks, eager and
  traced collectives): CP GatedDeltaNet vs single device (cp 2/4, both token
  layouts, packed-document resets, outputs and every gradient), the CP debug
  model vs single device (loss and all parameter gradients), recipes,
  transform and validation.
- test_train_minimal_distributed: qwen3_5_simplefsdp_cp2_{all_gather,ulysses}
  and their per-layer torch.compile + Tokamax Splash variants (parity vs the
  CPU reference).
- test_train_distributed: Qwen3.5 dense / MoE + Splash CP cases and FSDP2 CP.
qwen35_moe_35b_split_tokamax_ep_{allgather,ulysses}_cp2: CP degree 2 on top of
qwen35_moe_35b_split_tokamax_ep_benchmark (EP=4; override with
--parallelism.expert_parallel_degree). EP and CP compose: CP shards each
sequence's tokens over the cp axis, and the routed experts dispatch every
rank's local tokens over the ep axis (ep must divide dp_shard * cp). The EP
recipe settings (by_layer split, one layer per executable, tpu_selective save
regions, SparseCore offload) are kept; a CPU test checks both.

test_train_distributed trains the debug MoE with EP 2 + CP 2 under both CP
strategies (ep groups == cp groups on 4 ranks).
With the EP regular-trainer compile fix below this commit, `--compile.enable
--compile.backend=tpu` (per-block `torch.compile(fullgraph=True)`) works with
EP, so EP + CP also runs on the regular `TPUTrainer`, not only on
`TpuGraphTrainer`.

- `qwen35_moe_35b_ep_{allgather,ulysses}_cp2`: `qwen35_moe_35b_ep` (EP=4,
  Tokamax GatedDeltaNet) with CP degree 2. They compile every block by default
  because eager Qwen3.5 CP with dp_shard > 1 is unreliable on the tpu7x
  torch_tpu build (see test_train_minimal_distributed); `--compile.no-enable`
  runs them eagerly.
- test_cp_gated_delta: the recipes keep EP and the Tokamax GatedDeltaNet, use
  spmd_types, compile every block and convert every GatedDeltaNet.
- test_train_distributed: no per-layer-compiled EP + CP debug cases. On
  tpu7x, per-block-compiled CP on the debug MoE model intermittently gives
  non-finite steps 1-2 (with or without EP or the Tokamax GatedDeltaNet; launch
  blocking does not help), so they would be flaky. 35B EP2 + CP2 runs trained
  cleanly (2/2); `TpuGraphTrainer` remains the stable CP path.
@jialei777
jialei777 force-pushed the feat/tpu-qwen35-cp-on-ep branch from 2f61d1e to ac466a4 Compare November 18, 2026 00:30
@jialei777
jialei777 changed the base branch from qwen3_5-35b-a3b-tpu-moe-ep-fast to fix/tpu-qwen35-ep-regular-trainer-compile November 18, 2026 00:31
jialei777 pushed a commit to jialei777/verl-hardware-plugin that referenced this pull request Nov 20, 2026
… v7x part-slice mesh

- Replace the hardcoded Qwen3.5 GatedDeltaNet CP import in the generic engine with engines/cp_layer_transforms.py:
  a registry of non-attention sequence mixers keyed by layer config type (model.traverse). A model whose layers
  need a CP transform the installed torchtitan lacks now raises instead of silently running them on token shards;
  availability uses importlib.util.find_spec so real ImportErrors propagate.
- Add TPUEngineOptions.context_parallel_gdn_strategy / context_parallel_gdn_reduce_dtype (validated), passed to
  GatedDeltaNetContextParallelTransform as strategy / reduce_dtype; rejected when the transform lacks the field.
- Reject gdn_row_slot_len with all_gather (head-tail) CP: the two token layouts disagree.
- Derive the mesh of a Ray v7x pool of whole hosts from the hosts' slice coordinates (ray.io/tpu-worker-id,
  ray.io/tpu-topology); non-block pools must set TORCH_TPU_TOPOLOGY; TPU_V7X_TOPOLOGY_MAP is only the fallback.
- Docs: CP support matrix and minimum torchtitan (google-pytorch/torchtitan#201); tests for fail-loud, ulysses
  (load_balancer=None), options, and topology derivation.

Signed-off-by: x <x@y>
jialei777 added a commit to jialei777/verl-hardware-plugin that referenced this pull request Nov 20, 2026
- Under context parallelism, also apply torchtitan's GatedDeltaNetContextParallelTransform (when the installed
  torchtitan provides it, google-pytorch/torchtitan#201) so the Qwen3.5 GatedDeltaNet layers run head-parallel over
  the CP token layout, as torchtitan's qwen3_5 _qwen35_cp config does. No-op for older torchtitan and for models
  without TpuInnerGatedDeltaNet.
- Add the v7x 256-TensorCore (4x4x8 slice) entry to TPU_V7X_TOPOLOGY_MAP.

Signed-off-by: Jialei Chen <96352824+jialei777@users.noreply.github.com>
jialei777 pushed a commit to jialei777/verl-hardware-plugin that referenced this pull request Nov 20, 2026
… v7x part-slice mesh

- Replace the hardcoded Qwen3.5 GatedDeltaNet CP import in the generic engine with engines/cp_layer_transforms.py:
  a registry of non-attention sequence mixers keyed by layer config type (model.traverse). A model whose layers
  need a CP transform the installed torchtitan lacks now raises instead of silently running them on token shards;
  availability uses importlib.util.find_spec so real ImportErrors propagate.
- Add TPUEngineOptions.context_parallel_gdn_strategy / context_parallel_gdn_reduce_dtype (validated), passed to
  GatedDeltaNetContextParallelTransform as strategy / reduce_dtype; rejected when the transform lacks the field.
- Reject gdn_row_slot_len with all_gather (head-tail) CP: the two token layouts disagree.
- Derive the mesh of a Ray v7x pool of whole hosts from the hosts' slice coordinates (ray.io/tpu-worker-id,
  ray.io/tpu-topology); non-block pools must set TORCH_TPU_TOPOLOGY; TPU_V7X_TOPOLOGY_MAP is only the fallback.
- Docs: CP support matrix and minimum torchtitan (google-pytorch/torchtitan#201); tests for fail-loud, ulysses
  (load_balancer=None), options, and topology derivation.

Signed-off-by: x <x@y>
jialei777 added a commit to verl-project/verl-hardware-plugin that referenced this pull request Nov 20, 2026
#170)

### What

Two changes needed to run Qwen3.5 context parallelism with the
torchtitan TPU fork at google-pytorch/torchtitan#201, and to run on a
256-TensorCore (32-host) TPU v7x pool:

1. **CP for non-attention sequence mixers (Qwen3.5 GatedDeltaNet), with
no model names in the engine.**
- New `engines/cp_layer_transforms.py`: a small registry of
`CPLayerTransform(layer_config, transform, requirement, option_fields)`.
`_enable_context_parallel` calls
`context_parallel_layer_transforms(model, load_balancer, tpu_options)`
and applies whatever it returns after core `ContextParallelTransform`,
before `validate_tpu_context_parallel`.
- A layer is found by its config type
(`model.traverse(GatedDeltaNet.Config)`). The engine only looks at types
whose module is already imported, so it imports nothing for models that
don't use them. The built-in entry maps torchtitan's
`torchtitan.models.qwen3_5.gdn:GatedDeltaNet.Config` to
`torchtitan.experiments.tpu.qwen3_5.cp_gated_delta:GatedDeltaNetContextParallelTransform`,
the same transform torchtitan's `qwen3_5/config_registry.py::_qwen35_cp`
applies. It gets the CP strategy's load balancer (`"headtail"` for
`all_gather`, `None` for `ulysses`).
- **Fails loudly.** If the model has GatedDeltaNet layers, CP > 1, and
the installed torchtitan has no GDN CP transform, the engine raises
`RuntimeError` naming the missing transform and the torchtitan it needs.
It does not run the recurrence on token shards. Availability is checked
with `importlib.util.find_spec`: only a missing module or missing parent
package counts as unavailable, so an `ImportError` raised inside an
existing module (for example a broken dependency) propagates. Models
without these layers are unaffected.
- New `TPUEngineOptions` fields (validated, documented):
`context_parallel_gdn_strategy` (`all_to_all` |
`allgather_reducescatter`) and `context_parallel_gdn_reduce_dtype`
(`float32` | `bfloat16`). They are passed to the transform as `strategy`
/ `reduce_dtype`. Unset, they keep the transform's defaults. If you set
one and the installed transform has no such field, the engine raises; it
doesn't drop the option silently.
- `gdn_row_slot_len` with `all_gather` (head-tail) CP is now rejected.
The load balancer shards each packed micro-batch as one sequence, while
the CP GatedDeltaNet reads it as `gdn_row_slot_len` rows. Ulysses
(contiguous shards) is allowed.
2. **v7x mesh of a pool that uses part of a slice: derived instead of
hardcoded.**
- `get_tpu_env_vars` now derives the mesh of a Ray pool of whole v7x
hosts from the hosts' slice coordinates. It reads the Ray node labels
`ray.io/tpu-worker-id`, `ray.io/tpu-topology` and
`ray.io/tpu-slice-name`, which Ray sets from GKE's `TPU_WORKER_ID` /
`TPU_TOPOLOGY`. The mesh is the bounding box of the pool's hosts times
the 2x2x1 chips per host, plus the core axis (`tpu_pool_chip_topology` /
`ray_tpu_pool_chip_topology`). This assumes worker ids count x-fastest
over the slice's host grid. If the hosts don't fill a box (they aren't a
sub-mesh), the engine raises and asks for `TORCH_TPU_TOPOLOGY`. If the
labels are missing or the hosts span several slices, it falls back to
the table.
- Resolution order: `TORCH_TPU_TOPOLOGY` > whole GKE slice (webhook) >
derived block of hosts > `TPU_V7X_TOPOLOGY_MAP`. `256: "4,4,8,2"` stays
in the table only as the fallback shape, and the comment now lists
`4x4x8 = 32` hosts.

### Defaults / behavior changes

- CP off: unchanged.
- Dense / MoE models under CP: unchanged (no extra transform is
applied).
- Qwen3.5 under CP with a torchtitan that has #201: same transform as
the previous revision of this PR.
- Qwen3.5 under CP with an older torchtitan: **now raises.** Before, it
ran silently, with wrong GDN numerics.
- v7x Ray pools whose hosts carry Ray TPU labels: the mesh now comes
from their coordinates. For a whole slice this is the slice shape. For a
part-slice pool it can differ from the table's guess, e.g. 32 hosts
holding z=0..3 of a 4x8x8 slice give `4,8,4,2` instead of `4,4,8,2`.
This is a correctness fix (mesh = the ICI block), but it changes the
shape for part-slice pools.

### Minimum torchtitan

Qwen3.5 CP needs google-pytorch/torchtitan#201 (branch
`feat/tpu-qwen35-cp-on-ep`; the transform landed in `36643b4c9`). The
`context_parallel_gdn_strategy` / `_reduce_dtype` options need a later
torchtitan whose `GatedDeltaNetContextParallelTransform` has `strategy`
/ `reduce_dtype` fields (in progress upstream). Documented in
`docs/accelerators/tpu/engine.md` (new "Context Parallelism" section
with a support matrix) and in `README.md`.

### Tests (CPU, torch 2.15 cpu + verl)

New or reworked tests in `tests/accelerators/tpu/test_tpu_engine.py`,
sharing one `_fake_cp_modules` helper instead of duplicated
`sys.modules` fakes:
- GDN transform is applied after `ContextParallelTransform` and before
validate/patches, for `all_gather` (`load_balancer="headtail"`) and
`ulysses` (`load_balancer=None`).
- Fail-loud: a GDN model without the torchtitan transform raises before
any transform is applied.
- A model without GDN layers gets no GDN transform, even when the GDN
module is imported.
- `module_available` does not hide an `ImportError` raised inside an
existing package.
- GDN options are passed through. Unset options keep the transform
defaults. Options set on an old transform are rejected. Bad values are
rejected by option parsing.
- `gdn_row_slot_len` is rejected with `all_gather` CP and allowed with
`ulysses`.
- Topology: `tpu_pool_chip_topology` (whole slice, half slices, a 4x4x8
block of a 4x8x8 slice, one host, L-shape rejected, bad ids),
`ray_tpu_pool_chip_topology` (labels / no labels / two slices /
non-block), `resolve_tpu_v7x_topology` precedence, and
`get_tpu_env_vars` end to end for a 256-device pool (derived `4,8,4,2` /
`4,4,8,2`, table fallback without labels).

Results: `pytest tests/accelerators/tpu`: 484 passed, 3 skipped (the new
CP/topology subset: 26 passed). `ruff check`, `ruff format --check`,
`mypy --ignore-missing-imports verl_hardware_plugin/`,
`scripts/check_license.py`: clean.

### Measured vs. not measured on TPU

- **Measured earlier, on the previous revision of this PR** (same GDN
transform, table topology): Qwen3.5-35B-A3B 64k DeepSWE RL with FSDP8 x
CP2 (Runs 18a/18d/18g), and Qwen3.5-397B-A17B on a 4x4x8 pool (Runs
19/20/21). Those runs show it runs end to end. No numbers are reported
here.
- **Not measured (TPU validation still needed):**
- CP=1 vs CP=2 log-prob / entropy / loss parity on Qwen3.5 with packed
multi-document micro-batches, plus step time and HBM.
  - The new fail-loud and option plumbing on hardware.
  - Ulysses + GDN CP.
- GDN CP with Tokamax `kda` kernels and `set_packed_document_budget`. By
design the CP GatedDeltaNet runs the kernel on whole sequences after the
all-to-all, with all-gathered positions, so the budget per micro-batch
is unchanged. This is unvalidated.
  - `gdn_row_slot_len` + Ulysses CP.
- The derived topology on a real part-slice pool, including the
x-fastest worker-id assumption. `TORCH_TPU_TOPOLOGY` overrides it if
needed.
  - None of the changes in this revision were run on TPU.

### Follow-ups

- Upstream: torchtitan's TPU CP has a validator registry
(`register_tpu_cp_validator`) but no transform registry. Once torchtitan
exposes one (or a model-spec hook for CP transforms),
`cp_layer_transforms` should defer to it and drop its own entry.
- Hardware parity numbers (CP=1 vs CP=2) before this leaves draft.
- Optionally split the topology change into its own PR. It is kept here
because both changes are needed together for the 397B 4x4x8 runs.

---------

Signed-off-by: Jialei Chen <96352824+jialei777@users.noreply.github.com>
Signed-off-by: x <x@y>
Co-authored-by: x <x@y>
jialei777 added a commit to verl-project/verl-hardware-plugin that referenced this pull request Nov 20, 2026
… GDN projections & slot sorting via TPUEngineOptions (#172)

## Summary

Replaces the prototype Qwen3.5 monkeypatches and `VERL_TPU_*`
environment variables with clean, model-agnostic scheduling and
input-preparation support in `verl-hardware-plugin`, paired with
upstream model/CP capabilities in
[google-pytorch/torchtitan#216](google-pytorch/torchtitan#216)
and GDN CP strategy options in #170:

- **No torchtitan monkeypatches or `VERL_TPU_*` env vars**: Deletes
`_install_fast_cp_gdn_patches` and exposes five validated
`TPUEngineOptions` (`engine_kwargs.tpu.*`) fields:
- `cp_full_positions`: Passes pre-`cp_shard` unsharded 1-D `positions`
as `cp_full_positions` (`DP: V, CP: R, TP: R`) to `model.forward`,
eliminating per-layer `all_gather(positions)` collectives across both
Ulysses and KV all-gather CP (`google-pytorch/torchtitan#216`), while
preserving `cp_shard`'s `attention_masks` / load-balancer metadata.
Rejects multimodal and 3-D MRoPE inputs explicitly instead of flattening
positions.
- `gdn_auto_row_slot_len`: Dynamic per-micro-batch row-slot planner
(`gdn_slot_schedule.py`) for any model whose `forward()` accepts
`gated_delta_document_resets` (`google-pytorch/torchtitan#216`):
- Micro-batches whose single-document sequences fit
`gdn_auto_row_slot_len` rows are permuted into slots
(`permute_packed_for_row_slots`) and run unsegmented
(`set_unflatten_seq_len(slot_len)`,
`gated_delta_document_resets=False`), restoring 1-D fused `log_probs`
and `entropy` after optional CP gather (for both CP and non-CP).
- Single-document micro-batches (`n_seqs == 1`) run unsegmented at
`padded_len`.
- Multi-document micro-batches (verified by checking exact `position ==
0` indices against `cu_seqlens[:-1]`, catching continuation sequences
with interior resets) run segmented; Tokamax `packed_document_budget` is
sized over segmented micro-batches only.
- All SPMD ranks synchronize eligibility masks via `dist.all_reduce(...,
ReduceOp.BAND)` so every rank takes the same branch and compiled shape.
- `fuse_gdn_projections`: Walks `model_spec.model` before build and sets
`fuse_input_projections = True` (`google-pytorch/torchtitan#216`),
failing loudly if zero sub-configs declare `fuse_input_projections`.
- `sort_micro_batches_by_slot`: Orders micro-batches by `(slot_category,
-total_tokens, original_idx)` using the exact same
`classify_gdn_micro_batch` classifier as `plan_micro_batch_gdn_slots` so
identical slot/segmentation shapes execute back to back.
- `static_log_prob_micro_batch_len`: Overrides `static_micro_batch_len`
on forward-only (`old_log_probs` / `ref_log_prob`) passes.
- **Head-replicated AG/RS GatedDeltaNet CP**: Moved upstream into
`CPTpuGatedDeltaNet` (`strategy="allgather_reducescatter"`) in
[google-pytorch/torchtitan#216](google-pytorch/torchtitan#216)
for any `cp` degree and both contiguous and `headtail` load balancers,
configured via #170's `engine_kwargs.tpu.context_parallel_gdn_strategy:
"allgather_reducescatter"` and `context_parallel_gdn_reduce_dtype`.

## Dependencies

- Model-side kwargs (`cp_full_positions`, `gated_delta_document_resets`)
and `TpuGatedDeltaNet.Config.fuse_input_projections`:
[google-pytorch/torchtitan#216](google-pytorch/torchtitan#216)
(stacked on `google-pytorch/torchtitan#201`). Fails loudly at engine
initialization if any of these options are enabled against an older
torchtitan build.
- GatedDeltaNet `allgather_reducescatter` CP transform wiring: #170
(`context_parallel_gdn_strategy` / `context_parallel_gdn_reduce_dtype`).

## Verification

- **CPU unit & integration tests**:
- `pytest tests/accelerators/tpu/test_tpu_gdn_slot_schedule.py
tests/accelerators/tpu/test_tpu_options.py` (exact interior reset
detection including continuation sequences, shared classifier/sorter
category order, differentiated-entropy and 3-D MRoPE / multimodal
exclusions, `BAND` cross-rank slot negotiation, non-CP and 2-way CP
row-slot permutation + `cp_full_positions` round-trip log-prob/entropy
parity, config walker & model signature fail-loud checks).
- Full `tests/accelerators/tpu` suite, `ruff format --check`, `ruff
check`, `mypy`, and license checks.
- **TPU benchmark status**:
- Initial prototype measurements on Qwen3.5-35B-A3B DeepSWE RL (64k
context, TPU v7x, FSDP8 x CP2, on top of #171): pure trainer step time
dropped from 141–156 s/step (Ulysses baseline, Run 18a) to 6.35
s/micro-batch with full positions + 32k row slots + AG/RS GDN (Run 18d;
val pass@1 0.414 -> 0.700 over 70 steps) and 6.18 s/micro-batch mean /
128 s minimum trainer step with slot sorting (Run 18g).
- **TODO before marking ready for review**: Re-run isolated per-option
A/B throughput and log-prob parity validation on TPU v7x using the
refactored `google-pytorch/torchtitan#216` + #170 + #172 stack.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant