Repository navigation
Conversation
jialei777
force-pushed
the
feat/tpu-qwen35-cp-on-ep
branch
from
November 17, 2026 18:14
abb6079 to
2f61d1e
Compare
…DeltaNet Stacked on the fast dropless EP branch (#142). TPU CP (#2) only rewrote attention, so on Qwen3.5 the GatedDeltaNet layers (30 of 40 in 35B-A3B) would run their causal conv and delta-rule recurrence on each rank's token shard as if it were a whole sequence, silently. - qwen3_5/cp_gated_delta.py: CPTpuInnerGatedDeltaNet runs the GatedDeltaNet head-parallel: all-to-alls take q/k/v/a/b from token- to head-sharded (one per input: packing them measured slower), the tokens go back in sequence order (head-tail undone for KV all-gather CP), conv + recurrence run over whole sequences on the rank's heads with its slice of the conv weights / A_log / dt_bias (index_select on the cached CP-rank tensor), and one all-to-all brings the output back. Exact; works with both full-attention strategies. Plus GatedDeltaNetContextParallelTransform and a registered validator that rejects CP configs whose GatedDeltaNet layers are not converted. - muse_glimmer/attention.py: Ulysses / KV all-gather CP versions of SlidingWindowScaledDotProductInnerAttention (Qwen3.5's TPU full attention). - qwen3_5/model.py: the EP branch's TpuQwen35Model also shards CP inputs through the CP attention's cp_shard (per-sequence head-tail, CP-rank cache), as core Decoder does; upstream Qwen35Model calls prepare_context_parallel_input. - qwen3_5/infra/parallelize.py: TPUSelectiveAC under CP; upstream's parallelize steps without its CP NotImplementedError. - cp_attention.py: public cp_all_to_all / cp_all_gather / cp_rank_tensor; register_tpu_cp_inner_attention and register_tpu_cp_validator registries. - qwen3_5/config_registry.py: _qwen35_cp (also selects the spmd_types SPMD backend, which CP requires on this base) and debug / 35B CP recipes.
- graph_trainer/qwen3_5.py: allow CP (still rejects TP / PP / EP). - graph_trainer/graph_split.py: fsdp_pipeline raised for any backward all-gather / reduce-scatter that is neither a parameter gather nor a gradient reduction, including KV all-gather CP's recomputed K/V gathers and dK/dV reduce-scatters. Raise only when such a collective runs on the process group of a gradient reduction (gathers of saved activations are also fed only by graph inputs, so only reductions name the FSDP groups); on other groups (CP) it stays with its inputs, the existing fallback.
…n tests
- qwen3_5/tests/test_cp_gated_delta.py (CPU, emulated CP ranks, eager and
traced collectives): CP GatedDeltaNet vs single device (cp 2/4, both token
layouts, packed-document resets, outputs and every gradient), the CP debug
model vs single device (loss and all parameter gradients), recipes,
transform and validation.
- test_train_minimal_distributed: qwen3_5_simplefsdp_cp2_{all_gather,ulysses}
and their per-layer torch.compile + Tokamax Splash variants (parity vs the
CPU reference).
- test_train_distributed: Qwen3.5 dense / MoE + Splash CP cases and FSDP2 CP.
qwen35_moe_35b_split_tokamax_ep_{allgather,ulysses}_cp2: CP degree 2 on top of
qwen35_moe_35b_split_tokamax_ep_benchmark (EP=4; override with
--parallelism.expert_parallel_degree). EP and CP compose: CP shards each
sequence's tokens over the cp axis, and the routed experts dispatch every
rank's local tokens over the ep axis (ep must divide dp_shard * cp). The EP
recipe settings (by_layer split, one layer per executable, tpu_selective save
regions, SparseCore offload) are kept; a CPU test checks both.
test_train_distributed trains the debug MoE with EP 2 + CP 2 under both CP
strategies (ep groups == cp groups on 4 ranks).
With the EP regular-trainer compile fix below this commit, `--compile.enable
--compile.backend=tpu` (per-block `torch.compile(fullgraph=True)`) works with
EP, so EP + CP also runs on the regular `TPUTrainer`, not only on
`TpuGraphTrainer`.
- `qwen35_moe_35b_ep_{allgather,ulysses}_cp2`: `qwen35_moe_35b_ep` (EP=4,
Tokamax GatedDeltaNet) with CP degree 2. They compile every block by default
because eager Qwen3.5 CP with dp_shard > 1 is unreliable on the tpu7x
torch_tpu build (see test_train_minimal_distributed); `--compile.no-enable`
runs them eagerly.
- test_cp_gated_delta: the recipes keep EP and the Tokamax GatedDeltaNet, use
spmd_types, compile every block and convert every GatedDeltaNet.
- test_train_distributed: no per-layer-compiled EP + CP debug cases. On
tpu7x, per-block-compiled CP on the debug MoE model intermittently gives
non-finite steps 1-2 (with or without EP or the Tokamax GatedDeltaNet; launch
blocking does not help), so they would be flaky. 35B EP2 + CP2 runs trained
cleanly (2/2); `TpuGraphTrainer` remains the stable CP path.
jialei777
force-pushed
the
feat/tpu-qwen35-cp-on-ep
branch
from
November 18, 2026 00:30
2f61d1e to
ac466a4
Compare
jialei777
changed the base branch from
qwen3_5-35b-a3b-tpu-moe-ep-fast
to
fix/tpu-qwen35-ep-regular-trainer-compile
November 18, 2026 00:31
This was referenced Nov 19, 2026
Draft
jialei777
pushed a commit
to jialei777/verl-hardware-plugin
that referenced
this pull request
Nov 20, 2026
… v7x part-slice mesh - Replace the hardcoded Qwen3.5 GatedDeltaNet CP import in the generic engine with engines/cp_layer_transforms.py: a registry of non-attention sequence mixers keyed by layer config type (model.traverse). A model whose layers need a CP transform the installed torchtitan lacks now raises instead of silently running them on token shards; availability uses importlib.util.find_spec so real ImportErrors propagate. - Add TPUEngineOptions.context_parallel_gdn_strategy / context_parallel_gdn_reduce_dtype (validated), passed to GatedDeltaNetContextParallelTransform as strategy / reduce_dtype; rejected when the transform lacks the field. - Reject gdn_row_slot_len with all_gather (head-tail) CP: the two token layouts disagree. - Derive the mesh of a Ray v7x pool of whole hosts from the hosts' slice coordinates (ray.io/tpu-worker-id, ray.io/tpu-topology); non-block pools must set TORCH_TPU_TOPOLOGY; TPU_V7X_TOPOLOGY_MAP is only the fallback. - Docs: CP support matrix and minimum torchtitan (google-pytorch/torchtitan#201); tests for fail-loud, ulysses (load_balancer=None), options, and topology derivation. Signed-off-by: x <x@y>
jialei777
added a commit
to jialei777/verl-hardware-plugin
that referenced
this pull request
Nov 20, 2026
- Under context parallelism, also apply torchtitan's GatedDeltaNetContextParallelTransform (when the installed torchtitan provides it, google-pytorch/torchtitan#201) so the Qwen3.5 GatedDeltaNet layers run head-parallel over the CP token layout, as torchtitan's qwen3_5 _qwen35_cp config does. No-op for older torchtitan and for models without TpuInnerGatedDeltaNet. - Add the v7x 256-TensorCore (4x4x8 slice) entry to TPU_V7X_TOPOLOGY_MAP. Signed-off-by: Jialei Chen <96352824+jialei777@users.noreply.github.com>
jialei777
pushed a commit
to jialei777/verl-hardware-plugin
that referenced
this pull request
Nov 20, 2026
… v7x part-slice mesh - Replace the hardcoded Qwen3.5 GatedDeltaNet CP import in the generic engine with engines/cp_layer_transforms.py: a registry of non-attention sequence mixers keyed by layer config type (model.traverse). A model whose layers need a CP transform the installed torchtitan lacks now raises instead of silently running them on token shards; availability uses importlib.util.find_spec so real ImportErrors propagate. - Add TPUEngineOptions.context_parallel_gdn_strategy / context_parallel_gdn_reduce_dtype (validated), passed to GatedDeltaNetContextParallelTransform as strategy / reduce_dtype; rejected when the transform lacks the field. - Reject gdn_row_slot_len with all_gather (head-tail) CP: the two token layouts disagree. - Derive the mesh of a Ray v7x pool of whole hosts from the hosts' slice coordinates (ray.io/tpu-worker-id, ray.io/tpu-topology); non-block pools must set TORCH_TPU_TOPOLOGY; TPU_V7X_TOPOLOGY_MAP is only the fallback. - Docs: CP support matrix and minimum torchtitan (google-pytorch/torchtitan#201); tests for fail-loud, ulysses (load_balancer=None), options, and topology derivation. Signed-off-by: x <x@y>
jialei777
added a commit
to verl-project/verl-hardware-plugin
that referenced
this pull request
Nov 20, 2026
#170) ### What Two changes needed to run Qwen3.5 context parallelism with the torchtitan TPU fork at google-pytorch/torchtitan#201, and to run on a 256-TensorCore (32-host) TPU v7x pool: 1. **CP for non-attention sequence mixers (Qwen3.5 GatedDeltaNet), with no model names in the engine.** - New `engines/cp_layer_transforms.py`: a small registry of `CPLayerTransform(layer_config, transform, requirement, option_fields)`. `_enable_context_parallel` calls `context_parallel_layer_transforms(model, load_balancer, tpu_options)` and applies whatever it returns after core `ContextParallelTransform`, before `validate_tpu_context_parallel`. - A layer is found by its config type (`model.traverse(GatedDeltaNet.Config)`). The engine only looks at types whose module is already imported, so it imports nothing for models that don't use them. The built-in entry maps torchtitan's `torchtitan.models.qwen3_5.gdn:GatedDeltaNet.Config` to `torchtitan.experiments.tpu.qwen3_5.cp_gated_delta:GatedDeltaNetContextParallelTransform`, the same transform torchtitan's `qwen3_5/config_registry.py::_qwen35_cp` applies. It gets the CP strategy's load balancer (`"headtail"` for `all_gather`, `None` for `ulysses`). - **Fails loudly.** If the model has GatedDeltaNet layers, CP > 1, and the installed torchtitan has no GDN CP transform, the engine raises `RuntimeError` naming the missing transform and the torchtitan it needs. It does not run the recurrence on token shards. Availability is checked with `importlib.util.find_spec`: only a missing module or missing parent package counts as unavailable, so an `ImportError` raised inside an existing module (for example a broken dependency) propagates. Models without these layers are unaffected. - New `TPUEngineOptions` fields (validated, documented): `context_parallel_gdn_strategy` (`all_to_all` | `allgather_reducescatter`) and `context_parallel_gdn_reduce_dtype` (`float32` | `bfloat16`). They are passed to the transform as `strategy` / `reduce_dtype`. Unset, they keep the transform's defaults. If you set one and the installed transform has no such field, the engine raises; it doesn't drop the option silently. - `gdn_row_slot_len` with `all_gather` (head-tail) CP is now rejected. The load balancer shards each packed micro-batch as one sequence, while the CP GatedDeltaNet reads it as `gdn_row_slot_len` rows. Ulysses (contiguous shards) is allowed. 2. **v7x mesh of a pool that uses part of a slice: derived instead of hardcoded.** - `get_tpu_env_vars` now derives the mesh of a Ray pool of whole v7x hosts from the hosts' slice coordinates. It reads the Ray node labels `ray.io/tpu-worker-id`, `ray.io/tpu-topology` and `ray.io/tpu-slice-name`, which Ray sets from GKE's `TPU_WORKER_ID` / `TPU_TOPOLOGY`. The mesh is the bounding box of the pool's hosts times the 2x2x1 chips per host, plus the core axis (`tpu_pool_chip_topology` / `ray_tpu_pool_chip_topology`). This assumes worker ids count x-fastest over the slice's host grid. If the hosts don't fill a box (they aren't a sub-mesh), the engine raises and asks for `TORCH_TPU_TOPOLOGY`. If the labels are missing or the hosts span several slices, it falls back to the table. - Resolution order: `TORCH_TPU_TOPOLOGY` > whole GKE slice (webhook) > derived block of hosts > `TPU_V7X_TOPOLOGY_MAP`. `256: "4,4,8,2"` stays in the table only as the fallback shape, and the comment now lists `4x4x8 = 32` hosts. ### Defaults / behavior changes - CP off: unchanged. - Dense / MoE models under CP: unchanged (no extra transform is applied). - Qwen3.5 under CP with a torchtitan that has #201: same transform as the previous revision of this PR. - Qwen3.5 under CP with an older torchtitan: **now raises.** Before, it ran silently, with wrong GDN numerics. - v7x Ray pools whose hosts carry Ray TPU labels: the mesh now comes from their coordinates. For a whole slice this is the slice shape. For a part-slice pool it can differ from the table's guess, e.g. 32 hosts holding z=0..3 of a 4x8x8 slice give `4,8,4,2` instead of `4,4,8,2`. This is a correctness fix (mesh = the ICI block), but it changes the shape for part-slice pools. ### Minimum torchtitan Qwen3.5 CP needs google-pytorch/torchtitan#201 (branch `feat/tpu-qwen35-cp-on-ep`; the transform landed in `36643b4c9`). The `context_parallel_gdn_strategy` / `_reduce_dtype` options need a later torchtitan whose `GatedDeltaNetContextParallelTransform` has `strategy` / `reduce_dtype` fields (in progress upstream). Documented in `docs/accelerators/tpu/engine.md` (new "Context Parallelism" section with a support matrix) and in `README.md`. ### Tests (CPU, torch 2.15 cpu + verl) New or reworked tests in `tests/accelerators/tpu/test_tpu_engine.py`, sharing one `_fake_cp_modules` helper instead of duplicated `sys.modules` fakes: - GDN transform is applied after `ContextParallelTransform` and before validate/patches, for `all_gather` (`load_balancer="headtail"`) and `ulysses` (`load_balancer=None`). - Fail-loud: a GDN model without the torchtitan transform raises before any transform is applied. - A model without GDN layers gets no GDN transform, even when the GDN module is imported. - `module_available` does not hide an `ImportError` raised inside an existing package. - GDN options are passed through. Unset options keep the transform defaults. Options set on an old transform are rejected. Bad values are rejected by option parsing. - `gdn_row_slot_len` is rejected with `all_gather` CP and allowed with `ulysses`. - Topology: `tpu_pool_chip_topology` (whole slice, half slices, a 4x4x8 block of a 4x8x8 slice, one host, L-shape rejected, bad ids), `ray_tpu_pool_chip_topology` (labels / no labels / two slices / non-block), `resolve_tpu_v7x_topology` precedence, and `get_tpu_env_vars` end to end for a 256-device pool (derived `4,8,4,2` / `4,4,8,2`, table fallback without labels). Results: `pytest tests/accelerators/tpu`: 484 passed, 3 skipped (the new CP/topology subset: 26 passed). `ruff check`, `ruff format --check`, `mypy --ignore-missing-imports verl_hardware_plugin/`, `scripts/check_license.py`: clean. ### Measured vs. not measured on TPU - **Measured earlier, on the previous revision of this PR** (same GDN transform, table topology): Qwen3.5-35B-A3B 64k DeepSWE RL with FSDP8 x CP2 (Runs 18a/18d/18g), and Qwen3.5-397B-A17B on a 4x4x8 pool (Runs 19/20/21). Those runs show it runs end to end. No numbers are reported here. - **Not measured (TPU validation still needed):** - CP=1 vs CP=2 log-prob / entropy / loss parity on Qwen3.5 with packed multi-document micro-batches, plus step time and HBM. - The new fail-loud and option plumbing on hardware. - Ulysses + GDN CP. - GDN CP with Tokamax `kda` kernels and `set_packed_document_budget`. By design the CP GatedDeltaNet runs the kernel on whole sequences after the all-to-all, with all-gathered positions, so the budget per micro-batch is unchanged. This is unvalidated. - `gdn_row_slot_len` + Ulysses CP. - The derived topology on a real part-slice pool, including the x-fastest worker-id assumption. `TORCH_TPU_TOPOLOGY` overrides it if needed. - None of the changes in this revision were run on TPU. ### Follow-ups - Upstream: torchtitan's TPU CP has a validator registry (`register_tpu_cp_validator`) but no transform registry. Once torchtitan exposes one (or a model-spec hook for CP transforms), `cp_layer_transforms` should defer to it and drop its own entry. - Hardware parity numbers (CP=1 vs CP=2) before this leaves draft. - Optionally split the topology change into its own PR. It is kept here because both changes are needed together for the 397B 4x4x8 runs. --------- Signed-off-by: Jialei Chen <96352824+jialei777@users.noreply.github.com> Signed-off-by: x <x@y> Co-authored-by: x <x@y>
jialei777
added a commit
to verl-project/verl-hardware-plugin
that referenced
this pull request
Nov 20, 2026
… GDN projections & slot sorting via TPUEngineOptions (#172) ## Summary Replaces the prototype Qwen3.5 monkeypatches and `VERL_TPU_*` environment variables with clean, model-agnostic scheduling and input-preparation support in `verl-hardware-plugin`, paired with upstream model/CP capabilities in [google-pytorch/torchtitan#216](google-pytorch/torchtitan#216) and GDN CP strategy options in #170: - **No torchtitan monkeypatches or `VERL_TPU_*` env vars**: Deletes `_install_fast_cp_gdn_patches` and exposes five validated `TPUEngineOptions` (`engine_kwargs.tpu.*`) fields: - `cp_full_positions`: Passes pre-`cp_shard` unsharded 1-D `positions` as `cp_full_positions` (`DP: V, CP: R, TP: R`) to `model.forward`, eliminating per-layer `all_gather(positions)` collectives across both Ulysses and KV all-gather CP (`google-pytorch/torchtitan#216`), while preserving `cp_shard`'s `attention_masks` / load-balancer metadata. Rejects multimodal and 3-D MRoPE inputs explicitly instead of flattening positions. - `gdn_auto_row_slot_len`: Dynamic per-micro-batch row-slot planner (`gdn_slot_schedule.py`) for any model whose `forward()` accepts `gated_delta_document_resets` (`google-pytorch/torchtitan#216`): - Micro-batches whose single-document sequences fit `gdn_auto_row_slot_len` rows are permuted into slots (`permute_packed_for_row_slots`) and run unsegmented (`set_unflatten_seq_len(slot_len)`, `gated_delta_document_resets=False`), restoring 1-D fused `log_probs` and `entropy` after optional CP gather (for both CP and non-CP). - Single-document micro-batches (`n_seqs == 1`) run unsegmented at `padded_len`. - Multi-document micro-batches (verified by checking exact `position == 0` indices against `cu_seqlens[:-1]`, catching continuation sequences with interior resets) run segmented; Tokamax `packed_document_budget` is sized over segmented micro-batches only. - All SPMD ranks synchronize eligibility masks via `dist.all_reduce(..., ReduceOp.BAND)` so every rank takes the same branch and compiled shape. - `fuse_gdn_projections`: Walks `model_spec.model` before build and sets `fuse_input_projections = True` (`google-pytorch/torchtitan#216`), failing loudly if zero sub-configs declare `fuse_input_projections`. - `sort_micro_batches_by_slot`: Orders micro-batches by `(slot_category, -total_tokens, original_idx)` using the exact same `classify_gdn_micro_batch` classifier as `plan_micro_batch_gdn_slots` so identical slot/segmentation shapes execute back to back. - `static_log_prob_micro_batch_len`: Overrides `static_micro_batch_len` on forward-only (`old_log_probs` / `ref_log_prob`) passes. - **Head-replicated AG/RS GatedDeltaNet CP**: Moved upstream into `CPTpuGatedDeltaNet` (`strategy="allgather_reducescatter"`) in [google-pytorch/torchtitan#216](google-pytorch/torchtitan#216) for any `cp` degree and both contiguous and `headtail` load balancers, configured via #170's `engine_kwargs.tpu.context_parallel_gdn_strategy: "allgather_reducescatter"` and `context_parallel_gdn_reduce_dtype`. ## Dependencies - Model-side kwargs (`cp_full_positions`, `gated_delta_document_resets`) and `TpuGatedDeltaNet.Config.fuse_input_projections`: [google-pytorch/torchtitan#216](google-pytorch/torchtitan#216) (stacked on `google-pytorch/torchtitan#201`). Fails loudly at engine initialization if any of these options are enabled against an older torchtitan build. - GatedDeltaNet `allgather_reducescatter` CP transform wiring: #170 (`context_parallel_gdn_strategy` / `context_parallel_gdn_reduce_dtype`). ## Verification - **CPU unit & integration tests**: - `pytest tests/accelerators/tpu/test_tpu_gdn_slot_schedule.py tests/accelerators/tpu/test_tpu_options.py` (exact interior reset detection including continuation sequences, shared classifier/sorter category order, differentiated-entropy and 3-D MRoPE / multimodal exclusions, `BAND` cross-rank slot negotiation, non-CP and 2-way CP row-slot permutation + `cp_full_positions` round-trip log-prob/entropy parity, config walker & model signature fail-loud checks). - Full `tests/accelerators/tpu` suite, `ruff format --check`, `ruff check`, `mypy`, and license checks. - **TPU benchmark status**: - Initial prototype measurements on Qwen3.5-35B-A3B DeepSWE RL (64k context, TPU v7x, FSDP8 x CP2, on top of #171): pure trainer step time dropped from 141–156 s/step (Ulysses baseline, Run 18a) to 6.35 s/micro-batch with full positions + 32k row slots + AG/RS GDN (Run 18d; val pass@1 0.414 -> 0.700 over 70 steps) and 6.18 s/micro-batch mean / 128 s minimum trainer step with slot sorting (Run 18g). - **TODO before marking ready for review**: Re-run isolated per-option A/B throughput and log-prob parity validation on TPU v7x using the refactored `google-pytorch/torchtitan#216` + #170 + #172 stack.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #206 (regular-trainer per-layer compile with EP), which is stacked on #142 (fast dropless EP) and #6. This is #200 (Qwen3.5 context parallelism on TPU) rebased onto the EP stack, so expert parallelism and context parallelism can be enabled together. See #200 for the CP design, the CP-only results and the known issues; this PR only lists what differs.
What changed vs #200
mainat 2026-11-06, before the TPU experiment root was reorganized into subpackages. The CP code uses the old paths here:experiments/tpu/cp_attention.py,experiments/tpu/unflatten_shim.pyandexperiments/tpu/activation_checkpoint.py.TpuQwen35Model. [TPU][Qwen3.5] Expert parallelism (EP > 1) for Qwen3.5 MoE #6 already adds aTpuQwen35Model(EPupdate_from_configandparallelize). The CPpreprocess_inputs, which shards the inputs through the CP attention'scp_shard, is added to that class.parallelism.spmd_backenddefaults topartial_dtensor, and CP requiresspmd_types._qwen35_cpsets it, asqwen3/config_registry.py::_qwen3_cpdoes here._parallelize_text_qwen3_5_with_cpandpreprocess_inputsmirror this base's upstream branches onspmd_backend.qwen35_moe_35b_split_tokamax_ep_{allgather,ulysses}_cp2. These areqwen35_moe_35b_split_tokamax_ep_benchmark(EP=4; override with--parallelism.expert_parallel_degree) plus CP degree 2. They keep the EP recipe settings:by_layersplit, one layer per executable,tpu_selectivesave regions and SparseCore offload.test_train_distributedtrains the debug MoE with EP 2 + CP 2 under both CP strategies;test_cp_gated_deltachecks that the EP + CP recipes keep the EP settings.qwen35_moe_35b_ep_{allgather,ulysses}_cp2areqwen35_moe_35b_epfrom [TPU] Qwen3.5 EP on the regular trainer with per-layer torch.compile #206 (EP=4, Tokamax GatedDeltaNet) plus CP degree 2, on the regularTPUTrainer. They compile every block (--compile.enable --compile.backend=tpu) by default, because eager CP withdp_shard > 1is unreliable (feat(tpu): context parallelism for Qwen3.5 (head-parallel GatedDeltaNet + CP full attention) #200).test_cp_gated_deltachecks that they keep EP, use Tokamax and spmd_types, compile every block and convert every GatedDeltaNet. See "Regular trainer" below for the results and the known issue.How EP and CP compose
CP shards each sequence's tokens over the
cpmesh axis. The routed experts are token-wise, so EP dispatches each rank's local, CP-sharded tokens over theepaxis.epmust dividedp_shard * cp.epis the innermost axis of the expert mesh(dp_replicate, efsdp, ep), andcpthe innermost data axis of(batch, cp, tp): withcp=2, ep=2the EP groups are the CP groups, and withcp=2, ep=4one EP group spans two CP groups. Each token's routed-expert contribution is computed once, on the EP rank that owns the expert, and the expert gradients are summed overefsdp.Tests
torchtitan-container:2026-11-13):test_cp_gated_delta(incl. the EP + CP recipe checks),test_cp_attentionandtest_graph_splitpass.test_train_distributedEP 2 + CP 2 (all-gather and Ulysses, 4 gloo ranks) pass.test_qwen3_5_tpu,test_token_dispatcher,test_workarounds,test_cp_attentionandtest_graph_splitgive the same CPU results as the EP basec56f441fc. The only difference is the renamed graph-trainer rejection test; the failures are the TPU-onlyQwen35Testcases, in both.test_cp_gated_delta,test_cp_attention,test_graph_split,test_token_dispatcher,test_workarounds): 137 passed.test_train_minimal_distributed, SimpleFSDP, CP2 all-gather and Ulysses, eager and compiled): 5/5.test_train_distributed: 9/9. Covers CP2 (both strategies), FSDP2 + CP, MoE + CP with Splash, EP2, EP2 with the TPU MoE kernels, and EP2 + CP2 (both strategies).test_qwen3_5_tpu, including [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142's EP tests: 59/59.test_qwen3_5_graph_trainer_matches_eagerpasses.test_train_distributed3/3.torch.compilewith EP used to fail in Dynamo (_tpu_generation()inkernels/ragged_gather_tc.py). [TPU] Qwen3.5 EP on the regular trainer with per-layer torch.compile #206 fixes that: compiled EP2 + CP2 now trains on the regular trainer; see below.Results: Qwen3.5-35B-A3B, EP and EP + CP
Setup: the
TpuGraphTrainer35B benchmark recipes of this branch on TPU7x (NAP cluster).torchtitan-container:2026-11-13+ this branch.num_tokens_per_microbatch_per_dp_ranktocp x 65536.--splash_attention_kernel.no-sa_use_document_aware_block_skipping --splash_attention_kernel.no-sa_use_fused_bwd_kernel, as in feat(tpu): context parallelism for Qwen3.5 (head-parallel GatedDeltaNet + CP full attention) #200.qwen35_moe_35b_split_tokamax_ep_benchmarkwith--parallelism.expert_parallel_degree=2. EP + CP usesqwen35_moe_35b_split_tokamax_ep_{ulysses,allgather}_cp2. EP=1 and CP-only use the feat(tpu): context parallelism for Qwen3.5 (head-parallel GatedDeltaNet + CP full attention) #200 recipes on this branch.dp 16)dp 8 x cp 2)Other notes:
main(same loss and grad norm at every logged step). EP=2 alone reproduces [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142's 35B number (14.30%, 8.74 s, 23.9 GiB there).Regular trainer (per-layer compile) with EP + CP
Setup: Qwen3.5-35B-A3B on the regular
TPUTraineron tpu7x-2x2x2 (16 ranks).torch.compile(fullgraph=True), the Tokamax GatedDeltaNet with document resets, full activation checkpointing, and FP32 master weights.torchtitan-container:2026-11-13+ this branch (4534b94cf).qwen35_moe_35b+ Tokamax, compiled)qwen35_moe_35b_ep, #206)qwen35_moe_35b_ulysses_cp2+ Tokamax, compiled)qwen35_moe_35b_ep_ulysses_cp2)qwen35_moe_35b_ep_allgather_cp2)--parallelism.expert_parallel_degree=2throughout. CP runs set--training.num_tokens_per_microbatch_per_dp_rank=65536.Known issue: per-block-compiled CP is intermittently non-finite on this torch_tpu build.
SLICE_FAILURE_SW_INJECT_ERRORabort, one stopped at step 1 with a non-finite loss / grad norm, and one completed (the row above).--tpu_config.eager_mode=DEFER_NEVER_AND_LAUNCH_BLOCKINGdoes not remove it (2/2 failed).TpuGraphTrainerCP runs are deterministic. The failures hit configs without EP and without the code changed here. My guess is torch_tpu's per-block split compilation, which runs 59-117 partitions per block, but this is not confirmed.TpuGraphTrainerrecipes for CP. The regular-trainer EP + CP recipes work in the 35B runs above but should not be relied on for long runs until this is fixed.test_train_distributedtherefore has no compiled EP + CP debug cases.EP + CP loss at step 20: rounding sensitivity, not an EP + CP bug
At 8K, EP=2 + CP2 ends at 2.652 (Ulysses) and 2.644 (KV all-gather) at step 20, against 2.562 for CP2 alone on the same data. Per-step logs show that this is one excursion of a trajectory that already wanders around the CP-only one. Deltas are against CP2 alone:
Why it happens:
warmup_steps=200, whichlr_scheduler.buildclamps totraining.steps. A 20-step benchmark therefore ramps a randomly initialized model to the peak LR (3e-4) at step 20. Grad norms reach 10-17 in the first 10 steps.lr * sign(g), so rounding-level gradient differences flip the sign of near-zero elements. Top-k routing then turns small weight differences into discrete expert changes for near-tie tokens.ragged_gather_tc.py,total.astype(x.dtype)) before_EpReduceScattersums the two EP ranks. EP=1 sums all K routes in FP32 and rounds once._LocalReorderdocstring intoken_dispatcher.pyalready records that reordering equal-expert routes, a pure accumulation-order change, "produced an unacceptable training-trajectory shift".Evidence:
Exact on CPU. CPU fp32 run with the parity harness, 4 gloo ranks against a single device. EP2 + CP2's step-1 loss and gradients match the single-device run as closely as plain data parallelism does. Both leave the same residual: shared-expert and
lm_headgradients within 5e-5. After one AdamW step, every MoE configuration differs from the single-device run by 8e-4 to 9e-3 in loss, plain DP included. The dense model with CP2 matches to < 1e-5.Deterministic, and not the SPMD backend. Rerunning EP=2 + Ulysses CP2 reproduces every logged step exactly. EP=2 without CP under
spmd_types, the backend CP forces, is bit-identical to EP=2 underpartial_dtensor.No accumulating error. With
--lr_scheduler.total_steps=200the warmup is not clamped and LR stays <= 3e-5, while the loss still falls from 12.9 to 3.8. Under that schedule EP=2 + CP2 stays within 3.5e-3 of CP2 alone for 20 steps and ends 1e-4 apart (last row of the table).Growth matches the perturbation size. Over 20 steps, each perturbation grows by roughly the same factor:
No consistent direction. At 64K and 128K, EP=2 + CP2 ends below CP2 alone at step 10 (-0.065 and -0.037). EP=2 alone deviates from EP=1 by -0.022 at step 10 at 8K, then closes to +0.005 by step 20.
Equivalent variants cluster by family. Five EP=2 + CP2 rounding variants all end at 2.644-2.660 at step 20, with the same stall at step 18: Ulysses or KV all-gather, 4 or 2 token chunks, routing weights on the activation or in the combine. CP2 alone, the EP recipe at EP=1 and the kernel variants all end at 2.561-2.565. Within a family, runs differ by at most 2e-5 at step 1 and end within 0.016. Between families they differ by 5.5e-4 and end 0.09 apart. The separation grows with the initial perturbation.
The gap closes after the LR peak. A 40-step pair with
--lr_scheduler.warmup_steps=20has the same LR as the 20-step runs for steps 1-20 and reproduces them exactly. As the LR then decays, EP=2 + CP2 minus CP2 alone goes from +0.090 at step 20 to +0.019 at step 24, +0.005 at step 30 and +0.0065 at step 40 (1.90070 vs 1.89416).Code checked for EP + CP interactions:
cap = ep * T * K), so nothing is dropped.efsdp.global_valid_tokensmatches CP alone.Nothing EP + CP-specific was wrong. To compare EP + CP convergence against CP alone, use runs longer than the warmup, or set
--lr_scheduler.total_steps, rather than 20-step benchmarks.