Skip to content

feat(tpu): Qwen3.5 CP speedups: cp_full_positions, per-batch GDN document resets, AG/RS GatedDeltaNet CP, fused GDN projections (stacked on #201) - #216

Draft
jialei777 wants to merge 1 commit into
feat/tpu-qwen35-cp-on-epfrom
feat/tpu-qwen35-fast-ulysses-cp
Draft

jialei777 wants to merge 1 commit into
feat/tpu-qwen35-cp-on-epfrom
feat/tpu-qwen35-fast-ulysses-cp

Conversation

@jialei777

@jialei777 jialei777 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #201 (Qwen3.5 CP on the EP stack). This PR moves the Qwen3.5 Ulysses CP speedups from verl-project/verl-hardware-plugin#172 into torchtitan, in a general form. With it, the plugin no longer has to monkeypatch TpuQwen35TransformerBlock, UlyssesCPInnerAttention, CPTpuInnerGatedDeltaNet or TpuGatedDeltaNet.

What changes

1. cp_full_positions: no per-layer all-gather of positions under CP

Before this PR, every CP attention layer (via _full_sequence_segment_ids / _full_sequence_segment_ids_all_gather) and every head-parallel GatedDeltaNet layer (cp_all_gather(positions)) all-gathered the sharded positions to rebuild the whole sequence's document boundaries. On 35B-A3B that is 40 all-gathers per forward, plus their recompute under AC.

  • TpuQwen35Model.forward takes an optional cp_full_positions: the unsharded, sequence-order positions. It passes them to every layer as full_positions.
  • UlyssesCPInnerAttention and KVAllGatherCPInnerAttention build segment_ids from full_positions locally, with no collective. KV all-gather numbers documents per row, like its cp_shard (the new KVAllGatherCPInnerAttention.full_segment_ids is shared by both paths). Ulysses numbers them globally, as before.
  • CPTpuInnerGatedDeltaNet uses full_positions directly: no all-gather, and no head-tail reorder of the positions.
  • TpuQwen35Model.preprocess_inputs (the torchtitan trainer) sets cp_full_positions automatically and gives it an SPMD layout (DP: V, CP: R). External trainers that call cp_shard themselves (verl) pass cp_full_positions=<positions before cp_shard>.
  • Without it, everything falls back to the existing paths. The tensor-attribute _cp_full_segment_ids path also stays.

This works with both CP strategies (Ulysses and KV all-gather / head-tail), unlike the plugin version, which was Ulysses-only.

2. gated_delta_document_resets: a per-batch document-reset override

TpuQwen35Model.forward(..., gated_delta_document_resets=False) overrides document_resets.enabled on every GatedDeltaNet for that batch. Use it when each [B, S] row (set_unflatten_seq_len) holds a single document, so the unsegmented kernels run. This is what the plugin's CP row-slot packing needs. TpuGatedDeltaNet.forward takes the same override as document_resets=. It is a Python bool, so it is part of the compiled program's key (two programs at most).

3. GatedDeltaNet CP strategy allgather_reducescatter

GatedDeltaNetContextParallelTransform(load_balancer=..., strategy="allgather_reducescatter", reduce_dtype="float32") turns every TpuGatedDeltaNet into a new CPTpuGatedDeltaNet. Instead of five all-to-alls (q/k/v/a/b in, output out) it does:

  1. All-gather the layer input x (T/cp → T tokens). The backward reduce-scatters its gradient in reduce_dtype.
  2. Project with this rank's rows of the six input projections (its 1/cp of the heads), then run the conv, the recurrence and the gated norm over whole sequences.
  3. Multiply by this rank's columns of out_proj, then reduce-scatter (sum, in reduce_dtype) back to T/cp.

Compared with the plugin's VERL_TPU_GDN_AG_RS_CP2, it generalizes in five ways:

  • It works for any cp that divides n_key_heads and n_value_heads (not only cp=2).
  • It works for either load balancer (head-tail is reordered around the local computation).
  • The reductions run in fp32 by default, which is configurable.
  • It calls self.norm instead of a copied RMSNorm fallback.
  • It reuses CPTpuInnerGatedDeltaNet.local_forward.

The default strategy="all_to_all" leaves configs unchanged.

New public collectives in cp_attention.py: cp_all_gather_with_grad (all-gather, with a reduce-scatter backward) and cp_reduce_scatter (reduce-scatter, with an all-gather backward).

4. TpuGatedDeltaNet.Config.fuse_input_projections

This runs the six bias-free input projections as one matmul over their concatenated weights. It works with and without CP, and the plugin's VERL_TPU_FUSE_GDN_PROJ becomes a config field. It is off by default. It rejects biases, and rejects non-plain nn.Linear projections (for example Float8) instead of silently bypassing them.

Tests (CPU, emulated CP groups, torch 2.15 CPU)

  • test_cp_gated_delta.py:
    • The whole-model parity test now covers {all_gather, ulysses} × {eager, traced} × {all_to_all, allgather_reducescatter} × {with/without cp_full_positions} × {default, gated_delta_document_resets=False}: 32 cases, cp=2, B=2, packed documents. Loss and every parameter gradient match a single device.
    • With cp_full_positions, the test fails if any layer all-gathers positions.
    • New: allgather_reducescatter compiles with fullgraph=True to graphs that contain only all-gather/reduce-scatter, one wait per collective, and no all-to-all (fake PG; both load balancers; fused and unfused).
    • New: fused projections match the unfused ones and reject biases. The document-reset override equals the config switch and changes the result on packed documents. Transform strategy conversion, idempotence and rejection of unknown strategies are covered. preprocess_inputs emits cp_full_positions.
  • test_cp_attention.py: the emulator now also patches functional collectives in eager mode (used by cp_reduce_scatter).
  • python -m pytest torchtitan/experiments/tpu/qwen3_5/tests/test_cp_gated_delta.py torchtitan/experiments/tpu/tests/test_cp_attention.py: 125 passed. The base (feat(tpu): Qwen3.5 context parallelism on top of fast dropless EP (stacked on #142) #201) has 87, all still passing.
  • python -m torchtitan.experiments.tpu.qwen3_5.tests.test_qwen3_5_tpu: 60 tests OK (5 skipped), same as the base.

Not yet validated on TPU

The plugin's TPU numbers (Qwen3.5-35B-A3B, 64k DeepSWE, v7x, FSDP8 × CP2) were measured with the plugin patches, not this code. A TPU A/B with this PR plus the slimmed plugin is still needed: step time, and log-prob parity on/off per feature.

Plugin follow-up

verl-project/verl-hardware-plugin#172 will drop its patch closure and its six env vars. Instead it will:

  • pass cp_full_positions / gated_delta_document_resets;
  • set the transform's strategy / reduce_dtype and fuse_input_projections from TPUEngineOptions fields;
  • keep only the verl-side scheduling (row-slot planning, micro-batch sorting, static log-prob length) as options.

…ment resets, all-gather/reduce-scatter GDN CP strategy, fused GDN input projections

- TpuQwen35Model.forward(cp_full_positions=...) threads the unsharded positions
  to every layer so CP attention and the head-parallel GatedDeltaNet build
  document boundaries without per-layer positions all-gathers (both CP
  strategies); preprocess_inputs sets it automatically.
- gated_delta_document_resets per-batch override (single-document rows).
- CPTpuGatedDeltaNet + GatedDeltaNetContextParallelTransform(strategy=
  'allgather_reducescatter'): AG(x) + head-sliced projections/GDN/norm +
  RS(out_proj) for any cp dividing the head counts and either load balancer.
- TpuGatedDeltaNet.Config.fuse_input_projections.
- cp_all_gather_with_grad / cp_reduce_scatter collectives.
- Tests: model parity over strategies x eager/traced x GDN strategy x
  cp_full_positions x document-reset override; fullgraph compile test for the
  new strategy; transform/projection/preprocess tests.
jialei777 added a commit to verl-project/verl-hardware-plugin that referenced this pull request Nov 20, 2026
… GDN projections & slot sorting via TPUEngineOptions (#172)

## Summary

Replaces the prototype Qwen3.5 monkeypatches and `VERL_TPU_*`
environment variables with clean, model-agnostic scheduling and
input-preparation support in `verl-hardware-plugin`, paired with
upstream model/CP capabilities in
[google-pytorch/torchtitan#216](google-pytorch/torchtitan#216)
and GDN CP strategy options in #170:

- **No torchtitan monkeypatches or `VERL_TPU_*` env vars**: Deletes
`_install_fast_cp_gdn_patches` and exposes five validated
`TPUEngineOptions` (`engine_kwargs.tpu.*`) fields:
- `cp_full_positions`: Passes pre-`cp_shard` unsharded 1-D `positions`
as `cp_full_positions` (`DP: V, CP: R, TP: R`) to `model.forward`,
eliminating per-layer `all_gather(positions)` collectives across both
Ulysses and KV all-gather CP (`google-pytorch/torchtitan#216`), while
preserving `cp_shard`'s `attention_masks` / load-balancer metadata.
Rejects multimodal and 3-D MRoPE inputs explicitly instead of flattening
positions.
- `gdn_auto_row_slot_len`: Dynamic per-micro-batch row-slot planner
(`gdn_slot_schedule.py`) for any model whose `forward()` accepts
`gated_delta_document_resets` (`google-pytorch/torchtitan#216`):
- Micro-batches whose single-document sequences fit
`gdn_auto_row_slot_len` rows are permuted into slots
(`permute_packed_for_row_slots`) and run unsegmented
(`set_unflatten_seq_len(slot_len)`,
`gated_delta_document_resets=False`), restoring 1-D fused `log_probs`
and `entropy` after optional CP gather (for both CP and non-CP).
- Single-document micro-batches (`n_seqs == 1`) run unsegmented at
`padded_len`.
- Multi-document micro-batches (verified by checking exact `position ==
0` indices against `cu_seqlens[:-1]`, catching continuation sequences
with interior resets) run segmented; Tokamax `packed_document_budget` is
sized over segmented micro-batches only.
- All SPMD ranks synchronize eligibility masks via `dist.all_reduce(...,
ReduceOp.BAND)` so every rank takes the same branch and compiled shape.
- `fuse_gdn_projections`: Walks `model_spec.model` before build and sets
`fuse_input_projections = True` (`google-pytorch/torchtitan#216`),
failing loudly if zero sub-configs declare `fuse_input_projections`.
- `sort_micro_batches_by_slot`: Orders micro-batches by `(slot_category,
-total_tokens, original_idx)` using the exact same
`classify_gdn_micro_batch` classifier as `plan_micro_batch_gdn_slots` so
identical slot/segmentation shapes execute back to back.
- `static_log_prob_micro_batch_len`: Overrides `static_micro_batch_len`
on forward-only (`old_log_probs` / `ref_log_prob`) passes.
- **Head-replicated AG/RS GatedDeltaNet CP**: Moved upstream into
`CPTpuGatedDeltaNet` (`strategy="allgather_reducescatter"`) in
[google-pytorch/torchtitan#216](google-pytorch/torchtitan#216)
for any `cp` degree and both contiguous and `headtail` load balancers,
configured via #170's `engine_kwargs.tpu.context_parallel_gdn_strategy:
"allgather_reducescatter"` and `context_parallel_gdn_reduce_dtype`.

## Dependencies

- Model-side kwargs (`cp_full_positions`, `gated_delta_document_resets`)
and `TpuGatedDeltaNet.Config.fuse_input_projections`:
[google-pytorch/torchtitan#216](google-pytorch/torchtitan#216)
(stacked on `google-pytorch/torchtitan#201`). Fails loudly at engine
initialization if any of these options are enabled against an older
torchtitan build.
- GatedDeltaNet `allgather_reducescatter` CP transform wiring: #170
(`context_parallel_gdn_strategy` / `context_parallel_gdn_reduce_dtype`).

## Verification

- **CPU unit & integration tests**:
- `pytest tests/accelerators/tpu/test_tpu_gdn_slot_schedule.py
tests/accelerators/tpu/test_tpu_options.py` (exact interior reset
detection including continuation sequences, shared classifier/sorter
category order, differentiated-entropy and 3-D MRoPE / multimodal
exclusions, `BAND` cross-rank slot negotiation, non-CP and 2-way CP
row-slot permutation + `cp_full_positions` round-trip log-prob/entropy
parity, config walker & model signature fail-loud checks).
- Full `tests/accelerators/tpu` suite, `ruff format --check`, `ruff
check`, `mypy`, and license checks.
- **TPU benchmark status**:
- Initial prototype measurements on Qwen3.5-35B-A3B DeepSWE RL (64k
context, TPU v7x, FSDP8 x CP2, on top of #171): pure trainer step time
dropped from 141–156 s/step (Ulysses baseline, Run 18a) to 6.35
s/micro-batch with full positions + 32k row slots + AG/RS GDN (Run 18d;
val pass@1 0.414 -> 0.700 over 70 steps) and 6.18 s/micro-batch mean /
128 s minimum trainer step with slot sorting (Run 18g).
- **TODO before marking ready for review**: Re-run isolated per-option
A/B throughput and log-prob parity validation on TPU v7x using the
refactored `google-pytorch/torchtitan#216` + #170 + #172 stack.
jialei777 added a commit that referenced this pull request Nov 21, 2026
…ment resets, all-gather/reduce-scatter GDN CP strategy, fused GDN input projections

- TpuQwen35Model.forward(cp_full_positions=...) threads the unsharded positions
  to every layer so CP attention and the head-parallel GatedDeltaNet build
  document boundaries without per-layer positions all-gathers (both CP
  strategies); preprocess_inputs sets it automatically.
- gated_delta_document_resets per-batch override (single-document rows).
- CPTpuGatedDeltaNet + GatedDeltaNetContextParallelTransform(strategy=
  'allgather_reducescatter'): AG(x) + head-sliced projections/GDN/norm +
  RS(out_proj) for any cp dividing the head counts and either load balancer.
- TpuGatedDeltaNet.Config.fuse_input_projections.
- cp_all_gather_with_grad / cp_reduce_scatter collectives.
- Tests: model parity over strategies x eager/traced x GDN strategy x
  cp_full_positions x document-reset override; fullgraph compile test for the
  new strategy; transform/projection/preprocess tests.

Ported from #216 (stacked on the EP branch) onto main: main's module layout
(distributed/cp_attention.py, components/unflatten_shim.py), main's
TpuQwen35Model (EP config) and the #200 review fixes (layout checks also on
the all-gather / reduce-scatter path; revert_tpu_context_parallel also
reverts CPTpuGatedDeltaNet).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant