Repository navigation
feat(tpu): Qwen3.5 CP speedups: cp_full_positions, per-batch GDN document resets, AG/RS GatedDeltaNet CP, fused GDN projections (stacked on #201) - #216
Draft
jialei777 wants to merge 1 commit into
Conversation
…ment resets, all-gather/reduce-scatter GDN CP strategy, fused GDN input projections - TpuQwen35Model.forward(cp_full_positions=...) threads the unsharded positions to every layer so CP attention and the head-parallel GatedDeltaNet build document boundaries without per-layer positions all-gathers (both CP strategies); preprocess_inputs sets it automatically. - gated_delta_document_resets per-batch override (single-document rows). - CPTpuGatedDeltaNet + GatedDeltaNetContextParallelTransform(strategy= 'allgather_reducescatter'): AG(x) + head-sliced projections/GDN/norm + RS(out_proj) for any cp dividing the head counts and either load balancer. - TpuGatedDeltaNet.Config.fuse_input_projections. - cp_all_gather_with_grad / cp_reduce_scatter collectives. - Tests: model parity over strategies x eager/traced x GDN strategy x cp_full_positions x document-reset override; fullgraph compile test for the new strategy; transform/projection/preprocess tests.
jialei777
added a commit
to verl-project/verl-hardware-plugin
that referenced
this pull request
Nov 20, 2026
… GDN projections & slot sorting via TPUEngineOptions (#172) ## Summary Replaces the prototype Qwen3.5 monkeypatches and `VERL_TPU_*` environment variables with clean, model-agnostic scheduling and input-preparation support in `verl-hardware-plugin`, paired with upstream model/CP capabilities in [google-pytorch/torchtitan#216](google-pytorch/torchtitan#216) and GDN CP strategy options in #170: - **No torchtitan monkeypatches or `VERL_TPU_*` env vars**: Deletes `_install_fast_cp_gdn_patches` and exposes five validated `TPUEngineOptions` (`engine_kwargs.tpu.*`) fields: - `cp_full_positions`: Passes pre-`cp_shard` unsharded 1-D `positions` as `cp_full_positions` (`DP: V, CP: R, TP: R`) to `model.forward`, eliminating per-layer `all_gather(positions)` collectives across both Ulysses and KV all-gather CP (`google-pytorch/torchtitan#216`), while preserving `cp_shard`'s `attention_masks` / load-balancer metadata. Rejects multimodal and 3-D MRoPE inputs explicitly instead of flattening positions. - `gdn_auto_row_slot_len`: Dynamic per-micro-batch row-slot planner (`gdn_slot_schedule.py`) for any model whose `forward()` accepts `gated_delta_document_resets` (`google-pytorch/torchtitan#216`): - Micro-batches whose single-document sequences fit `gdn_auto_row_slot_len` rows are permuted into slots (`permute_packed_for_row_slots`) and run unsegmented (`set_unflatten_seq_len(slot_len)`, `gated_delta_document_resets=False`), restoring 1-D fused `log_probs` and `entropy` after optional CP gather (for both CP and non-CP). - Single-document micro-batches (`n_seqs == 1`) run unsegmented at `padded_len`. - Multi-document micro-batches (verified by checking exact `position == 0` indices against `cu_seqlens[:-1]`, catching continuation sequences with interior resets) run segmented; Tokamax `packed_document_budget` is sized over segmented micro-batches only. - All SPMD ranks synchronize eligibility masks via `dist.all_reduce(..., ReduceOp.BAND)` so every rank takes the same branch and compiled shape. - `fuse_gdn_projections`: Walks `model_spec.model` before build and sets `fuse_input_projections = True` (`google-pytorch/torchtitan#216`), failing loudly if zero sub-configs declare `fuse_input_projections`. - `sort_micro_batches_by_slot`: Orders micro-batches by `(slot_category, -total_tokens, original_idx)` using the exact same `classify_gdn_micro_batch` classifier as `plan_micro_batch_gdn_slots` so identical slot/segmentation shapes execute back to back. - `static_log_prob_micro_batch_len`: Overrides `static_micro_batch_len` on forward-only (`old_log_probs` / `ref_log_prob`) passes. - **Head-replicated AG/RS GatedDeltaNet CP**: Moved upstream into `CPTpuGatedDeltaNet` (`strategy="allgather_reducescatter"`) in [google-pytorch/torchtitan#216](google-pytorch/torchtitan#216) for any `cp` degree and both contiguous and `headtail` load balancers, configured via #170's `engine_kwargs.tpu.context_parallel_gdn_strategy: "allgather_reducescatter"` and `context_parallel_gdn_reduce_dtype`. ## Dependencies - Model-side kwargs (`cp_full_positions`, `gated_delta_document_resets`) and `TpuGatedDeltaNet.Config.fuse_input_projections`: [google-pytorch/torchtitan#216](google-pytorch/torchtitan#216) (stacked on `google-pytorch/torchtitan#201`). Fails loudly at engine initialization if any of these options are enabled against an older torchtitan build. - GatedDeltaNet `allgather_reducescatter` CP transform wiring: #170 (`context_parallel_gdn_strategy` / `context_parallel_gdn_reduce_dtype`). ## Verification - **CPU unit & integration tests**: - `pytest tests/accelerators/tpu/test_tpu_gdn_slot_schedule.py tests/accelerators/tpu/test_tpu_options.py` (exact interior reset detection including continuation sequences, shared classifier/sorter category order, differentiated-entropy and 3-D MRoPE / multimodal exclusions, `BAND` cross-rank slot negotiation, non-CP and 2-way CP row-slot permutation + `cp_full_positions` round-trip log-prob/entropy parity, config walker & model signature fail-loud checks). - Full `tests/accelerators/tpu` suite, `ruff format --check`, `ruff check`, `mypy`, and license checks. - **TPU benchmark status**: - Initial prototype measurements on Qwen3.5-35B-A3B DeepSWE RL (64k context, TPU v7x, FSDP8 x CP2, on top of #171): pure trainer step time dropped from 141–156 s/step (Ulysses baseline, Run 18a) to 6.35 s/micro-batch with full positions + 32k row slots + AG/RS GDN (Run 18d; val pass@1 0.414 -> 0.700 over 70 steps) and 6.18 s/micro-batch mean / 128 s minimum trainer step with slot sorting (Run 18g). - **TODO before marking ready for review**: Re-run isolated per-option A/B throughput and log-prob parity validation on TPU v7x using the refactored `google-pytorch/torchtitan#216` + #170 + #172 stack.
jialei777
added a commit
that referenced
this pull request
Nov 21, 2026
…ment resets, all-gather/reduce-scatter GDN CP strategy, fused GDN input projections - TpuQwen35Model.forward(cp_full_positions=...) threads the unsharded positions to every layer so CP attention and the head-parallel GatedDeltaNet build document boundaries without per-layer positions all-gathers (both CP strategies); preprocess_inputs sets it automatically. - gated_delta_document_resets per-batch override (single-document rows). - CPTpuGatedDeltaNet + GatedDeltaNetContextParallelTransform(strategy= 'allgather_reducescatter'): AG(x) + head-sliced projections/GDN/norm + RS(out_proj) for any cp dividing the head counts and either load balancer. - TpuGatedDeltaNet.Config.fuse_input_projections. - cp_all_gather_with_grad / cp_reduce_scatter collectives. - Tests: model parity over strategies x eager/traced x GDN strategy x cp_full_positions x document-reset override; fullgraph compile test for the new strategy; transform/projection/preprocess tests. Ported from #216 (stacked on the EP branch) onto main: main's module layout (distributed/cp_attention.py, components/unflatten_shim.py), main's TpuQwen35Model (EP config) and the #200 review fixes (layout checks also on the all-gather / reduce-scatter path; revert_tpu_context_parallel also reverts CPTpuGatedDeltaNet).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #201 (Qwen3.5 CP on the EP stack). This PR moves the Qwen3.5 Ulysses CP speedups from verl-project/verl-hardware-plugin#172 into torchtitan, in a general form. With it, the plugin no longer has to monkeypatch
TpuQwen35TransformerBlock,UlyssesCPInnerAttention,CPTpuInnerGatedDeltaNetorTpuGatedDeltaNet.What changes
1.
cp_full_positions: no per-layer all-gather ofpositionsunder CPBefore this PR, every CP attention layer (via
_full_sequence_segment_ids/_full_sequence_segment_ids_all_gather) and every head-parallel GatedDeltaNet layer (cp_all_gather(positions)) all-gathered the shardedpositionsto rebuild the whole sequence's document boundaries. On 35B-A3B that is 40 all-gathers per forward, plus their recompute under AC.TpuQwen35Model.forwardtakes an optionalcp_full_positions: the unsharded, sequence-order positions. It passes them to every layer asfull_positions.UlyssesCPInnerAttentionandKVAllGatherCPInnerAttentionbuildsegment_idsfromfull_positionslocally, with no collective. KV all-gather numbers documents per row, like itscp_shard(the newKVAllGatherCPInnerAttention.full_segment_idsis shared by both paths). Ulysses numbers them globally, as before.CPTpuInnerGatedDeltaNetusesfull_positionsdirectly: no all-gather, and no head-tail reorder of the positions.TpuQwen35Model.preprocess_inputs(the torchtitan trainer) setscp_full_positionsautomatically and gives it an SPMD layout (DP: V, CP: R). External trainers that callcp_shardthemselves (verl) passcp_full_positions=<positions before cp_shard>._cp_full_segment_idspath also stays.This works with both CP strategies (Ulysses and KV all-gather / head-tail), unlike the plugin version, which was Ulysses-only.
2.
gated_delta_document_resets: a per-batch document-reset overrideTpuQwen35Model.forward(..., gated_delta_document_resets=False)overridesdocument_resets.enabledon every GatedDeltaNet for that batch. Use it when each[B, S]row (set_unflatten_seq_len) holds a single document, so the unsegmented kernels run. This is what the plugin's CP row-slot packing needs.TpuGatedDeltaNet.forwardtakes the same override asdocument_resets=. It is a Python bool, so it is part of the compiled program's key (two programs at most).3. GatedDeltaNet CP strategy
allgather_reducescatterGatedDeltaNetContextParallelTransform(load_balancer=..., strategy="allgather_reducescatter", reduce_dtype="float32")turns everyTpuGatedDeltaNetinto a newCPTpuGatedDeltaNet. Instead of five all-to-alls (q/k/v/a/b in, output out) it does:x(T/cp → Ttokens). The backward reduce-scatters its gradient inreduce_dtype.1/cpof the heads), then run the conv, the recurrence and the gated norm over whole sequences.out_proj, then reduce-scatter (sum, inreduce_dtype) back toT/cp.Compared with the plugin's
VERL_TPU_GDN_AG_RS_CP2, it generalizes in five ways:cpthat dividesn_key_headsandn_value_heads(not only cp=2).self.norminstead of a copied RMSNorm fallback.CPTpuInnerGatedDeltaNet.local_forward.The default
strategy="all_to_all"leaves configs unchanged.New public collectives in
cp_attention.py:cp_all_gather_with_grad(all-gather, with a reduce-scatter backward) andcp_reduce_scatter(reduce-scatter, with an all-gather backward).4.
TpuGatedDeltaNet.Config.fuse_input_projectionsThis runs the six bias-free input projections as one matmul over their concatenated weights. It works with and without CP, and the plugin's
VERL_TPU_FUSE_GDN_PROJbecomes a config field. It is off by default. It rejects biases, and rejects non-plainnn.Linearprojections (for example Float8) instead of silently bypassing them.Tests (CPU, emulated CP groups, torch 2.15 CPU)
test_cp_gated_delta.py:cp_full_positions} × {default,gated_delta_document_resets=False}: 32 cases, cp=2, B=2, packed documents. Loss and every parameter gradient match a single device.cp_full_positions, the test fails if any layer all-gathers positions.allgather_reducescattercompiles withfullgraph=Trueto graphs that contain only all-gather/reduce-scatter, one wait per collective, and no all-to-all (fake PG; both load balancers; fused and unfused).strategyconversion, idempotence and rejection of unknown strategies are covered.preprocess_inputsemitscp_full_positions.test_cp_attention.py: the emulator now also patches functional collectives in eager mode (used bycp_reduce_scatter).python -m pytest torchtitan/experiments/tpu/qwen3_5/tests/test_cp_gated_delta.py torchtitan/experiments/tpu/tests/test_cp_attention.py: 125 passed. The base (feat(tpu): Qwen3.5 context parallelism on top of fast dropless EP (stacked on #142) #201) has 87, all still passing.python -m torchtitan.experiments.tpu.qwen3_5.tests.test_qwen3_5_tpu: 60 tests OK (5 skipped), same as the base.Not yet validated on TPU
The plugin's TPU numbers (Qwen3.5-35B-A3B, 64k DeepSWE, v7x, FSDP8 × CP2) were measured with the plugin patches, not this code. A TPU A/B with this PR plus the slimmed plugin is still needed: step time, and log-prob parity on/off per feature.
Plugin follow-up
verl-project/verl-hardware-plugin#172 will drop its patch closure and its six env vars. Instead it will:
cp_full_positions/gated_delta_document_resets;strategy/reduce_dtypeandfuse_input_projectionsfromTPUEngineOptionsfields;