Reshard KV pools between pipeline-parallel engines - #1064
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
A pipeline-parallel engine splits its layers over its stages: each stage registers only the pools of its own layers, every decode stage plans the same request once, and a decode stage's plan names a single prefill stage. Pool resharding assumed the opposite in four places: 1. Source and destination pools paired by position and the two pool tables had to be equal, so planning failed with "Canonical pool manifest mismatch" as soon as a producer stage registered a layer subset. 2. A pushed pool was named by the sender's own pool index, which the receiver resolved against its own table, so every stage but the first had its pushes rejected with "No transfer chunks found for block N". 3. A request's block snapshot could be claimed by one planning attempt, so the first decode stage claimed it and the other stages were refused with "already claimed by another planning attempt" and timed out. 4. A source was keyed by its position in the plan's source list instead of its transfer rank, so a plan naming one source ran on worker_0 and the receiver found no schedule for the sender's node id. Changes: - Pool pairing: pools pair by tag. A source may register a subset of the destination's tags and a destination a subset of the source's; tags on both sides must agree on pool count and dtype. A sender's start-transfer request is rewritten into its own pool index space. - Wire pool index: StartTransferRequest carries wire_pool_indices (sender index -> destination index, the identity for equal pool tables), carried through the transfer-program reshard binding; a sender's push names the destination's block array. - Shared claims: planning attempts that name the same source unit set share one request claim; a different unit set is still refused; the claim is freed when the last attempt leaves. Dtype-tag mismatch errors name the plan's dtype list and the local pool table. - Rank keys: each source's schedule and each sender's worker are keyed by the source's registered transfer rank. Validation: the OSS build passes all 7 reshard, session and transport test targets (reshard_service_test 29 cases incl. SubsetSourceManifestsPairPoolsByTag, SourceTagsAbsentOnTheDestinationAreIgnored, PipelinedDestinationStagesShareOneRequestClaim, AbandonedStageLeavesTheSiblingClaimInPlace and AbandoningTheLastStageFreesTheClaim; block_transport_test 30 cases incl. PushNamesTheReceiversArrayByWireLayerIndex; transfer_program_reshard_test incl. WirePoolIndexMapRoundTrips). End to end, PP8 -> DP8 and PP8 -> PP8 pairs on Qwen3.5-35B-A3B-FP8 complete every request and answer every greedy probe whose KV crosses the reshard (vllm-torchtpu PR 871). PiperOrigin-RevId: 988715760
copybara-service
Bot
force-pushed
the
test_988715760
branch
from
September 26, 2026 07:54
78f2c08 to
0976a5f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reshard KV pools between pipeline-parallel engines
A pipeline-parallel engine splits its layers over its stages: each stage
registers only the pools of its own layers, every decode stage plans the
same request once, and a decode stage's plan names a single prefill stage.
Pool resharding assumed the opposite in four places:
had to be equal, so planning failed with "Canonical pool manifest
mismatch" as soon as a producer stage registered a layer subset.
receiver resolved against its own table, so every stage but the first
had its pushes rejected with "No transfer chunks found for block N".
the first decode stage claimed it and the other stages were refused with
"already claimed by another planning attempt" and timed out.
its transfer rank, so a plan naming one source ran on worker_0 and the
receiver found no schedule for the sender's node id.
Changes:
destination's tags and a destination a subset of the source's; tags on
both sides must agree on pool count and dtype. A sender's start-transfer
request is rewritten into its own pool index space.
index -> destination index, the identity for equal pool tables), carried
through the transfer-program reshard binding; a sender's push names the
destination's block array.
one request claim; a different unit set is still refused; the claim is
freed when the last attempt leaves. Dtype-tag mismatch errors name the
plan's dtype list and the local pool table.
the source's registered transfer rank.
Validation: the OSS build passes all 7 reshard, session and transport
test targets (reshard_service_test 29 cases incl.
SubsetSourceManifestsPairPoolsByTag,
SourceTagsAbsentOnTheDestinationAreIgnored,
PipelinedDestinationStagesShareOneRequestClaim,
AbandonedStageLeavesTheSiblingClaimInPlace and
AbandoningTheLastStageFreesTheClaim; block_transport_test 30 cases incl.
PushNamesTheReceiversArrayByWireLayerIndex; transfer_program_reshard_test
incl. WirePoolIndexMapRoundTrips). End to end, PP8 -> DP8 and PP8 -> PP8
pairs on Qwen3.5-35B-A3B-FP8 complete every request and answer every
greedy probe whose KV crosses the reshard (vllm-torchtpu PR 871).