Skip to content

Reshard KV pools between pipeline-parallel engines - #1064

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_988715760
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_988715760

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Reshard KV pools between pipeline-parallel engines

A pipeline-parallel engine splits its layers over its stages: each stage
registers only the pools of its own layers, every decode stage plans the
same request once, and a decode stage's plan names a single prefill stage.
Pool resharding assumed the opposite in four places:

  1. Source and destination pools paired by position and the two pool tables
    had to be equal, so planning failed with "Canonical pool manifest
    mismatch" as soon as a producer stage registered a layer subset.
  2. A pushed pool was named by the sender's own pool index, which the
    receiver resolved against its own table, so every stage but the first
    had its pushes rejected with "No transfer chunks found for block N".
  3. A request's block snapshot could be claimed by one planning attempt, so
    the first decode stage claimed it and the other stages were refused with
    "already claimed by another planning attempt" and timed out.
  4. A source was keyed by its position in the plan's source list instead of
    its transfer rank, so a plan naming one source ran on worker_0 and the
    receiver found no schedule for the sender's node id.

Changes:

  • Pool pairing: pools pair by tag. A source may register a subset of the
    destination's tags and a destination a subset of the source's; tags on
    both sides must agree on pool count and dtype. A sender's start-transfer
    request is rewritten into its own pool index space.
  • Wire pool index: StartTransferRequest carries wire_pool_indices (sender
    index -> destination index, the identity for equal pool tables), carried
    through the transfer-program reshard binding; a sender's push names the
    destination's block array.
  • Shared claims: planning attempts that name the same source unit set share
    one request claim; a different unit set is still refused; the claim is
    freed when the last attempt leaves. Dtype-tag mismatch errors name the
    plan's dtype list and the local pool table.
  • Rank keys: each source's schedule and each sender's worker are keyed by
    the source's registered transfer rank.

Validation: the OSS build passes all 7 reshard, session and transport
test targets (reshard_service_test 29 cases incl.
SubsetSourceManifestsPairPoolsByTag,
SourceTagsAbsentOnTheDestinationAreIgnored,
PipelinedDestinationStagesShareOneRequestClaim,
AbandonedStageLeavesTheSiblingClaimInPlace and
AbandoningTheLastStageFreesTheClaim; block_transport_test 30 cases incl.
PushNamesTheReceiversArrayByWireLayerIndex; transfer_program_reshard_test
incl. WirePoolIndexMapRoundTrips). End to end, PP8 -> DP8 and PP8 -> PP8
pairs on Qwen3.5-35B-A3B-FP8 complete every request and answer every
greedy probe whose KV crosses the reshard (vllm-torchtpu PR 871).

A pipeline-parallel engine splits its layers over its stages: each stage
registers only the pools of its own layers, every decode stage plans the
same request once, and a decode stage's plan names a single prefill stage.
Pool resharding assumed the opposite in four places:

1. Source and destination pools paired by position and the two pool tables
   had to be equal, so planning failed with "Canonical pool manifest
   mismatch" as soon as a producer stage registered a layer subset.
2. A pushed pool was named by the sender's own pool index, which the
   receiver resolved against its own table, so every stage but the first
   had its pushes rejected with "No transfer chunks found for block N".
3. A request's block snapshot could be claimed by one planning attempt, so
   the first decode stage claimed it and the other stages were refused with
   "already claimed by another planning attempt" and timed out.
4. A source was keyed by its position in the plan's source list instead of
   its transfer rank, so a plan naming one source ran on worker_0 and the
   receiver found no schedule for the sender's node id.

Changes:
- Pool pairing: pools pair by tag. A source may register a subset of the
  destination's tags and a destination a subset of the source's; tags on
  both sides must agree on pool count and dtype. A sender's start-transfer
  request is rewritten into its own pool index space.
- Wire pool index: StartTransferRequest carries wire_pool_indices (sender
  index -> destination index, the identity for equal pool tables), carried
  through the transfer-program reshard binding; a sender's push names the
  destination's block array.
- Shared claims: planning attempts that name the same source unit set share
  one request claim; a different unit set is still refused; the claim is
  freed when the last attempt leaves. Dtype-tag mismatch errors name the
  plan's dtype list and the local pool table.
- Rank keys: each source's schedule and each sender's worker are keyed by
  the source's registered transfer rank.

Validation: the OSS build passes all 7 reshard, session and transport
test targets (reshard_service_test 29 cases incl.
SubsetSourceManifestsPairPoolsByTag,
SourceTagsAbsentOnTheDestinationAreIgnored,
PipelinedDestinationStagesShareOneRequestClaim,
AbandonedStageLeavesTheSiblingClaimInPlace and
AbandoningTheLastStageFreesTheClaim; block_transport_test 30 cases incl.
PushNamesTheReceiversArrayByWireLayerIndex; transfer_program_reshard_test
incl. WirePoolIndexMapRoundTrips). End to end, PP8 -> DP8 and PP8 -> PP8
pairs on Qwen3.5-35B-A3B-FP8 complete every request and answer every
greedy probe whose KV crosses the reshard (vllm-torchtpu PR 871).
PiperOrigin-RevId: 988715760
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant