[Feat]: first impliment for async GPU connector - #239
Draft
specture724 wants to merge 18 commits into
Draft
Conversation
specture724
requested review from
hsliuustc0106 and
jiangkuaixue123
as code owners
August 13, 2026 06:21
specture724
force-pushed
the
AsyncGPU
branch
2 times, most recently
from
August 13, 2026 06:38
21aa48b to
107a002
Compare
specture724
marked this pull request as draft
August 13, 2026 08:31
hsliuustc0106
requested changes
Aug 15, 2026
hsliuustc0106
left a comment
hsliuustc0106
left a comment
Collaborator
There was a problem hiding this comment.
Requesting changes based on review of head 906ee58. The new connector has blocking correctness and execution-contract issues in FP16 shared scaling, quantized expert execution, ubatch slot ownership, expert placement, runtime validation, symmetric allocation setup, and the supplied launch recipes. CPU CI and focused unit tests are green, but the PR still records no production GPU launch result. Please also rebase onto current main while retaining the newer connector-lifecycle and async-CAM deadlock fixes.
| ) | ||
|
|
||
| num_local_experts = counts.numel() | ||
| if num_rows == 0: |
Collaborator
There was a problem hiding this comment.
[P1] Preserve shared scaling on empty routes. When num_rows == 0, this returns before the common FP16 scaling below. If this rank owns shared tokens and routed_scaling_factor != 1, its contribution is too large; with factor 2, the path returns 3.0 instead of 1.5. Merge the empty case into the common post-processing path or apply the same scaling before returning.
| device=hidden_states.device, | ||
| ) | ||
|
|
||
| routed_output = fused_experts( |
Collaborator
There was a problem hiding this comment.
[P1] Preserve vLLM's quantization contract. Calling fused_experts directly with raw weights and no quant_config selects the unquantized path: FP8 scales are ignored, while packed quantized formats can fail or produce incorrect output. Execute through RoutedExperts.quant_method, or reject quantized models during startup.
| f"got {tuple(topk_ids.shape)}", | ||
| ) | ||
|
|
||
| stage_idx = metadata.stage_idx |
Collaborator
There was a problem hiding this comment.
[P1] Prevent cross-stage slot overwrite. _free_rings is keyed by stage, but every stage's pool starts at physical ring 0. Native two-stage ubatching can yield after stage 0 sends, allowing stage 1 to overwrite the same slot before stage 0 receives; the receive path also cannot demultiplex that later response. Use a global ring allocator plus response demux, or reject native GPU ubatching.
| # single-token decode hits, since both the routed segment and the | ||
| # round-robin shared slice can come out empty for one rank. | ||
| expected_ffn = list(range(self.ffn_size)) | ||
| for ffn_rank in range(self.ffn_size): |
Collaborator
There was a problem hiding this comment.
[P1] Route using actual expert ownership. This fixed contiguous-block calculation is incompatible with disabled expert parallelism and round-robin placement, neither of which is rejected. The FFN helper then renumbers rows to local expert IDs and can execute the wrong weights. Derive destinations from vLLM's expert map, or validate enabled linear EP and a matching topology.
| f"{SUPPORTED_AFD_CONNECTORS!r}, got {config.connector!r}", | ||
| ) | ||
| if config.async_dp and config.connector != AFD_ASYNC_CONNECTOR: | ||
| if config.async_dp and config.connector not in AFD_ASYNC_CONNECTORS: |
Collaborator
There was a problem hiding this comment.
[P1] Validate the async-GPU execution contract. This only checks that async=true uses an async connector; it does not require GpuAsyncAFDConnector to use async mode. Configurations without async mode, Attention-side gating, eager execution, or with native DBO are accepted and later hang, overwrite slots, or fail for missing top-k payloads. Add GPU-specific validation before either role initializes the connector.
| self.total_bytes = self._flag_bytes + self.num_flags * layout.slot_bytes | ||
|
|
||
| nvshmem_rt.init(group, rank, world_size) | ||
| self._base = nvshmem_rt.malloc(self.total_bytes) |
Collaborator
There was a problem hiding this comment.
[P1] Verify symmetric layouts before allocation. NVSHMEM requires matching allocation sizes across PEs, but total_bytes is independently derived from each role's scheduler, dtype, model, and connector configuration. Separate role settings can therefore hang allocation or create invalid peer views. Collectively compare a layout fingerprint before calling nvshmem_malloc.
| --gpu-memory-utilization "$GPU_MEM_UTIL" \ | ||
| --enforce-eager \ | ||
| --host 127.0.0.1 \ | ||
| --port "$API_PORT" \ |
Collaborator
There was a problem hiding this comment.
[P1] Use a separate FFN API port. The Attention process already binds API_PORT, and both commands request one API server without port reuse, so the second frontend fails with EADDRINUSE. Introduce distinct Attention and FFN ports here and in the 2A2F recipe; health and completion requests should target Attention.
specture724
changed the title
Signed-off-by: specture724 <specture724@gmail.com>
Signed-off-by: specture724 <specture724@gmail.com>
… stream Two transport-path costs on the async GPU connector, both invisible to the numerics. send_attn_output materialized each peer's routed and shared rows with index_select and then copied that staging tensor into the peer's window, so every payload byte was written and re-read locally before it went on the wire. write_slot now takes the row index instead and gathers straight into the destination view. At 2A2F that is ~650 MiB of local bandwidth per forward. SymmWindow.poll and read_header ran their D2H on the compute stream, so a rank could not even look at a flag until its own previous kernel had retired -- the receive was serialized against exactly the compute it exists to overlap with. They now run on a dedicated poll stream. This needs no ordering against local work: flags and headers are written by a peer, never by anything this rank queued. Verified on 4x L20X, DeepSeek-V2-Lite: - tests/unit: 619 passed - async_gpu_window_roundtrip, async_gpu_connector_e2e: pass - async_gpu_moe_equivalence: max rel 4.5367e-03, unchanged - 2A2F greedy probe byte-for-byte identical to the same stack before this change, so the remaining async/sync divergence is the pre-existing bf16 reduction-order difference Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
Each FFN work item forced four full pipeline drains before it could run, which in eager mode also cost the host its run-ahead: - the per-expert group list was rebuilt with a pageable H2D from the decoded header. The counts already sit in device memory as the header's trailing words, so SymmWindow.local_expert_counts hands the grouped GEMM a view of them and the copy disappears. - send_ffn_output echoed that group list back through .cpu().tolist(), reading back numbers the receive had decoded on the host thirty lines earlier. They are kept on GpuAsyncTransferState instead. - the gate validated group_list with int(counts.sum()). - repeat_interleave sized its output by reading the counts back to the host. Passing output_size does both jobs at once: no readback, and it raises if the counts disagree with the row count, which is what the removed check tested. The header's own consistency is now checked where it is free, on the decoded host values in recv_attn_output. What is left on the layer path is the counts D2H in send_attn_output, which is intrinsic to slicing the per-rank segments on the host; the comment there says what removes it. Verified on L20X, DeepSeek-V2-Lite: tests/unit 619 passed, async_gpu_connector_e2e passes, async_gpu_moe_equivalence unchanged at max rel 4.5367e-03. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
Dispatch sent one row per (token, topk slot), so a token whose topk landed several times on the same FFN rank crossed the wire several times. With DeepSeek-V2-Lite's topk=6 over 2 FFN ranks that was measured at 1536 rows per (A,F) pair for a 512-token batch against 512 distinct tokens: 3x the traffic, for rows the receiver could rebuild locally. Dispatch now carries the distinct tokens plus, per partial, a 4-byte row index and its topk weight. The FFN side expands the rows back with a local gather before the grouped GEMM, then weights and sums each token's partials in float32 before replying, so the return trip carries the same distinct rows. Combine is left a plain scatter-add; combine_scatter and the route table are gone from both the wire and the code. Measured on 2A2F, 512-token batches: 512 rows per pair per direction instead of 1536. Against the sync connector's 512, the routed traffic goes from 3.0x to 1.0x, and 1.5x with the shared-expert rows counted. The window shrinks with it. Payload rows are bounded by the batch no matter how skewed the gate is, so the payload is sized by token_cap and only the 4-byte index arrays carry the every-partial-to-one-rank worst case. A 2A2F slot goes from 14.0 MiB to 4.0 MiB, capacity no longer grows with ffn_size**2, and routed_cap_multiplier -- which had no safe value above ffn_size=2 -- is deleted rather than retuned. Reduction order changes: a token's partials are now summed in float32 before the single narrowing to the payload dtype, where each partial used to be narrowed separately and summed on the Attention side. That is one rounding instead of many, so greedy output shifts slightly while getting marginally more accurate. Verified on 4x L20X, DeepSeek-V2-Lite: tests/unit 619 passed, async_gpu_window_roundtrip and async_gpu_connector_e2e pass on the new format, async_gpu_moe_equivalence at max rel 4.5367e-03, and a 2A2F server answers the greedy probe sensibly, matching the sync connector on 3 of 5 prompts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
Both roles spun on poll() with their own copy of the loop. One helper now owns it, and its docstring carries the measurement that says to leave it hot. Profiling 2A2F under load showed the poll costing more GPU copy-engine time than the expert GEMM: ~25k D2H polls a second per rank, 42.8 ms of Memcpy DtoH against 1.8 ms of fused_moe_kernel in a 1 s window on an FFN rank, which sits 95.4% idle. Backing off looked like the fix. It is not: every layer is a serialized A->F->A round trip, so detection latency is paid twice per layer on the critical path, and on the FFN side it also delays picking up the next rank's dispatch. Sleeping between attempts (4 hot tries, then 50us doubling to 1 ms) made mean TTFT worse at every rate -- 346 -> 431 ms at 32 rps, 1869 -> 3906 ms at 64 rps -- so the backoff is not here, only the note saying why. No behaviour change: reverting to the hot spin reproduces the baseline sweep (335/445/1899 ms at 32/48/64 rps against 346/425/1869 before the experiment). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
The two-stage schedule was written for CAM and never ran on CUDA: the switch existed, the model-side schedule was already platform-neutral, but nothing on the GPU side planned the stages. It does now, and four things had to be fixed before a request survived the path. Stage planning reuses plan_async_moe_stages and vLLM's own per-ubatch metadata build, so the split is one extra _build_attention_metadata call over the same inputs. Batches with fewer than two requests, dummy runs and graph capture fall back to the unstaged schedule -- a synthetic batch has no peer waiting for its stages. The four fixes: - vLLM sizes the per-ubatch metadata builder pool from its own ubatching switch, which AFD leaves off, so only one builder existed and building the second stage asserted. initialize_metadata_builders now asks for one per stage, plus one for the full batch: the dense prefix keeps running whole, so its metadata is live at the same time as both stages' and cannot share a builder with them. Sharing one showed up as an illegal memory access inside attention. - Each stage was handed the same ring list, so both stages of a layer claimed the same window slot and the second dispatch overwrote the first's payload and flag. Rings are now partitioned across stages. This was a latent bug in the connector, not in the schedule. - The stage forward context kept the full batch's slot_mapping, so MLA wrote each stage's rows into the whole batch's KV-cache slots. - Restoring the stage outputs splits a concatenation along the last dimension, and CUDA's fused_add_rms_norm aborts on the resulting non-contiguous views. Verified on 4x L20X, DeepSeek-V2-Lite 2A2F: tests/unit 619 passed, and the greedy probe is character-for-character identical to the unstaged schedule on all five prompts. It is slower, so it stays off by default and lives in its own recipe. Mean TTFT against the unstaged schedule: 144 vs 144 ms at 8 rps, 224 vs 185 at 16, 546 vs 346 at 32, 798 vs 426 at 48, no failures at any rate. Halving the batch halves the work per stage but doubles the per-layer fixed cost -- two dispatches, two waits and two host synchronizes instead of one each -- and at these batch sizes that costs more than the overlap returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
A profile of a 6A2F FFN rank under load put the scatter machinery at 2.2 s of a 6.6 s busy window against 2.7 s for fused_moe_kernel itself: the reduce from partials back to one row per shipped token widened the expert output to float32, scaled it, and scattered it, which is three passes over [partials, hidden] with the last two moving twice the bytes they need to. The Attention-side combine did the same thing, widening every arriving block and narrowing the result on the way out; its scatter and gather together cost 60 ms against FlashAttention's 24 ms in the same window. Both now stay in the payload dtype, so the reduce is a multiply and a scatter. What float32 was protecting is a sum of at most topk terms on the FFN side and one term per FFN rank plus the shared one on the Attention side, and the result crosses the wire narrowed either way. Separately, the gate scaled its output by routed_scaling_factor in a pass over [num_partials, hidden]. fused_experts multiplies every row by its topk weight in an epilogue it already runs, so the factor goes in there instead and the pass disappears. The topk weighting itself deliberately stays in the connector: an earlier version of this change handed the per-partial weights to the gate and dropped the multiply from the connector, which left send_ffn_output silently depending on its caller having applied them -- the end-to-end test's own FFN loop had not, and the failure mode was a wrong answer rather than an error. Numerics: the equivalence test moves from 4.54e-3 to 6.60e-3 max relative error, about one bf16 ulp and still an order inside its tolerance. Verified on L20X: tests/unit 619 passed, async_gpu_connector_e2e passes over the wire. The end-to-end effect is not measured yet -- every GPU on the box is busy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
Combine polled for its replies: copy the flag word down, look at it, try again. That cost a synchronize per attempt, but the real damage was that it left the GPU with nothing queued behind the wait. A profile of a 2A2F attention rank found the host running 2.3us ahead of the GPU at the median, with 76.9% of kernel launches executing within 5us of being issued -- the device was idle waiting to be fed one kernel at a time, because every layer blocked the host twice. cuStreamWaitValue32 moves the wait onto the stream, so the host enqueues "block until this word reaches N" and goes straight on to the next layer. For that to work combine has to know, before anything arrives, which slot each reply lands in, how many rows it carries, and what value its flag will take: - the slot is the region the FFN rank owns, on the ring the dispatch used; - the row counts are the ones the dispatch shipped, so they are recorded in the pending record instead of read back out of the reply header; - a reply now stamps the flag with the dispatch sequence it answers rather than the sender's own counter, which is a number the waiting rank already holds. The comparison is GEQ, and dispatch sequence numbers only ever increase, so a stale reply in the slot cannot satisfy it. The per-arrival header checks go with the poll, and the stream wait subsumes them: it blocks on one specific slot reaching one specific sequence, where the poll took whatever had landed and had to check afterwards that it was the thing it wanted. The shutdown branch went too; announce_shutdown has no callers. Measured on 4x L20X, DeepSeek-V2-Lite 2A2F throughout. Mean TTFT against the previous commit: 144 -> 129 ms at 8 rps, 185 -> 163 at 16, 346 -> 261 at 32, 426 -> 347 at 48, 1869 -> 951 at 64, with achieved throughput at offered 64 going 55.6 -> 56.6 rps. Against the synchronous connector async now wins by 21% at 8 rps and 25% at 16, and the gap at 64 rps closes from 4.8x to 2.5x. On the same attention rank at rate 32, queue lag moves from a 13.2us p90 to 660us and launches into an empty queue from 76.9% to 59.5% -- what is left is the counts readback that still blocks the host once per dispatch. async_gpu_connector_e2e passes over the wire, tests/unit 619 passed, and the greedy probe still answers sensibly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The yield hook imported afd_plugin.v1.worker.npu.ubatching inside the custom op's body, which runs once per MoE layer. On a CUDA build that module pulls in torch_npu and cannot import, and Python does not cache a failed import: every call re-walked the import machinery. A profile of a 2A2F Attention rank put vllm::manual_dbo_yield at 833us per call at the median with nothing nested inside it -- 258 ms of a 1403 ms window, the largest single host cost on the layer path, all of it spent failing to import a module that can never be there. The import is now resolved once at module import, so the op does what it says: check whether DBO is on, and return. Measured on 4x L20X, DeepSeek-V2-Lite 2A2F, mean TTFT against the previous commit: 129 -> 114 ms at 8 rps, 163 -> 147 at 16, 261 -> 228 at 32, 951 -> 826 at 64. Against the synchronous connector, async is now ahead at 8 rps (114 vs 143) and 16 rps (147 vs 155). The 48 rps point moved the wrong way in this run, 347 -> 624 ms with a p50 of 383 against a mean of 624, which reads as one slow stretch inside a 25 s window rather than a change in the steady state; the box is shared and that point wants re-running. The unit tests patched sys.modules to exercise the per-call import, so they now patch the resolved names instead. Same behaviour under test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
With the poll gone, the host stopped being blocked and started being busy: a profile of an Attention rank attributed only 7.8 ms of a 1403 ms window to synchronizes, while roughly half the window was plain Python between recorded ops. Three things on the dispatch path, all of them per peer per MoE layer: - encode_header assigned fourteen fields into a fresh tensor one setitem at a time. It builds a numpy array and wraps it instead, which shares the buffer. - the caller summed each rank's expert counts and prefix-summed all of them in Python to find segment bounds, a 64-iteration loop per layer for the counts this model has. plan_dispatch returns the per-rank totals and starts, which ride back in the readback that already happens. - shared-expert tokens were split round robin, so each peer needed an arange, a gather through it, and a whole slot field to carry the index. A contiguous split gives the same balance -- the shared expert treats every token alike -- and makes a rank's slice a view: no index built, none gathered through, none sent. The slot field is gone with it, and combine adds a slice instead of scattering. cudaMemcpyAsync was costing 25us of host time per call, 4000 calls in that window, so dropping one of the five copies per peer per layer is worth more than its bytes suggest. Measured on 4x L20X, DeepSeek-V2-Lite 2A2F, SECS=25, mean TTFT against the previous commit: 116 -> 106 ms at 8 rps, 146 -> 129 at 16, 245 -> 205 at 32, 355 -> 308 at 48, 870 -> 727 at 64. Against the synchronous connector async is now ahead by 25% at 8 rps and 17% at 16, within 11% at 32, and achieved throughput at offered 64 is 58.4 rps against 59.8. The shared split is a wire change but not a numerical one: a token's shared output is the same function wherever it runs. async_gpu_window_roundtrip and async_gpu_connector_e2e pass, tests/unit 619 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
The readback used .cpu(), which allocates a pageable destination. A pageable device-to-host copy cannot be handed straight to the DMA engine, so the driver stages it through its own buffer and blocks inside the memcpy: a profile of an Attention rank measured 466us of host time per call over 468 calls, 218 ms of a 1163 ms window, against 8us for the device-to-device peer writes and 16us for the same direction into pinned memory. Copying into a pre-allocated pinned buffer instead makes it an ordinary transfer. The buffer is sized once from the topology, and the concatenation writes into a matching device buffer rather than allocating one per layer. This is the readback I said earlier was worth about 2 ms. That was wrong: I had measured its cudaStreamSynchronize and missed that a pageable copy does its waiting inside cudaMemcpyAsync, where it does not show up as a synchronize at all. Measured on 4x L20X, DeepSeek-V2-Lite 2A2F, SECS=25, mean TTFT against the previous commit: 106 -> 101 ms at 8 rps, 129 -> 125 at 16, 205 -> 191 at 32, 308 -> 298 at 48, 727 -> 724 at 64. Against the synchronous connector that is 29% faster at 8 rps, 20% at 16, and within 4% at 32. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
torch.bincount sizes its output from the maximum value in the data, so it reads that maximum back to the host -- into pageable memory, which blocks inside the copy. A profile of an Attention rank at 48 rps measured 783us of host time per call over 260 calls, 203 ms of a 1170 ms window: the largest single host cost, inside a call whose output size we already knew. The partials are already sorted by global expert id for the dispatch, so the boundaries are one searchsorted over that sorted array against the expert range. It needs no readback, and the lower boundaries are the offsets that the cumulative sum used to produce, so a pass goes with it. Measured on 4x L20X, DeepSeek-V2-Lite 2A2F, SECS=25, mean TTFT against the previous commit: 101 -> 95 ms at 8 rps, 125 -> 118 at 16, 191 -> 178 at 32, 298 -> 276 at 48, 724 -> 517 at 64. Achieved throughput at offered 64 goes 58.4 -> 59.4 rps against the synchronous connector's 59.8. Async is now ahead of synchronous at 8, 16 and 32 rps (95 vs 143, 118 vs 155, 178 vs 185) and behind at 48 and 64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
Rebasing onto upstream/main brought together upstream's connector-lifecycle test and this branch's constructor, which reads connector.extra_info to decide whether async MoE ubatching is configured. Every real connector has one from AFDConnectorBase; the fake did not, so construction raised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The send path read the routing back to the host once per MoE layer to size each destination's slice. A profile put that synchronize at 392us a call -- not the copy, but the host waiting for everything queued ahead of it -- and it capped run-ahead at one layer, turning the forward into a sum of per-layer maxima instead of a max of sums. Write every slot at capacity instead: the payload carries the whole batch and the index arrays carry every partial, with a new segment_start header word telling a destination which run is its own. The header's routing tail is filled straight from the plan on the device, so nothing is read back. The bytes are nearly free -- with topk slots over ffn_size destinations a token misses a given destination only (1 - 1/ffn_size) ** topk of the time, 1.6% at 2A2F -- and deduplication, its presence grid and the combine scatter all go away with it. Signed-off-by: specture724 <specture724@gmail.com>
39 tasks
This file predates the ruff format hook and never satisfied it: a bare `pre-commit run` on it fails both ruff-check (E501 at the torch.randn line) and ruff-format. Touching it for anything else drags the whole reflow into that commit, so it goes in on its own here. No behaviour change -- the parsed AST is byte-identical before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
The slot reserved `token_cap` rows for `shared_x`, the same as `routed_x`, but nothing can ever fill them. Shared-expert tokens are split across the FFN ranks as contiguous chunks, so a slot holds at most `ceil(token_cap / ffn_size)` of them, in both directions: the dispatch ships one rank's chunk, and the reply answers that same chunk. A model with no shared experts does not need the field at all -- and was still being sent one every layer to every peer, because the send path never consulted `has_shared_experts`. Sizing the field by `shared_cap` cuts the payload half of every slot by `(1 - 1/ffn_size)`. Measured on a 2A2F stack (DeepSeek-V2-Lite, 4x L20X, max-num-batched-tokens 8192), the window logged at startup goes from 64.4 MiB per slot / 128.8 MiB total to 48.4 / 96.8, a 25% saving that comes straight back to the KV cache budget. It grows with the FFN rank count: 41% at 2A6F, and 50% for a model with no shared experts, which now ships no shared rows at all. The split arithmetic lived in two places -- `_headers_for`, which puts `shared_tokens` on the wire, and `send_attn_output`, which writes the rows it describes. Those two have to agree or the receiver reads a count that does not match the payload, so they now come from one `_shared_slice` rather than from two copies that can drift. Verified beyond the unit suite, because neither GPU e2e exercises this path: both run 1A1F, where the split is the whole batch and `shared_cap` collapses back to `token_cap`. A 4-GPU 2A2F run with real inference is what covers `shared_cap != token_cap`; greedy output stays coherent, a 90k-token prompt chunk-prefills without tripping the new bound, and no capacity error appears in either log. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
`announce_shutdown` passed `shared_idx=`, which `write_slot` has no parameter for, and omitted `expand_idx` and `weights`, which it requires. Any call raised TypeError before reaching the window. Nothing calls it -- the FFN loop leaves on the `recv_poll_timeout_ms` timeout instead, and the receive half is fully wired (`recv_attn_output` raises ConnectorShutdown on the header bit) -- so the wire protocol's shutdown path has never actually run. The signature drifted when the slot layout replaced a per-destination shared index with a contiguous range; every live call site was updated and this one, having no caller, was not. Since no runtime path exercises it, the regression guard is a signature bind: the test captures the arguments announce_shutdown really sends and binds them against `SymmWindow.write_slot`, which fails on both a stale keyword and a missing required one. Restoring the old call makes it fail with the TypeError it was hiding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: specture724 <specture724@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Impliment an async GPU connector based on NVSHMEM.
Issue
Scope
Test Plan
test_async_gpu_connector.pyis the main unit test added.async_gpu_connector_e2e.pyis for e2e GPU test. 1a1f and 2a2f eager recipe added.Test Result
TBD
Docs Impact
Essential PR Checklist