[WIP][NPU]Add DSV4 NPU ubatch async connector by bjf-frz · Pull Request #264 · vllm-project/afd-plugin

bjf-frz · 2026-08-24T08:09:41Z

PLEASE FILL IN THE PR DESCRIPTION AND MAKE SURE THE CHECKLIST ITEMS HAVE BEEN CONSIDERED.

Purpose

Issue

Related issue(s): #
Closing keyword, only if fully resolved: Closes #

Scope

In scope:
Out of scope:

Implementation Notes

Test Plan

Test Result

Docs Impact

Files updated:
If none, reason:

Essential PR Checklist

Purpose is clear and linked to public context when possible.
Scope is bounded.
Compatibility with vLLM v0.26.0 is considered.
No changes are made to the vLLM source checkout.
Plugin-owned classes or explicit dotted class paths are preferred over monkey patches.
Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested.
Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated.
Validation evidence is included, including skipped GPU tests when applicable.
Documentation impact is stated.

jiangkuaixue123

Please address the inline comments.

jiangkuaixue123 · 2026-08-24T08:20:54Z

 _logger = logging.getLogger(__name__)
 _registered = False

+


Why do we need this process-wide multiprocessing override here? It does not appear necessary for the AFD plugin itself, and mutating Python's global multiprocessing context (including the private _concrete_contexts registry) is quite intrusive. Please remove this unless there is a concrete AFD-specific requirement and reproducer that cannot be addressed in the affected NPU runtime path.

jiangkuaixue123 · 2026-08-24T08:20:54Z

+        # fail_if_unsupported_npu_afd_features; the AFD async MoE ubatching
+        # path drives stage counts through connector_extra_config
+        # (async_moe_num_ubatches), which is validated there as well.
+        if self.vllm_config.parallel_config.use_ubatching:


This check does not look correct. Rejecting use_ubatching in the wrapper disables native ubatching for ordinary decode as well, rather than only restricting the CAM async path. Please remove this change so normal decode ubatching can still be enabled.

jiangkuaixue123 · 2026-08-24T08:20:54Z

+from vllm.platforms import current_platform
+
+if current_platform.device_type == "npu":
+    from afd_plugin.model_executor.models.deepseek_v4_npu import (


Could we place the GPU and NPU implementations under dedicated gpu/ and npu/ directories, respectively, and keep the two backend implementations isolated? Please avoid sharing/reusing backend-specific DeepSeek-V4 code across GPU and NPU.

jiangkuaixue123 · 2026-08-24T08:20:54Z

@@ -0,0 +1,164 @@
+# SPDX-License-Identifier: Apache-2.0


I do not understand why this KV-cache patch is necessary. What concrete upstream issue does it address, and why must it be patched in AFD rather than fixed in vLLM-Ascend? Please provide a minimal reproducer and explain the necessity of this patch; otherwise, please remove it.

jiangkuaixue123 · 2026-08-24T08:20:54Z

@@ -0,0 +1,119 @@
+# SPDX-License-Identifier: Apache-2.0


I do not understand why this ModelSlim patch is necessary. What exact incompatibility requires AFD to override the upstream prefix mapping, and why cannot it be handled in vLLM-Ascend? Please provide a concrete reproducer and justify the patch; otherwise, please remove it.

jiangkuaixue123 · 2026-08-24T08:39:29Z

Please fix conflict.

ShwStone · 2026-08-24T09:17:32Z

It seems that there are a lot of documents and operation manuals in the repository about specific test environments. Please make sure that the documents you submit are general guidance and avoid submitting irrelevant documents and startup scripts.

ShwStone · 2026-08-25T01:55:02Z

+        # Keep EngineCore's synthetic KV-cache profile local so that workspace
+        # is not mistaken for reclaimable KV capacity.  The first real request
+        # performs the normal CAM dispatch/receive sequence below.
+        if forward_context.in_profile_run:


Could you explain why the startup profile must skip CAM dispatch/combine here?

PR #248 intentionally removed an equivalent guard to fix #247. Matched profile traffic ensures that every Attention DP rank enters the CAM operators before live serving; skipping it appears to defer lazy CAM/HCCL initialization to the first real request, potentially reintroducing the DP>1 first-request deadlock.

Has the CAM 209 initialization contract changed? If so, how does this PR initialize all Attention ranks before serving, and has the DP2 single-cold-request scenario from #247 been validated?

This also appears to conflict with the existing test_async_cam_profile_forward_runs_matched_connector_io regression test added by #248.

ShwStone · 2026-08-25T02:34:05Z

Could you clarify the motivation for replacing the previous global-DP topology with multiple independent DP1 Attention services?

The previous setup already supported a cross-node DP3TP8 + EP8 topology (for example, local DP2 on one node and DP1 on another), while vLLM handled request routing across the Attention DP replicas automatically. With this change, request routing must be moved to an external load balancer, and the FFN side additionally needs multiple CAM/HCCL communicators plus a TCPStore scheduler.

What concrete limitation or failure in the previous global CAM world requires this higher-complexity design? Is this mandated by a DSV4/CAM 209 contract, or does the old topology exhibit a reproducible correctness or performance issue under concurrent DP traffic?

It would also be helpful to explain why the existing cross-node DP mechanism cannot satisfy the intended deployment and what benefit justifies giving up vLLM built-in DP routing.

Signed-off-by: bjf-frz <frz123db@gmail.com>

jiangkuaixue123 · 2026-08-25T07:14:52Z

    depends_on: upload-merge-pipeline
    steps:
-      - label: "E2E · DeepSeekV2-Lite"
+      - label: "E2E · DeepSeekV2-Lite · Legacy"


This file change is not needed.

jiangkuaixue123 · 2026-08-25T07:15:03Z

    depends_on: upload-ready-pipeline
    steps:
-      - label: "E2E · DeepSeekV2-Lite"
+      - label: "E2E · DeepSeekV2-Lite · Legacy"


This file change is not needed.

Signed-off-by: bjf-frz <frz123db@gmail.com>

jiangkuaixue123 · 2026-08-25T07:44:01Z

 ):
+    # Install DSV4's Ascend KV grouping compatibility before an EngineCore can
+    # ask vLLM to construct its cache configuration.
+    from afd_plugin.compat.patches.npu.deepseek_v4_kv_cache import (


Remove these two patches if there is no need.

jiangkuaixue123 · 2026-08-25T07:45:03Z

            llama_4_scaling,
        )

+        # CAM allocates persistent operator workspace on its first dispatch.


Please remove this.

Signed-off-by: bjf-frz <frz123db@gmail.com>

jiangkuaixue123 · 2026-08-25T09:08:21Z

+        # small dummy batch with an FFN receive-capacity buffer and leave the
+        # external communicator in a broken state.  Keep the profile local;
+        # live forwards below retain the real remote-FFN exchange.
+        if bool(getattr(forward_context, "in_profile_run", False)):


There is no need. Please remove.

jiangkuaixue123 · 2026-08-25T12:51:49Z

+                # dispatch-recv for all following MoE layers can reuse CAM's
+                # bounded buffers while the prior combine is still in flight.
+                # The target baseline is deliberately serialized per layer;
+                # async MoE ubatching will introduce explicit stage credits.


Need remove.

…m-patch [NPU] Remove redundant DSV4 ModelSlim patch

jiangkuaixue123 · 2026-08-27T11:29:15Z

        if not self.connector.is_initialized:
            self.connector.init_afd_connector()

+    def _is_dsv4_model(self) -> bool:


这个函数多余

bjf-frz requested review from hsliuustc0106 and jiangkuaixue123 as code owners August 24, 2026 08:09

bjf-frz force-pushed the v0.26.0_camasync_dsv4 branch from 5bbeadc to 5399493 Compare August 24, 2026 08:19

jiangkuaixue123 reviewed Aug 24, 2026

View reviewed changes

bjf-frz force-pushed the v0.26.0_camasync_dsv4 branch from 2cc1aa8 to 053b808 Compare August 24, 2026 09:11

ShwStone reviewed Aug 25, 2026

View reviewed changes

bjf-frz added 4 commits August 25, 2026 14:56

Add DSV4 CAM async connector

7c3b85d

Signed-off-by: bjf-frz <frz123db@gmail.com>

Add DSV4 recipes and experiment tools

8784835

Signed-off-by: bjf-frz <frz123db@gmail.com>

Support cross-node DSV4 CAM async topology

76c76d1

Signed-off-by: bjf-frz <frz123db@gmail.com>

Add DSV4 itask launch scripts

88d1d87

Signed-off-by: bjf-frz <frz123db@gmail.com>

bjf-frz force-pushed the v0.26.0_camasync_dsv4 branch from eaada78 to ca2eacb Compare August 25, 2026 07:06

bjf-frz changed the base branch from main to v0.26.0_camasync_dsv4 August 25, 2026 07:12

jiangkuaixue123 reviewed Aug 25, 2026

View reviewed changes

bjf-frz added 3 commits August 25, 2026 15:18

Fix CPU-only lint for DSV4 connector

00ca98b

Signed-off-by: bjf-frz <frz123db@gmail.com>

Remove local experiment docs and scripts from PR

b8ce020

Signed-off-by: bjf-frz <frz123db@gmail.com>

Fix NPU worker import lint

ea44030

Signed-off-by: bjf-frz <frz123db@gmail.com>

bjf-frz force-pushed the v0.26.0_camasync_dsv4 branch from ca2eacb to ea44030 Compare August 25, 2026 07:22

jiangkuaixue123 reviewed Aug 25, 2026

View reviewed changes

Remove standalone CAM async DP rank offsets

b0fa922

Signed-off-by: bjf-frz <frz123db@gmail.com>

jiangkuaixue123 reviewed Aug 25, 2026

View reviewed changes

Add cross-node DSV4 AFD launch script

6d24183

jiangkuaixue123 reviewed Aug 25, 2026

View reviewed changes

bjf-frz and others added 3 commits August 26, 2026 09:13

Use Ascend DSV4 KV cache grouping

71fe223

Remove redundant DSV4 ModelSlim patch

eca8007

Merge pull request #1 from jiangkuaixue123/codex/remove-dsv4-modelsli…

59020e6

…m-patch [NPU] Remove redundant DSV4 ModelSlim patch

refactor: unify async CAM control store

ff23e6d

bjf-frz force-pushed the v0.26.0_camasync_dsv4 branch from d7d3b20 to ff23e6d Compare August 27, 2026 08:04

bjf-frz added 2 commits August 27, 2026 18:44

fix(npu): restore profile guard and simplify async CAM world

793db3a

refactor(npu): isolate DeepSeek V4 implementation

501a331

jiangkuaixue123 reviewed Aug 27, 2026

View reviewed changes

Uh oh!

Conversation

bjf-frz commented Aug 24, 2026

Purpose

Issue

Scope

Implementation Notes

Test Plan

Test Result

Docs Impact

Uh oh!

jiangkuaixue123 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiangkuaixue123 commented Aug 24, 2026

Uh oh!

ShwStone commented Aug 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShwStone commented Aug 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants