refactor(vllm): use native IPC engine for colocated weight sync#332
refactor(vllm): use native IPC engine for colocated weight sync#332miracle0517 wants to merge 1 commit into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the colocated vLLM weight synchronization mechanism to leverage native IPC transfer engines (such as NPUIPCWeightTransferEngine and IPCWeightTransferEngine) instead of custom IPC gathering and serialization logic. The custom update_weights_from_tensor and update_weights_chunk methods have been replaced by a single update_weights method that posts native vLLM payloads, and serialization has been switched from cloudpickle to standard pickle. Unit tests, scripts, and Docker patches have been updated to align with these native APIs. No review comments were provided, so there is no additional feedback to address.
Documentation build overview
44 files changed ·
|
9c9d3d4 to
01d77cd
Compare
Replace the custom colocated IPC handle gathering and worker chunk RPC path with vLLM/vLLM-Ascend native IPC weight transfer engines. Route generated update_info through VLLMEngine.update_weights and vLLM's /update_weights endpoint, initialize colocated transfer engines once during connection, and update unit tests for the native colocated transfer flow. Signed-off-by: wuxiang <498160096@qq.com>
01d77cd to
38f431a
Compare
|
The content of this PR has been merged into another PR. #345 |
1 participant
Summary
Replace VIME's custom colocated IPC weight-sync implementation with the native weight transfer engines provided by vLLM and vLLM-Ascend.
This removes the manual IPC handle gathering, serialization, and worker-side chunk loading path while aligning colocated weight updates with vLLM's native
/update_weightslifecycle.What changed
NPUIPCWeightTransferEngineon Ascend to send colocated weights.update_infopayloads throughVLLMEngine.update_weightsand vLLM's/update_weightsendpoint.start_weight_update/finish_weight_updatestate machine with checkpoint-format loading.update_weights_from_tensor/update_weights_chunkRPC pathWhy
The previous implementation duplicated functionality already supported by vLLM's native IPC transfer engines and required VIME to maintain custom GPU-slot gathering, IPC handle reconstruction, and worker RPC code.
Using the native transfer path reduces VIME-specific integration code and keeps colocated weight synchronization aligned with upstream vLLM and vLLM-Ascend behavior.
Testing
pytest tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py -qpytest tests/unit/backends/vllm_utils/test_vllm_engine.py -q