[vLLM] Feature support for DEP/TP

[liuzijing2014]

Hi friends,

Thanks for working on Foundry to speed up llm inference startup time. I wonder if currently we support DEP or TP on vLLM? Any plan/interest to integrate foundry into the the upstream vLLM?

Thanks!

[xenshinu]

Thanks for your interests, both DP and EP are supported in the current version. We expect to release another version by the end of Sept. to support TP via torch symmetric memory. Right now I have another job occupied.

We are happy to integrate into upstream serving engine. Willing to connect to discuss how to align the codebase.

Best,
Xueshen

[xenshinu]

Hi @liuzijing2014, please refer to vLLM recipe for how to run use Foundry in TP with torch symm-memory. Let me know if you have any feedbacks. The implementation is still a prototype so any bug report is appreciated.