[TP][sglang] When/how will TP be supported?

[lvhoaa]

Hi team,

This is a great work to address the tough CUDA graph init coldstart, and I believe it's going to be much more impactful if Foundry could support TP since a great number of LLM engine deployment uses TP.

I understand that supporting TP is not straightforward as other parallelism schemes (DP, EP) since it involves NCCL all-reduce buffer/communicator issues and VMM conflicts...

So I want to ask if the team has any concrete solution / strategy / plan to support TP in Foundry.

Having this capability would make this work truly impressive in covering almost all parallel deployment configs today!

Thanks!

[xenshinu]

Thanks for your interest @lvhoaa.
As I mentioned in #5, we have TP support via torch symmetric memory prototype. Consider such high demands, I will push a demo code later this week. Please take that as a initial reference. Also, feel free to contribute via submitting PRs.
Best,

[xenshinu]

Hi @lvhoaa, Foundry v0.0.3 supports TP via torch symmetric memory, please refer to SGLang recipes for instructions.