[AMD/ROCM] ATOM DS R1 FP8 MTP 3 tokens support #984
Conversation
Signed-off-by: seungrokj <seungrok.jung@amd.com>
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you |
1 similar comment
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you |
|
Thanks! @seungrokj can u update atom recipes for this? |
yes it's updated here https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-R1.md#fp8-with-mtp-speculative-decoding-recommended |
Hi @functionstackx @cquil11
This PR include this config
Now atom supports mtp 3 tokens.
Regards,
Seungrok
cc. @ChuanLi1101 @andyluo7 @chunfangamd