Repository navigation
Keep the SGLang radix cache for agentic runs on MI300X/MI355X - #123
Closed
zoroyihan7 wants to merge 1 commit into
Closed
zoroyihan7 wants to merge 1 commit into
zoroyihan7 wants to merge 1 commit into
Conversation
sglang_mi300x.sh and sglang_mi355x.sh append --disable-radix-cache unless EXTRA_SGLANG_ARGS mentions it. Agentic trace replay resends long shared prefixes (about 97% theoretical prefix hit on the AgentX traces), so a radix-off baseline measures an unrepresentative server: TTFT is dominated by recomputing ~100K-token prefixes, and any later run that simply turns the cache back on looks like a large optimization. When IS_AGENTIC is 1 or true (the variable Magpie's AgentX mode and the InferenceX agentic recipes already use), skip the --disable-radix-cache default. Fixed-length runs and explicit EXTRA_SGLANG_ARGS are unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: zoroyihan7 <Yihan.Wang@amd.com>
zoroyihan7
requested review from
a team,
haofrank and
sinarafati-amd
as code owners
October 14, 2026 09:33
Author
|
Closing: per maintainer feedback, AgentX benchmarking and profiling will go through Magpie's native AgentX path (Hyperloom PR in flight), so this switch is not needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
sglang_mi300x.shandsglang_mi355x.shadd--disable-radix-cacheby default unlessEXTRA_SGLANG_ARGSmentions it. That default suits fixed-length random-prompt runs, but agentic trace replay resends long shared prefixes. The AgentX traces have a theoretical prefix-cache hit rate of about 97% and inputs averaging about 143K tokens.With the radix cache off, the server recomputes every prefix. We saw this on an AgentX trace replay on MI355X, where the launcher was booted as the server phase. The server log reported 0 cached tokens and TTFT p50 was about 68 s. A later run whose only real change was turning the radix cache back on reported about +210% output throughput and about +2300% on
e2e_norm_intvty_p50. The baseline measured an unrepresentative server. The upstream InferenceX agentic recipes keep the prefix cache on.Change
IS_AGENTICis1ortrue(case-insensitive), both launchers skip the--disable-radix-cachedefault. Magpie's AgentX mode and the InferenceX agentic scripts already use this variable.IS_AGENTICis unset, empty or0.--mem-fraction-static=0.8is still added. An explicitEXTRA_SGLANG_ARGSstill takes precedence.test_sglang_agentic_runs_keep_radix_cachechecks the server argv of both scripts for unset,0, empty,1,trueandTRUE.Verification
python -m pytest -q tests/test_client_tokenizer_mode.py: 97 passed, 76 skipped (the packaged-InferenceX cases skip below Python 3.12).main's scripts in all 6 agentic cases and passes with this change.ruff check/ruff format --checkare clean on the touched test file.shellcheckreports no new warnings on the two scripts.🤖 Generated with Claude Code