Skip to content

docs(qwen35): add 9B and 27B single-GH200 serving numbers#12

Open
FeathBow wants to merge 1 commit into
openinfer-project:mainfrom
FeathBow:feat/qwen35-9b-27b-serving
Open

docs(qwen35): add 9B and 27B single-GH200 serving numbers#12
FeathBow wants to merge 1 commit into
openinfer-project:mainfrom
FeathBow:feat/qwen35-9b-27b-serving

Conversation

@FeathBow

Copy link
Copy Markdown
Contributor

Description

  • Adds Qwen3.5-9B Serving Load and Qwen3.5-27B Serving Load to /models/qwen35-4b/, measured on 1x GH200 120GB (aarch64, sm_90) during the openinfer 509/510 rerun campaign: full c/QPS ladders with TTFT/TPOT p50+p99, long-context and c=120 overload behavior, tool-call round-trips, and greedy-vs-HF plus per-size logits-gate summaries.
    • 9B: openinfer main ffb959c4, Python vllm bench serve client, random 1024/128, greedy, default seed. Saturation ~6.7 req/s / ~857 out tok/s.
    • 27B: ffb959c4. Plateau ~2.7 req/s / ~340 out tok/s at qps≥8; the load-time budget line is quoted and explained — 17.2 GB KV, 5.5 GB prefill scratch, 18.8 GB recurrent-state reserve (two ~147 MB states per decode slot across the 64-slot capacity).
  • Rewrites the Notes section for the family: per-request recurrent-state sizes (~49 MB at 9B / ~147 MB at 27B), the load-time reserve and the --max-batch capacity/KV trade-off, the decodable-vocab selection bound (248077 of the 248320 padded lm_head rows), and a corrected CUDA-Graph statement — the server always captures and rejects --cuda-graph=false, so the previous "disable for debugging" note no longer holds.
  • Links the overview table row: Qwen3.5-4B / 9B / 27B → /models/qwen35-4b/.
  • Validation: npm run build + static tests 12/12.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant