[krisztian-gajdar]

Qwen/Qwen3.6-27B:h100 does not start in the published 0.10.1 cuda12-sglang image on an H100 80GB. SGLang exits with Not enough memory. Please try to increase --mem-fraction-static. Current value: mem_fraction_static=0.93 while sizing its pools, and the server reports exit code -9. The compatibility matrix for 0.10.1 records the profile as unsupported for this reason.

Cause. NEXTN on this hybrid Gated-DeltaNet model reserves intermediate mamba state for every running request and draft token, about 0.14 GB per request per token. The profile set no running-request limit, so SGLang used its speculative default of 48. With four draft tokens that reservation is about 27.5 GB. Only 26.93 GB is left after the 51.05 GB of BF16 weights, so no fraction can fit it.

Measurements. All runs used one H100 80GB, with SGLang 0.5.10.post1, torch 2.9.1 and transformers 5.3.0, the versions in the 0.10.1 image. Each was served through sie-server from this branch.

mem_fraction_static	Running requests	Result
0.93	48 (default)	Pool sizing fails: Not enough memory
0.93	8	Pools fit, but the NEXTN draft needs 5.53 GB to load with 4.64 GB free. Its embedding and head are allocated before they are shared.
0.88	8	Serves. 42 mamba slots (6.17 GB), 5.16 GB of speculative state, a 113,971-token KV pool, the draft loaded with 2.54 GB to spare.

With the committed YAML, the profile served:

• a first chat;
• 8 concurrent chats, none empty;
• a 7,027-token prompt alongside 7 short requests.

Change. The h100 profile now uses mem_fraction_static: 0.88 and --max-running-requests 8, and the measurement is recorded next to it in the YAML. Nothing else changes: the model-card 3/4 NEXTN draft, the 8K context and the 32,768-token KV budget stay as they were. The cap is far above what this profile uses. Speculative decoding here is tuned for latency and does not batch, and the KV pool still holds more than three times the profile's KV budget. Qwen3.8-27B-FP8 uses the same flag to bound its launch shapes.

Validation.

• The Qwen profile, generation-mode, grammar-routing, SGLang adapter, loader and engine launch-contract tests pass, 557 in all.
• The launch-flag snapshot test passes. --max-running-requests is in the SGLang 0.5.10.post1 flag snapshot.

Summary by CodeRabbit

• Performance
  • Adjusted the H100 serving configuration to use less static memory and limit concurrent running requests to 8. Added measured memory-use and serving-result details for the 3/4 NEXTN configuration.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Team
• Run ID: 998d488f-7383-4613-9f90-84ded83f931e

📥 Commits

Reviewing files that changed from the base of the PR and between 3186eb0 and d3d839f.

📒 Files selected for processing: 1

• packages/sie_server/models/Qwen__Qwen3.6-27B.yaml

📝 Walkthrough

The H100 profile now sets mem_fraction_static to 0.88 and caps running requests at 8. Its comments document memory measurements and tested concurrency for the BF16 3/4 NEXTN configuration.

Changes: H100 serving profile

Layer / File(s)	Summary
H100 memory and request limits packages/sie_server/models/Qwen__Qwen3.6-27B.yaml	The profile retains its 32,768-token KV budget, lowers mem_fraction_static from 0.93 to 0.88, and adds a limit of 8 running requests. Comments document memory measurements and tested concurrency.

Suggested reviewers: dragosboca

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to d3d83

No actionable merge-blocking risk was identified in the H100 profile adjustment.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies the main change: updating the Qwen3.6-27B H100 NEXTN profile to fit on one H100.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}