[ShaiShmuel]

Summary

• Adds a loadtime option, runtime_headroom_gib, to SGLangGenerationAdapter. The vision-extraction, CUDA 13 and Gemma SGLang adapters inherit it.
• SGLangMemoryBudget.required_bytes adds the value in both modes:
  • explicit mem_fraction_static: total × fraction + 1 GiB + headroom, capped at the device total
  • automatic sizing: static + 2 GiB + headroom
• reserved_bytes / memory_footprint() include the headroom. Load staging evicts idle eligible models until that much memory is free. An automatically sized engine does not launch without it.
• The headroom stays outside the static pool. The derived --mem-fraction-static and the KV cap do not change.
• The default is 0, so no existing profile's reservation changes.

Why

This was split out of #875. CodeRabbit asked there to reserve the vision runtime headroom in shared-device admission. The measurement was on one L4 with Xiaomi-OCR-0. One 4096×4096 page is 16,384 visual tokens and needs about 3.6 GiB of GPU memory above the static pool. At mem_fraction_static 0.85 the engine runs out of memory. At 0.70 it passes, with a GPU peak of 21.1 of 23.0 GB. SGLang admission reserved only the static pool plus a fixed 1–2 GiB allowance. A vision model loading onto a shared device could therefore be admitted without room for its own peak.

Scope

• Covered: the vision engine loads onto a device that already holds other models. Load staging now requires the declared headroom to be free and evicts idle models until it is.
• Not covered: a model loaded after a resident vision engine. That model is still admitted on actual device usage, so it can take headroom the engine has not allocated yet. The README states this limit. Closing this case needs core memory admission to account for the reserves of resident models. That is a separate design change.
• No checked-in profile sets a value yet. Xiaomi-OCR-0 (feat(models): onboard xiaomi-research/Xiaomi-OCR-0 #875) will set runtime_headroom_gib: 4, the measured 3.6 GiB rounded up. On an L4 that gives 0.70 × total + 5 GiB ≈ 20.7 GiB, which still fits. GLM-OCR, LightOnOCR-2/3, PaddleOCR-VL, Qwen3-VL and Qwen3.5/3.6 have the same exposure, but each needs a measured peak before it gets a value.

Tests

• tests/adapters/test_sglang_memory.py:
  • headroom is added in both modes and capped at the device total
  • it stays outside the static pool (same derived fraction, and the launch is refused when free memory lacks the headroom)
  • invalid values are rejected
  • load staging evicts when free memory fits the plain reservation but not the headroom
• tests/config/test_sglang_memory_profiles.py: a profile's adapter_options.loadtime.runtime_headroom_gib reaches the adapter's load reservation, and the checked-in GLM-OCR profile keeps its reservation.
• Local runs:
  • mise run test -- packages/sie_server/tests/config: 1685 passed, 525 skipped
  • SGLang adapter suites (test_sglang*.py): all passed
  • mise run lint and mise run typecheck: clean

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[ShaiShmuel]

Closing without merging. Checking the pinned engine's sizing shows this option is the wrong lever:

• With an explicit --mem-fraction-static F, sglang 0.5.10.post1 sizes weights + KV as F × free memory at launch (model_runner_kv_cache_mixin.py: rest_memory = post_model_load_memory - pre_model_load_memory * (1 - F)), so the engine keeps (1 − F) × free-at-launch as its own runtime headroom. mem_fraction_static is already the per-profile headroom knob.
• This option only raises the pre-launch free-memory floor. In explicit mode the engine splits the extra memory F:(1 − F), so a declared 4 GiB at F = 0.70 yields ~1.2 GiB of headroom and ~2.8 GiB of KV pool. The README sentence about the headroom staying outside the static pool holds only in automatic mode, and no measured profile uses that mode.
• It changes nothing in the ordering the original review comment described (engine resident first, a later model admitted on actual free memory). In the ordering it does affect, the existing floor (int(total × F) + 1 GiB) already leaves ~(1 − F) × floor of engine headroom.

The remaining gap (a later load admitted into a resident engine's not-yet-allocated headroom) is an admission-accounting question that no profile field can express; the discussion continues on #875.