Repository navigation
Description
Why
vLLM serves one model per server process — this is upstream's design, not a vllm-helper limitation. Upstream's guidance is to run multiple instances, each serving a different model, with a routing layer in front (their FAQ, and vllm#5899 / vllm#21481).
vllm-helper currently assumes exactly one instance: one systemd unit / launchd job, one port, one state/active.env, one "active profile". Everything else follows from that assumption — switch unloads the old model to load the new one, and the Coder sync disables every other registered model so users can't pick something that isn't loaded.
On a box with enough memory (a DGX Spark, or any host that fits several small models), that's more restrictive than the hardware requires. A user with a 7B chat model and a small embedding/reranker model has to choose between them for no good reason.
What this would mean
Roughly, per-profile instances instead of one global instance:
- one service unit per profile (
vllm-<profile>.service) rather than a singlevllm.service - a port per instance, allocated and recorded rather than a single global
PORT start/stop/restart/logs/statusgaining a notion of which instance- something in front so callers still see one endpoint — either a documented per-model port, or a router. (A router is the bigger question; it overlaps web UI #2 and would put vllm-helper in the data path, which it deliberately isn't today.)
- memory budgeting across concurrent instances —
--gpu-memory-utilizationis currently computed as if the process owns the machine
Interaction with the integrations work
This is the reason the "should several models stay enabled?" question came up while designing the plugin system (#1). Today the Coder plugin disables all non-active models because picking one would fail — with several instances actually running, that stops being true and the behavior should be revisited.
Deliberately not blocking #1 on this. The plugin seam should stay agnostic about how many models are live.
Open questions
- Single endpoint (router) or documented per-model ports? The former is nicer and much larger.
- How are instances supervised — one unit per profile, or a template unit (
vllm@<profile>.service)? - What decides which models are running: explicit
up/downverbs, or a declarative "these are enabled" set? - How is memory divided, and what happens when the next model doesn't fit?
Activity
DevelopmentCats commentedon Nov 10, 2026
Done in #63: every running model is its own instance (
llm-helper@<profile>.serviceon Linux, one launchd job per profile on macOS), with per-instance ports,up/downverbs, memory-fit checks across instances, an opt-in router on one endpoint, and the Coder sync enabling every running model. Existing installs keep serving on the legacy unit untilllm-helper migrateruns. Seedocs/multi-instance.md.