Skip to content

Run several models at once by managing one vLLM instance per model #15

Closed

Description

@DevelopmentCats

Why

vLLM serves one model per server process — this is upstream's design, not a vllm-helper limitation. Upstream's guidance is to run multiple instances, each serving a different model, with a routing layer in front (their FAQ, and vllm#5899 / vllm#21481).

vllm-helper currently assumes exactly one instance: one systemd unit / launchd job, one port, one state/active.env, one "active profile". Everything else follows from that assumption — switch unloads the old model to load the new one, and the Coder sync disables every other registered model so users can't pick something that isn't loaded.

On a box with enough memory (a DGX Spark, or any host that fits several small models), that's more restrictive than the hardware requires. A user with a 7B chat model and a small embedding/reranker model has to choose between them for no good reason.

What this would mean

Roughly, per-profile instances instead of one global instance:

  • one service unit per profile (vllm-<profile>.service) rather than a single vllm.service
  • a port per instance, allocated and recorded rather than a single global PORT
  • start/stop/restart/logs/status gaining a notion of which instance
  • something in front so callers still see one endpoint — either a documented per-model port, or a router. (A router is the bigger question; it overlaps web UI #2 and would put vllm-helper in the data path, which it deliberately isn't today.)
  • memory budgeting across concurrent instances — --gpu-memory-utilization is currently computed as if the process owns the machine

Interaction with the integrations work

This is the reason the "should several models stay enabled?" question came up while designing the plugin system (#1). Today the Coder plugin disables all non-active models because picking one would fail — with several instances actually running, that stops being true and the behavior should be revisited.

Deliberately not blocking #1 on this. The plugin seam should stay agnostic about how many models are live.

Open questions

  • Single endpoint (router) or documented per-model ports? The former is nicer and much larger.
  • How are instances supervised — one unit per profile, or a template unit (vllm@<profile>.service)?
  • What decides which models are running: explicit up/down verbs, or a declarative "these are enabled" set?
  • How is memory divided, and what happens when the next model doesn't fit?

Activity

  1. DevelopmentCats commented on Nov 10, 2026

    @DevelopmentCats
    CollaboratorAuthor

    Done in #63: every running model is its own instance (llm-helper@<profile>.service on Linux, one launchd job per profile on macOS), with per-instance ports, up/down verbs, memory-fit checks across instances, an opt-in router on one endpoint, and the Coder sync enabling every running model. Existing installs keep serving on the legacy unit until llm-helper migrate runs. See docs/multi-instance.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions