Skip to content

instances: run several models at once - #63

Merged
DevelopmentCats merged 4 commits into
mainfrom
devcats/multi-instance
Nov 10, 2026
Merged

DevelopmentCats merged 4 commits into
mainfrom
devcats/multi-instance

Conversation

@DevelopmentCats

@DevelopmentCats DevelopmentCats commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #15.

Every running model is now an instance, keyed by its profile, with its own port, service and log.
llm-helper up <profile> starts one next to the ones already running, and down <profile> stops it.

What changed

  • Each instance is a unit of the llm-helper@.service template on Linux, and its own launchd job (com.coder.llm-helper.<profile>) on macOS.
  • up checks the model fits in the memory the running instances on the same GPUs leave (the Metal limit on Apple silicon), then takes the next free port from PORT unless --port is given.
    --mem-util and --kv-cache give it a smaller share.
    Concurrent up runs, or up and the panel, never pick the same port.
  • After a reboot instances start oldest first; one that no longer fits exits with status 78 instead of loading.
  • New commands:
    • default <profile> sets the model Coder selects first (DEFAULT_PROFILE).
    • migrate moves a model started by an earlier release from the single vllm.service (or com.coder.vllm) unit to an instance on the same port.
    • router (opt-in) serves every instance on one port and dispatches by the request's model, with install-service, serve and uninstall-service.
  • stop is another name for down.
    restart, logs and status take a profile name and act on the only instance when exactly one runs.
  • switch is deprecated and hidden.
  • new, search and the panel's "Pull and run" start an instance next to the running ones instead of replacing them.
  • Coder sync offers every running instance.
    Without the router only the one on PORT is enabled, and sync names the ones it left out.
  • setup no longer installs a service unit; up installs each instance's.
  • The recipe health watchdog keeps its launch and stopped markers per profile.
  • Panel: one row per instance with Restart, Stop and Make default, and Run for saved profiles.
    New endpoints GET /api/instances and POST /api/instances/{name}/up|down|restart|default; /api/logs takes ?instance=.
    The sudoers line covers llm-helper@*.service.

Removed

  • state/active.env, state/active-profile and the single active profile. migrate converts them.
  • /api/switch, /api/restart and /api/stop.

Upgrading

The old unit keeps serving until llm-helper migrate runs.
doctor reports when it is still needed, and up and the panel refuse to start a model until then.

Docs

New page "Several models at once" (docs/multi-instance.md), and CLI reference pages for up, down, default, migrate and router.

Every running model is an instance keyed by its profile, with its own port, unit (llm-helper@<profile>.service on Linux, one plist per instance on macOS) and log. Adds up, down, default, migrate and an opt-in model router; switch is deprecated. Coder sync offers every running instance.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 01:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical findings permit overly broad sudo access and unauthenticated router access, while launch and migration defects can disrupt running services or delete instance records.

12 open issues
What changed in this PR

Enables concurrent, per-profile model instances across the CLI, web panel, recipe services, and Coder integration, replacing the single-active-model workflow.

Changes:

  • Adds instance-specific services, ports, logs, lifecycle commands, memory admission, and legacy migration.
  • Introduces model-aware routing and synchronization of multiple running models with Coder.
  • Updates panel controls, regression tests, guides, and generated CLI documentation.

This pull request requires changes. Unresolved security, routing, lifecycle, resource-allocation, integration, and accessibility findings block approval.

File Description
README.md Introduces concurrent models and instance commands.
internal/​web/​static/​models.js Updates model actions for concurrent startup.
internal/​web/​static/​models.html Updates the model-start button label.
internal/​web/​static/​index.html Adds instance listings and log selection.
internal/​web/​static/​app.css Styles instance controls and log selection.
internal/​web/​models.go Starts pulled models and protects running profiles.
internal/​web/​docssite.go Updates the documentation-site summary.
internal/​web/​activate.go Implements panel-driven instance startup.
internal/​service/​service.go Adds service arguments and no-fit exit status.
internal/​service/​service_other.go Adds unsupported-platform instance interfaces.
internal/​service/​service_linux.go Implements systemd instance templates and removal.
internal/​service/​service_linux_internal_test.go Tests instance templates and router units.
internal/​service/​service_darwin.go Implements per-instance launchd jobs and removal.
internal/​service/​service_darwin_internal_test.go Tests instance labels and launch arguments.
internal/​router/​models.go Adds model discovery and request routing.
internal/​router/​models_internal_test.go Tests discovery, routing, and backend failures.
internal/​profile/​profile.go Validates service-compatible instance names.
internal/​profile/​profile_test.go Tests instance-name validation.
internal/​plugin/​registry_internal_test.go Updates lifecycle-event dispatch tests.
internal/​plugin/​plugin.go Adds instance lifecycle events and live-instance state.
internal/​plugin/​coder/​sync.go Synchronizes reachable models and default selection.
internal/​plugin/​coder/​sync_internal_test.go Updates shared-entry ownership coverage.
internal/​engine/​registry.go Updates environment-layering documentation.
internal/​control/​instances_internal_test.go Tests instance state, defaults, and port allocation.
internal/​control/​gpus.go Separates global GPU selection from profile selection.
internal/​control/​gguf.go Clarifies plugin effects on running instances.
internal/​control/​fit.go Adds aggregate memory-share admission checks.
internal/​control/​fit_internal_test.go Tests memory admission limits and refusal messages.
internal/​control/​control.go Adds instance-aware integration notifications.
internal/​config/​envfile.go Updates environment-file documentation.
internal/​config/​config.go Adds instance paths, defaults, and router settings.
internal/​config/​config_test.go Tests instance paths and legacy state access.
internal/​cli/​watch.go Watches each instance’s runner and endpoint.
internal/​cli/​watch_internal_test.go Tests instance-specific boot targets.
internal/​cli/​vllmargs.go Renames environment helpers for instance startup.
internal/​cli/​vllmargs_internal_test.go Updates environment-helper tests.
internal/​cli/​status_internal_test.go Updates crash-loop diagnostic expectations.
internal/​cli/​start.go Adds instance startup and legacy compatibility.
internal/​cli/​setup.go Defers service installation until instance startup.
internal/​cli/​service.go Makes restart and logs instance-specific.
internal/​cli/​serve.go Supplies instance-specific cluster lookups to the panel.
internal/​cli/​search.go Starts selected models alongside existing instances.
internal/​cli/​router.go Adds router serving and service management.
internal/​cli/​router_internal_test.go Tests router configuration and service operations.
internal/​cli/​root.go Registers instance commands and runner selection.
internal/​cli/​root_internal_test.go Updates command-set and grouping expectations.
internal/​cli/​recipeservice.go Generates profile-specific recipe service units.
internal/​cli/​recipeservice_internal_test.go Tests profile-specific recipe units.
internal/​cli/​recipes.go Registers recipe service lifecycle commands.
internal/​cli/​recipelaunch.go Records recipe launches as instances.
internal/​cli/​recipelaunch_internal_test.go Updates recipe launch assertions and terminology.
internal/​cli/​recipehealth.go Adds profile-specific watchdog targeting.
internal/​cli/​recipehealth_internal_test.go Tests profile-aware watchdog commands.
internal/​cli/​plugins.go Removes the single-endpoint helper.
internal/​cli/​memfit.go Shares the unified-memory limit with control.
internal/​cli/​ggufplugin.go Updates plugin activation messaging.
internal/​cli/​editset.go Gives instance-specific restart instructions.
internal/​cli/​clusterrunner.go Removes obsolete single-active-runner switching logic.
internal/​cli/​clusterrunner_internal_test.go Updates cluster-runner coverage for instance semantics.
docs/​troubleshooting.md Documents instance logs and memory-admission failures.
docs/​reference/​cli/​up.md Documents instance startup and resource overrides.
docs/​reference/​cli/​status.md Documents multi-instance status.
docs/​reference/​cli/​serve.md Updates panel capabilities and limitations.
docs/​reference/​cli/​search.md Updates search-and-start terminology.
docs/​reference/​cli/​router/​uninstall-service.md Documents router service removal.
docs/​reference/​cli/​router/​serve.md Documents foreground router options.
docs/​reference/​cli/​router/​install-service.md Documents router service installation.
docs/​reference/​cli/​router.md Introduces model-aware routing commands.
docs/​reference/​cli/​restart.md Documents instance-specific restart.
docs/​reference/​cli/​recipes/​launch.md Documents recipe instance launches.
docs/​reference/​cli/​recipes/​install-service.md Documents profile-specific recipe services.
docs/​reference/​cli/​new.md Updates model-creation and startup semantics.
docs/​reference/​cli/​migrate.md Documents legacy migration.
docs/​reference/​cli/​logs.md Documents instance-specific logs.
docs/​reference/​cli/​down.md Documents instance shutdown.
docs/​reference/​cli/​default.md Documents default-instance selection.
docs/​reference/​cli.md Updates the generated command index.
docs/​recipes.md Updates recipe startup and instance-state terminology.
docs/​README.md Introduces instances and optional routing.
docs/​quickstart.md Introduces concurrent model startup and shutdown.
docs/​providers.md Updates provider workflows for concurrent serving.
docs/​profiles.md Explains applying changes to individual instances.
docs/​panel.md Documents instance controls and service permissions.
docs/​multi-node.md Documents recipe instances and per-profile services.
docs/​multi-instance.md Adds the concurrent-instance operations guide.
docs/​models.md Updates model-start and plugin workflows.
docs/​memory.md Clarifies profile-specific GPU selection.
docs/​manifest.json Registers new guides and command-reference pages.
docs/​macos.md Documents per-model launchd jobs and migration.
docs/​install.md Documents instance configuration and upgrades.
docs/​hardware.md Updates GPU-selection documentation.
docs/​coder.md Explains multi-model synchronization and routing.
CHANGELOG.md Records instance, router, migration, and compatibility changes.

Review effort: Balanced

Comment thread docs/panel.md Resolved
Comment thread internal/cli/recipelaunch.go Resolved
Comment thread internal/cli/recipeservice.go Resolved
Comment thread internal/cli/router.go Resolved
Comment thread internal/control/fit.go Resolved
Comment thread internal/cli/recipehealth.go Resolved
Comment thread internal/cli/router.go Resolved
Comment thread internal/control/fit.go Resolved
Comment thread internal/web/static/index.html Resolved
Comment thread docs/quickstart.md Resolved
@DevelopmentCats
DevelopmentCats merged commit 230d4e9 into main Nov 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run several models at once by managing one vLLM instance per model

2 participants