Repository navigation
instances: run several models at once - #63
Merged
Merged
Conversation
Every running model is an instance keyed by its profile, with its own port, unit (llm-helper@<profile>.service on Linux, one plist per instance on macOS) and log. Adds up, down, default, migrate and an opt-in model router; switch is deprecated. Coder sync offers every running instance.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical findings permit overly broad sudo access and unauthenticated router access, while launch and migration defects can disrupt running services or delete instance records.
12 open issues
Restrict sudoers rules to validated commands and arguments · New Check port conflicts before removing existing containers · New Migrate existing units to profile-specific commands · New Require router authentication before discovery or proxying · New Allow the default vllm-metal profile within the memory budget · New Authenticate backend discovery requests · New Serialize port allocation and instance reservation · New Scope watchdog markers to each profile · New Route remote recipe instances to their head endpoint · New Track memory limits per overlapping GPU · New Add an accessible label to the log-instance selector · New Remove the duplicate comma after “port” · New
What changed in this PR
Enables concurrent, per-profile model instances across the CLI, web panel, recipe services, and Coder integration, replacing the single-active-model workflow.
Changes:
- Adds instance-specific services, ports, logs, lifecycle commands, memory admission, and legacy migration.
- Introduces model-aware routing and synchronization of multiple running models with Coder.
- Updates panel controls, regression tests, guides, and generated CLI documentation.
This pull request requires changes. Unresolved security, routing, lifecycle, resource-allocation, integration, and accessibility findings block approval.
| File | Description |
|---|---|
README.md |
Introduces concurrent models and instance commands. |
internal/web/static/models.js |
Updates model actions for concurrent startup. |
internal/web/static/models.html |
Updates the model-start button label. |
internal/web/static/index.html |
Adds instance listings and log selection. |
internal/web/static/app.css |
Styles instance controls and log selection. |
internal/web/models.go |
Starts pulled models and protects running profiles. |
internal/web/docssite.go |
Updates the documentation-site summary. |
internal/web/activate.go |
Implements panel-driven instance startup. |
internal/service/service.go |
Adds service arguments and no-fit exit status. |
internal/service/service_other.go |
Adds unsupported-platform instance interfaces. |
internal/service/service_linux.go |
Implements systemd instance templates and removal. |
internal/service/service_linux_internal_test.go |
Tests instance templates and router units. |
internal/service/service_darwin.go |
Implements per-instance launchd jobs and removal. |
internal/service/service_darwin_internal_test.go |
Tests instance labels and launch arguments. |
internal/router/models.go |
Adds model discovery and request routing. |
internal/router/models_internal_test.go |
Tests discovery, routing, and backend failures. |
internal/profile/profile.go |
Validates service-compatible instance names. |
internal/profile/profile_test.go |
Tests instance-name validation. |
internal/plugin/registry_internal_test.go |
Updates lifecycle-event dispatch tests. |
internal/plugin/plugin.go |
Adds instance lifecycle events and live-instance state. |
internal/plugin/coder/sync.go |
Synchronizes reachable models and default selection. |
internal/plugin/coder/sync_internal_test.go |
Updates shared-entry ownership coverage. |
internal/engine/registry.go |
Updates environment-layering documentation. |
internal/control/instances_internal_test.go |
Tests instance state, defaults, and port allocation. |
internal/control/gpus.go |
Separates global GPU selection from profile selection. |
internal/control/gguf.go |
Clarifies plugin effects on running instances. |
internal/control/fit.go |
Adds aggregate memory-share admission checks. |
internal/control/fit_internal_test.go |
Tests memory admission limits and refusal messages. |
internal/control/control.go |
Adds instance-aware integration notifications. |
internal/config/envfile.go |
Updates environment-file documentation. |
internal/config/config.go |
Adds instance paths, defaults, and router settings. |
internal/config/config_test.go |
Tests instance paths and legacy state access. |
internal/cli/watch.go |
Watches each instance’s runner and endpoint. |
internal/cli/watch_internal_test.go |
Tests instance-specific boot targets. |
internal/cli/vllmargs.go |
Renames environment helpers for instance startup. |
internal/cli/vllmargs_internal_test.go |
Updates environment-helper tests. |
internal/cli/status_internal_test.go |
Updates crash-loop diagnostic expectations. |
internal/cli/start.go |
Adds instance startup and legacy compatibility. |
internal/cli/setup.go |
Defers service installation until instance startup. |
internal/cli/service.go |
Makes restart and logs instance-specific. |
internal/cli/serve.go |
Supplies instance-specific cluster lookups to the panel. |
internal/cli/search.go |
Starts selected models alongside existing instances. |
internal/cli/router.go |
Adds router serving and service management. |
internal/cli/router_internal_test.go |
Tests router configuration and service operations. |
internal/cli/root.go |
Registers instance commands and runner selection. |
internal/cli/root_internal_test.go |
Updates command-set and grouping expectations. |
internal/cli/recipeservice.go |
Generates profile-specific recipe service units. |
internal/cli/recipeservice_internal_test.go |
Tests profile-specific recipe units. |
internal/cli/recipes.go |
Registers recipe service lifecycle commands. |
internal/cli/recipelaunch.go |
Records recipe launches as instances. |
internal/cli/recipelaunch_internal_test.go |
Updates recipe launch assertions and terminology. |
internal/cli/recipehealth.go |
Adds profile-specific watchdog targeting. |
internal/cli/recipehealth_internal_test.go |
Tests profile-aware watchdog commands. |
internal/cli/plugins.go |
Removes the single-endpoint helper. |
internal/cli/memfit.go |
Shares the unified-memory limit with control. |
internal/cli/ggufplugin.go |
Updates plugin activation messaging. |
internal/cli/editset.go |
Gives instance-specific restart instructions. |
internal/cli/clusterrunner.go |
Removes obsolete single-active-runner switching logic. |
internal/cli/clusterrunner_internal_test.go |
Updates cluster-runner coverage for instance semantics. |
docs/troubleshooting.md |
Documents instance logs and memory-admission failures. |
docs/reference/cli/up.md |
Documents instance startup and resource overrides. |
docs/reference/cli/status.md |
Documents multi-instance status. |
docs/reference/cli/serve.md |
Updates panel capabilities and limitations. |
docs/reference/cli/search.md |
Updates search-and-start terminology. |
docs/reference/cli/router/uninstall-service.md |
Documents router service removal. |
docs/reference/cli/router/serve.md |
Documents foreground router options. |
docs/reference/cli/router/install-service.md |
Documents router service installation. |
docs/reference/cli/router.md |
Introduces model-aware routing commands. |
docs/reference/cli/restart.md |
Documents instance-specific restart. |
docs/reference/cli/recipes/launch.md |
Documents recipe instance launches. |
docs/reference/cli/recipes/install-service.md |
Documents profile-specific recipe services. |
docs/reference/cli/new.md |
Updates model-creation and startup semantics. |
docs/reference/cli/migrate.md |
Documents legacy migration. |
docs/reference/cli/logs.md |
Documents instance-specific logs. |
docs/reference/cli/down.md |
Documents instance shutdown. |
docs/reference/cli/default.md |
Documents default-instance selection. |
docs/reference/cli.md |
Updates the generated command index. |
docs/recipes.md |
Updates recipe startup and instance-state terminology. |
docs/README.md |
Introduces instances and optional routing. |
docs/quickstart.md |
Introduces concurrent model startup and shutdown. |
docs/providers.md |
Updates provider workflows for concurrent serving. |
docs/profiles.md |
Explains applying changes to individual instances. |
docs/panel.md |
Documents instance controls and service permissions. |
docs/multi-node.md |
Documents recipe instances and per-profile services. |
docs/multi-instance.md |
Adds the concurrent-instance operations guide. |
docs/models.md |
Updates model-start and plugin workflows. |
docs/memory.md |
Clarifies profile-specific GPU selection. |
docs/manifest.json |
Registers new guides and command-reference pages. |
docs/macos.md |
Documents per-model launchd jobs and migration. |
docs/install.md |
Documents instance configuration and upgrades. |
docs/hardware.md |
Updates GPU-selection documentation. |
docs/coder.md |
Explains multi-model synchronization and routing. |
CHANGELOG.md |
Records instance, router, migration, and compatibility changes. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #15.
Every running model is now an instance, keyed by its profile, with its own port, service and log.
llm-helper up <profile>starts one next to the ones already running, anddown <profile>stops it.What changed
llm-helper@.servicetemplate on Linux, and its own launchd job (com.coder.llm-helper.<profile>) on macOS.upchecks the model fits in the memory the running instances on the same GPUs leave (the Metal limit on Apple silicon), then takes the next free port fromPORTunless--portis given.--mem-utiland--kv-cachegive it a smaller share.Concurrent
upruns, orupand the panel, never pick the same port.default <profile>sets the model Coder selects first (DEFAULT_PROFILE).migratemoves a model started by an earlier release from the singlevllm.service(orcom.coder.vllm) unit to an instance on the same port.router(opt-in) serves every instance on one port and dispatches by the request'smodel, withinstall-service,serveanduninstall-service.stopis another name fordown.restart,logsandstatustake a profile name and act on the only instance when exactly one runs.switchis deprecated and hidden.new,searchand the panel's "Pull and run" start an instance next to the running ones instead of replacing them.Without the router only the one on
PORTis enabled, and sync names the ones it left out.setupno longer installs a service unit;upinstalls each instance's.New endpoints
GET /api/instancesandPOST /api/instances/{name}/up|down|restart|default;/api/logstakes?instance=.The sudoers line covers
llm-helper@*.service.Removed
state/active.env,state/active-profileand the single active profile.migrateconverts them./api/switch,/api/restartand/api/stop.Upgrading
The old unit keeps serving until
llm-helper migrateruns.doctorreports when it is still needed, andupand the panel refuse to start a model until then.Docs
New page "Several models at once" (
docs/multi-instance.md), and CLI reference pages forup,down,default,migrateandrouter.