diff --git a/CHANGELOG.md b/CHANGELOG.md index f80a506..2f604f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,35 @@ the fact. catalogue file. - `glm-5.3-flash-tf-ablit-2spark` recipe: the GLM-5.3 Flash TensorFold launch across two DGX Sparks with the abliterated checkpoint, thinking kept on. +- Several models at once: every running model is an instance with its own + port, service and log. `llm-helper up ` starts one next to the + others on the next free port from `PORT` (or `--port`), after checking it + fits in the memory the running ones on the same GPUs leave (on Apple + silicon, the Metal limit); `--mem-util` and `--kv-cache` give it a smaller + share. Two `up` runs at once, or `up` and the panel, never pick the same + port. `down ` stops one. On Linux each instance + is a unit of the `llm-helper@.service` template; on macOS each is its own + launchd job, `com.coder.llm-helper.`. After a reboot they start + oldest first, and one that no longer fits exits with status 78 instead of + loading. See the new "Several models at once" page. +- `llm-helper default ` picks the model Coder selects first, + recorded as `DEFAULT_PROFILE`. Without it, the instance started first is + the default. +- `llm-helper router`: an opt-in router that serves every instance on one + port (8000 by default) and dispatches by the request's `model`. + `router install-service` runs it as a service and records `ROUTER_PORT`; + point the Coder AI provider at it to offer every running model. With + `VLLM_API_KEY` set it requires that key as a bearer token, and a recipe + instance is reached on the machine that runs its head. +- `llm-helper migrate` moves a model started by an earlier release from the + single `vllm.service` (or `com.coder.vllm`) unit to an instance on the same + port. That unit keeps serving until `migrate` runs, `doctor` reports when it + is still needed, and `up` refuses to start a model until then. For a recipe + profile it also rewrites the recipe's service and health watchdog units. +- Panel: one row per instance with Restart, Stop and Make default, and Run + for saved profiles that are not running. New API: `GET /api/instances` and + `POST /api/instances/{name}/up|down|restart|default`; `/api/logs` takes + `?instance=`. ### Changed @@ -42,6 +71,30 @@ the fact. - The chat template fix for a system message after the first turn applies to Qwen3.5 repos (`qwen3_5` and `qwen3_5_moe`, which includes Ornith), not to any repo whose template contains the same text. +- `stop` is another name for `down`. `restart`, `logs` and `status` take a + profile name, and act on the only instance when exactly one runs; `status` + lists every instance. +- `switch` is deprecated and hidden: it stops every other instance and runs + the one named. Use `up` and `down`. It will be removed in a later release. +- `new`, `search` and the panel's "Pull and run" start the model as an + instance next to the running ones instead of replacing them. +- Coder sync offers every running instance. Without the router, only the one + on `PORT` is enabled, since the provider has one base URL, and sync names + the ones it left out. +- `setup` no longer installs a service unit; `up` installs each instance's. +- A running profile can't be removed; stop it first. +- The panel's sudoers line covers `llm-helper@*.service`, one unit per + command. +- The recipe health watchdog keeps its launch and stopped markers per + profile, so stopping or launching one recipe no longer pauses another's + watchdog. Markers left by an earlier release are ignored. + +### Removed + +- The single-model state: `state/active.env`, `state/active-profile` and the + idea of one active profile. `migrate` converts them. +- The panel API's `/api/switch`, `/api/restart` and `/api/stop`, replaced by + the per-instance endpoints. ### Fixed diff --git a/README.md b/README.md index 0635199..5a0d50f 100644 --- a/README.md +++ b/README.md @@ -3,29 +3,29 @@

llm-helper

-Serve large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and switch models with one command. +Serve large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and run several models at once. [Quickstart](#quickstart) | [Docs][docs] | [CLI reference][docs-cli-reference] | [Changelog](CHANGELOG.md) -llm-helper manages an OpenAI-compatible inference server on one machine or across several: [vLLM](https://github.com/vllm-project/vllm) by default, [mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple Silicon, or [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF models. Each model's working settings are saved as a profile, and switching to another profile restarts the server on it, whether that model runs on this machine alone or spans a group of them. It was built for unified-memory machines like the NVIDIA DGX Spark, where one machine holds one big model and a bigger model needs two or more working together. It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on the GPU through [vllm-metal](https://github.com/vllm-project/vllm-metal), mlx-lm or llama.cpp. +llm-helper manages an OpenAI-compatible inference server on one machine or across several: [vLLM](https://github.com/vllm-project/vllm) by default, [mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple Silicon, or [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF models. Each model's working settings are saved as a profile, and every profile you start runs as its own server on its own port, whether that model runs on this machine alone or spans a group of them. It was built for unified-memory machines like the NVIDIA DGX Spark, where one machine holds one big model and a bigger model needs two or more working together. It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on the GPU through [vllm-metal](https://github.com/vllm-project/vllm-metal), mlx-lm or llama.cpp. llm-helper was called vllm-helper until it served more than vLLM. The old command name, config directory and service names all keep working; see [Providers][docs-providers] for what changes per server. -- Save a model's serving settings as a profile, and switch between profiles with one command +- Save a model's serving settings as a profile, and run as many profiles at once as the memory holds, each with one command - Search HuggingFace with the picked model's weight size checked against the machine's memory, and vLLM flags read from its model card - Adopt a vLLM server that is already running, without restarting it - Get machine details, like how a GPU's memory is shared, from a hardware catalogue, and record a new one with `hardware report` - Register other machines once, then launch tested recipes across them: configurations run and measured on named hardware, pinned by image digest - Pool replicas of one model behind a session-sticky router, or shard a model too big for one machine across several - Control a machine from a web panel that also serves these docs -- Keep a [Coder](https://coder.com) deployment's model picker in step with whatever is loaded +- Keep a [Coder](https://coder.com) deployment's model picker in step with the models running > [!NOTE] > llm-helper is experimental. It is a useful way to launch local LLMs, but use it on dedicated hardware that runs only your models, on a safe network. > -> A bug in llm-helper can affect what it controls: the service that setup installs, the server restart when you switch profiles, the web panel, and the model list that it syncs to Coder. A bug cannot break a request in flight, because llm-helper is a control plane and inference traffic goes straight to the inference server on its own port. The one exception is a pool, because its router sits in the request path. +> A bug in llm-helper can affect what it controls: the service units it installs, the server restarts when you start and stop models, the web panel, and the model list that it syncs to Coder. A bug cannot break a request in flight, because llm-helper is a control plane and inference traffic goes straight to the inference server on its own port. The exceptions are a pool and the optional router that offers every running model through one port, because each sits in the request path. ## Quickstart @@ -40,17 +40,17 @@ sudo ln -sf llm-helper /usr/local/bin/vllm-helper # the old name, for scripts llm-helper setup # or: --provider mlx, --provider llamacpp ``` -Setup detects the hardware and the server, generates an API key where the server takes one, writes config, and installs a service unit (systemd on Linux, launchd on macOS). It never overwrites existing config or restarts a running server. If `vllm` isn't on your `PATH`, set `VLLM_BIN` in `~/.config/vllm-helper/config.env`. +Setup detects the hardware and the server, generates an API key where the server takes one, and writes config. Each model you start runs as its own service (systemd on Linux, launchd on macOS). It never overwrites existing config or restarts a running server. If `vllm` isn't on your `PATH`, set `VLLM_BIN` in `~/.config/vllm-helper/config.env`. Pull a model and serve it, or take over a server that is already running: ```shell -llm-helper new Qwen/Qwen3-32B # save a profile and switch to it +llm-helper new Qwen/Qwen3-32B # save a profile and start serving it llm-helper adopt # or: manage the vLLM server already running here -llm-helper status # service state, active model, endpoint health +llm-helper status # running models, their ports, endpoint health ``` -Point any OpenAI-compatible client at `http://:8001/v1` with the `VLLM_API_KEY` from `~/.config/vllm-helper/secrets.env`. A first switch downloads the weights, so it takes as long as the download does. +Point any OpenAI-compatible client at `http://:8001/v1` with the `VLLM_API_KEY` from `~/.config/vllm-helper/secrets.env`. A second model started with `llm-helper up ` takes the next free port; [Several models at once][docs-multi-instance] covers ports, memory and the router. A first start downloads the weights, so it takes as long as the download does. To serve across machines, register every machine a recipe will use on the one that will serve the API, that one included, then launch. llm-helper drives the other nodes over SSH and runs the recipe's container on each with Docker: @@ -65,31 +65,32 @@ llm-helper recipes launch --nodes spark1,spark2 ## How it works -| Piece | What it is | Where it lives | -| ------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------- | -| Config | Settings for every profile: port, `VLLM_BIN`, service name, Coder sync | `~/.config/vllm-helper/config.env` | -| Secrets | The endpoint's API key, and tokens for Coder and HuggingFace | `~/.config/vllm-helper/secrets.env` | -| Profile | One model's serving settings, as a small env file | `~/.config/vllm-helper/models/.env` | -| Runner | What serves the active profile: the native service unit, or recipe containers | `vllm.service` / `com.coder.vllm`, `vllm-cluster.service` | -| Node registry | Other machines, by name, for recipes, pools and shards | `~/.config/vllm-helper/nodes.json` | -| Endpoint | The engine's OpenAI-compatible API | port `8001` by default | +| Piece | What it is | Where it lives | +| ------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | +| Config | Settings for every profile: first port, `VLLM_BIN`, service scope, Coder sync | `~/.config/vllm-helper/config.env` | +| Secrets | The endpoint's API key, and tokens for Coder and HuggingFace | `~/.config/vllm-helper/secrets.env` | +| Profile | One model's serving settings, as a small env file | `~/.config/vllm-helper/models/.env` | +| Instance | A running profile, served by its own service unit or by recipe containers | `llm-helper@.service` / `com.coder.llm-helper.`, `vllm-cluster.service` | +| Node registry | Other machines, by name, for recipes, pools and shards | `~/.config/vllm-helper/nodes.json` | +| Endpoint | The engine's OpenAI-compatible API, one per instance | ports from `8001` up by default | -`restart`, `stop`, `status` and `logs` act on whichever runner the active profile belongs to, and `switch` stops the outgoing runner before starting the new one, so they work the same for a native model and for a multi-machine recipe. +`up`, `down`, `restart`, `status` and `logs` act on whichever runner an instance belongs to, so they work the same for a native model and for a multi-machine recipe. Everyday commands: ```shell llm-helper list # saved profiles -llm-helper switch # load a different model +llm-helper up # serve another model beside the running ones +llm-helper down # stop one llm-helper search # find a model that fits this machine llm-helper set K=V # change a profile setting -llm-helper logs # follow the server log +llm-helper logs # follow a model's server log llm-helper doctor # hardware, running servers, integrations, drift llm-helper serve # web panel, with these docs under Docs llm-helper recipes list # tested configurations, and what verified them ``` -`status`, `list`, `search`, `inspect`, `doctor` and `recipes list` take `--json`, and `switch`, `new` and `restart` take `--wait`, so scripts and agents can drive llm-helper without parsing prose. +`status`, `list`, `search`, `inspect`, `doctor` and `recipes list` take `--json`, and `up`, `new` and `restart` take `--wait`, so scripts and agents can drive llm-helper without parsing prose. ## Documentation @@ -97,6 +98,7 @@ The docs are compiled into the binary: run `llm-helper serve` and open the **Doc - [**Quickstart**][docs-quickstart]: From nothing to serving a model - [**Profiles**][docs-profiles]: What a profile holds, and how to edit one +- [**Several models at once**][docs-multi-instance]: Ports, memory, and one port for every model - [**Providers**][docs-providers]: vLLM, mlx-lm, llama.cpp, and what each one changes - [**Models and quantizations**][docs-models]: Finding a model that fits, and choosing a quant - [**Memory budget**][docs-memory]: How llm-helper decides what this machine can hold @@ -141,6 +143,7 @@ Notable changes go in [CHANGELOG.md](CHANGELOG.md). Maintainers cutting a releas [docs-multi-node]: https://vllm-helper-docs.up.railway.app/docs/multi-node [docs-quickstart]: https://vllm-helper-docs.up.railway.app/docs/quickstart [docs-profiles]: https://vllm-helper-docs.up.railway.app/docs/profiles +[docs-multi-instance]: https://vllm-helper-docs.up.railway.app/docs/multi-instance [docs-providers]: https://vllm-helper-docs.up.railway.app/docs/providers [docs-models]: https://vllm-helper-docs.up.railway.app/docs/models [docs-memory]: https://vllm-helper-docs.up.railway.app/docs/memory diff --git a/docs/README.md b/docs/README.md index b7c8ea24..3891f9e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,7 +3,7 @@ title: Introduction description: What llm-helper is, and when to reach for it. --- -llm-helper serves large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and switches models with one command. +llm-helper serves large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and runs one model or several at once. It wraps an OpenAI-compatible inference server, [vLLM](https://github.com/vllm-project/vllm) by default, [mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple Silicon or [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF models, with saved model profiles, HuggingFace search, tested multi-machine recipes, and an optional [Coder](https://coder.com) Agents sync. It was called vllm-helper until it served more than vLLM, and the old name still works. @@ -13,14 +13,14 @@ It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on t ## How it fits together - **A profile** is a small env file holding the settings that serve one model, for the provider the profile names. -- **A service unit** runs the active profile: a systemd unit on Linux, a launchd job on macOS. -- **The CLI** switches profiles, pulls models, and diagnoses the machine. +- **An instance** is one running model: a profile served by its own service unit, a systemd unit on Linux or a launchd job on macOS. +- **The CLI** starts and stops instances, pulls models, and diagnoses the machine. - **The web panel** is what `llm-helper serve` serves: the same controls in a browser. -- **The endpoint** is the inference server's own OpenAI-compatible API, on port 8001 by default. +- **The endpoint** is the inference server's own OpenAI-compatible API, one per instance, on port 8001 and up. Inference traffic goes straight to the inference server on its own port. llm-helper is a control plane, so nothing here sits in the path of a request — a bug in it cannot break inference. -The one exception is a [pool](./multi-node.md), whose router sits in front of the replicas by design. +The exceptions are opt-in: the [model router](./multi-instance.md), which gives several instances one address, and a [pool](./multi-node.md), whose router sits in front of the replicas by design. The binary is self-contained: one file to put on a machine, with the docs you are reading compiled in. @@ -28,6 +28,7 @@ The binary is self-contained: one file to put on a machine, with the docs you ar - Follow the [Quickstart](./quickstart.md) to set the machine up and serve a first model. - See what a profile holds in [Profiles](./profiles.md). +- Run several models side by side with [Several models at once](./multi-instance.md). - Serve with mlx-lm or llama.cpp instead of vLLM with [Providers](./providers.md). - Serve a model measured across several machines with [Recipes](./recipes.md). - Pool replicas or shard one model across machines with [More than one machine](./multi-node.md). diff --git a/docs/coder.md b/docs/coder.md index 2b0bc6f..229a8dc 100644 --- a/docs/coder.md +++ b/docs/coder.md @@ -3,7 +3,7 @@ title: Coder integration description: Keep a Coder deployment's model picker truthful. --- -If this machine backs an `openai-compat` AI provider in a [Coder](https://coder.com) deployment, llm-helper can keep the deployment's model picker truthful: every profile gets a model-config entry, and only the loaded model is enabled. +If this machine backs an `openai-compat` AI provider in a [Coder](https://coder.com) deployment, llm-helper can keep the deployment's model picker truthful: every profile gets a model-config entry, and only the models running here are enabled. ## Configure it @@ -11,13 +11,22 @@ If this machine backs an `openai-compat` AI provider in a [Coder](https://coder. 2. Set `CODER_API_TOKEN` in `~/.config/vllm-helper/secrets.env`. 3. On a deployment with more than one organization, set `CODER_ORGANIZATION` in `config.env` to yours. It defaults to `default`. -Sync failures never block a local switch: a deployment that cannot be reached is reported, and the model loads anyway. +Sync failures never block a model from starting: a deployment that cannot be reached is reported, and the model loads anyway. ## What syncs when -Every switch updates the deployment. +Starting or stopping a model updates the deployment, and so does choosing a new default. Removing a profile removes its entry. +## Several models + +A Coder AI provider has one base URL, so it reaches one port. +Without the [model router](./multi-instance.md), only the model on `PORT` is enabled in the picker, and sync names the others it left out. +With the router installed, point the provider's base URL at the router's port, and every running model is enabled. + +The default model, the one Coder selects first, is the profile set with `llm-helper default `. +When that profile isn't running, the model that has been running longest is the default. + `llm-helper coder-sync` also takes verbs of its own: ```sh @@ -43,10 +52,6 @@ Coder caps a reply at its own server-side default of 32,000 tokens. Set `CODER_MAX_OUTPUT_TOKENS` in `config.env` to raise it for the models this machine serves. Left unset, llm-helper sends nothing, so whatever is configured in Coder stays. -## One note on stopping - -Stopping the server with `llm-helper stop` does not disable the active model in Coder's picker — it stays enabled until the next switch. - ## Next steps - [Profiles](./profiles.md) — what gets an entry. diff --git a/docs/hardware.md b/docs/hardware.md index 376ba53..35e5257 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -11,7 +11,7 @@ An NVIDIA GPU with no definition still works: its memory comes from `nvidia-smi` - **NVIDIA GPUs** — every GPU `nvidia-smi` lists, with its memory, compute capability and addressing mode. `nvidia-smi` gets 10 seconds; a wedged driver is treated as `nvidia-smi` failing, rather than hanging the command. -- **`CUDA_VISIBLE_DEVICES`** — from the environment and from `config.env`, `secrets.env` and the active profile, so the GPUs sized are the ones the server will get. +- **`CUDA_VISIBLE_DEVICES`** — from the environment and from `config.env`, `secrets.env` and the profile being served, so the GPUs sized are the ones the server will get. - **Jetson markers** — `/etc/nv_tegra_release`, `/sys/class/tegra-firmware` and the device-tree model. - **The PCI bus** — every display controller from a known vendor, and every processing accelerator, so an AMD GPU or an NPU is named even though llm-helper does not size for it yet. - **The cgroup memory limit** — in a container or a limited systemd unit, the budget is held to the limit rather than the host's RAM. diff --git a/docs/install.md b/docs/install.md index a2cefbb..55682a0 100644 --- a/docs/install.md +++ b/docs/install.md @@ -28,7 +28,7 @@ If `vllm` isn't on your PATH, set `VLLM_BIN` in `~/.config/vllm-helper/config.en `setup` writes two files under `~/.config/vllm-helper/`: -- **config.env** — settings that apply to every profile, sourced into the server's environment on every switch. +- **config.env** — settings that apply to every profile, sourced into the server's environment whenever a model starts. - **secrets.env** — credentials, generated or filled in by you. Never commit this file anywhere. The same directory can also hold **hardware.d/**, which `setup` does not create: local hardware definitions for a machine llm-helper does not recognize yet. @@ -36,13 +36,15 @@ See [Hardware detection](./hardware.md). | Key | Where | Meaning | | ------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -| `PORT` | config.env | Port the OpenAI-compatible endpoint listens on. Default `8001`. | +| `PORT` | config.env | First port a model listens on. Each model running at the same time takes the next free port. Default `8001`. | | `PROVIDER` | config.env | The inference server new profiles are made for: `vllm` (default), `mlx` or `llamacpp`. See [Providers](./providers.md). | | `VLLM_BIN` | config.env | Full path to the `vllm` executable. Required for vLLM profiles. | | `MLX_LM_BIN` | config.env | Full path to the `mlx_lm.server` executable. Required for MLX profiles. | | `LLAMA_SERVER_BIN` | config.env | Full path to the `llama-server` executable. Required for llama.cpp profiles. | -| `SERVICE_NAME` | config.env | Service unit name: `vllm.service` on Linux, `com.coder.vllm` on macOS. | +| `SERVICE_NAME` | config.env | The single service unit earlier releases ran: `vllm.service` on Linux, `com.coder.vllm` on macOS. Read only by `migrate`. | | `SERVICE_SCOPE` | config.env | `system` on Linux (needs sudo to control) or `user`. macOS defaults to `user`; use `system` on a headless Mac. | +| `DEFAULT_PROFILE` | config.env | The model Coder selects first. Written by `llm-helper default`. See [Several models at once](./multi-instance.md). | +| `ROUTER_PORT` | config.env | Port the model router listens on. Written by `llm-helper router install-service`. | | `MODEL_BUDGET_GB` | config.env | Overrides the measured model budget. See [Memory budget](./memory.md). | | `KV_CONTEXT_TARGET` | config.env | How many full-length requests each profile's KV cache is sized to hold. Default `4`. | | `METAL_WIRED_LIMIT_MB` | config.env | macOS only. GPU wired memory limit in MiB, applied at boot. Written by `llm-helper metal-limit --set`. See [Running on macOS](./macos.md). | @@ -81,13 +83,15 @@ On the dev channel every build shares one moving release tag, so the commit is w ## Upgrading from vllm-helper llm-helper is vllm-helper under a new name, and an upgrade from vllm-helper changes nothing on disk. -`vllm-helper upgrade` replaces the binary where it is, and the old name keeps working: config and state stay in `~/.config/vllm-helper` and `~/.local/state/vllm-helper`, the `VLLM_HELPER_*` overrides are read as before, and the service is still `vllm.service` or `com.coder.vllm`, running the same path. +`vllm-helper upgrade` replaces the binary where it is, and the old name keeps working: config and state stay in `~/.config/vllm-helper` and `~/.local/state/vllm-helper`, and the `VLLM_HELPER_*` overrides are read as before. Commands now print the new name in their hints, so link it beside the old one: ```sh sudo ln -s vllm-helper "$(dirname "$(command -v vllm-helper)")/llm-helper" ``` +The single service unit from earlier releases, `vllm.service` or `com.coder.vllm`, keeps running until you run `llm-helper migrate`, which moves the model it serves to its own [instance](./multi-instance.md). + ## Next steps - Follow the [Quickstart](./quickstart.md) to serve a first model. diff --git a/docs/macos.md b/docs/macos.md index 1f4cac7..97c6f1b 100644 --- a/docs/macos.md +++ b/docs/macos.md @@ -84,35 +84,29 @@ Mainline llama.cpp does run it on the GPU, from Unsloth's GGUF. - **`doctor` skips the build toolchain.** vllm-metal ships prebuilt wheels and Metal kernels, so there is nothing to compile. - **Logs are a file, not a journal.** - The launchd job writes to a log file; see [Logs](#logs). + Each model's launchd job writes to a log file; see [Logs](#logs). ## Run on a headless Mac -By default the service is a LaunchAgent in `~/Library/LaunchAgents`, so controlling it needs no root. +Each running model is its own launchd job, labelled `com.coder.llm-helper.`. +By default each is a LaunchAgent in `~/Library/LaunchAgents`, so controlling it needs no root. A LaunchAgent needs a GUI login session. Over SSH on a Mac nobody is logged in to, loading it fails with launchd error 125, and llm-helper says why. On a headless or server Mac, run it as a LaunchDaemon instead: -1. Set `SERVICE_SCOPE=system` in `~/.config/vllm-helper/config.env`. -2. Run `llm-helper setup` again. +1. Stop any model that is running with `llm-helper down `. +2. Set `SERVICE_SCOPE=system` in `~/.config/vllm-helper/config.env`. +3. Run `llm-helper up ` for each model. -That installs `/Library/LaunchDaemons/com.coder.vllm.plist`. +`up` installs `/Library/LaunchDaemons/com.coder.llm-helper..plist`. It starts at boot and runs as the user who installed it, so it reads that user's config and model cache. `llm-helper status` and `doctor` read its state without sudo. -Starting, stopping, restarting, and switching it need sudo, as system scope does on Linux. +Starting, stopping and restarting a model need sudo, as system scope does on Linux. -A LaunchDaemon installed by vllm-helper 0.5.0 or earlier runs as root, and `setup` keeps a plist that already exists. -To replace it with one that runs as you, unload and remove it, then run setup again: - -```sh -sudo launchctl bootout system/com.coder.vllm -sudo rm /Library/LaunchDaemons/com.coder.vllm.plist -llm-helper setup -``` - -Use your `SERVICE_NAME` in place of `com.coder.vllm` if you changed it. -The server is down from the `bootout` until setup loads the new daemon. +Earlier releases ran one job, `com.coder.vllm`, for whichever model was active. +`llm-helper migrate` moves that model to its own job and removes the old plist. +A LaunchDaemon installed by vllm-helper 0.5.0 or earlier runs as root; the job `migrate` installs runs as you. ## Give the GPU more memory @@ -123,7 +117,7 @@ The suggestion leaves 16 GiB for macOS on a Mac with 128 GiB or more, and 8 GiB ```sh llm-helper metal-limit # what it is now, and a suggestion llm-helper metal-limit --set auto # apply the suggestion, or pass a size in MiB -llm-helper restart # vllm-metal reads the working set at startup +llm-helper restart # vllm-metal reads the working set at startup llm-helper metal-limit --reset # back to the macOS default ``` @@ -146,19 +140,19 @@ What happens to the KV cache depends on the vllm-metal build: Planning again only changes a cache that was cut short to fit; one that already holds `KV_CONTEXT_TARGET` full contexts comes out the same. To use the extra room in the cache, either set the size yourself with `llm-helper set KV_CACHE_MEMORY_BYTES=`, or plan the profile again. `llm-helper new ` reuses a profile that already exists, so remove it first with `llm-helper remove `. - The active profile cannot be removed, so switch to another one first. + A running profile cannot be removed, so stop it with `llm-helper down ` first. A higher limit gives the GPU more of the RAM and leaves less headroom for macOS. That is a good trade on a dedicated server, and a worse one on a desktop you also work on. ## Logs -The launchd job writes to `