Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,35 @@ the fact.
catalogue file.
- `glm-5.3-flash-tf-ablit-2spark` recipe: the GLM-5.3 Flash TensorFold launch
across two DGX Sparks with the abliterated checkpoint, thinking kept on.
- Several models at once: every running model is an instance with its own
port, service and log. `llm-helper up <profile>` starts one next to the
others on the next free port from `PORT` (or `--port`), after checking it
fits in the memory the running ones on the same GPUs leave (on Apple
silicon, the Metal limit); `--mem-util` and `--kv-cache` give it a smaller
share. Two `up` runs at once, or `up` and the panel, never pick the same
port. `down <profile>` stops one. On Linux each instance
is a unit of the `llm-helper@.service` template; on macOS each is its own
launchd job, `com.coder.llm-helper.<profile>`. After a reboot they start
oldest first, and one that no longer fits exits with status 78 instead of
loading. See the new "Several models at once" page.
- `llm-helper default <profile>` picks the model Coder selects first,
recorded as `DEFAULT_PROFILE`. Without it, the instance started first is
the default.
- `llm-helper router`: an opt-in router that serves every instance on one
port (8000 by default) and dispatches by the request's `model`.
`router install-service` runs it as a service and records `ROUTER_PORT`;
point the Coder AI provider at it to offer every running model. With
`VLLM_API_KEY` set it requires that key as a bearer token, and a recipe
instance is reached on the machine that runs its head.
- `llm-helper migrate` moves a model started by an earlier release from the
single `vllm.service` (or `com.coder.vllm`) unit to an instance on the same
port. That unit keeps serving until `migrate` runs, `doctor` reports when it
is still needed, and `up` refuses to start a model until then. For a recipe
profile it also rewrites the recipe's service and health watchdog units.
- Panel: one row per instance with Restart, Stop and Make default, and Run
for saved profiles that are not running. New API: `GET /api/instances` and
`POST /api/instances/{name}/up|down|restart|default`; `/api/logs` takes
`?instance=`.

### Changed

Expand All @@ -42,6 +71,30 @@ the fact.
- The chat template fix for a system message after the first turn applies to
Qwen3.5 repos (`qwen3_5` and `qwen3_5_moe`, which includes Ornith), not to
any repo whose template contains the same text.
- `stop` is another name for `down`. `restart`, `logs` and `status` take a
profile name, and act on the only instance when exactly one runs; `status`
lists every instance.
- `switch` is deprecated and hidden: it stops every other instance and runs
the one named. Use `up` and `down`. It will be removed in a later release.
- `new`, `search` and the panel's "Pull and run" start the model as an
instance next to the running ones instead of replacing them.
- Coder sync offers every running instance. Without the router, only the one
on `PORT` is enabled, since the provider has one base URL, and sync names
the ones it left out.
- `setup` no longer installs a service unit; `up` installs each instance's.
- A running profile can't be removed; stop it first.
- The panel's sudoers line covers `llm-helper@*.service`, one unit per
command.
- The recipe health watchdog keeps its launch and stopped markers per
profile, so stopping or launching one recipe no longer pauses another's
watchdog. Markers left by an earlier release are ignored.

### Removed

- The single-model state: `state/active.env`, `state/active-profile` and the
idea of one active profile. `migrate` converts them.
- The panel API's `/api/switch`, `/api/restart` and `/api/stop`, replaced by
the per-instance endpoints.

### Fixed

Expand Down
45 changes: 24 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,29 @@

<h1>llm-helper</h1>

Serve large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and switch models with one command.
Serve large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and run several models at once.

[Quickstart](#quickstart) | [Docs][docs] | [CLI reference][docs-cli-reference] | [Changelog](CHANGELOG.md)

</div>

llm-helper manages an OpenAI-compatible inference server on one machine or across several: [vLLM](https://github.com/vllm-project/vllm) by default, [mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple Silicon, or [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF models. Each model's working settings are saved as a profile, and switching to another profile restarts the server on it, whether that model runs on this machine alone or spans a group of them. It was built for unified-memory machines like the NVIDIA DGX Spark, where one machine holds one big model and a bigger model needs two or more working together. It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on the GPU through [vllm-metal](https://github.com/vllm-project/vllm-metal), mlx-lm or llama.cpp.
llm-helper manages an OpenAI-compatible inference server on one machine or across several: [vLLM](https://github.com/vllm-project/vllm) by default, [mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple Silicon, or [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF models. Each model's working settings are saved as a profile, and every profile you start runs as its own server on its own port, whether that model runs on this machine alone or spans a group of them. It was built for unified-memory machines like the NVIDIA DGX Spark, where one machine holds one big model and a bigger model needs two or more working together. It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on the GPU through [vllm-metal](https://github.com/vllm-project/vllm-metal), mlx-lm or llama.cpp.

llm-helper was called vllm-helper until it served more than vLLM. The old command name, config directory and service names all keep working; see [Providers][docs-providers] for what changes per server.

- Save a model's serving settings as a profile, and switch between profiles with one command
- Save a model's serving settings as a profile, and run as many profiles at once as the memory holds, each with one command
- Search HuggingFace with the picked model's weight size checked against the machine's memory, and vLLM flags read from its model card
- Adopt a vLLM server that is already running, without restarting it
- Get machine details, like how a GPU's memory is shared, from a hardware catalogue, and record a new one with `hardware report`
- Register other machines once, then launch tested recipes across them: configurations run and measured on named hardware, pinned by image digest
- Pool replicas of one model behind a session-sticky router, or shard a model too big for one machine across several
- Control a machine from a web panel that also serves these docs
- Keep a [Coder](https://coder.com) deployment's model picker in step with whatever is loaded
- Keep a [Coder](https://coder.com) deployment's model picker in step with the models running

> [!NOTE]
> llm-helper is experimental. It is a useful way to launch local LLMs, but use it on dedicated hardware that runs only your models, on a safe network.
>
> A bug in llm-helper can affect what it controls: the service that setup installs, the server restart when you switch profiles, the web panel, and the model list that it syncs to Coder. A bug cannot break a request in flight, because llm-helper is a control plane and inference traffic goes straight to the inference server on its own port. The one exception is a pool, because its router sits in the request path.
> A bug in llm-helper can affect what it controls: the service units it installs, the server restarts when you start and stop models, the web panel, and the model list that it syncs to Coder. A bug cannot break a request in flight, because llm-helper is a control plane and inference traffic goes straight to the inference server on its own port. The exceptions are a pool and the optional router that offers every running model through one port, because each sits in the request path.

## Quickstart

Expand All @@ -40,17 +40,17 @@ sudo ln -sf llm-helper /usr/local/bin/vllm-helper # the old name, for scripts
llm-helper setup # or: --provider mlx, --provider llamacpp
```

Setup detects the hardware and the server, generates an API key where the server takes one, writes config, and installs a service unit (systemd on Linux, launchd on macOS). It never overwrites existing config or restarts a running server. If `vllm` isn't on your `PATH`, set `VLLM_BIN` in `~/.config/vllm-helper/config.env`.
Setup detects the hardware and the server, generates an API key where the server takes one, and writes config. Each model you start runs as its own service (systemd on Linux, launchd on macOS). It never overwrites existing config or restarts a running server. If `vllm` isn't on your `PATH`, set `VLLM_BIN` in `~/.config/vllm-helper/config.env`.

Pull a model and serve it, or take over a server that is already running:

```shell
llm-helper new Qwen/Qwen3-32B # save a profile and switch to it
llm-helper new Qwen/Qwen3-32B # save a profile and start serving it
llm-helper adopt # or: manage the vLLM server already running here
llm-helper status # service state, active model, endpoint health
llm-helper status # running models, their ports, endpoint health
```

Point any OpenAI-compatible client at `http://<machine>:8001/v1` with the `VLLM_API_KEY` from `~/.config/vllm-helper/secrets.env`. A first switch downloads the weights, so it takes as long as the download does.
Point any OpenAI-compatible client at `http://<machine>:8001/v1` with the `VLLM_API_KEY` from `~/.config/vllm-helper/secrets.env`. A second model started with `llm-helper up <profile>` takes the next free port; [Several models at once][docs-multi-instance] covers ports, memory and the router. A first start downloads the weights, so it takes as long as the download does.

To serve across machines, register every machine a recipe will use on the one that will serve the API, that one included, then launch. llm-helper drives the other nodes over SSH and runs the recipe's container on each with Docker:

Expand All @@ -65,38 +65,40 @@ llm-helper recipes launch <id> --nodes spark1,spark2

## How it works

| Piece | What it is | Where it lives |
| ------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------- |
| Config | Settings for every profile: port, `VLLM_BIN`, service name, Coder sync | `~/.config/vllm-helper/config.env` |
| Secrets | The endpoint's API key, and tokens for Coder and HuggingFace | `~/.config/vllm-helper/secrets.env` |
| Profile | One model's serving settings, as a small env file | `~/.config/vllm-helper/models/<name>.env` |
| Runner | What serves the active profile: the native service unit, or recipe containers | `vllm.service` / `com.coder.vllm`, `vllm-cluster.service` |
| Node registry | Other machines, by name, for recipes, pools and shards | `~/.config/vllm-helper/nodes.json` |
| Endpoint | The engine's OpenAI-compatible API | port `8001` by default |
| Piece | What it is | Where it lives |
| ------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Config | Settings for every profile: first port, `VLLM_BIN`, service scope, Coder sync | `~/.config/vllm-helper/config.env` |
| Secrets | The endpoint's API key, and tokens for Coder and HuggingFace | `~/.config/vllm-helper/secrets.env` |
| Profile | One model's serving settings, as a small env file | `~/.config/vllm-helper/models/<name>.env` |
| Instance | A running profile, served by its own service unit or by recipe containers | `llm-helper@<profile>.service` / `com.coder.llm-helper.<profile>`, `vllm-cluster.service` |
| Node registry | Other machines, by name, for recipes, pools and shards | `~/.config/vllm-helper/nodes.json` |
| Endpoint | The engine's OpenAI-compatible API, one per instance | ports from `8001` up by default |

`restart`, `stop`, `status` and `logs` act on whichever runner the active profile belongs to, and `switch` stops the outgoing runner before starting the new one, so they work the same for a native model and for a multi-machine recipe.
`up`, `down`, `restart`, `status` and `logs` act on whichever runner an instance belongs to, so they work the same for a native model and for a multi-machine recipe.

Everyday commands:

```shell
llm-helper list # saved profiles
llm-helper switch <profile> # load a different model
llm-helper up <profile> # serve another model beside the running ones
llm-helper down <profile> # stop one
llm-helper search <query> # find a model that fits this machine
llm-helper set <profile> K=V # change a profile setting
llm-helper logs # follow the server log
llm-helper logs <profile> # follow a model's server log
llm-helper doctor # hardware, running servers, integrations, drift
llm-helper serve # web panel, with these docs under Docs
llm-helper recipes list # tested configurations, and what verified them
```

`status`, `list`, `search`, `inspect`, `doctor` and `recipes list` take `--json`, and `switch`, `new` and `restart` take `--wait`, so scripts and agents can drive llm-helper without parsing prose.
`status`, `list`, `search`, `inspect`, `doctor` and `recipes list` take `--json`, and `up`, `new` and `restart` take `--wait`, so scripts and agents can drive llm-helper without parsing prose.

## Documentation

The docs are compiled into the binary: run `llm-helper serve` and open the **Docs** tab. The same pages are on the [docs site][docs], and their Markdown source is in [`docs/`](docs/README.md):

- [**Quickstart**][docs-quickstart]: From nothing to serving a model
- [**Profiles**][docs-profiles]: What a profile holds, and how to edit one
- [**Several models at once**][docs-multi-instance]: Ports, memory, and one port for every model
- [**Providers**][docs-providers]: vLLM, mlx-lm, llama.cpp, and what each one changes
- [**Models and quantizations**][docs-models]: Finding a model that fits, and choosing a quant
- [**Memory budget**][docs-memory]: How llm-helper decides what this machine can hold
Expand Down Expand Up @@ -141,6 +143,7 @@ Notable changes go in [CHANGELOG.md](CHANGELOG.md). Maintainers cutting a releas
[docs-multi-node]: https://vllm-helper-docs.up.railway.app/docs/multi-node
[docs-quickstart]: https://vllm-helper-docs.up.railway.app/docs/quickstart
[docs-profiles]: https://vllm-helper-docs.up.railway.app/docs/profiles
[docs-multi-instance]: https://vllm-helper-docs.up.railway.app/docs/multi-instance
[docs-providers]: https://vllm-helper-docs.up.railway.app/docs/providers
[docs-models]: https://vllm-helper-docs.up.railway.app/docs/models
[docs-memory]: https://vllm-helper-docs.up.railway.app/docs/memory
Expand Down
11 changes: 6 additions & 5 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Introduction
description: What llm-helper is, and when to reach for it.
---

llm-helper serves large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and switches models with one command.
llm-helper serves large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and runs one model or several at once.
It wraps an OpenAI-compatible inference server, [vLLM](https://github.com/vllm-project/vllm) by default, [mlx-lm](https://github.com/ml-explore/mlx-lm) on Apple Silicon or [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF models, with saved model profiles, HuggingFace search, tested multi-machine recipes, and an optional [Coder](https://coder.com) Agents sync.
It was called vllm-helper until it served more than vLLM, and the old name still works.

Expand All @@ -13,21 +13,22 @@ It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on t
## How it fits together

- **A profile** is a small env file holding the settings that serve one model, for the provider the profile names.
- **A service unit** runs the active profile: a systemd unit on Linux, a launchd job on macOS.
- **The CLI** switches profiles, pulls models, and diagnoses the machine.
- **An instance** is one running model: a profile served by its own service unit, a systemd unit on Linux or a launchd job on macOS.
- **The CLI** starts and stops instances, pulls models, and diagnoses the machine.
- **The web panel** is what `llm-helper serve` serves: the same controls in a browser.
- **The endpoint** is the inference server's own OpenAI-compatible API, on port 8001 by default.
- **The endpoint** is the inference server's own OpenAI-compatible API, one per instance, on port 8001 and up.

Inference traffic goes straight to the inference server on its own port.
llm-helper is a control plane, so nothing here sits in the path of a request — a bug in it cannot break inference.
The one exception is a [pool](./multi-node.md), whose router sits in front of the replicas by design.
The exceptions are opt-in: the [model router](./multi-instance.md), which gives several instances one address, and a [pool](./multi-node.md), whose router sits in front of the replicas by design.

The binary is self-contained: one file to put on a machine, with the docs you are reading compiled in.

## Next steps

- Follow the [Quickstart](./quickstart.md) to set the machine up and serve a first model.
- See what a profile holds in [Profiles](./profiles.md).
- Run several models side by side with [Several models at once](./multi-instance.md).
- Serve with mlx-lm or llama.cpp instead of vLLM with [Providers](./providers.md).
- Serve a model measured across several machines with [Recipes](./recipes.md).
- Pool replicas or shard one model across machines with [More than one machine](./multi-node.md).
Expand Down
Loading
Loading