Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,18 @@ the fact.

### Added

- Warm-up: a profile's `WARMUP_FILE` names a chat request that llm-helper
sends when `up`, `restart` or `switch` sees the server healthy, and that the router sends when a
server comes back, so the server has the client's fixed prompt cached
before the first chat. `llm-helper warmup capture` records one from a real
request through the router, and `llm-helper warmup` sends it and reports
how much came from the cache.
- The router counts prompt tokens and cached prompt tokens per model.
`status`, the panel and `/router/stats` show how much of each model's
prompts came from the cache. vLLM is started with
`--enable-prompt-tokens-details` so it reports them.
- Docs: how llama.cpp's prompt cache works on hybrid models and how to size
it with `--cache-ram`.
- Model types: what an architecture needs to run correctly is now data, one
file per HuggingFace `model_type` in `catalogue/model-types/`. `new` applies
the file for the repo's `model_type` and the profile's engine.
Expand Down
3 changes: 2 additions & 1 deletion catalogue/engines/vllm.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
"--gpu-memory-utilization",
"{mem_utilization}",
"--api-key",
"{api_key}"
"{api_key}",
"--enable-prompt-tokens-details"
],
"optional_args": [["--kv-cache-memory-bytes", "{kv_cache_bytes}"]],
"api_key": "flag:--api-key",
Expand Down
12 changes: 12 additions & 0 deletions docs/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,18 @@
"title": "llm-helper version",
"description": "Print the version",
"path": "./reference/cli/version.md"
},
{
"title": "llm-helper warmup",
"description": "Send a profile's warm-up request so its prompt is cached",
"path": "./reference/cli/warmup.md",
"children": [
{
"title": "llm-helper warmup capture",
"description": "Record the next chat request through the router as the profile's warm-up",
"path": "./reference/cli/warmup/capture.md"
}
]
}
]
}
Expand Down
4 changes: 4 additions & 0 deletions docs/multi-instance.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,10 @@ llm-helper router install-service
A request for a model whose instance is still loading gets 503 with `Retry-After`, so OpenAI-compatible clients, Coder among them, wait and retry instead of failing.
A model no instance serves gets 404 `model_not_found`.

The router counts the prompt tokens of each model's replies and how many of them the server read from its cache.
`status` and the panel show the share for each running instance, and `/router/stats` returns the counts as JSON.
They cover only requests that went through the router, and only servers that report cached tokens: llama.cpp does, and llm-helper starts vLLM with `--enable-prompt-tokens-details` so it does too, except in recipe containers.

The instances keep their own ports, so clients that already talk to one directly are unaffected.

## Upgrading from a single-model release
Expand Down
24 changes: 24 additions & 0 deletions docs/profiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ EXTRA_ARGS="--enable-auto-tool-choice --tool-call-parser qwen3_coder"
| `EXTRA_ARGS` | Extra flags passed to the profile's server verbatim. |
| `PROVIDER` | The server that runs this profile, `vllm` when absent. See [Providers](./providers.md). |
| `THINKING` | `on` or `off` for a [thinking model](./model-types.md), written by `new`. Shown by `list` and `status`. |
| `WARMUP_FILE` | A chat request sent after each start so the server caches its prompt. See below. |

Any other `KEY=VALUE` in the file is exported into the server's environment as-is.

Expand All @@ -40,6 +41,29 @@ A flag value with spaces in it — a JSON payload, for example — can be quoted
To turn thinking off in an existing profile, add `--default-chat-template-kwargs '{"enable_thinking":false}'` to its `EXTRA_ARGS` (with the model's own kwarg, which `model-types show <model_type>` prints) and set `THINKING=off`.
Profiles written before this key existed have no `THINKING` line, and `list` shows nothing for them until they are re-created.

### WARMUP_FILE

`WARMUP_FILE` is the path to a JSON body for `/v1/chat/completions`: the system prompt and tools your client sends, ending in a short user message.
llm-helper sends it, asking for one token, when `up`, `restart` or `switch` sees the server become healthy.
With `--no-follow`, or when the server is started from the panel, the command does not wait, so only the [router](./multi-instance.md) sends it, as it does whenever the server comes back after a reboot or crash.
Without the router, run `llm-helper warmup <profile>` yourself.
The server then has that prompt in its cache before the first real chat, which on a long agent prompt saves seconds on the first turn.
It works with every provider, since it is an ordinary chat request.

The easiest way to get the file is to record a real request through the router:

```sh
llm-helper warmup capture <profile>
```

Start a chat from your client while it waits.
It runs on the machine that serves the router, since the router only answers capture requests from the local machine, and only one capture per model can wait at a time.
It keeps everything before the first user message, replaces that message with `Hi`, saves the result under the data directory and sets `WARMUP_FILE`.
The request holds your client's system prompt, so keep it private.

`llm-helper warmup <profile>` sends the request twice and prints how many prompt tokens each send read from the cache.
When the second send reads almost nothing from the cache, the server is not keeping that prompt; see the llama.cpp part of [Providers](./providers.md).

### KV cache sizing

`KV_CACHE_MEMORY_BYTES` is sized when the profile is written, from the model's own attention shape and `KV_CONTEXT_TARGET` in `config.env` — four full contexts by default.
Expand Down
19 changes: 19 additions & 0 deletions docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,25 @@ Things to know about `llama-server`:
llm-helper set <profile> -- EXTRA_ARGS='-c 65536'
```

### Prompt cache

`llama-server` keeps the prompts of finished requests in host memory, up to `--cache-ram` MiB (8192 by default), and reuses the longest matching start of a new prompt instead of reading it again.
On a coding agent, every chat begins with the same long system prompt and tool list, so a hit saves most of the first turn.
The cache is empty after the server starts, and old prompts are dropped when it is full.

Hybrid models, which mix attention layers with recurrent or sliding-window ones (GLM-5.3-Flash, Qwen3-Next and others), can only go back to a saved checkpoint, not to any point in the prompt.
llama.cpp saves those checkpoints at the start of user messages, so two prompts share cached work only when everything before the first user message is identical.
Chats with different tools, skills or rules differ there and read the whole prompt again.

Set the size with `--cache-ram` in `EXTRA_ARGS`, or `LLAMA_ARG_CACHE_RAM` in the profile:

```sh
llm-helper set <profile> -- LLAMA_ARG_CACHE_RAM=16384
```

On a machine whose GPU shares memory with the CPU, such as a Mac or a DGX Spark, the cache comes out of the same memory as the model and its context.
A warm-up request (`WARMUP_FILE`, see [Profiles](./profiles.md)) fills the cache after every start, so the first chat does not pay for it.

### Example: DeepSeek V4 Flash Vision on a Mac Studio

This ran on an M5 Ultra with 256 GB, llama.cpp 0.8.0 from Homebrew, and the wired limit raised to 240 GiB.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,4 @@ sharding, replica pools, recipes, optional Coder Agents sync).
| [`up`](cli/up.md) | Start serving a profile alongside anything already running |
| [`upgrade`](cli/upgrade.md) | Update llm-helper itself from a published release |
| [`version`](cli/version.md) | Print the version |
| [`warmup`](cli/warmup.md) | Send a profile's warm-up request so its prompt is cached |
16 changes: 16 additions & 0 deletions docs/reference/cli/warmup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
# Code generated by go run ./scripts/gendocs. DO NOT EDIT.
title: "llm-helper warmup"
description: "Send a profile's warm-up request so its prompt is cached"
---

<!-- DO NOT EDIT | GENERATED CONTENT -->

Sends the chat request named by the profile's WARMUP_FILE twice and reports how much of each prompt the server served from its cache.
The warm-up also runs on its own when up, restart or switch waits for the server, and whenever the router sees the server come back.

## Usage

```console
llm-helper warmup [profile]
```
16 changes: 16 additions & 0 deletions docs/reference/cli/warmup/capture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
# Code generated by go run ./scripts/gendocs. DO NOT EDIT.
title: "llm-helper warmup capture"
description: "Record the next chat request through the router as the profile's warm-up"
---

<!-- DO NOT EDIT | GENERATED CONTENT -->

Waits for the next chat request the router passes to the profile's model, keeps everything before its first user message, and saves that as the profile's warm-up request.
Needs the router on this machine; start a chat from your client while this waits.

## Usage

```console
llm-helper warmup capture [profile]
```
1 change: 1 addition & 0 deletions internal/cli/editset.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ var standardProfileKeys = map[string]bool{
"EXTRA_ARGS": true,
profile.RecipeKey: true,
profile.ThinkingKey: true,
profile.WarmupKey: true,
engine.Key: true,
}

Expand Down
12 changes: 12 additions & 0 deletions internal/cli/profiles.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"github.com/coder/llm-helper/internal/hf"
"github.com/coder/llm-helper/internal/profile"
"github.com/coder/llm-helper/internal/resolve"
"github.com/coder/llm-helper/internal/router"
"github.com/coder/llm-helper/internal/vllm"
)

Expand Down Expand Up @@ -326,6 +327,8 @@ type statusReport struct {
LastLog string `json:"last_log,omitempty"`
Thinking string `json:"thinking,omitempty"`
MemUtil string `json:"mem_util,omitempty"`
// Cache is the router's prompt-cache counts, when the router is running.
Cache *router.CacheStats `json:"prompt_cache,omitempty"`
}

func gatherStatus(cfg *config.Config, inst *control.Instance, def string) statusReport {
Expand Down Expand Up @@ -396,6 +399,12 @@ func runStatus(args []string, jsonOut bool) error {
for i, inst := range list {
reports[i] = gatherStatus(cfg, inst, def)
}
cache := routerCache(cfg, list)
for i := range reports {
if c, ok := cache[reports[i].Profile]; ok {
reports[i].Cache = &c
}
}
if jsonOut {
if len(args) > 0 {
return printJSON(reports[0])
Expand Down Expand Up @@ -456,6 +465,9 @@ func printStatusDetail(cfg *config.Config, r statusReport) {
}
if r.Healthy {
fmt.Printf("endpoint: healthy (http %d on port %s)\n", r.HTTPStatus, r.Port)
if r.Cache != nil {
fmt.Printf("cache: %s\n", describeCache(*r.Cache))
}
return
}
if r.Crashed {
Expand Down
1 change: 1 addition & 0 deletions internal/cli/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ func newRootCmd() *cobra.Command {
newLogsCmd(),
newServeCmd(),
newRouterCmd(),
newWarmupCmd(),

newSetupCmd(),
newDoctorCmd(),
Expand Down
1 change: 1 addition & 0 deletions internal/cli/root_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ func TestTheCommandSetIsPinned(t *testing.T) {
"adopt", "coder-sync", "default", "doctor", "down", "edit", "engines", "hardware", "inspect", "install-gguf", "list", "logs",
"metal-limit", "migrate", "model-types", "new", "node", "pool", "recipes", "remove", "restart", "router", "search", "serve", "set",
"setup", "shard", "spark", "start", "status", "switch", "up", "upgrade", "version",
"warmup",
}

var got []string
Expand Down
65 changes: 56 additions & 9 deletions internal/cli/router.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package cli
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"net/http"
Expand All @@ -17,6 +18,8 @@ import (
"github.com/coder/llm-helper/internal/control"
"github.com/coder/llm-helper/internal/router"
"github.com/coder/llm-helper/internal/service"
"github.com/coder/llm-helper/internal/warmup"
"github.com/coder/llm-helper/internal/web"
)

const defaultRouterPort = "8000"
Expand Down Expand Up @@ -61,21 +64,28 @@ func instanceBackends(cfg *config.Config) func() []router.Backend {
}
out := make([]router.Backend, 0, len(list))
for _, inst := range list {
model := inst.Env.Get("MODEL")
if c := clusterRunnerForName(cfg, inst.Name); c != nil {
if u, err := c.endpointURL(); err == nil {
out = append(out, router.Backend{URL: u, Model: model})
continue
}
}
if inst.Port != "" {
out = append(out, router.Backend{URL: "http://localhost:" + inst.Port, Model: model})
if u := instanceURL(cfg, inst); u != "" {
out = append(out, router.Backend{URL: u, Model: inst.Env.Get("MODEL"), Warmup: warmupPath(cfg, inst)})
}
}
return out
}
}

// instanceURL is where the instance answers: its cluster endpoint, or its
// local port.
func instanceURL(cfg *config.Config, inst *control.Instance) string {
if c := clusterRunnerForName(cfg, inst.Name); c != nil {
if u, err := c.endpointURL(); err == nil {
return u
}
}
if inst.Port != "" {
return "http://localhost:" + inst.Port
}
return ""
}

func newRouterServeCmd() *cobra.Command {
var port, bind string
c := &cobra.Command{
Expand All @@ -94,6 +104,13 @@ func newRouterServeCmd() *cobra.Command {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
models := router.NewModels(instanceBackends(cfg), cfg.APIKey())
models.OnWarm = func(model string, res warmup.Result, err error) {
if err != nil {
fmt.Fprintf(os.Stderr, "warm-up for %s failed: %v\n", model, err)
return
}
fmt.Printf("Warmed up %s: %s\n", model, res.Describe())
}
models.StartRefresh(ctx, 5*time.Second)
mux := routerMux(models, cfg.APIKey())

Expand Down Expand Up @@ -133,6 +150,36 @@ func routerMux(models *router.Models, key string) *http.ServeMux {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(models.Routes())
})))
mux.Handle("/router/stats", router.RequireKey(key, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(models.Report())
})))
mux.Handle("/router/capture", router.RequireKey(key, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
Comment thread
DevelopmentCats marked this conversation as resolved.
Resolved
if !web.LoopbackOnly(r.RemoteAddr) {
http.Error(w, "capture is only available from this machine", http.StatusForbidden)
return
}
model := r.URL.Query().Get("model")
if _, ok := models.Routes()[model]; !ok {
http.Error(w, fmt.Sprintf("the router does not serve %q", model), http.StatusNotFound)
return
}
raw, err := models.Capture(r.Context(), model)
if errors.Is(err, router.ErrCaptureBusy) {
http.Error(w, err.Error(), http.StatusConflict)
return
}
if err != nil {
return
}
body, err := warmup.FromCapture(raw)
if err != nil {
http.Error(w, err.Error(), http.StatusUnprocessableEntity)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(body)
})))
return mux
}

Expand Down
10 changes: 10 additions & 0 deletions internal/cli/serve.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (

"github.com/spf13/cobra"

"github.com/coder/llm-helper/internal/control"
"github.com/coder/llm-helper/internal/web"
)

Expand Down Expand Up @@ -55,6 +56,15 @@ func runServe(cmd *cobra.Command, addr string) error {
}
return nil
})
srv.SetPromptCache(func(list []*control.Instance) map[string]int {
out := map[string]int{}
for name, c := range routerCache(cfg, list) {
if p := c.Percent(); p >= 0 {
out[name] = p
}
}
return out
})

handler := srv.Handler(web.Static())
// A token in the query string is exchanged for a cookie on first load,
Expand Down
2 changes: 1 addition & 1 deletion internal/cli/vllmargs_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ func TestServeSpecFromActivePicksTheProvider(t *testing.T) {
active: "VLLM_BIN=" + vllmBin + "\nMLX_LM_BIN=" + mlxBin + "\nPORT=8001\nVLLM_API_KEY=k\n" +
"MODEL=org/m\nMAX_MODEL_LEN=4096\nGPU_MEMORY_UTILIZATION=0.5\nKV_CACHE_MEMORY_BYTES=1000\nEXTRA_ARGS=--x y\n",
want: []string{vllmBin, "serve", "org/m", "--port", "8001", "--max-model-len", "4096",
"--gpu-memory-utilization", "0.5", "--api-key", "k", "--kv-cache-memory-bytes", "1000", "--x", "y"},
"--gpu-memory-utilization", "0.5", "--api-key", "k", "--enable-prompt-tokens-details", "--kv-cache-memory-bytes", "1000", "--x", "y"},
}, {
name: "mlx",
active: "VLLM_BIN=" + vllmBin + "\nMLX_LM_BIN=" + mlxBin + "\nPORT=8001\nVLLM_API_KEY=k\n" +
Expand Down
Loading
Loading