diff --git a/CHANGELOG.md b/CHANGELOG.md index f8fcdf8..c60386f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,18 @@ the fact. ### Added +- Warm-up: a profile's `WARMUP_FILE` names a chat request that llm-helper + sends when `up`, `restart` or `switch` sees the server healthy, and that the router sends when a + server comes back, so the server has the client's fixed prompt cached + before the first chat. `llm-helper warmup capture` records one from a real + request through the router, and `llm-helper warmup` sends it and reports + how much came from the cache. +- The router counts prompt tokens and cached prompt tokens per model. + `status`, the panel and `/router/stats` show how much of each model's + prompts came from the cache. vLLM is started with + `--enable-prompt-tokens-details` so it reports them. +- Docs: how llama.cpp's prompt cache works on hybrid models and how to size + it with `--cache-ram`. - Model types: what an architecture needs to run correctly is now data, one file per HuggingFace `model_type` in `catalogue/model-types/`. `new` applies the file for the repo's `model_type` and the profile's engine. diff --git a/catalogue/engines/vllm.json b/catalogue/engines/vllm.json index 5caf3ae..baf3e3c 100644 --- a/catalogue/engines/vllm.json +++ b/catalogue/engines/vllm.json @@ -22,7 +22,8 @@ "--gpu-memory-utilization", "{mem_utilization}", "--api-key", - "{api_key}" + "{api_key}", + "--enable-prompt-tokens-details" ], "optional_args": [["--kv-cache-memory-bytes", "{kv_cache_bytes}"]], "api_key": "flag:--api-key", diff --git a/docs/manifest.json b/docs/manifest.json index 8e9701b..0cd09ff 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -398,6 +398,18 @@ "title": "llm-helper version", "description": "Print the version", "path": "./reference/cli/version.md" + }, + { + "title": "llm-helper warmup", + "description": "Send a profile's warm-up request so its prompt is cached", + "path": "./reference/cli/warmup.md", + "children": [ + { + "title": "llm-helper warmup capture", + "description": "Record the next chat request through the router as the profile's warm-up", + "path": "./reference/cli/warmup/capture.md" + } + ] } ] } diff --git a/docs/multi-instance.md b/docs/multi-instance.md index eb94cf9..ef19d19 100644 --- a/docs/multi-instance.md +++ b/docs/multi-instance.md @@ -96,6 +96,10 @@ llm-helper router install-service A request for a model whose instance is still loading gets 503 with `Retry-After`, so OpenAI-compatible clients, Coder among them, wait and retry instead of failing. A model no instance serves gets 404 `model_not_found`. +The router counts the prompt tokens of each model's replies and how many of them the server read from its cache. +`status` and the panel show the share for each running instance, and `/router/stats` returns the counts as JSON. +They cover only requests that went through the router, and only servers that report cached tokens: llama.cpp does, and llm-helper starts vLLM with `--enable-prompt-tokens-details` so it does too, except in recipe containers. + The instances keep their own ports, so clients that already talk to one directly are unaffected. ## Upgrading from a single-model release diff --git a/docs/profiles.md b/docs/profiles.md index 09010d9..6370b83 100644 --- a/docs/profiles.md +++ b/docs/profiles.md @@ -26,6 +26,7 @@ EXTRA_ARGS="--enable-auto-tool-choice --tool-call-parser qwen3_coder" | `EXTRA_ARGS` | Extra flags passed to the profile's server verbatim. | | `PROVIDER` | The server that runs this profile, `vllm` when absent. See [Providers](./providers.md). | | `THINKING` | `on` or `off` for a [thinking model](./model-types.md), written by `new`. Shown by `list` and `status`. | +| `WARMUP_FILE` | A chat request sent after each start so the server caches its prompt. See below. | Any other `KEY=VALUE` in the file is exported into the server's environment as-is. @@ -40,6 +41,29 @@ A flag value with spaces in it — a JSON payload, for example — can be quoted To turn thinking off in an existing profile, add `--default-chat-template-kwargs '{"enable_thinking":false}'` to its `EXTRA_ARGS` (with the model's own kwarg, which `model-types show ` prints) and set `THINKING=off`. Profiles written before this key existed have no `THINKING` line, and `list` shows nothing for them until they are re-created. +### WARMUP_FILE + +`WARMUP_FILE` is the path to a JSON body for `/v1/chat/completions`: the system prompt and tools your client sends, ending in a short user message. +llm-helper sends it, asking for one token, when `up`, `restart` or `switch` sees the server become healthy. +With `--no-follow`, or when the server is started from the panel, the command does not wait, so only the [router](./multi-instance.md) sends it, as it does whenever the server comes back after a reboot or crash. +Without the router, run `llm-helper warmup ` yourself. +The server then has that prompt in its cache before the first real chat, which on a long agent prompt saves seconds on the first turn. +It works with every provider, since it is an ordinary chat request. + +The easiest way to get the file is to record a real request through the router: + +```sh +llm-helper warmup capture +``` + +Start a chat from your client while it waits. +It runs on the machine that serves the router, since the router only answers capture requests from the local machine, and only one capture per model can wait at a time. +It keeps everything before the first user message, replaces that message with `Hi`, saves the result under the data directory and sets `WARMUP_FILE`. +The request holds your client's system prompt, so keep it private. + +`llm-helper warmup ` sends the request twice and prints how many prompt tokens each send read from the cache. +When the second send reads almost nothing from the cache, the server is not keeping that prompt; see the llama.cpp part of [Providers](./providers.md). + ### KV cache sizing `KV_CACHE_MEMORY_BYTES` is sized when the profile is written, from the model's own attention shape and `KV_CONTEXT_TARGET` in `config.env` — four full contexts by default. diff --git a/docs/providers.md b/docs/providers.md index 0bab677..4cbd6fc 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -134,6 +134,25 @@ Things to know about `llama-server`: llm-helper set -- EXTRA_ARGS='-c 65536' ``` +### Prompt cache + +`llama-server` keeps the prompts of finished requests in host memory, up to `--cache-ram` MiB (8192 by default), and reuses the longest matching start of a new prompt instead of reading it again. +On a coding agent, every chat begins with the same long system prompt and tool list, so a hit saves most of the first turn. +The cache is empty after the server starts, and old prompts are dropped when it is full. + +Hybrid models, which mix attention layers with recurrent or sliding-window ones (GLM-5.3-Flash, Qwen3-Next and others), can only go back to a saved checkpoint, not to any point in the prompt. +llama.cpp saves those checkpoints at the start of user messages, so two prompts share cached work only when everything before the first user message is identical. +Chats with different tools, skills or rules differ there and read the whole prompt again. + +Set the size with `--cache-ram` in `EXTRA_ARGS`, or `LLAMA_ARG_CACHE_RAM` in the profile: + +```sh +llm-helper set -- LLAMA_ARG_CACHE_RAM=16384 +``` + +On a machine whose GPU shares memory with the CPU, such as a Mac or a DGX Spark, the cache comes out of the same memory as the model and its context. +A warm-up request (`WARMUP_FILE`, see [Profiles](./profiles.md)) fills the cache after every start, so the first chat does not pay for it. + ### Example: DeepSeek V4 Flash Vision on a Mac Studio This ran on an M5 Ultra with 256 GB, llama.cpp 0.8.0 from Homebrew, and the wired limit raised to 240 GiB. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 9baf059..4a50162 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -56,3 +56,4 @@ sharding, replica pools, recipes, optional Coder Agents sync). | [`up`](cli/up.md) | Start serving a profile alongside anything already running | | [`upgrade`](cli/upgrade.md) | Update llm-helper itself from a published release | | [`version`](cli/version.md) | Print the version | +| [`warmup`](cli/warmup.md) | Send a profile's warm-up request so its prompt is cached | diff --git a/docs/reference/cli/warmup.md b/docs/reference/cli/warmup.md new file mode 100644 index 0000000..b9f4bbd --- /dev/null +++ b/docs/reference/cli/warmup.md @@ -0,0 +1,16 @@ +--- +# Code generated by go run ./scripts/gendocs. DO NOT EDIT. +title: "llm-helper warmup" +description: "Send a profile's warm-up request so its prompt is cached" +--- + + + +Sends the chat request named by the profile's WARMUP_FILE twice and reports how much of each prompt the server served from its cache. +The warm-up also runs on its own when up, restart or switch waits for the server, and whenever the router sees the server come back. + +## Usage + +```console +llm-helper warmup [profile] +``` diff --git a/docs/reference/cli/warmup/capture.md b/docs/reference/cli/warmup/capture.md new file mode 100644 index 0000000..5460c0c --- /dev/null +++ b/docs/reference/cli/warmup/capture.md @@ -0,0 +1,16 @@ +--- +# Code generated by go run ./scripts/gendocs. DO NOT EDIT. +title: "llm-helper warmup capture" +description: "Record the next chat request through the router as the profile's warm-up" +--- + + + +Waits for the next chat request the router passes to the profile's model, keeps everything before its first user message, and saves that as the profile's warm-up request. +Needs the router on this machine; start a chat from your client while this waits. + +## Usage + +```console +llm-helper warmup capture [profile] +``` diff --git a/internal/cli/editset.go b/internal/cli/editset.go index 50e8afa..f123648 100644 --- a/internal/cli/editset.go +++ b/internal/cli/editset.go @@ -26,6 +26,7 @@ var standardProfileKeys = map[string]bool{ "EXTRA_ARGS": true, profile.RecipeKey: true, profile.ThinkingKey: true, + profile.WarmupKey: true, engine.Key: true, } diff --git a/internal/cli/profiles.go b/internal/cli/profiles.go index a7c6238..eaa5ab1 100644 --- a/internal/cli/profiles.go +++ b/internal/cli/profiles.go @@ -16,6 +16,7 @@ import ( "github.com/coder/llm-helper/internal/hf" "github.com/coder/llm-helper/internal/profile" "github.com/coder/llm-helper/internal/resolve" + "github.com/coder/llm-helper/internal/router" "github.com/coder/llm-helper/internal/vllm" ) @@ -326,6 +327,8 @@ type statusReport struct { LastLog string `json:"last_log,omitempty"` Thinking string `json:"thinking,omitempty"` MemUtil string `json:"mem_util,omitempty"` + // Cache is the router's prompt-cache counts, when the router is running. + Cache *router.CacheStats `json:"prompt_cache,omitempty"` } func gatherStatus(cfg *config.Config, inst *control.Instance, def string) statusReport { @@ -396,6 +399,12 @@ func runStatus(args []string, jsonOut bool) error { for i, inst := range list { reports[i] = gatherStatus(cfg, inst, def) } + cache := routerCache(cfg, list) + for i := range reports { + if c, ok := cache[reports[i].Profile]; ok { + reports[i].Cache = &c + } + } if jsonOut { if len(args) > 0 { return printJSON(reports[0]) @@ -456,6 +465,9 @@ func printStatusDetail(cfg *config.Config, r statusReport) { } if r.Healthy { fmt.Printf("endpoint: healthy (http %d on port %s)\n", r.HTTPStatus, r.Port) + if r.Cache != nil { + fmt.Printf("cache: %s\n", describeCache(*r.Cache)) + } return } if r.Crashed { diff --git a/internal/cli/root.go b/internal/cli/root.go index d81dea8..2b44ec5 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -122,6 +122,7 @@ func newRootCmd() *cobra.Command { newLogsCmd(), newServeCmd(), newRouterCmd(), + newWarmupCmd(), newSetupCmd(), newDoctorCmd(), diff --git a/internal/cli/root_internal_test.go b/internal/cli/root_internal_test.go index bb50033..abd4abe 100644 --- a/internal/cli/root_internal_test.go +++ b/internal/cli/root_internal_test.go @@ -16,6 +16,7 @@ func TestTheCommandSetIsPinned(t *testing.T) { "adopt", "coder-sync", "default", "doctor", "down", "edit", "engines", "hardware", "inspect", "install-gguf", "list", "logs", "metal-limit", "migrate", "model-types", "new", "node", "pool", "recipes", "remove", "restart", "router", "search", "serve", "set", "setup", "shard", "spark", "start", "status", "switch", "up", "upgrade", "version", + "warmup", } var got []string diff --git a/internal/cli/router.go b/internal/cli/router.go index c047411..3af8e9d 100644 --- a/internal/cli/router.go +++ b/internal/cli/router.go @@ -3,6 +3,7 @@ package cli import ( "context" "encoding/json" + "errors" "fmt" "net" "net/http" @@ -17,6 +18,8 @@ import ( "github.com/coder/llm-helper/internal/control" "github.com/coder/llm-helper/internal/router" "github.com/coder/llm-helper/internal/service" + "github.com/coder/llm-helper/internal/warmup" + "github.com/coder/llm-helper/internal/web" ) const defaultRouterPort = "8000" @@ -61,21 +64,28 @@ func instanceBackends(cfg *config.Config) func() []router.Backend { } out := make([]router.Backend, 0, len(list)) for _, inst := range list { - model := inst.Env.Get("MODEL") - if c := clusterRunnerForName(cfg, inst.Name); c != nil { - if u, err := c.endpointURL(); err == nil { - out = append(out, router.Backend{URL: u, Model: model}) - continue - } - } - if inst.Port != "" { - out = append(out, router.Backend{URL: "http://localhost:" + inst.Port, Model: model}) + if u := instanceURL(cfg, inst); u != "" { + out = append(out, router.Backend{URL: u, Model: inst.Env.Get("MODEL"), Warmup: warmupPath(cfg, inst)}) } } return out } } +// instanceURL is where the instance answers: its cluster endpoint, or its +// local port. +func instanceURL(cfg *config.Config, inst *control.Instance) string { + if c := clusterRunnerForName(cfg, inst.Name); c != nil { + if u, err := c.endpointURL(); err == nil { + return u + } + } + if inst.Port != "" { + return "http://localhost:" + inst.Port + } + return "" +} + func newRouterServeCmd() *cobra.Command { var port, bind string c := &cobra.Command{ @@ -94,6 +104,13 @@ func newRouterServeCmd() *cobra.Command { ctx, cancel := context.WithCancel(context.Background()) defer cancel() models := router.NewModels(instanceBackends(cfg), cfg.APIKey()) + models.OnWarm = func(model string, res warmup.Result, err error) { + if err != nil { + fmt.Fprintf(os.Stderr, "warm-up for %s failed: %v\n", model, err) + return + } + fmt.Printf("Warmed up %s: %s\n", model, res.Describe()) + } models.StartRefresh(ctx, 5*time.Second) mux := routerMux(models, cfg.APIKey()) @@ -133,6 +150,36 @@ func routerMux(models *router.Models, key string) *http.ServeMux { w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode(models.Routes()) }))) + mux.Handle("/router/stats", router.RequireKey(key, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(models.Report()) + }))) + mux.Handle("/router/capture", router.RequireKey(key, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if !web.LoopbackOnly(r.RemoteAddr) { + http.Error(w, "capture is only available from this machine", http.StatusForbidden) + return + } + model := r.URL.Query().Get("model") + if _, ok := models.Routes()[model]; !ok { + http.Error(w, fmt.Sprintf("the router does not serve %q", model), http.StatusNotFound) + return + } + raw, err := models.Capture(r.Context(), model) + if errors.Is(err, router.ErrCaptureBusy) { + http.Error(w, err.Error(), http.StatusConflict) + return + } + if err != nil { + return + } + body, err := warmup.FromCapture(raw) + if err != nil { + http.Error(w, err.Error(), http.StatusUnprocessableEntity) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write(body) + }))) return mux } diff --git a/internal/cli/serve.go b/internal/cli/serve.go index ebf3d82..253b4b8 100644 --- a/internal/cli/serve.go +++ b/internal/cli/serve.go @@ -12,6 +12,7 @@ import ( "github.com/spf13/cobra" + "github.com/coder/llm-helper/internal/control" "github.com/coder/llm-helper/internal/web" ) @@ -55,6 +56,15 @@ func runServe(cmd *cobra.Command, addr string) error { } return nil }) + srv.SetPromptCache(func(list []*control.Instance) map[string]int { + out := map[string]int{} + for name, c := range routerCache(cfg, list) { + if p := c.Percent(); p >= 0 { + out[name] = p + } + } + return out + }) handler := srv.Handler(web.Static()) // A token in the query string is exchanged for a cookie on first load, diff --git a/internal/cli/vllmargs_internal_test.go b/internal/cli/vllmargs_internal_test.go index 1233e48..daffba7 100644 --- a/internal/cli/vllmargs_internal_test.go +++ b/internal/cli/vllmargs_internal_test.go @@ -80,7 +80,7 @@ func TestServeSpecFromActivePicksTheProvider(t *testing.T) { active: "VLLM_BIN=" + vllmBin + "\nMLX_LM_BIN=" + mlxBin + "\nPORT=8001\nVLLM_API_KEY=k\n" + "MODEL=org/m\nMAX_MODEL_LEN=4096\nGPU_MEMORY_UTILIZATION=0.5\nKV_CACHE_MEMORY_BYTES=1000\nEXTRA_ARGS=--x y\n", want: []string{vllmBin, "serve", "org/m", "--port", "8001", "--max-model-len", "4096", - "--gpu-memory-utilization", "0.5", "--api-key", "k", "--kv-cache-memory-bytes", "1000", "--x", "y"}, + "--gpu-memory-utilization", "0.5", "--api-key", "k", "--enable-prompt-tokens-details", "--kv-cache-memory-bytes", "1000", "--x", "y"}, }, { name: "mlx", active: "VLLM_BIN=" + vllmBin + "\nMLX_LM_BIN=" + mlxBin + "\nPORT=8001\nVLLM_API_KEY=k\n" + diff --git a/internal/cli/warmup.go b/internal/cli/warmup.go new file mode 100644 index 0000000..4efa5c3 --- /dev/null +++ b/internal/cli/warmup.go @@ -0,0 +1,215 @@ +package cli + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "os" + "os/signal" + "path/filepath" + "strings" + "time" + + "github.com/spf13/cobra" + + "github.com/coder/llm-helper/internal/config" + "github.com/coder/llm-helper/internal/control" + "github.com/coder/llm-helper/internal/detect" + "github.com/coder/llm-helper/internal/profile" + "github.com/coder/llm-helper/internal/router" + "github.com/coder/llm-helper/internal/warmup" +) + +func newWarmupCmd() *cobra.Command { + c := &cobra.Command{ + Use: "warmup [profile]", + Short: "Send a profile's warm-up request so its prompt is cached", + Long: "Sends the chat request named by the profile's " + profile.WarmupKey + " twice and reports how much of each prompt the server served from its cache.\n" + + "The warm-up also runs on its own when up, restart or switch waits for the server, and whenever the router sees the server come back.", + Args: cobra.MaximumNArgs(1), + GroupID: groupServer, + RunE: func(_ *cobra.Command, args []string) error { + cfg, err := loadConfig() + if err != nil { + return err + } + inst, err := pickInstance(cfg, args) + if err != nil { + return err + } + path := warmupPath(cfg, inst) + if path == "" { + return fmt.Errorf("'%s' has no %s; record one with '%s warmup capture %s'", inst.Name, profile.WarmupKey, cmdName, inst.Name) + } + base, model := warmupTarget(cfg, inst) + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() + for _, label := range []string{"First send", "Second send"} { + res, err := warmup.Run(ctx, base, cfg.APIKey(), model, path) + if err != nil { + return err + } + fmt.Printf("%s: %s\n", label, res.Describe()) + } + return nil + }, + } + c.AddCommand(newWarmupCaptureCmd()) + return c +} + +func newWarmupCaptureCmd() *cobra.Command { + return &cobra.Command{ + Use: "capture [profile]", + Short: "Record the next chat request through the router as the profile's warm-up", + Long: "Waits for the next chat request the router passes to the profile's model, keeps everything before its first user message, and saves that as the profile's warm-up request.\n" + + "Needs the router on this machine; start a chat from your client while this waits.", + Args: cobra.MaximumNArgs(1), + RunE: func(_ *cobra.Command, args []string) error { + cfg, err := loadConfig() + if err != nil { + return err + } + inst, err := pickInstance(cfg, args) + if err != nil { + return err + } + _, model := warmupTarget(cfg, inst) + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() + fmt.Printf("Waiting for the next chat request to %s through the router (Ctrl+C to stop)...\n", model) + body, err := captureRequest(ctx, cfg, model) + if err != nil { + return err + } + path, err := saveWarmup(cfg, inst.Name, body) + if err != nil { + return err + } + fmt.Printf("Saved %s\n", path) + fmt.Printf("%s: %s=%s\n", inst.Name, profile.WarmupKey, path) + return nil + }, + } +} + +// warmupPath is the instance's warm-up request: the profile's current value, +// so a file set after the instance started still counts. +func warmupPath(cfg *config.Config, inst *control.Instance) string { + path := inst.Env.Get(profile.WarmupKey) + if p, err := profile.Load(cfg.Dirs, inst.Name); err == nil { + path = p.Warmup() + } + return expandHome(strings.TrimSpace(path)) +} + +// warmupTarget is the instance's OpenAI-compatible root and the model id it +// serves, which can differ from the profile's MODEL. +func warmupTarget(cfg *config.Config, inst *control.Instance) (base, model string) { + root := instanceURL(cfg, inst) + model = inst.Env.Get("MODEL") + if ep, ok := detect.ProbeURL(root, cfg.APIKey()); ok && len(ep.Models) > 0 { + model = ep.Models[0] + } + return strings.TrimSuffix(root, "/") + "/v1", model +} + +// warmAfterBoot sends the warm-up once the server is healthy. A failure only +// warns: the server is up either way. +func warmAfterBoot(cfg *config.Config, inst *control.Instance) { + path := warmupPath(cfg, inst) + if path == "" { + return + } + base, model := warmupTarget(cfg, inst) + fmt.Println("Warming up the prompt cache...") + res, err := warmup.Run(context.Background(), base, cfg.APIKey(), model, path) + if err != nil { + fmt.Fprintf(os.Stderr, "warm-up failed (the server is still up): %v\n", err) + return + } + fmt.Printf("Warmed up: %s\n", res.Describe()) +} + +func routerURL(cfg *config.Config, path string) string { + return "http://localhost:" + routerPort(cfg, "") + path +} + +func routerRequest(ctx context.Context, cfg *config.Config, path string) (*http.Response, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, routerURL(cfg, path), nil) + if err != nil { + return nil, err + } + if key := cfg.APIKey(); key != "" { + req.Header.Set("Authorization", "Bearer "+key) + } + return http.DefaultClient.Do(req) +} + +func captureRequest(ctx context.Context, cfg *config.Config, model string) ([]byte, error) { + resp, err := routerRequest(ctx, cfg, "/router/capture?model="+url.QueryEscape(model)) + if err != nil { + if errors.Is(ctx.Err(), context.Canceled) { + return nil, errors.New("stopped before a request arrived") + } + return nil, fmt.Errorf("the router is not answering on port %s (start it with '%s router install-service'): %w", routerPort(cfg, ""), cmdName, err) + } + defer resp.Body.Close() + data, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20)) + if err != nil { + return nil, err + } + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("router answered %s: %s", resp.Status, strings.TrimSpace(string(data))) + } + return data, nil +} + +// saveWarmup writes body under the data directory and points the profile at it. +func saveWarmup(cfg *config.Config, name string, body []byte) (string, error) { + p, err := profile.Load(cfg.Dirs, name) + if err != nil { + return "", err + } + path := filepath.Join(cfg.Dirs.Data, "warmup", name+".json") + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return "", err + } + if err := os.WriteFile(path, append(body, '\n'), 0o600); err != nil { + return "", err + } + p.File.Set(profile.WarmupKey, path) + if err := p.File.Write(cfg.Dirs.ProfileFile(name), 0o644); err != nil { + return "", err + } + return path, nil +} + +// routerCache is the router's prompt-cache counts for each instance it has +// passed replies for, by profile name. It is empty when the router is not +// running. +func routerCache(cfg *config.Config, list []*control.Instance) map[string]router.CacheStats { + out := map[string]router.CacheStats{} + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + stats, err := router.FetchStats(ctx, routerURL(cfg, ""), cfg.APIKey()) + if err != nil { + return out + } + for _, inst := range list { + if s, ok := router.StatsAt(stats, instanceURL(cfg, inst)); ok && s.Reported+s.Unknown > 0 { + out[inst.Name] = s + } + } + return out +} + +func describeCache(s router.CacheStats) string { + if p := s.Percent(); p >= 0 { + return fmt.Sprintf("%d%% of %d prompt tokens came from the cache (router traffic only)", p, s.PromptTokens) + } + return "the server does not report cached tokens" +} diff --git a/internal/cli/warmup_internal_test.go b/internal/cli/warmup_internal_test.go new file mode 100644 index 0000000..c7935b3 --- /dev/null +++ b/internal/cli/warmup_internal_test.go @@ -0,0 +1,319 @@ +package cli + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/coder/llm-helper/internal/config" + "github.com/coder/llm-helper/internal/control" + "github.com/coder/llm-helper/internal/profile" + "github.com/coder/llm-helper/internal/router" +) + +// fakeEngine answers like an OpenAI-compatible server serving "served" and +// keeps the last chat request it got. +type fakeEngine struct { + *httptest.Server + mu sync.Mutex + last map[string]any +} + +func newFakeEngine(t *testing.T) *fakeEngine { + t.Helper() + f := &fakeEngine{} + f.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/health": + case "/v1/models": + _, _ = io.WriteString(w, `{"data":[{"id":"served"}]}`) + case "/v1/chat/completions": + var body map[string]any + _ = json.NewDecoder(r.Body).Decode(&body) + f.mu.Lock() + f.last = body + f.mu.Unlock() + _, _ = io.WriteString(w, `{"usage":{"prompt_tokens":100,"prompt_tokens_details":{"cached_tokens":90}}}`) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(f.Close) + return f +} + +func (f *fakeEngine) lastRequest() map[string]any { + f.mu.Lock() + defer f.mu.Unlock() + return f.last +} + +func writeWarmupFile(t *testing.T, dir string) string { + t.Helper() + path := filepath.Join(dir, "warmup.json") + body := `{"model":"x","messages":[{"role":"system","content":"rules"},{"role":"user","content":"Hi"}],"stream":true}` + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatal(err) + } + return path +} + +func loadInstance(t *testing.T, cfg *config.Config, name string) *control.Instance { + t.Helper() + inst, err := control.LoadInstance(cfg, name) + if err != nil { + t.Fatal(err) + } + return inst +} + +func TestWarmupPathPrefersProfile(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\n") + writeInstance(t, cfg.Dirs, "x", "PORT=9123\nMODEL=org/m\n"+profile.WarmupKey+"=/started.json\n") + if got := warmupPath(cfg, loadInstance(t, cfg, "x")); got != "/started.json" { + t.Errorf("without a profile: %q", got) + } + if err := os.WriteFile(cfg.Dirs.ProfileFile("x"), []byte("MODEL=org/m\n"+profile.WarmupKey+"=/edited.json\n"), 0o600); err != nil { + t.Fatal(err) + } + if got := warmupPath(cfg, loadInstance(t, cfg, "x")); got != "/edited.json" { + t.Errorf("with a profile: %q", got) + } +} + +func TestWarmupTargetUsesServedModel(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\n") + engine := newFakeEngine(t) + writeInstance(t, cfg.Dirs, "x", "PORT="+portOf(t, engine.URL)+"\nMODEL=org/m\n") + base, model := warmupTarget(cfg, loadInstance(t, cfg, "x")) + if base != "http://localhost:"+portOf(t, engine.URL)+"/v1" || model != "served" { + t.Errorf("target: %q %q", base, model) + } + + writeInstance(t, cfg.Dirs, "y", "PORT="+closedPort(t)+"\nMODEL=org/m\n") + if _, model := warmupTarget(cfg, loadInstance(t, cfg, "y")); model != "org/m" { + t.Errorf("down server: %q", model) + } +} + +func TestWarmAfterBootSendsTheRequest(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\n") + engine := newFakeEngine(t) + path := writeWarmupFile(t, t.TempDir()) + writeInstance(t, cfg.Dirs, "x", "PORT="+portOf(t, engine.URL)+"\nMODEL=org/m\n"+profile.WarmupKey+"="+path+"\n") + out, _ := captureStdout(t, func() error { + warmAfterBoot(cfg, loadInstance(t, cfg, "x")) + return nil + }) + if !strings.Contains(out, "Warmed up: 100 prompt tokens, 90 from cache") { + t.Errorf("output: %q", out) + } + got := engine.lastRequest() + if got["model"] != "served" || got["max_tokens"] != float64(1) || got["stream"] != false { + t.Errorf("request: %v", got) + } +} + +func TestWarmAfterBootSkipsWithoutFile(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\n") + engine := newFakeEngine(t) + writeInstance(t, cfg.Dirs, "x", "PORT="+portOf(t, engine.URL)+"\nMODEL=org/m\n") + out, _ := captureStdout(t, func() error { + warmAfterBoot(cfg, loadInstance(t, cfg, "x")) + return nil + }) + if out != "" || engine.lastRequest() != nil { + t.Errorf("warmed without a file: %q", out) + } +} + +func TestSaveWarmupSetsTheProfileKey(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\n") + if err := os.WriteFile(cfg.Dirs.ProfileFile("x"), []byte("MODEL=org/m\n"), 0o600); err != nil { + t.Fatal(err) + } + path, err := saveWarmup(cfg, "x", []byte(`{"messages":[]}`)) + if err != nil { + t.Fatal(err) + } + if want := filepath.Join(cfg.Dirs.Data, "warmup", "x.json"); path != want { + t.Errorf("path %q, want %q", path, want) + } + if data, err := os.ReadFile(path); err != nil || string(data) != "{\"messages\":[]}\n" { + t.Errorf("file: %q %v", data, err) + } + p, err := profile.Load(cfg.Dirs, "x") + if err != nil { + t.Fatal(err) + } + if p.Warmup() != path || p.File.Get("MODEL") != "org/m" { + t.Errorf("profile: %q %q", p.Warmup(), p.File.Get("MODEL")) + } +} + +func TestInstanceBackendsCarryWarmup(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\n") + writeInstance(t, cfg.Dirs, "x", "PORT=9123\nMODEL=org/m\n"+profile.WarmupKey+"=/w.json\n") + got := instanceBackends(cfg)() + if len(got) != 1 || got[0].Warmup != "/w.json" { + t.Errorf("backends: %+v", got) + } +} + +func fakeRouter(t *testing.T, h http.HandlerFunc) *config.Config { + t.Helper() + srv := httptest.NewServer(h) + t.Cleanup(srv.Close) + return routerTestConfig(t, "PORT=8001\nVLLM_API_KEY=k\nROUTER_PORT="+portOf(t, srv.URL)+"\n") +} + +func TestCaptureRequestReadsTheRouter(t *testing.T) { + cfg := fakeRouter(t, func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer k" { + http.Error(w, "no", http.StatusUnauthorized) + return + } + if r.URL.Path != "/router/capture" || r.URL.Query().Get("model") != "org/m" { + http.Error(w, "the router does not serve it", http.StatusNotFound) + return + } + _, _ = io.WriteString(w, `{"messages":[]}`) + }) + got, err := captureRequest(context.Background(), cfg, "org/m") + if err != nil || string(got) != `{"messages":[]}` { + t.Errorf("capture: %q %v", got, err) + } + if _, err := captureRequest(context.Background(), cfg, "org/other"); err == nil || !strings.Contains(err.Error(), "does not serve") { + t.Errorf("unrouted model: %v", err) + } +} + +func TestCaptureRequestWithoutRouter(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\nROUTER_PORT="+closedPort(t)+"\n") + if _, err := captureRequest(context.Background(), cfg, "org/m"); err == nil || !strings.Contains(err.Error(), "not answering") { + t.Errorf("err: %v", err) + } +} + +func TestRouterCacheMatchesInstances(t *testing.T) { + cfg := fakeRouter(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/router/stats" || r.Header.Get("Authorization") != "Bearer k" { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(map[string]router.ModelStats{ + "org/m": {CacheStats: router.CacheStats{Reported: 2, PromptTokens: 200, CachedTokens: 150}, URL: "http://localhost:9123"}, + "org/old": {CacheStats: router.CacheStats{Reported: 1, PromptTokens: 50}}, + }) + }) + writeInstance(t, cfg.Dirs, "x", "PORT=9123\nMODEL=org/m\n") + writeInstance(t, cfg.Dirs, "y", "PORT=9124\nMODEL=org/n\n") + got := routerCache(cfg, []*control.Instance{loadInstance(t, cfg, "x"), loadInstance(t, cfg, "y")}) + if len(got) != 1 || got["x"].Percent() != 75 { + t.Errorf("cache: %+v", got) + } +} + +func TestRouterCacheWithoutRouter(t *testing.T) { + cfg := routerTestConfig(t, "PORT=8001\nROUTER_PORT="+closedPort(t)+"\n") + writeInstance(t, cfg.Dirs, "x", "PORT=9123\nMODEL=org/m\n") + if got := routerCache(cfg, []*control.Instance{loadInstance(t, cfg, "x")}); len(got) != 0 { + t.Errorf("cache: %+v", got) + } +} + +func TestRouterMuxStatsAndCapture(t *testing.T) { + mux := routerMux(router.NewModels(func() []router.Backend { return nil }, "k"), "k") + cases := []struct { + path, remote string + want int + }{ + {"/router/stats", "192.0.2.1:1234", http.StatusOK}, + {"/router/capture?model=org/m", "192.0.2.1:1234", http.StatusForbidden}, + {"/router/capture?model=org/m", "127.0.0.1:1234", http.StatusNotFound}, + {"/router/capture?model=org/m", "[::1]:1234", http.StatusNotFound}, + } + for _, c := range cases { + for auth, code := range map[string]int{"": http.StatusUnauthorized, "Bearer k": c.want} { + req := httptest.NewRequest(http.MethodGet, c.path, nil) + req.RemoteAddr = c.remote + if auth != "" { + req.Header.Set("Authorization", auth) + } + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, req) + if rec.Code != code { + t.Errorf("%s from %s with %q: %d, want %d", c.path, c.remote, auth, rec.Code, code) + } + } + } +} + +func TestRouterMuxCaptureReturnsSharedPrefix(t *testing.T) { + backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = io.WriteString(w, `{"data":[{"id":"org/m"}]}`) + return + } + _, _ = io.WriteString(w, `{}`) + })) + defer backend.Close() + models := router.NewModels(func() []router.Backend { return []router.Backend{{URL: backend.URL}} }, "") + models.Refresh(context.Background()) + mux := routerMux(models, "") + + first := make(chan *httptest.ResponseRecorder) + go func() { + req := httptest.NewRequest(http.MethodGet, "/router/capture?model=org/m", nil) + req.RemoteAddr = "127.0.0.1:1234" + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, req) + first <- rec + }() + chat := `{"model":"org/m","messages":[{"role":"system","content":"rules"},{"role":"user","content":"secret task"}]}` + var got *httptest.ResponseRecorder + for deadline := time.Now().Add(5 * time.Second); got == nil && time.Now().Before(deadline); { + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(chat)) + mux.ServeHTTP(httptest.NewRecorder(), req) + select { + case got = <-first: + case <-time.After(10 * time.Millisecond): + } + } + if got == nil { + t.Fatal("nothing captured") + } + if got.Code != http.StatusOK || strings.Contains(got.Body.String(), "secret task") || !strings.Contains(got.Body.String(), "rules") { + t.Fatalf("capture: %d %s", got.Code, got.Body) + } +} + +func TestStatusShowsPromptCache(t *testing.T) { + r := statusReport{Profile: "x", Healthy: true, HTTPStatus: 200, Port: "9123", Cache: &router.CacheStats{Reported: 1, PromptTokens: 200, CachedTokens: 50}} + data, err := json.Marshal(r) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(data), `"prompt_cache":{"reported":1,"prompt_tokens":200,"cached_tokens":50,"unknown":0}`) { + t.Errorf("json: %s", data) + } + cfg := routerTestConfig(t, "PORT=8001\n") + out, _ := captureStdout(t, func() error { + printStatusDetail(cfg, r) + return nil + }) + if !strings.Contains(out, "cache: 25% of 200 prompt tokens came from the cache (router traffic only)") { + t.Errorf("output: %q", out) + } + if got := describeCache(router.CacheStats{Unknown: 3}); got != "the server does not report cached tokens" { + t.Errorf("unknown: %q", got) + } +} diff --git a/internal/cli/watch.go b/internal/cli/watch.go index 8ae6c61..ec36f26 100644 --- a/internal/cli/watch.go +++ b/internal/cli/watch.go @@ -37,7 +37,11 @@ func watchBoot(cfg *config.Config, name string, opts watchOpts) error { if err != nil { return err } - return watchTarget(t, name, opts) + if err := watchTarget(t, name, opts); err != nil { + return err + } + warmAfterBoot(cfg, inst) + return nil } // bootTarget is what a boot is watched through: the runner that reports diff --git a/internal/engine/registry_test.go b/internal/engine/registry_test.go index 324d7b8..6a0f2f0 100644 --- a/internal/engine/registry_test.go +++ b/internal/engine/registry_test.go @@ -52,7 +52,7 @@ func TestVLLMCommand(t *testing.T) { } want := []string{"/opt/vllm/bin/vllm", "serve", "Qwen/Qwen3-32B", "--port", "8001", "--max-model-len", "32768", "--gpu-memory-utilization", "0.85", - "--api-key", "sk-x", "--kv-cache-memory-bytes", "1000", + "--api-key", "sk-x", "--enable-prompt-tokens-details", "--kv-cache-memory-bytes", "1000", "--tool-call-parser", "hermes", "--chat-template", "/a b.jinja"} if !reflect.DeepEqual(got, want) { t.Fatalf("argv\n got %q\nwant %q", got, want) diff --git a/internal/profile/profile.go b/internal/profile/profile.go index 77f856c..bb3b404 100644 --- a/internal/profile/profile.go +++ b/internal/profile/profile.go @@ -76,6 +76,13 @@ const ThinkingKey = "THINKING" // Thinking returns "on", "off" or "". func (p *Profile) Thinking() string { return strings.TrimSpace(p.File.Get(ThinkingKey)) } +// WarmupKey names a JSON chat request whose prompt is sent to the server once +// it is healthy, so the client's fixed prefix is already cached. +const WarmupKey = "WARMUP_FILE" + +// Warmup returns the warm-up request path, or "". +func (p *Profile) Warmup() string { return strings.TrimSpace(p.File.Get(WarmupKey)) } + // Recipe returns the recipe this profile was generated from, or "". func (p *Profile) Recipe() string { return strings.TrimSpace(p.File.Get(RecipeKey)) } diff --git a/internal/router/cache.go b/internal/router/cache.go new file mode 100644 index 0000000..c4bd6bd --- /dev/null +++ b/internal/router/cache.go @@ -0,0 +1,314 @@ +package router + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "sync" + + "github.com/coder/llm-helper/internal/warmup" +) + +// CacheStats counts the prompt tokens of the replies the router passed on +// for one model, and how many of them the server took from its cache. +type CacheStats struct { + // Reported is the number of replies that said how many tokens were cached. + Reported int64 `json:"reported"` + PromptTokens int64 `json:"prompt_tokens"` + CachedTokens int64 `json:"cached_tokens"` + // Unknown is the number of replies that did not. + Unknown int64 `json:"unknown"` +} + +// Percent is the share of prompt tokens served from cache, or -1 when no +// reply reported it. +func (s CacheStats) Percent() int { + if s.Reported == 0 || s.PromptTokens == 0 { + return -1 + } + return int(s.CachedTokens * 100 / s.PromptTokens) +} + +// Stats reports the cache counts of every model the router has passed a +// reply for since it started. +func (m *Models) Stats() map[string]CacheStats { + m.mu.RLock() + defer m.mu.RUnlock() + out := make(map[string]CacheStats, len(m.stats)) + for id, s := range m.stats { + out[id] = *s + } + return out +} + +// ModelStats is one model's line in the router's stats endpoint. +type ModelStats struct { + CacheStats + // URL is the backend serving the model, empty when it is not routed now. + URL string `json:"url,omitempty"` +} + +// Report is Stats with the backend each model is routed to. +func (m *Models) Report() map[string]ModelStats { + stats := m.Stats() + m.mu.RLock() + defer m.mu.RUnlock() + out := make(map[string]ModelStats, len(stats)) + for id, s := range stats { + e := ModelStats{CacheStats: s} + if u := m.routes[id]; u != nil { + e.URL = u.String() + } + out[id] = e + } + return out +} + +// FetchStats reads the stats of the router listening on base. +func FetchStats(ctx context.Context, base, key string) (map[string]ModelStats, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(base, "/")+"/router/stats", nil) + if err != nil { + return nil, err + } + if key != "" { + req.Header.Set("Authorization", "Bearer "+key) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("router stats: %s", resp.Status) + } + var out map[string]ModelStats + if err := json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&out); err != nil { + return nil, err + } + return out, nil +} + +// StatsAt adds up the stats of every model served from backend. +func StatsAt(stats map[string]ModelStats, backend string) (CacheStats, bool) { + var sum CacheStats + found := false + for _, e := range stats { + if e.URL == "" || strings.TrimRight(e.URL, "/") != strings.TrimRight(backend, "/") { + continue + } + found = true + sum.Reported += e.Reported + sum.PromptTokens += e.PromptTokens + sum.CachedTokens += e.CachedTokens + sum.Unknown += e.Unknown + } + return sum, found +} + +func (m *Models) count(model string, u *warmup.Usage) { + m.mu.Lock() + defer m.mu.Unlock() + s := m.stats[model] + if s == nil { + s = &CacheStats{} + m.stats[model] = s + } + prompt, cached := -1, -1 + if u != nil { + prompt, cached = u.Counts() + } + if prompt < 0 || cached < 0 { + s.Unknown++ + return + } + s.Reported++ + s.PromptTokens += int64(prompt) + s.CachedTokens += int64(cached) +} + +// modelAt is the first model id served from u. +func (m *Models) modelAt(u *url.URL) string { + m.mu.RLock() + defer m.mu.RUnlock() + best := "" + for id, r := range m.routes { + if r.String() == u.String() && (best == "" || id < best) { + best = id + } + } + return best +} + +func countsTokens(path string) bool { + switch path { + case "/v1/chat/completions", "/v1/completions", "/v1/responses": + return true + } + return false +} + +// watchUsage wraps a proxied reply so its usage block is counted as it +// passes, without holding the reply back. +func (m *Models) watchUsage(resp *http.Response, model string) { + if resp.StatusCode != http.StatusOK || !countsTokens(resp.Request.URL.Path) { + return + } + sse := strings.HasPrefix(resp.Header.Get("Content-Type"), "text/event-stream") + resp.Body = &usageBody{ReadCloser: resp.Body, sse: sse, done: func(u *warmup.Usage) { m.count(model, u) }} +} + +const maxScan = 8 << 20 + +type usageBody struct { + io.ReadCloser + sse bool + buf []byte + skip bool + eof bool + usage *warmup.Usage + done func(*warmup.Usage) + once sync.Once +} + +func (b *usageBody) Read(p []byte) (int, error) { + n, err := b.ReadCloser.Read(p) + b.feed(p[:n]) + if err == io.EOF { + b.eof = true + b.finish() + } + return n, err +} + +func (b *usageBody) Close() error { + b.finish() + return b.ReadCloser.Close() +} + +func (b *usageBody) feed(p []byte) { + if !b.sse { + if b.skip || len(b.buf)+len(p) > maxScan { + b.skip, b.buf = true, nil + return + } + b.buf = append(b.buf, p...) + return + } + for len(p) > 0 { + i := bytes.IndexByte(p, '\n') + if i < 0 { + if len(b.buf)+len(p) > maxScan { + b.skip = true + } + if !b.skip { + b.buf = append(b.buf, p...) + } + return + } + if !b.skip { + b.buf = append(b.buf, p[:i]...) + line := bytes.TrimSpace(bytes.TrimPrefix(bytes.TrimSpace(b.buf), []byte("data:"))) + b.parse(line) + } + b.buf, b.skip = b.buf[:0], false + p = p[i+1:] + } +} + +func (b *usageBody) parse(data []byte) { + if !bytes.Contains(data, []byte(`"usage"`)) { + return + } + var reply struct { + Usage *warmup.Usage `json:"usage"` + Response *struct { + Usage *warmup.Usage `json:"usage"` + } `json:"response"` + } + if json.Unmarshal(data, &reply) != nil { + return + } + switch { + case reply.Usage != nil: + b.usage = reply.Usage + case reply.Response != nil && reply.Response.Usage != nil: + b.usage = reply.Response.Usage + } +} + +func (b *usageBody) finish() { + b.once.Do(func() { + if !b.sse && !b.skip && b.eof { + b.parse(b.buf) + } + b.buf = nil + if b.usage != nil || b.eof { + b.done(b.usage) + } + }) +} + +// ErrCaptureBusy is returned when a capture for the model is already waiting. +var ErrCaptureBusy = errors.New("a capture for this model is already waiting") + +// Capture waits for the next chat request for model and returns its body. +func (m *Models) Capture(ctx context.Context, model string) ([]byte, error) { + ch := make(chan []byte, 1) + m.mu.Lock() + if m.captures[model] != nil { + m.mu.Unlock() + return nil, ErrCaptureBusy + } + m.captures[model] = ch + m.mu.Unlock() + defer func() { + m.mu.Lock() + if m.captures[model] == ch { + delete(m.captures, model) + } + m.mu.Unlock() + }() + select { + case body := <-ch: + return body, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (m *Models) offer(model string, body []byte) { + m.mu.Lock() + ch := m.captures[model] + delete(m.captures, model) + m.mu.Unlock() + if ch != nil { + ch <- body + } +} + +// startWarm sends path's warm-up request to the backend at u, unless one is +// already on its way for model. +func (m *Models) startWarm(u *url.URL, model, path string) { + m.mu.Lock() + if m.warming[model] { + m.mu.Unlock() + return + } + m.warming[model] = true + m.mu.Unlock() + go func() { + res, err := warmup.Run(context.Background(), u.String()+"/v1", m.key, model, path) + m.mu.Lock() + delete(m.warming, model) + m.mu.Unlock() + if m.OnWarm != nil { + m.OnWarm(model, res, err) + } + }() +} diff --git a/internal/router/cache_internal_test.go b/internal/router/cache_internal_test.go new file mode 100644 index 0000000..71625e0 --- /dev/null +++ b/internal/router/cache_internal_test.go @@ -0,0 +1,331 @@ +package router + +import ( + "compress/gzip" + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/coder/llm-helper/internal/warmup" +) + +func replying(t *testing.T, contentType, reply string) *httptest.Server { + t.Helper() + s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = io.WriteString(w, `{"data":[{"id":"glm"}]}`) + return + } + w.Header().Set("Content-Type", contentType) + _, _ = io.WriteString(w, reply) + })) + t.Cleanup(s.Close) + return s +} + +func TestModelsCountsCachedTokens(t *testing.T) { + cases := []struct { + name, contentType, reply string + want CacheStats + }{ + {"json", "application/json", `{"usage":{"prompt_tokens":100,"prompt_tokens_details":{"cached_tokens":80}}}`, + CacheStats{Reported: 1, PromptTokens: 100, CachedTokens: 80}}, + {"sse", "text/event-stream", "data: {\"choices\":[],\"usage\":null}\n\ndata: {\"choices\":[],\"usage\":{\"prompt_tokens\":50,\"prompt_tokens_details\":{\"cached_tokens\":10}}}\n\ndata: [DONE]\n\n", + CacheStats{Reported: 1, PromptTokens: 50, CachedTokens: 10}}, + {"responses sse", "text/event-stream", "event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"usage\":{\"input_tokens\":40,\"input_tokens_details\":{\"cached_tokens\":40}}}}\n\n", + CacheStats{Reported: 1, PromptTokens: 40, CachedTokens: 40}}, + {"no usage", "application/json", `{"choices":[]}`, CacheStats{Unknown: 1}}, + {"no cached field", "application/json", `{"usage":{"prompt_tokens":100}}`, CacheStats{Unknown: 1}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + s := replying(t, c.contentType, c.reply) + m := NewModels(urls(s.URL), "") + m.Refresh(context.Background()) + code, got := do(t, m, http.MethodPost, "/v1/chat/completions", `{"model":"glm"}`) + if code != 200 || got != c.reply { + t.Fatalf("reply changed: %d %q", code, got) + } + if st := m.Stats()["glm"]; st != c.want { + t.Errorf("stats %+v, want %+v", st, c.want) + } + }) + } +} + +func TestModelsCountsOnlyCompletions(t *testing.T) { + s := replying(t, "application/json", `{"usage":{"prompt_tokens":1,"prompt_tokens_details":{"cached_tokens":1}}}`) + m := NewModels(urls(s.URL), "") + m.Refresh(context.Background()) + do(t, m, http.MethodPost, "/v1/embeddings", `{"model":"glm"}`) + if st := m.Stats(); len(st) != 0 { + t.Errorf("embeddings counted: %+v", st) + } + do(t, m, http.MethodPost, "/v1/completions", `{}`) + if st := m.Stats()["glm"]; st.Reported != 1 { + t.Errorf("a request naming no model should count for the only one: %+v", m.Stats()) + } +} + +func TestCacheStatsPercent(t *testing.T) { + if p := (CacheStats{}).Percent(); p != -1 { + t.Errorf("empty: %d", p) + } + if p := (CacheStats{Reported: 2, PromptTokens: 200, CachedTokens: 150}).Percent(); p != 75 { + t.Errorf("75%%: %d", p) + } +} + +func TestModelsCapture(t *testing.T) { + s := replying(t, "application/json", `{}`) + m := NewModels(urls(s.URL), "") + m.Refresh(context.Background()) + + got := make(chan []byte) + go func() { + body, _ := m.Capture(context.Background(), "glm") + got <- body + }() + deadline := time.Now().Add(5 * time.Second) + for { + m.mu.RLock() + armed := m.captures["glm"] != nil + m.mu.RUnlock() + if armed || time.Now().After(deadline) { + break + } + time.Sleep(5 * time.Millisecond) + } + do(t, m, http.MethodPost, "/v1/completions", `{"model":"glm","prompt":"x"}`) + first := `{"model":"glm","messages":[{"role":"user","content":"one"}]}` + do(t, m, http.MethodPost, "/v1/chat/completions", first) + do(t, m, http.MethodPost, "/v1/chat/completions", `{"model":"glm","messages":[{"role":"user","content":"two"}]}`) + select { + case body := <-got: + if string(body) != first { + t.Errorf("captured %q", body) + } + case <-time.After(5 * time.Second): + t.Fatal("nothing captured") + } + m.mu.RLock() + defer m.mu.RUnlock() + if len(m.captures) != 0 { + t.Error("capture still armed") + } +} + +func TestModelsCaptureRefusesSecond(t *testing.T) { + m := NewModels(urls(), "") + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + _, _ = m.Capture(ctx, "glm") + close(done) + }() + deadline := time.Now().Add(5 * time.Second) + for { + m.mu.RLock() + armed := m.captures["glm"] != nil + m.mu.RUnlock() + if armed || time.Now().After(deadline) { + break + } + time.Sleep(5 * time.Millisecond) + } + if _, err := m.Capture(context.Background(), "glm"); !errors.Is(err, ErrCaptureBusy) { + t.Fatalf("second capture: %v", err) + } + cancel() + <-done + ctx2, cancel2 := context.WithTimeout(context.Background(), 10*time.Millisecond) + defer cancel2() + if _, err := m.Capture(ctx2, "glm"); errors.Is(err, ErrCaptureBusy) { + t.Fatal("capture still busy after the first gave up") + } +} + +func TestModelsCountsGzippedReplies(t *testing.T) { + reply := `{"usage":{"prompt_tokens":100,"prompt_tokens_details":{"cached_tokens":80}}}` + s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = io.WriteString(w, `{"data":[{"id":"glm"}]}`) + return + } + w.Header().Set("Content-Type", "application/json") + if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") { + w.Header().Set("Content-Encoding", "gzip") + zw := gzip.NewWriter(w) + _, _ = io.WriteString(zw, reply) + _ = zw.Close() + return + } + _, _ = io.WriteString(w, reply) + })) + t.Cleanup(s.Close) + m := NewModels(urls(s.URL), "") + m.Refresh(context.Background()) + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"glm"}`)) + req.Header.Set("Accept-Encoding", "gzip") + rec := httptest.NewRecorder() + m.ServeHTTP(rec, req) + if rec.Code != 200 { + t.Fatalf("status %d", rec.Code) + } + body := rec.Body.String() + if rec.Header().Get("Content-Encoding") == "gzip" { + zr, err := gzip.NewReader(rec.Body) + if err != nil { + t.Fatal(err) + } + data, _ := io.ReadAll(zr) + body = string(data) + } + if body != reply { + t.Fatalf("reply changed: %q", body) + } + if st := m.Stats()["glm"]; st.Reported != 1 || st.CachedTokens != 80 { + t.Fatalf("stats %+v", st) + } +} + +func TestModelsCaptureGivesUp(t *testing.T) { + m := NewModels(urls(), "") + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond) + defer cancel() + if _, err := m.Capture(ctx, "glm"); err == nil { + t.Error("expected the context's error") + } +} + +func TestModelsWarmsBackendThatComesUp(t *testing.T) { + path := filepath.Join(t.TempDir(), "warmup.json") + if err := os.WriteFile(path, []byte(`{"messages":[{"role":"system","content":"tools"},{"role":"user","content":"Hi"}]}`), 0o600); err != nil { + t.Fatal(err) + } + var up atomic.Bool + sent := make(chan map[string]any, 4) + s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if !up.Load() { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + if r.URL.Path == "/v1/models" { + _, _ = io.WriteString(w, `{"data":[{"id":"glm"}]}`) + return + } + if r.Header.Get("Authorization") != "Bearer k" { + t.Errorf("warm-up without the key") + } + var body map[string]any + _ = json.NewDecoder(r.Body).Decode(&body) + sent <- body + _, _ = io.WriteString(w, `{"usage":{"prompt_tokens":9,"prompt_tokens_details":{"cached_tokens":0}}}`) + })) + t.Cleanup(s.Close) + + m := NewModels(func() []Backend { return []Backend{{URL: s.URL, Model: "org/glm", Warmup: path}} }, "k") + warmed := make(chan warmup.Result, 4) + m.OnWarm = func(model string, res warmup.Result, err error) { + if model != "glm" || err != nil { + t.Errorf("OnWarm(%q, %v)", model, err) + } + warmed <- res + } + m.Refresh(context.Background()) + up.Store(true) + m.Refresh(context.Background()) + select { + case body := <-sent: + if body["model"] != "glm" || body["max_tokens"] != float64(1) { + t.Errorf("warm-up body %v", body) + } + case <-time.After(5 * time.Second): + t.Fatal("no warm-up sent") + } + if res := <-warmed; res.PromptTokens != 9 { + t.Errorf("result %+v", res) + } + m.Refresh(context.Background()) + select { + case <-sent: + t.Error("warmed again while the backend stayed up") + case <-time.After(100 * time.Millisecond): + } +} + +func TestModelsNoWarmupForBackendAlreadyUp(t *testing.T) { + sent := make(chan struct{}, 1) + s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = io.WriteString(w, `{"data":[{"id":"glm"}]}`) + return + } + sent <- struct{}{} + })) + t.Cleanup(s.Close) + m := NewModels(func() []Backend { return []Backend{{URL: s.URL, Warmup: "/nonexistent"}} }, "") + m.Refresh(context.Background()) + m.Refresh(context.Background()) + select { + case <-sent: + t.Error("warmed a backend that was already up when the router started") + case <-time.After(100 * time.Millisecond): + } +} + +func TestModelsReportAndFetchStats(t *testing.T) { + s := replying(t, "application/json", `{"usage":{"prompt_tokens":100,"prompt_tokens_details":{"cached_tokens":60}}}`) + m := NewModels(urls(s.URL), "") + m.Refresh(context.Background()) + do(t, m, http.MethodPost, "/v1/chat/completions", `{"model":"glm"}`) + + rep := m.Report() + if got := rep["glm"]; got.URL != s.URL || got.CachedTokens != 60 { + t.Fatalf("report %+v", rep) + } + + stats := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/router/stats" || r.Header.Get("Authorization") != "Bearer k" { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(m.Report()) + })) + t.Cleanup(stats.Close) + got, err := FetchStats(context.Background(), stats.URL+"/", "k") + if err != nil { + t.Fatal(err) + } + if got["glm"] != rep["glm"] { + t.Errorf("fetched %+v, want %+v", got, rep) + } + if _, err := FetchStats(context.Background(), stats.URL, "wrong"); err == nil { + t.Error("a refused request should be an error") + } +} + +func TestStatsAtAddsUpOneBackend(t *testing.T) { + stats := map[string]ModelStats{ + "a": {CacheStats: CacheStats{Reported: 1, PromptTokens: 10, CachedTokens: 5}, URL: "http://h:8001"}, + "b": {CacheStats: CacheStats{Reported: 2, PromptTokens: 30, CachedTokens: 15, Unknown: 1}, URL: "http://h:8001/"}, + "c": {CacheStats: CacheStats{Reported: 9, PromptTokens: 90}, URL: "http://h:8002"}, + "old": {CacheStats: CacheStats{Reported: 9, PromptTokens: 90}}, + } + got, ok := StatsAt(stats, "http://h:8001/") + if want := (CacheStats{Reported: 3, PromptTokens: 40, CachedTokens: 20, Unknown: 1}); !ok || got != want { + t.Errorf("got %+v %v, want %+v", got, ok, want) + } + if _, ok := StatsAt(stats, "http://h:9999"); ok { + t.Error("no model is served there") + } +} diff --git a/internal/router/models.go b/internal/router/models.go index d9dd8dd..5c863fa 100644 --- a/internal/router/models.go +++ b/internal/router/models.go @@ -14,6 +14,8 @@ import ( "strings" "sync" "time" + + "github.com/coder/llm-helper/internal/warmup" ) // Models sends each request to the backend that serves the model it names. @@ -30,13 +32,21 @@ type Models struct { entries []json.RawMessage urls []*url.URL tried time.Time + down map[string]bool + warming map[string]bool + captures map[string]chan []byte + stats map[string]*CacheStats + + // OnWarm, when set, is told the outcome of every warm-up the router sends. + OnWarm func(model string, res warmup.Result, err error) } -// Backend is one instance: its base URL (scheme://host:port) and the model -// it is configured to serve, "" when unknown. +// Backend is one instance: its base URL (scheme://host:port), the model it +// is configured to serve ("" when unknown) and its warm-up request ("" for none). type Backend struct { - URL string - Model string + URL string + Model string + Warmup string } // RetryAfter is how long a client is told to wait for a starting instance. @@ -51,6 +61,9 @@ func NewModels(backends func() []Backend, key string) *Models { key: key, client: &http.Client{Timeout: 5 * time.Second}, routes: map[string]*url.URL{}, + warming: map[string]bool{}, + captures: map[string]chan []byte{}, + stats: map[string]*CacheStats{}, } } @@ -67,13 +80,19 @@ type modelID struct { // does not answer yet is starting, as are the ids it last advertised. func (m *Models) Refresh(ctx context.Context) { m.mu.RLock() - prev := m.learned + prev, prevDown := m.learned, m.down m.mu.RUnlock() learned := map[string][]string{} routes := map[string]*url.URL{} var entries []json.RawMessage var urls []*url.URL var waiting []string + down := map[string]bool{} + type ready struct { + u *url.URL + model, path string + } + var wake []ready for _, b := range m.backends() { u, err := url.Parse(strings.TrimRight(b.URL, "/")) if err != nil { @@ -82,6 +101,7 @@ func (m *Models) Refresh(ctx context.Context) { urls = append(urls, u) key := u.String() + "\x00" + b.Model list, ok := m.fetch(ctx, u) + down[key] = !ok if !ok && b.Model != "" { waiting = append(waiting, b.Model) waiting = append(waiting, prev[key]...) @@ -99,6 +119,9 @@ func (m *Models) Refresh(ctx context.Context) { routes[id.ID] = u entries = append(entries, e) } + if ok && prevDown[key] && b.Warmup != "" && len(learned[key]) > 0 { + wake = append(wake, ready{u, learned[key][0], b.Warmup}) + } } starting := map[string]bool{} for _, id := range waiting { @@ -108,7 +131,11 @@ func (m *Models) Refresh(ctx context.Context) { } m.mu.Lock() m.routes, m.starting, m.learned, m.entries, m.urls, m.tried = routes, starting, learned, entries, urls, time.Now() + m.down = down m.mu.Unlock() + for _, r := range wake { + m.startWarm(r.u, r.model, r.path) + } } func (m *Models) fetch(ctx context.Context, u *url.URL) ([]json.RawMessage, bool) { @@ -214,6 +241,9 @@ func (m *Models) ServeHTTP(w http.ResponseWriter, r *http.Request) { model = req.Model r.Body = io.NopCloser(bytes.NewReader(body)) r.ContentLength = int64(len(body)) + if model != "" && r.URL.Path == "/v1/chat/completions" { + m.offer(model, body) + } } } @@ -241,8 +271,20 @@ func (m *Models) ServeHTTP(w http.ResponseWriter, r *http.Request) { rx.Director = func(r *http.Request) { direct(r) m.authorize(r) + if countsTokens(r.URL.Path) { + // Let the transport ask for and undo compression so usage can be read. + r.Header.Del("Accept-Encoding") + } } rx.FlushInterval = -1 + counted := model + if counted == "" { + counted = m.modelAt(u) + } + rx.ModifyResponse = func(resp *http.Response) error { + m.watchUsage(resp, counted) + return nil + } rx.ErrorHandler = func(w http.ResponseWriter, _ *http.Request, err error) { unavailable(w, fmt.Sprintf("the instance serving %s is not answering; it may be restarting: %v", model, err)) } diff --git a/internal/warmup/warmup.go b/internal/warmup/warmup.go new file mode 100644 index 0000000..a498ded --- /dev/null +++ b/internal/warmup/warmup.go @@ -0,0 +1,289 @@ +// Package warmup sends a client's fixed prompt prefix to a freshly started +// server so the server's prompt cache holds it before the first real request. +package warmup + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "strings" + "time" +) + +// Timeout bounds one warm-up request; a long prefix on a slow machine takes a while. +const Timeout = 5 * time.Minute + +// Result is what the server reported for one warm-up request. +type Result struct { + PromptTokens int + // CachedTokens is -1 when the server did not report it. + CachedTokens int +} + +// Load reads a chat completions request body and rewrites it for the given +// served model so that it generates a single token. +func Load(path, model string) ([]byte, error) { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + return Prepare(raw, model) +} + +// Prepare rewrites a chat completions request body for the given model so +// that it generates a single token without streaming. +func Prepare(raw []byte, model string) ([]byte, error) { + body, err := parseObject(raw) + if err != nil { + return nil, fmt.Errorf("warm-up request is not a JSON object: %w", err) + } + var msgs []json.RawMessage + if v, ok := body.get("messages"); ok { + _ = json.Unmarshal(v, &msgs) + } + if len(msgs) == 0 { + return nil, fmt.Errorf("warm-up request has no messages") + } + if model != "" { + body.set("model", mustJSON(model)) + } + body.set("max_tokens", json.RawMessage("1")) + body.set("stream", json.RawMessage("false")) + body.del("max_completion_tokens") + body.del("stream_options") + body.del("n") + return body.marshal(), nil +} + +// Send posts body to base + "/chat/completions". base is the server's +// OpenAI-compatible root, ending in /v1. +func Send(ctx context.Context, client *http.Client, base, key string, body []byte) (Result, error) { + res := Result{CachedTokens: -1} + url := strings.TrimSuffix(base, "/") + "/chat/completions" + req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) + if err != nil { + return res, err + } + req.Header.Set("Content-Type", "application/json") + if key != "" { + req.Header.Set("Authorization", "Bearer "+key) + } + resp, err := client.Do(req) + if err != nil { + return res, err + } + defer resp.Body.Close() + data, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return res, err + } + if resp.StatusCode != http.StatusOK { + return res, fmt.Errorf("server answered %s: %s", resp.Status, strings.TrimSpace(string(data))) + } + var reply struct { + Usage *Usage `json:"usage"` + } + if err := json.Unmarshal(data, &reply); err != nil { + return res, fmt.Errorf("server reply is not JSON: %w", err) + } + if reply.Usage != nil { + res.PromptTokens, res.CachedTokens = reply.Usage.Counts() + } + return res, nil +} + +// Usage is the usage block of an OpenAI-compatible reply: chat completions +// name the fields prompt_*, the Responses API names them input_*. +type Usage struct { + PromptTokens *int `json:"prompt_tokens"` + InputTokens *int `json:"input_tokens"` + PromptTokensDetails *details `json:"prompt_tokens_details"` + InputTokensDetails *details `json:"input_tokens_details"` +} + +type details struct { + CachedTokens *int `json:"cached_tokens"` +} + +// Counts returns the prompt tokens and the cached prompt tokens, -1 for +// whichever the server did not report. +func (u *Usage) Counts() (prompt, cached int) { + prompt, cached = -1, -1 + switch { + case u.PromptTokens != nil: + prompt = *u.PromptTokens + case u.InputTokens != nil: + prompt = *u.InputTokens + } + switch { + case u.PromptTokensDetails != nil && u.PromptTokensDetails.CachedTokens != nil: + cached = *u.PromptTokensDetails.CachedTokens + case u.InputTokensDetails != nil && u.InputTokensDetails.CachedTokens != nil: + cached = *u.InputTokensDetails.CachedTokens + } + return prompt, cached +} + +// Run loads path and sends it once. +func Run(ctx context.Context, base, key, model, path string) (Result, error) { + body, err := Load(path, model) + if err != nil { + return Result{CachedTokens: -1}, err + } + ctx, cancel := context.WithTimeout(ctx, Timeout) + defer cancel() + return Send(ctx, http.DefaultClient, base, key, body) +} + +// Describe renders a result for people. +func (r Result) Describe() string { + if r.CachedTokens < 0 { + return fmt.Sprintf("%d prompt tokens (the server does not report cached tokens)", r.PromptTokens) + } + return fmt.Sprintf("%d prompt tokens, %d from cache", r.PromptTokens, r.CachedTokens) +} + +// Placeholder is the user message a captured request ends with. +const Placeholder = "Hi" + +// FromCapture turns a client's chat request into a warm-up request: it keeps +// what comes before the first user message, which every chat shares, and ends +// with Placeholder as the user message. Everything it keeps stays byte for +// byte, so the order of keys the chat template sees does not change. +func FromCapture(raw []byte) ([]byte, error) { + body, err := parseObject(raw) + if err != nil { + return nil, fmt.Errorf("captured request is not a JSON object: %w", err) + } + var msgs []json.RawMessage + if v, ok := body.get("messages"); ok { + _ = json.Unmarshal(v, &msgs) + } + first := -1 + for i, m := range msgs { + var role struct { + Role string `json:"role"` + } + if json.Unmarshal(m, &role) == nil && role.Role == "user" { + first = i + break + } + } + if first < 0 { + return nil, fmt.Errorf("captured request has no user message") + } + msgs = msgs[:first+1] + msgs[first] = mustJSON(struct { + Role string `json:"role"` + Content string `json:"content"` + }{"user", Placeholder}) + body.set("messages", joinArray(msgs)) + var out bytes.Buffer + if err := json.Indent(&out, body.marshal(), "", " "); err != nil { + return nil, err + } + return out.Bytes(), nil +} + +// object is a JSON object that keeps its keys in order and its values as +// they were written. +type object struct { + keys []string + vals map[string]json.RawMessage +} + +func parseObject(raw []byte) (*object, error) { + dec := json.NewDecoder(bytes.NewReader(raw)) + tok, err := dec.Token() + if err != nil { + return nil, err + } + if d, ok := tok.(json.Delim); !ok || d != '{' { + return nil, fmt.Errorf("not an object") + } + o := &object{vals: map[string]json.RawMessage{}} + for dec.More() { + tok, err := dec.Token() + if err != nil { + return nil, err + } + key, _ := tok.(string) + var v json.RawMessage + if err := dec.Decode(&v); err != nil { + return nil, err + } + o.set(key, v) + } + if _, err := dec.Token(); err != nil { + return nil, err + } + if _, err := dec.Token(); err != io.EOF { + return nil, fmt.Errorf("trailing data after the object") + } + return o, nil +} + +func (o *object) get(key string) (json.RawMessage, bool) { + v, ok := o.vals[key] + return v, ok +} + +func (o *object) set(key string, v json.RawMessage) { + if _, ok := o.vals[key]; !ok { + o.keys = append(o.keys, key) + } + o.vals[key] = v +} + +func (o *object) del(key string) { + if _, ok := o.vals[key]; !ok { + return + } + delete(o.vals, key) + for i, k := range o.keys { + if k == key { + o.keys = append(o.keys[:i], o.keys[i+1:]...) + break + } + } +} + +func (o *object) marshal() []byte { + var b bytes.Buffer + b.WriteByte('{') + for i, k := range o.keys { + if i > 0 { + b.WriteByte(',') + } + b.Write(mustJSON(k)) + b.WriteByte(':') + b.Write(o.vals[k]) + } + b.WriteByte('}') + return b.Bytes() +} + +func joinArray(items []json.RawMessage) json.RawMessage { + var b bytes.Buffer + b.WriteByte('[') + for i, v := range items { + if i > 0 { + b.WriteByte(',') + } + b.Write(v) + } + b.WriteByte(']') + return b.Bytes() +} + +func mustJSON(v any) json.RawMessage { + out, err := json.Marshal(v) + if err != nil { + panic(err) + } + return out +} diff --git a/internal/warmup/warmup_internal_test.go b/internal/warmup/warmup_internal_test.go new file mode 100644 index 0000000..ef2d6d4 --- /dev/null +++ b/internal/warmup/warmup_internal_test.go @@ -0,0 +1,152 @@ +package warmup + +import ( + "bytes" + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" +) + +func TestPrepareRewritesForOneToken(t *testing.T) { + raw := `{"model":"other","stream":true,"stream_options":{"include_usage":true},"max_completion_tokens":900,"tools":[{"type":"function"}],"messages":[{"role":"system","content":"s"},{"role":"user","content":"Hi"}]}` + out, err := Prepare([]byte(raw), "served") + if err != nil { + t.Fatal(err) + } + var got map[string]any + if err := json.Unmarshal(out, &got); err != nil { + t.Fatal(err) + } + if got["model"] != "served" || got["stream"] != false || got["max_tokens"] != float64(1) { + t.Fatalf("not rewritten: %v", got) + } + for _, k := range []string{"stream_options", "max_completion_tokens"} { + if _, ok := got[k]; ok { + t.Errorf("%s kept", k) + } + } + if _, ok := got["tools"]; !ok { + t.Error("tools dropped") + } +} + +func TestPrepareRejectsBodiesWithoutMessages(t *testing.T) { + for _, raw := range []string{`[]`, `{"model":"x"}`, `not json`} { + if _, err := Prepare([]byte(raw), "m"); err == nil { + t.Errorf("%s accepted", raw) + } + } +} + +func TestRunSendsKeyAndReportsCachedTokens(t *testing.T) { + var gotAuth, gotPath string + var gotBody map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAuth, gotPath = r.Header.Get("Authorization"), r.URL.Path + data, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(data, &gotBody) + _, _ = w.Write([]byte(`{"usage":{"prompt_tokens":9400,"prompt_tokens_details":{"cached_tokens":9390}}}`)) + })) + defer srv.Close() + path := filepath.Join(t.TempDir(), "w.json") + if err := os.WriteFile(path, []byte(`{"messages":[{"role":"user","content":"Hi"}]}`), 0o600); err != nil { + t.Fatal(err) + } + res, err := Run(context.Background(), srv.URL+"/v1", "k", "m", path) + if err != nil { + t.Fatal(err) + } + if gotAuth != "Bearer k" || gotPath != "/v1/chat/completions" || gotBody["model"] != "m" { + t.Fatalf("auth %q path %q body %v", gotAuth, gotPath, gotBody) + } + if res.PromptTokens != 9400 || res.CachedTokens != 9390 { + t.Fatalf("got %+v", res) + } +} + +func TestSendWithoutCachedTokens(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"usage":{"prompt_tokens":12}}`)) + })) + defer srv.Close() + res, err := Send(context.Background(), srv.Client(), srv.URL+"/v1/", "", []byte(`{}`)) + if err != nil { + t.Fatal(err) + } + if res.PromptTokens != 12 || res.CachedTokens != -1 { + t.Fatalf("got %+v", res) + } +} + +func TestSendReportsServerErrors(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Error(w, "no such model", http.StatusNotFound) + })) + defer srv.Close() + if _, err := Send(context.Background(), srv.Client(), srv.URL, "", []byte(`{}`)); err == nil { + t.Fatal("no error") + } +} + +func TestUsageCountsResponsesAPI(t *testing.T) { + var u Usage + if err := json.Unmarshal([]byte(`{"input_tokens":50,"input_tokens_details":{"cached_tokens":40}}`), &u); err != nil { + t.Fatal(err) + } + if p, c := u.Counts(); p != 50 || c != 40 { + t.Fatalf("got %d %d", p, c) + } +} + +func TestFromCaptureKeepsSharedPrefix(t *testing.T) { + raw := `{"model":"m","stream":true,"tools":[{"type":"function"}],"messages":[ + {"role":"system","content":"rules"}, + {"role":"user","content":"fix the bug in main.go"}, + {"role":"assistant","content":"ok"}, + {"role":"user","content":"thanks"}]}` + out, err := FromCapture([]byte(raw)) + if err != nil { + t.Fatal(err) + } + var got struct { + Tools []any `json:"tools"` + Messages []map[string]string `json:"messages"` + } + if err := json.Unmarshal(out, &got); err != nil { + t.Fatal(err) + } + if len(got.Tools) != 1 || len(got.Messages) != 2 { + t.Fatalf("got %s", out) + } + if got.Messages[0]["content"] != "rules" || got.Messages[1]["content"] != Placeholder { + t.Fatalf("got %s", out) + } + if _, err := FromCapture([]byte(`{"messages":[{"role":"system","content":"s"}]}`)); err == nil { + t.Fatal("request without a user message accepted") + } +} + +func TestRewritesKeepKeyOrder(t *testing.T) { + raw := `{"stream":true,"tools":[{"type":"function","function":{"name":"edit","parameters":{"type":"object","properties":{"path":{"type":"string"},"old":{"type":"string"},"new":{"type":"string"}}}}}],"messages":[{"role":"system","content":"s"},{"role":"user","content":"do it"}],"model":"m"}` + captured, err := FromCapture([]byte(raw)) + if err != nil { + t.Fatal(err) + } + out, err := Prepare(captured, "served") + if err != nil { + t.Fatal(err) + } + var compact bytes.Buffer + if err := json.Compact(&compact, out); err != nil { + t.Fatal(err) + } + want := `{"stream":false,"tools":[{"type":"function","function":{"name":"edit","parameters":{"type":"object","properties":{"path":{"type":"string"},"old":{"type":"string"},"new":{"type":"string"}}}}}],"messages":[{"role":"system","content":"s"},{"role":"user","content":"Hi"}],"model":"served","max_tokens":1}` + if compact.String() != want { + t.Fatalf("got %s\nwant %s", compact.String(), want) + } +} diff --git a/internal/web/api.go b/internal/web/api.go index 0321f93..781f954 100644 --- a/internal/web/api.go +++ b/internal/web/api.go @@ -81,6 +81,10 @@ type instanceStatus struct { ControlReason string `json:"control_reason,omitempty"` Default bool `json:"default"` + + // CachePercent is the share of prompt tokens served from the engine's + // cache, counted by the router; unset when it has nothing to report. + CachePercent *int `json:"cache_percent,omitempty"` } func (s *Server) handleStatus(w http.ResponseWriter, _ *http.Request) { @@ -122,9 +126,17 @@ func (s *Server) instances() []instanceStatus { names = append(names, inst.Name) } def := control.DefaultInstance(s.cfg, names) + var cache map[string]int + if s.cacheFor != nil && len(list) > 0 { + cache = s.cacheFor(list) + } out := make([]instanceStatus, 0, len(list)) for _, inst := range list { - out = append(out, s.instanceStatus(inst, inst.Name == def)) + r := s.instanceStatus(inst, inst.Name == def) + if p, ok := cache[inst.Name]; ok && r.Healthy { + r.CachePercent = &p + } + out = append(out, r) } return out } diff --git a/internal/web/server.go b/internal/web/server.go index b0af0f8..114e8c8 100644 --- a/internal/web/server.go +++ b/internal/web/server.go @@ -40,6 +40,10 @@ type Server struct { // its own unit does. The CLI owns that lookup, so it is passed in. clusterFor func(name string) Cluster + // cacheFor reports the share of prompt tokens each instance served from + // its cache, by profile name, as the router counted them. + cacheFor func(list []*control.Instance) map[string]int + // svcFor builds the unit for an instance; tests replace it. svcFor func(name string) service.Service @@ -224,6 +228,10 @@ const clusterControlReason = "A recipe cluster is serving this profile. The pane // instance. fn returns nil when the instance's own unit serves it. func (s *Server) SetCluster(fn func(name string) Cluster) { s.clusterFor = fn } +// SetPromptCache tells the panel how to read the router's prompt-cache share +// for the running instances. +func (s *Server) SetPromptCache(fn func(list []*control.Instance) map[string]int) { s.cacheFor = fn } + func (s *Server) cluster(name string) Cluster { if s.clusterFor == nil { return nil diff --git a/internal/web/server_internal_test.go b/internal/web/server_internal_test.go index 400f108..571632a 100644 --- a/internal/web/server_internal_test.go +++ b/internal/web/server_internal_test.go @@ -834,6 +834,39 @@ func TestClusterIsReportedAndNotDriven(t *testing.T) { } } +func TestPromptCacheIsShownForHealthyInstances(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer engine.Close() + + s := newTestServer(t) + writeInstance(t, s, "demo", portOf(t, engine.URL), "") + writeInstance(t, s, "other", "8002", "") + s.SetCluster(func(name string) Cluster { + if name != "demo" { + return nil + } + return fakeCluster{health: service.Health{State: "active", SubState: "running"}} + }) + s.SetPromptCache(func([]*control.Instance) map[string]int { + return map[string]int{"demo": 80, "other": 40} + }) + + got := s.status().Instances + if len(got) != 2 { + t.Fatalf("instances = %+v, want demo and other", got) + } + for _, inst := range got { + switch { + case inst.Profile == "demo" && (inst.CachePercent == nil || *inst.CachePercent != 80): + t.Errorf("demo cache = %v, want 80", inst.CachePercent) + case inst.Profile == "other" && inst.CachePercent != nil: + t.Errorf("other cache = %d, want none: it is not serving", *inst.CachePercent) + } + } +} + // Every running model is listed, and exactly one is the default Coder offers. func TestInstancesListsEveryRunningModel(t *testing.T) { s := newTestServer(t) diff --git a/internal/web/static/machine.js b/internal/web/static/machine.js index f489859..fd1e9a7 100644 --- a/internal/web/static/machine.js +++ b/internal/web/static/machine.js @@ -116,6 +116,7 @@ function instanceRow(i, privileged) { const bits = [`profile ${i.profile}`, `port ${i.port}`]; if (i.phase === "serving") bits.push(`http ${i.http_status}`); + if (i.cache_percent != null) bits.push(`cache ${i.cache_percent}%`); if (i.restarts > 0) bits.push(`${i.restarts} restarts`); // The field keeps the previous run's verdict, so it only explains a unit // that is failing now.