Skip to content

warmup: warm the prompt cache after start and show cache reuse - #67

Merged
DevelopmentCats merged 2 commits into
mainfrom
devcats/warmup
Nov 12, 2026
Merged

DevelopmentCats merged 2 commits into
mainfrom
devcats/warmup

Conversation

@DevelopmentCats

Copy link
Copy Markdown
Collaborator

Refs #65.

The first agent turn on a fresh or restarted server re-reads the whole fixed prompt. This sends that prompt ahead of time and shows how much of each model's prompts is served from cache. The cache size default is unchanged.

What changes

  • WARMUP_FILE (profile key): path to a JSON /v1/chat/completions body. It is sent with max_tokens: 1:
    • after up, restart and switch, once the server is healthy;
    • by the router when a backend goes from not answering to answering (reboot, crash restart).
      It is a plain chat request, so it works on every engine.
  • llm-helper warmup capture <profile>: the router saves the next chat request for that model, keeps everything before the first user message, replaces that message with Hi, writes it under the data directory and sets WARMUP_FILE.
  • llm-helper warmup [profile]: sends the request twice and prints prompt and cached tokens for each send.
  • Cache counts: the router reads usage (streamed or not) from chat replies and counts prompt and cached tokens per model. Shown in status, the panel (cache N%) and GET /router/stats (behind the API key like the rest of the router). Router traffic only.
  • vLLM's serve args gain --enable-prompt-tokens-details so it reports cached tokens.
  • Docs: a prompt cache section in the llama.cpp part of providers.md (host --cache-ram cache, the hybrid checkpoint limit, setting the size with LLAMA_ARG_CACHE_RAM or EXTRA_ARGS, unified memory), WARMUP_FILE in profiles.md, cache counts in multi-instance.md.

Not covered

  • Recipe launches (waitForLaunch) have no warm-up hook after boot; the router still warms them when they come up.
  • Recipe container vLLM launches don't get --enable-prompt-tokens-details, so the router shows no cache share for them.
  • Where llama.cpp puts its checkpoints is upstream; chats whose system prompt or tools differ still can't share a cached prefix on hybrid models.

Checks

  • gofmt -l ., go vet ./..., go test -race ./... (only internal/detect TestProbeReportsThisMachine fails, same as on main), tools/recipediff vet and tests, prettier --check ..
  • Manual, llama.cpp from Homebrew on CPU with LiquidAI/LFM2-350M-GGUF:Q4_K_M (hybrid) behind the router: the router warmed the backend when it came up, captured a streamed chat (~2.6k-token system prompt plus a tool), and both warm-ups through the router read 2647 of 2656 prompt tokens from the restored checkpoint. The router counts matched. Not tested on the Sparks or on vLLM.

A profile's WARMUP_FILE is a chat request sent after up, restart and switch, and by the router when a server comes back. 'warmup capture' records one from a real request through the router; 'warmup' sends it and reports cached tokens. The router counts prompt and cached tokens per model for status, the panel and /router/stats. vLLM is started with --enable-prompt-tokens-details. Docs cover llama.cpp's prompt cache on hybrid models.

Refs #65
Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The capture endpoint allows anyone with the shared inference API key to retrieve another client’s complete request, exposing messages, conversation history, and tool results.

5 open issues
What changed in this PR

Adds prompt-cache warm-up and reuse reporting to llm-helper, addressing the first-turn latency described in #65 without changing cache-size defaults.

Changes:

  • Adds WARMUP_FILE, capture/manual warm-up commands, and automatic warm-up hooks.
  • Tracks cached prompt tokens through the router and exposes them in status, the panel, and /router/stats.
  • Enables vLLM token details and documents cache behavior, configuration, and limitations.
File Description
internal/​web/​static/​machine.js Displays cache percentages.
internal/​web/​server.go Adds a cache-stat lookup callback.
internal/​web/​server_internal_test.go Tests cache display for healthy instances.
internal/​web/​api.go Includes cache percentages in instance status.
internal/​warmup/​warmup.go Implements request preparation, sending, capture trimming, and usage parsing.
internal/​warmup/​warmup_internal_test.go Tests warm-up requests and usage handling.
internal/​router/​models.go Integrates recovery warm-up, capture, and usage inspection.
internal/​router/​cache.go Implements cache accounting and capture coordination.
internal/​router/​cache_internal_test.go Tests accounting, capture, and recovery warm-up.
internal/​profile/​profile.go Defines the WARMUP_FILE profile setting.
internal/​engine/​registry_test.go Updates expected vLLM arguments.
internal/​cli/​watch.go Adds warm-up after watched startup.
internal/​cli/​warmup.go Adds warm-up commands and cache-reporting helpers.
internal/​cli/​warmup_internal_test.go Tests CLI warm-up and router integration.
internal/​cli/​vllmargs_internal_test.go Updates expected launch arguments.
internal/​cli/​serve.go Connects panel status to router cache counts.
internal/​cli/​router.go Adds stats/capture endpoints and backend warm-up configuration.
internal/​cli/​root.go Registers the warm-up command.
internal/​cli/​root_internal_test.go Updates the pinned command list.
internal/​cli/​profiles.go Adds cache counts to status output.
internal/​cli/​editset.go Recognizes WARMUP_FILE as a standard key.
docs/​reference/​cli/​warmup/​capture.md Adds generated capture-command reference.
docs/​reference/​cli/​warmup.md Adds generated warm-up-command reference.
docs/​reference/​cli.md Lists the warm-up command.
docs/​providers.md Explains llama.cpp cache behavior and sizing.
docs/​profiles.md Documents warm-up configuration and capture.
docs/​multi-instance.md Documents router cache reporting and limits.
docs/​manifest.json Routes the new reference pages.
CHANGELOG.md Records warm-up and cache-reporting features.
catalogue/​engines/​vllm.json Enables prompt-token cache details.

Review effort: Balanced

Comment thread internal/cli/router.go Resolved
Comment thread internal/cli/watch.go Resolved
Comment thread internal/router/cache.go Resolved
Comment thread internal/router/models.go Resolved
Comment thread internal/warmup/warmup.go Resolved
@DevelopmentCats

Copy link
Copy Markdown
Collaborator Author

Hardware test results on two DGX Sparks.

llama.cpp b12242 (CPU), Qwen3.5-0.8B Q4_K_M (hybrid), -np 1 --cache-ram 1024.
The test prompt is about 5.35k tokens of system prompt plus 20 tools; CPU prefill is about 130 tok/s.

Check Result
warmup with no WARMUP_FILE pass: clear error pointing at warmup capture
Second warmup capture while one waits pass: 409
Capture of a real request through the router pass: system + user "Hi", tools kept, max_tokens 8, profile updated
Capture of a request with no user message pass: 422, existing warm-up file untouched
warmup <profile> on a warm server pass: 5338 then 5345 of 5349 from cache
status cache: line pass
Container restart, router warms it, then a real request 5352 prompt, 5338 cached, 0.5 s
Same restart without WARMUP_FILE (baseline) 5352 prompt, 0 cached, 40.4 s
Two-turn chat after the warm-up 5338 of 5364 cached, 0.7 s

GLM-5.3-Flash on TensorFold, with a scratch router in front of the running server.
The test prompt is about 7.5k tokens of system prompt plus 20 tools.

Check Result
warmup with no WARMUP_FILE, second capture, capture with no user message pass: error, 409, 422
Capture of a real Coder Agents request pass: system (28,001 chars) + "Hi", 20 tools
warmup on a warm prefix pass: 7488 of 7506 from cache on both sends
Cold prefix: warmup, then a real request pass: first send 1728 cached, second 7488; real request 7488/7518
Router sees the backend come back and warms it pass: warmed about 6 s after it came up; next real request 7488/7516
/router/capture from a non-loopback address pass: 403
status cache: line pass: "37% of 49098 prompt tokens came from the cache (router traffic only)"

TensorFold keeps its own prefix cache in 64-token blocks and reuses everything before a change, so there the warm-up only saves the first cold read of each prefix (about 3.9 s down to 0.2 s when idle).
On llama.cpp hybrid models it also avoids the full re-read after a restart.

Not tested: auto warm-up from up/restart/switch (it would create systemd units on the boxes), the panel's cache N%, the vLLM --enable-prompt-tokens-details flag, and GPU llama.cpp.

@DevelopmentCats
DevelopmentCats merged commit e257005 into main Nov 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants