Skip to content

feat(http): GET /api/local-model/performance for the local model's speed (#996) - #1119

Open
sheepdogs wants to merge 5 commits into
AtomicBot-ai:mainfrom
sheepdogs:feat/996-local-perf-route
Open

sheepdogs wants to merge 5 commits into
AtomicBot-ai:mainfrom
sheepdogs:feat/996-local-perf-route

Conversation

@sheepdogs

@sheepdogs sheepdogs commented Nov 10, 2026 •

Copy link
Copy Markdown

Part of #996: the agent half. The desktop pane that reads this route is #1120, against rel/xp-build (merge this one first; the pane says "update the agent" against an agent without the route).

What

GET /api/local-model/performance — the local model server's recent per-request timings, so a client can show how fast the local model is running without reading llama-server.log.

{
  "local": true,
  "url": "http://127.0.0.1:9931",
  "capacity": 300,
  "samples": [
    { "at": 1794329429000, "modelId": "Qwen3.6-35B-A3B-MXFP4_MOE.gguf", "streamed": true,
      "promptTokens": 2046, "cacheTokens": 5748, "promptMs": 1628.1, "promptPerSecond": 1256.6,
      "predictedTokens": 56, "predictedMs": 1152.3, "generationPerSecond": 48.6,
      "ttftMs": 1727, "contextTokens": 7850, "concurrent": 1 }
  ],
  "slots": { "total": 1, "busy": 0, "contextPerSlot": 262144 }
}
  • Samples come from the final event's timings, one per completed complete() / completeStream() call, newest 300 in memory (nothing persisted, no prompt or reply text, and model cut to the file name so a local path never leaves the process).
  • ttftMs is measured by the client: from the call's start to its first content or reasoning delta, so it includes retries, backoff and queueing — the wait the person actually sees.
  • concurrent is the most calls open on the client at once during this one. One registration per call, opened before runWithRetry, so a retry never counts twice.
  • slots is one bounded (1.5 s) /slots read through the existing readSlotOccupancy. It is null on any failure, and /slots is not asked at all when the active text provider is not the local server.
  • A sample is recorded only when the call completes: a transport error after the stop event, a consumer walking away or aborting after the done chunk, and a stream that ends without a stop event record nothing.

recordThroughput / measuredTokensPerSecond (Fusion's worker time limits) are unchanged. They now share one generationRate helper with the samples, and tests pin the old behaviour, including numeric-string timings and ignoring a rate that overflows to Infinity.

Files

  • src/llm/llama-performance-samples.ts (new): mapper, ring, generationRate.
  • src/llm/llama-server-client.ts: registrations, first-token time, stage-then-commit.
  • src/runtime/runtime-contract.ts, src/runtime/bootstrap.ts: an optional localPerformance() wired to the same client the provider uses.
  • src/http/route-local-model-performance.ts (new) + route-table.ts, src/http/README.md.

Checks

  • npm run lint: clean.
  • npm run typecheck:tests: no new errors.
  • npm run docs:check, npm run imports:check: pass.
  • npx vitest run src/llm src/http src/runtime: 275 files, 3101 passed, 1 skipped (+48 tests).
  • The full CI sequence from test.yml (docs/imports/typecheck self-tests, quarantine check, scripts/run-tests.mjs): 1552 files, 19377 passed, 5 skipped, 0 failed. Re-run after rebasing onto the Wave 1 merge (Wave 1: merge the 78 checked fix PRs into main #1256).
  • Mutation checks (each test group shown to fail with its guard removed):
    • recording the sample at the stop event instead of on return;
    • one registration per attempt instead of per call;
    • the abort check on return;
    • the finite-rate guard;
    • lenient number parsing for the throughput mean.
  • Run live against llama.cpp b11028 serving Qwen3.6-35B-A3B on the native /completion endpoint, from a built serve:
    • the route answered before any chat with samples: [] and real slot data;
    • after a few chat turns, the samples carried real speeds, cache reuse and concurrent up to 3 (the agent's own side requests overlapping its turn);
    • a 0.7.0 agent answers 404 on the route, which the desktop reads as "update the agent".

Not covered by a direct test:

  • a complete() that throws after its timings arrived: nothing can be recorded there, because the commit runs only after runWithRetry returns;
  • the repetition-stop return when it has a staged stop payload. Nothing is staged on that path in practice.

AI assistance

Developed with AI assistance (Claude Code built it to a written spec; an independent Codex review ran three rounds before this PR) under my direction. I reviewed the diff, re-ran every check above myself, and ran it against a real llama.cpp server.

sheepdogs and others added 5 commits November 12, 2026 10:19
LlamaServerClient keeps the last 300 completed requests' timings for a
performance view (issue AtomicBot-ai#996): prompt and generation speed, token
counts, time to first token and how many calls shared the server. A
sample is staged where the final timings arrive and recorded only when
the call succeeds, so a stream that fails or is abandoned after its stop
event leaves nothing. One registration per call, retries included.

recordThroughput now takes its rate from the shared generationRate
helper; the throughput mean is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An optional AgentRuntime.localPerformance hands out the samples and a
bounded /slots read of the llama-server client the provider uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers { local, url, capacity, samples, slots } for the desktop's
Performance pane. /slots is read only while the chat is on the local
server, once, bounded at 1.5 s; any failure there is slots: null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared generationRate helper read numbers only, so the client's
rolling throughput mean (what Fusion sizes local worker time limits from)
stopped counting a timings object whose values arrive as strings. It
read them before through toNumber. The mean now passes lenientNum and
keeps that behaviour; the performance samples still read numbers only.
A caller that aborts after the done chunk ends a call that did not
complete, but a body that has already closed does not throw on the next
read, so the generator returned normally and recorded a sample. It now
asks the caller's signal and the request's own controller before
recording.

generationRate no longer returns Infinity when a tiny predicted_ms
overflows the division; the throughput mean never took a non-finite
rate before the shared helper, and does not now.
@sheepdogs
sheepdogs force-pushed the feat/996-local-perf-route branch from f907f0d to 9754a60 Compare November 12, 2026 16:21
@sheepdogs

Copy link
Copy Markdown
Author

Rebased onto main after the Wave 1 merge (#1256). It conflicted in one file, src/llm/llama-server-client.ts: the new stream-close logging and outcome tracking from #1252 sit in the same places as the per-call registration and the sample commit here. Both are kept. A sample is still committed only after outcome = "done", and the call is still closed after the new close log.

The full test.yml sequence on the rebased branch: 1552 files, 19377 passed, 5 skipped, 0 failed.

#1120 (desktop) still merges cleanly into rel/xp-build after Wave 1. Merged locally, its unit tests are 1620 / 1620.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant