Repository navigation
Conversation
LlamaServerClient keeps the last 300 completed requests' timings for a performance view (issue AtomicBot-ai#996): prompt and generation speed, token counts, time to first token and how many calls shared the server. A sample is staged where the final timings arrive and recorded only when the call succeeds, so a stream that fails or is abandoned after its stop event leaves nothing. One registration per call, retries included. recordThroughput now takes its rate from the shared generationRate helper; the throughput mean is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An optional AgentRuntime.localPerformance hands out the samples and a bounded /slots read of the llama-server client the provider uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers { local, url, capacity, samples, slots } for the desktop's
Performance pane. /slots is read only while the chat is on the local
server, once, bounded at 1.5 s; any failure there is slots: null.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared generationRate helper read numbers only, so the client's rolling throughput mean (what Fusion sizes local worker time limits from) stopped counting a timings object whose values arrive as strings. It read them before through toNumber. The mean now passes lenientNum and keeps that behaviour; the performance samples still read numbers only.
A caller that aborts after the done chunk ends a call that did not complete, but a body that has already closed does not throw on the next read, so the generator returned normally and recorded a sample. It now asks the caller's signal and the request's own controller before recording. generationRate no longer returns Infinity when a tiny predicted_ms overflows the division; the throughput mean never took a non-finite rate before the shared helper, and does not now.
sheepdogs
force-pushed
the
feat/996-local-perf-route
branch
from
November 12, 2026 16:21
f907f0d to
9754a60
Compare
Author
|
Rebased onto The full #1120 (desktop) still merges cleanly into |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #996: the agent half. The desktop pane that reads this route is #1120, against
rel/xp-build(merge this one first; the pane says "update the agent" against an agent without the route).What
GET /api/local-model/performance— the local model server's recent per-request timings, so a client can show how fast the local model is running without readingllama-server.log.{ "local": true, "url": "http://127.0.0.1:9931", "capacity": 300, "samples": [ { "at": 1794329429000, "modelId": "Qwen3.6-35B-A3B-MXFP4_MOE.gguf", "streamed": true, "promptTokens": 2046, "cacheTokens": 5748, "promptMs": 1628.1, "promptPerSecond": 1256.6, "predictedTokens": 56, "predictedMs": 1152.3, "generationPerSecond": 48.6, "ttftMs": 1727, "contextTokens": 7850, "concurrent": 1 } ], "slots": { "total": 1, "busy": 0, "contextPerSlot": 262144 } }timings, one per completedcomplete()/completeStream()call, newest 300 in memory (nothing persisted, no prompt or reply text, andmodelcut to the file name so a local path never leaves the process).ttftMsis measured by the client: from the call's start to its first content or reasoning delta, so it includes retries, backoff and queueing — the wait the person actually sees.concurrentis the most calls open on the client at once during this one. One registration per call, opened beforerunWithRetry, so a retry never counts twice.slotsis one bounded (1.5 s)/slotsread through the existingreadSlotOccupancy. It isnullon any failure, and/slotsis not asked at all when the active text provider is not the local server.recordThroughput/measuredTokensPerSecond(Fusion's worker time limits) are unchanged. They now share onegenerationRatehelper with the samples, and tests pin the old behaviour, including numeric-string timings and ignoring a rate that overflows toInfinity.Files
src/llm/llama-performance-samples.ts(new): mapper, ring,generationRate.src/llm/llama-server-client.ts: registrations, first-token time, stage-then-commit.src/runtime/runtime-contract.ts,src/runtime/bootstrap.ts: an optionallocalPerformance()wired to the same client the provider uses.src/http/route-local-model-performance.ts(new) +route-table.ts,src/http/README.md.Checks
npm run lint: clean.npm run typecheck:tests: no new errors.npm run docs:check,npm run imports:check: pass.npx vitest run src/llm src/http src/runtime: 275 files, 3101 passed, 1 skipped (+48 tests).test.yml(docs/imports/typecheck self-tests, quarantine check,scripts/run-tests.mjs): 1552 files, 19377 passed, 5 skipped, 0 failed. Re-run after rebasing onto the Wave 1 merge (Wave 1: merge the 78 checked fix PRs into main #1256)./completionendpoint, from a builtserve:samples: []and real slot data;concurrentup to 3 (the agent's own side requests overlapping its turn);Not covered by a direct test:
complete()that throws after its timings arrived: nothing can be recorded there, because the commit runs only afterrunWithRetryreturns;AI assistance
Developed with AI assistance (Claude Code built it to a written spec; an independent Codex review ran three rounds before this PR) under my direction. I reviewed the diff, re-ran every check above myself, and ran it against a real llama.cpp server.