Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/http/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ Step labels in history (ATO-1382): `GET /api/sessions/{id}` adds `done_line` to

[Local model recheck](route-local-model-recheck.ts) (ATO-873): `POST /api/local-model/recheck`, no body, reads `localModels.url` fresh from the config file, probes it the way saving it does and answers `{ url, verdict, ok, message, portHint }` (`ok`, `loading`, `ollama`, `atomic_chat`, `openai_compat`, `key_rejected`, `not_model_server`, `unreachable`). It never takes a URL from the client and writes nothing. A failed step's and turn's `error` frame carries `code` and plain `advice` for a local server ([fields](failure-reason-frame.ts)): `local_server_not_found` (404/405 at `localModels.url`, the case for a Reconnect button), `local_provider_not_found` (the same at another address), `local_context_exceeded`, `local_tools_unsupported` (a 400 saying the server takes no tool calls: llama-server without `--jinja`, an Ollama model without tool support), `local_request_invalid`, `atomic_chat_key_required` (the `atomic-chat` provider answered 401/403: its API key protection is on, ATO-1274; not reported to Sentry), `local_port_taken` (the managed engine's address answered 401 while our engine's process is not there: another program holds the port, ATO-1277); `local_engine_down` wins over them, and so does `atomic_chat_not_running` (ATO-1278: the Atomic Chat app refused the connection past a short start grace; the user opens it).

[Local model performance](route-local-model-performance.ts) (issue #996): `GET /api/local-model/performance` answers `{ local, url, capacity, samples, slots }` for the desktop's Performance pane. `samples` (oldest first, at most `capacity`, 300, kept in memory only) are the runtime's llama-server client's completed requests, one each: `at`, `modelId` (file name only), `streamed`, `promptTokens`, `cacheTokens`, `promptMs`, `promptPerSecond` (`null` for a fully cached prompt), `predictedTokens`, `predictedMs`, `generationPerSecond`, `ttftMs` (streams: call start to first delta, retries and queueing included), `contextTokens` and `concurrent` (most calls open at once during it). A failed or abandoned request records nothing; no prompt or reply text is kept. `local` and `url` come from the cached config the live turns use. Only when `local` is true does the route read the server's `/slots` once, bounded at 1.5 s: `slots` is `{ total, busy, contextPerSlot }`, `null` on any failure. Always `200`; an older agent answers `404`.

[Telegram pairing routes](route-telegram-pairing.ts) (ATO-859): `POST /api/telegram/pairing` opens a 5-minute window through `TelegramChannel.startPairing` and answers `{ status: "waiting", code, expiresAt, timeoutMs, botUsername, instruction }` with `cache-control: no-store`. That reply is the only place the one-time code leaves the process; it is never logged and never in the `GET`. While an owner is paired the POST answers `409 owner_already_paired` until it is repeated with `{ "replaceOwner": true }`; a channel that is not `up` answers `409 telegram_not_running`. `GET` reports `idle`, `waiting`, `claimed` (with `owner: { userId, username }` and `pairedLine`), `expired`, `cancelled` or `failed`; `DELETE` cancels the window. Bearer auth like every `/api` route.

[Approval rules routes](route-approval-rules.ts) (ATO-1141): `GET /api/approval-rules?workingDir=|sessionId=` returns `{ rules, globalPath, projectPath, projectRules }`: the global file's rules and the folder's project-file ask / deny rules (read-only). `DELETE /api/approval-rules/{id}` removes a global rule (`{ removed: true }`, `404` when there is none). Contract: [approval README](../approval/README.md).
Expand Down
211 changes: 211 additions & 0 deletions src/http/route-local-model-performance.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
import { createServer, type Server } from "node:http";
import { readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";

import { resetConfigCache } from "../config/index.js";
import type { PerformanceSample } from "../llm/llama-performance-samples.js";
import type { AgentRuntime } from "../runtime/bootstrap.js";
import { startTestHarness, type Harness } from "./test-harness.js";

type SlotsReply = { status: number; body: string } | "hang";

/** A stand-in llama-server that answers `GET /slots` and counts the requests. */
async function startSlotsServer(
reply: SlotsReply,
): Promise<{ url: string; hits: () => number; stop: () => Promise<void> }> {
let hits = 0;
const srv: Server = createServer((req, res) => {
if (req.url !== "/slots") {
res.writeHead(404).end();
return;
}
hits += 1;
if (reply === "hang") return;
res.writeHead(reply.status, { "content-type": "application/json" }).end(reply.body);
});
await new Promise<void>((resolve) => srv.listen(0, "127.0.0.1", resolve));
const address = srv.address();
const port = typeof address === "object" && address ? address.port : 0;
return {
url: `http://127.0.0.1:${port}`,
hits: () => hits,
stop: () =>
new Promise((resolve) => {
srv.closeAllConnections();
srv.close(() => resolve());
}),
};
}

const SAMPLE: PerformanceSample = {
at: 1794329429000,
modelId: "Qwen3.6-35B-A3B-MXFP4_MOE.gguf",
streamed: true,
promptTokens: 2,
cacheTokens: 0,
promptMs: 32.969,
promptPerSecond: 60.663047104856076,
predictedTokens: 40,
predictedMs: 1158.29,
generationPerSecond: 33.67032435745798,
ttftMs: 412,
contextTokens: 42,
concurrent: 1,
};

/** Rewrite the harness's config file with an `llm` block and drop the cache. */
function setProviders(
harness: Harness,
activeTextProvider: string,
providers: Array<Record<string, unknown>>,
): void {
const path = join(harness.stateDir, "config.json");
const current = JSON.parse(readFileSync(path, "utf8")) as Record<string, unknown>;
writeFileSync(
path,
JSON.stringify({ ...current, llm: { activeTextProvider, providers } }, null, 2),
"utf8",
);
resetConfigCache();
}

async function getPerformance(
harness: Harness,
headers: Record<string, string> = {},
): Promise<{ status: number; body: Record<string, unknown> }> {
const res = await fetch(`${harness.baseUrl}/api/local-model/performance`, { headers });
return { status: res.status, body: (await res.json()) as Record<string, unknown> };
}

describe("GET /api/local-model/performance (issue #996)", () => {
let harness: Harness | null = null;
let server: Awaited<ReturnType<typeof startSlotsServer>> | null = null;

afterEach(async () => {
if (harness) await harness.cleanup();
harness = null;
if (server) await server.stop();
server = null;
});

async function withSlots(reply: SlotsReply): Promise<Harness> {
server = await startSlotsServer(reply);
harness = await startTestHarness({ localModelsUrl: server.url });
return harness;
}

it("answers a fresh agent with no samples and the live slots", async () => {
const h = await withSlots({
status: 200,
body: JSON.stringify([
{ id: 0, n_ctx: 262144, is_processing: true },
{ id: 1, n_ctx: 262144, is_processing: false },
]),
});
const { status, body } = await getPerformance(h);
expect(status).toBe(200);
expect(body).toEqual({
local: true,
url: server!.url,
capacity: 300,
samples: [],
slots: { total: 2, busy: 1, contextPerSlot: 262144 },
});
expect(server!.hits()).toBe(1);
});

it("reports slots as null when /slots hangs past its bound", async () => {
const h = await withSlots("hang");
const started = Date.now();
const { status, body } = await getPerformance(h);
expect(status).toBe(200);
expect(body.slots).toBeNull();
expect(Date.now() - started).toBeGreaterThanOrEqual(1400);
expect(Date.now() - started).toBeLessThan(5000);
});

const unreadable: Array<[string, SlotsReply]> = [
["a 404", { status: 404, body: "{}" }],
["a non-array body", { status: 200, body: JSON.stringify({ slots: [] }) }],
["an array of no slots", { status: 200, body: JSON.stringify([null]) }],
];
for (const [name, reply] of unreadable) {
it(`reports slots as null for ${name}`, async () => {
const h = await withSlots(reply);
const { status, body } = await getPerformance(h);
expect(status).toBe(200);
expect(body.slots).toBeNull();
expect(body.local).toBe(true);
});
}

it("keeps the load when a slot has no n_ctx", async () => {
const h = await withSlots({
status: 200,
body: JSON.stringify([{ id: 0, is_processing: true }]),
});
const { body } = await getPerformance(h);
expect(body.slots).toEqual({ total: 1, busy: 1, contextPerSlot: null });
});

it("asks nothing of the local server while the chat is on a cloud model", async () => {
const h = await withSlots({ status: 200, body: JSON.stringify([{ id: 0, n_ctx: 8 }]) });
let slotsCalls = 0;
const runtime = h.runtime as AgentRuntime;
runtime.localPerformance = () => ({
capacity: 300,
samples: [SAMPLE],
fetchSlots: async () => {
slotsCalls += 1;
return [];
},
});
setProviders(h, "cloud", [
{ id: "local", kind: "llama-server" },
{ id: "cloud", kind: "openrouter" },
]);
const { status, body } = await getPerformance(h);
expect(status).toBe(200);
expect(body).toEqual({
local: false,
url: server!.url,
capacity: 300,
samples: [SAMPLE],
slots: null,
});
expect(slotsCalls).toBe(0);
expect(server!.hits()).toBe(0);
});

it("counts a custom provider of kind llama-server as local", async () => {
const h = await withSlots({ status: 200, body: JSON.stringify([{ id: 0, n_ctx: 4096 }]) });
setProviders(h, "box", [
{ id: "box", kind: "llama-server" },
{ id: "cloud", kind: "openrouter" },
]);
const { body } = await getPerformance(h);
expect(body.local).toBe(true);
expect(body.slots).toEqual({ total: 1, busy: 0, contextPerSlot: 4096 });
});

it("answers without samples or slots on a runtime that has no local performance", async () => {
const h = await withSlots({ status: 200, body: JSON.stringify([{ id: 0, n_ctx: 8 }]) });
const runtime = h.runtime as AgentRuntime;
delete runtime.localPerformance;
const { status, body } = await getPerformance(h);
expect(status).toBe(200);
expect(body).toMatchObject({ local: true, capacity: 300, samples: [], slots: null });
expect(server!.hits()).toBe(0);
});

it("sits behind the API key", async () => {
server = await startSlotsServer({ status: 200, body: "[]" });
harness = await startTestHarness({ apiKey: "secret", localModelsUrl: server.url });
const missing = await fetch(`${harness.baseUrl}/api/local-model/performance`);
expect(missing.status).toBe(401);
const { status, body } = await getPerformance(harness, { authorization: "Bearer secret" });
expect(status).toBe(200);
expect(body.capacity).toBe(300);
});
});
79 changes: 79 additions & 0 deletions src/http/route-local-model-performance.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import { getConfig } from "../config/index.js";
import {
activeTextProviderIsLlamaServer,
resolveLlmConfig,
} from "../llm/provider/registry/index.js";
import { PERFORMANCE_CAPACITY } from "../llm/llama-performance-samples.js";
import { readSlotOccupancy } from "../tools/fusion/worker-result.js";
import { sendJson, type HttpHandler } from "./request-context.js";

/** How long the route waits for the local server's `/slots`. */
export const PERFORMANCE_SLOTS_TIMEOUT_MS = 1500;

export interface PerformanceSlots {
total: number;
busy: number;
/** The first slot's `n_ctx`; `null` when the server does not report it. */
contextPerSlot: number | null;
}

/**
* `GET /api/local-model/performance` — the desktop's Performance pane
* (issue #996). Answers
*
* `{ local, url, capacity, samples, slots }`
*
* `samples` are the local model server's recent completed requests,
* oldest first: speeds, token counts, first-token wait and how many
* requests shared the server, never any prompt or reply text. `local`
* says whether the running agent's chat goes to llama-server at all; it
* and `url` come from the cached config the live turns read, which
* `POST /api/llm/active` resets. Only when `local` is true does the route
* ask the server's `/slots` (once, bounded at 1.5 s) for the live load
* and the context per slot; any failure there is `slots: null`, never an
* error status. Writes nothing. Behind the API key like every `/api`
* route.
*/
export function createLocalModelPerformanceHandler(): HttpHandler {
return async (_req, res, ctx) => {
const config = getConfig();
const local = activeTextProviderIsLlamaServer(resolveLlmConfig(config));
const source = ctx.runtime.localPerformance?.() ?? null;
const slots = local && source !== null ? await readSlots(source.fetchSlots) : null;
sendJson(res, 200, {
local,
url: config.localModels.url,
capacity: source?.capacity ?? PERFORMANCE_CAPACITY,
samples: source?.samples ?? [],
slots,
});
};
}

async function readSlots(
fetchSlots: (timeoutMs: number) => Promise<unknown>,
): Promise<PerformanceSlots | null> {
let body: unknown;
try {
body = await fetchSlots(PERFORMANCE_SLOTS_TIMEOUT_MS);
} catch {
return null;
}
return slotsFromBody(body);
}

/** The slot table's load and context size, or `null` when it is not a slot table. */
export function slotsFromBody(body: unknown): PerformanceSlots | null {
const occupancy = readSlotOccupancy(body);
if (occupancy === null || !Array.isArray(body)) return null;
const first = body.find(
(entry): entry is Record<string, unknown> => entry !== null && typeof entry === "object",
);
const nCtx = first?.n_ctx;
return {
total: occupancy.total,
busy: occupancy.busy,
contextPerSlot:
typeof nCtx === "number" && Number.isFinite(nCtx) && nCtx > 0 ? nCtx : null,
};
}
3 changes: 3 additions & 0 deletions src/http/route-table.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { createChatCompletionsHandler } from "./openai-chat-completions.js";
import { createCancelCompletionHandler } from "./openai-completions-cancel.js";
import { createHealthHandler } from "./route-health.js";
import { createLocalModelRecheckHandler } from "./route-local-model-recheck.js";
import { createLocalModelPerformanceHandler } from "./route-local-model-performance.js";
import { createModelTiersHandler } from "./route-model-tiers.js";
import { createModelsHandler } from "./route-models.js";
import { createCapabilitiesHandler } from "./route-capabilities.js";
Expand Down Expand Up @@ -226,6 +227,8 @@ export function buildRouteTable(): RouteDefinition[] {
{ method: "POST", path: "/api/context-preview", handler: createContextPreviewHandler() },
// ATO-873: the desktop's Reconnect button after a local server answered 404/405.
{ method: "POST", path: "/api/local-model/recheck", handler: createLocalModelRecheckHandler() },
// Issue #996: the desktop's Performance pane; recent local request timings and the live slots.
{ method: "GET", path: "/api/local-model/performance", handler: createLocalModelPerformanceHandler() },
// ATO-1021: the desktop's model pick, swapped live instead of restarting serve.
{ method: "POST", path: "/api/llm/active", handler: createSetActiveLlmHandler() },
// ATO-1024: the desktop starts serve first and attaches the embedding server once it is up.
Expand Down
Loading