Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,35 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.

## Unreleased

### A Bot is offered the tools its message needs, not every tool it holds

A model chooses the right tool reliably out of about ten and unreliably out of thirty, and it fails
quietly: it calls a plausible neighbour, or calls nothing and answers from what it already knew. Two
connectors is enough to cross that line, so a Bot holding more than twelve tools is now offered, for
each run, the tools of the skills that match the message.

Skills already declare the tools they need. That declaration is now what the offer is built from:
the deployment asks its own model which skills a message needs, and the Bot gets those skills' tools
plus every granted tool no skill has claimed. Nothing here can widen a Bot: the offer is intersected
with the grants, so naming a tool in a skill still grants nobody anything.

Nothing changes for a deployment that has not declared tools on any skill, or whose Bots hold twelve
tools or fewer. Those Bots are built exactly as before, with no extra model call.

There is a new audit event, `mcp.tools_discovered`, written before the run. It says how many tools
were offered out of how many granted, and why: the skills chosen, or that nothing was declared, or
that the choice could not be made. It answers "why did it call that", and the harder question, "why
did it not call anything at all" — which until now left no trace.

### The intent router works again behind a gateway

`OPENAI_BASE_URL` is documented ending in `/v1`, and the router appended `/v1/chat/completions` to
it, so every call went to `/v1/v1/chat/completions` and 404'd. The router reads a failure as "not
sure" and falls back to the default coworker, so on any deployment that set the variable — a
gateway, a proxy, a self-hosted model, which is the only reason to set it — untagged messages
silently stopped being routed and nothing said why. The version segment is now added only when the
configured URL does not already carry one.

## 0.0.2

### Upgrading
Expand Down
44 changes: 44 additions & 0 deletions app/src/routes/_authed/admin/audit.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,18 @@ function Row({
*/}
{event.eventType === "channel.routed" && event.targetId ? (
<span title={event.targetId}>{nameFor(event.targetId)}</span>
) : /*
* A discovery row's subject is the narrowing itself, so the numbers are the subject. A
* reader asking "why did it not call the tool" needs to see that eleven of thirty were
* offered before anything else on the row means anything.
*/
event.eventType === "mcp.tools_discovered" ? (
<span className="font-mono text-xs">
{typeof payload.offered === "number" &&
typeof payload.granted === "number"
? `${payload.offered} of ${payload.granted} tools`
: "-"}
</span>
) : /* Named targets and file paths are the audit subject before page elements. */
NAMED_TARGETS.has(event.targetType) && event.targetId ? (
<span className="font-mono text-xs">
Expand Down Expand Up @@ -267,6 +279,20 @@ function Row({
{payload.reason}
</div>
) : null}
{/*
* Why this run was offered what it was, which is the only part of a discovery row that
* cannot be worked out from the numbers. "Nothing declared" and "selector unavailable" both
* offer everything and mean entirely different things about the deployment.
*/}
{event.eventType === "mcp.tools_discovered" &&
typeof payload.reason === "string" ? (
<div className="mt-0.5 text-xs text-muted-foreground">
{DISCOVERY_REASONS[payload.reason] ?? payload.reason}
{Array.isArray(payload.skills) && payload.skills.length > 0
? `: ${payload.skills.join(", ")}`
: ""}
</div>
) : null}
{event.eventType === "mcp.callback_refused" &&
typeof payload.refusal === "string" ? (
<div className="mt-0.5 text-xs text-muted-foreground">
Expand Down Expand Up @@ -332,6 +358,21 @@ function Row({
*
* Anything else falls through to the element or file subject.
*/
/**
* Why a run was offered the tools it was, in words rather than in the slug the server writes.
*
* Every one of these looks the same from outside: the Bot was handed some tools. The distinction is
* the difference between a deployment that narrowed on purpose, one that has never declared a skill,
* and one whose selector could not be reached, and only the last is a fault.
*/
const DISCOVERY_REASONS: Record<string, string> = {
"under-floor": "Few enough tools to offer them all",
"nothing-declared": "No skill declares any of these tools",
unavailable: "Could not choose, so all were offered",
"nothing-chosen": "No skill applied, so all were offered",
selected: "Chosen by skill",
};

const NAMED_TARGETS = new Set([
"component",
"mcp_tool",
Expand Down Expand Up @@ -368,6 +409,9 @@ const DECISIONS: Record<string, string> = {
// A function failure is execution failure, not a policy refusal.
"component.function_failed": "Could not be read",

// Not a call and not a decision: the tools this run was allowed to see. Worded so nobody reads it
// as permission, which it is not — everything named was already granted.
"mcp.tools_discovered": "Tools offered for one run",
"mcp.call_succeeded": "Called on this Bot's behalf",
"mcp.call_rejected": "Blocked",
"mcp.call_failed": "The server did not answer",
Expand Down
1 change: 1 addition & 0 deletions bun.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,14 @@ A catalogue entry says whose credential a Bot reaches it with, which is a differ

Every MCP call checks the grant first, then evaluates the same action policy engine with MCP context, then audits the result.

### Which tools a run is offered

A model picks the right tool reliably out of about ten, and unreliably out of thirty. A deployment that connects two vendors passes that point on its first afternoon, so a Bot holding more than a handful of tools is offered, per run, only the tools of the skills that match the message.

A skill declares the tools it needs (`skill_tools`). Before the run starts, the deployment asks its own model which skills the message needs, and the Bot is built with those skills' tools plus every granted tool no skill claims. A declaration grants nothing: the offer is always intersected with what the Bot was already granted, so writing a skill can never hand anybody a tool.

This narrows the offer. It is not a boundary, and it never substitutes for one. The grant, the policy and the audit row decide what may happen; this decides only what the model can see. Every way it can fail — no skills declared, a model that cannot answer, a message that matches nothing, twelve tools or fewer — leaves the whole catalogue offered, because a narrowing that failed closed would remove capability an administrator granted, silently. `mcp.tools_discovered` records what was offered, out of how much, and why.

## Tenant package and knowledge

`TENANT_PACKAGE_DIR` points at the tenant package. The default is `../examples/fintech`.
Expand Down
1 change: 1 addition & 0 deletions server/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
"drizzle-orm": "^0.45.2",
"hono": "^4.10.0",
"postgres": "^3.4.9",
"rxjs": "7.8.1",
"yaml": "^2.9.0",
"zod": "^4.4.3"
},
Expand Down
18 changes: 18 additions & 0 deletions server/src/audit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,24 @@ export const auditEventTypes = [
* endpoint that dies mid-answer from one that never answers at all.
*/
"agent.stream_stalled",
/*
* Which of a Bot's tools were put in front of the model for one run, and why those.
*
* Discovery, recorded as its own fact, because a run is now offered a subset of what the Bot holds
* and every other row here answers a question about a call that happened. This one answers "why
* did it call that", and its harder twin, "why did it not call anything" — a Bot that had the
* right tool granted, was not offered it, and answered from memory leaves no other trace at all.
* Without this row that failure is indistinguishable from a model that simply chose badly.
*
* DISCOVERY IS NOT PERMISSION, and the row is not an authorization record. Everything named here
* was already granted; being offered is what changed. A tool still goes through the grant, the
* policy and `mcp.call_succeeded` or `mcp.call_rejected` before anything happens, so this row
* never appears in place of one of those, only before it.
*
* `reason` is the part worth reading. It separates a deployment that narrowed from one that never
* declared anything and one whose selector was unreachable, which look identical from outside.
*/
"mcp.tools_discovered",
"mcp.call_succeeded",
"mcp.call_rejected",
/*
Expand Down
Loading