Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,52 @@ Sessions survive and nobody signs in again.
next migration.** Repair it with
`update drizzle.__drizzle_migrations set created_at = 1787359000000 where created_at = 1787444747113;`
or start from a fresh database, where migrations all run in one pass and ordering cannot bite.
- **Every Bot ran a model two generations old, and it was costing tool calls.** The example package
shipped `gpt-4.1` as the default for every built-in Bot. Asked to open a page behind a sign-in,
those Bots answered "would you like me to prompt you to sign in?" and called nothing, three times
out of three, while the prompt forbids that sentence in as many words. On `gpt-5.6-terra` the same
question produces the tool call first try. The default is now `gpt-5.6-terra` across the package,
the compose services and both example Bots, and the Responses API is inferred from the model rather
than left to a separate switch, because `gpt-5.6-*` rejects function tools on chat completions and
a deployment that set the model without knowing that got a Bot which started, looked healthy, and
failed on its first tool call. It is a default, not a commitment: `BOT_MODEL` and the package's
`model.yaml` still decide. `agent-bot` stays on `gpt-5.5` on purpose, since the only ways to 5.6 on
the endpoint it writes by hand are a streaming rewrite or turning reasoning off, and it is the Bot
whose job includes deciding when to ask a person for help.
- **A Bot browsed to a vendor this deployment already connects to.** A Bot holding no grants was told
nothing about connectors at all, so it treated a connected vendor as an ordinary website: asked
about Google Drive it opened `drive.google.com`, met a sign-in page, and asked the person to sign
in to an account the deployment had already connected. Every Bot is now told which vendors exist
here, held or not, and says plainly which one it has not been granted rather than reaching for the
browser.
- **A conversation was destroyed by a declined take-the-wheel.** A Bot that asks for help with a
sign-in and never gets it left a tool call nothing ever answered, and every later turn in that
thread failed at the provider. This was fixed once for the framework Bot and not for the Bot in the
box, which is the one behind the Browser Bot, so it went on happening where most people would meet
it. Both now answer their own unanswered calls with the truth rather than a fake success.
- **A Bot refused because a person had the wheel was told its refs were stale.** The computer flags a
takeover, the surface branches on that flag, and the flag did not survive the server, so a Bot was
sent back round the same action against the person who had just taken the browser. Reported and
fixed by @beardthelion.
- **A person could not take the wheel unless the Bot offered it.** The button appeared only after a
Bot called for help, so the control a person needs depended on the Bot getting one instruction
right, and when it did not there was nothing to press. It is there whenever the Bot is driving now.
The Bot asking for help is still its own row, with its reason.
- **The first message of a new channel could be lost.** A new channel's thread does not exist until
its first run, so the join that restores history had nothing to settle against; the message was sent
anyway after a deadline, while that join was still in flight, and the join finishing replaced it
with the thread's messages, which were none. The deadline now ends the join and waits for it, so
nothing is left in flight to overwrite anything. The transcript also says it is loading rather than
showing an empty conversation, and the thinking line is visible for the first time: a CSS rule
blanked the colour a gradient was built from, so the glyphs were painted with nothing. Reported and
fixed by @zopeVaibhav.
- **The in-memory snapshot store disagreed with the table.** The database only ever moves a snapshot
forward; the in-memory one, which is what a test reaches for when it has no database, took whatever
arrived last. A test could therefore prove a boundary property that is false in a deployment.
Reported by @beardthelion, fixed by @NathanTarbert.
- **A computer that had opened nothing still reserved a browser-sized frame.** That put a placeholder
the height of a browser window into the middle of a conversation, above an answer that never
involved the browser.
- **A Bot named after a deployment route was served without its guard.** The computer router steps
aside for `/policy` and `/fleet`, which are its own paths and not about a Bot, because Hono matches
`/*` against zero segments and a single-segment path arrives as a Bot id. It stepped aside on the
Expand Down
19 changes: 16 additions & 3 deletions app/src/components/computer/computer-view.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -171,11 +171,22 @@ export function ComputerView({
return () => window.removeEventListener("keydown", onKey);
}, [expanded]);

// Sized from the ratio, never from the payload, so the frame is identical in all three states.
const frameStyle = { aspectRatio, minWidth, minHeight };

// Always render the card frame; help/secret controls live below the conditional picture.
const blankBrowser = shot ? isBlankBrowser(shot) : false;

/*
* Sized from the ratio, never from the payload, so the frame is identical while a screen is
* loading and once it arrives.
*
* A browser that has opened nothing is the exception. Reserving a screen-sized frame for it put a
* placeholder the height of a browser window into the middle of a conversation, above an answer
* that never involved the browser at all: a Bot asked about Google Drive rendered a full-size
* empty panel saying it had not opened a page. Nothing is loading there and nothing is coming, so
* there is no layout jump to protect against and no reason to take the room.
*/
const frameStyle = blankBrowser
? { minWidth }
: { aspectRatio, minWidth, minHeight };
/** Blank browser placeholders should not be opened as readable screens. */
const showScreen = shot !== null && !blankBrowser;

Expand Down Expand Up @@ -206,6 +217,8 @@ export function ComputerView({
{blankBrowser ? (
<ComputerPlaceholder className="absolute inset-0 h-full w-full" />
) : null}
{/* The blank state is a line of text, so it needs its own height rather than the frame's. */}
{blankBrowser ? <span className="block py-6" /> : null}

{showScreen ? null : (
<span
Expand Down
34 changes: 31 additions & 3 deletions server/src/copilot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ import {
COMPUTER_GUIDANCE,
PROVENANCE_GUIDANCE,
} from "../../shared/bot-prompt";
import { grantedToolGuidance } from "./plugins/tools";
import type { AgentActor } from "./agents/profile-types";
import type { StallGuard } from "./channels/stall-guard";
import type { DeploymentConfig } from "./config";
import type { GrantedTool } from "./plugins/tools";
import { grantedToolGuidance } from "./plugins/tools";

/**
* The CopilotKit runtime, always in Intelligence mode.
Expand Down Expand Up @@ -197,6 +197,13 @@ export function builtInAgentConfiguration(
* be promising something that does not exist.
*/
computerGuidance?: string,
/**
* Vendors this deployment connects to, whether or not this Bot holds any of their tools.
*
* A Bot holding nothing was told nothing, so it treated a connected vendor as an ordinary website
* and browsed to it. See `grantedToolGuidance`.
*/
connectedVendors: readonly string[] = [],
): BuiltInAgentConfiguration {
if (!apiKey) {
return {
Expand Down Expand Up @@ -229,7 +236,9 @@ export function builtInAgentConfiguration(
* it says comes from its own knowledge, and saying so is the only honest move available.
*/
PROVENANCE_GUIDANCE,
...(grantedToolGuidance(tools) ? [grantedToolGuidance(tools)] : []),
...(grantedToolGuidance(tools, connectedVendors)
? [grantedToolGuidance(tools, connectedVendors)]
: []),
...(computerGuidance ? [computerGuidance] : []),
].join("\n\n"),
apiKey,
Expand Down Expand Up @@ -271,7 +280,13 @@ export async function buildAgents(
signRun?: SignRun,
/** What every built-in Bot is told about the computer. Absent means this deployment has none. */
computerGuidance?: string,
/**
* Which vendors this deployment connects to. Asked once per build rather than per Bot, because it
* is a fact about the deployment; what differs per Bot is which of them it holds.
*/
loadVendors: () => Promise<readonly string[]> = async () => [],
): Promise<Record<string, AbstractAgent>> {
const vendors = await loadVendors().catch(() => [] as readonly string[]);
return Object.fromEntries(
await Promise.all(
agents.map(async (agent) => [
Expand All @@ -284,6 +299,7 @@ export async function buildAgents(
loadTools,
signRun,
computerGuidance,
vendors,
),
]),
),
Expand All @@ -298,6 +314,7 @@ async function buildAgent(
loadTools: LoadToolsForBot,
signRun?: SignRun,
computerGuidance?: string,
connectedVendors: readonly string[] = [],
): Promise<AbstractAgent> {
if (agent.type === "built_in") {
return new BuiltInAgent(
Expand All @@ -307,6 +324,7 @@ async function buildAgent(
apiKey,
await loadTools(agent.id),
computerGuidance,
connectedVendors,
),
);
}
Expand All @@ -318,6 +336,7 @@ async function buildAgent(
stallGuard,
await loadTools(agent.id),
signRun,
connectedVendors,
);
}

Expand Down Expand Up @@ -345,6 +364,8 @@ function remoteAgentWithStandingRole(
*/
tools: GrantedTool[] = [],
signRun?: SignRun,
/** As for the built-in path: what this deployment connects to, held or not. */
connectedVendors: readonly string[] = [],
) {
const remote = new HttpAgent({
url: agent.endpoint,
Expand All @@ -368,7 +389,7 @@ function remoteAgentWithStandingRole(
* prompt — a page about the browser that mentions connectors nowhere. That is the Bot that browsed
* to drive.google.com holding four Drive tools.
*/
const holdings = grantedToolGuidance(tools);
const holdings = grantedToolGuidance(tools, connectedVendors);
const holdingsMessage = holdings
? {
id: `granted-tools:${agent.id}`,
Expand Down Expand Up @@ -467,6 +488,7 @@ export async function resolveRuntimeAgents(
loadTools?: LoadToolsForBot,
signRun?: SignRun,
computerGuidance?: string,
loadVendors?: () => Promise<readonly string[]>,
): Promise<Record<string, AbstractAgent>> {
const registered = await loadAgents();
if (registered.length === 0) {
Expand All @@ -486,6 +508,7 @@ export async function resolveRuntimeAgents(
loadTools,
signRun,
computerGuidance,
loadVendors,
);
}

Expand Down Expand Up @@ -533,6 +556,8 @@ export function createRequestAgents(
signRunForActor?: (actorId: string) => SignRun,
/** What every built-in Bot is told about the computer. Absent means this deployment has none. */
computerGuidance?: string,
/** Which vendors this deployment connects to, held by a Bot or not. Absent means none. */
loadVendors?: () => Promise<readonly string[]>,
) {
return async ({ request }: { request: Request }) => {
const actor = await identifyActor(request);
Expand All @@ -544,6 +569,7 @@ export function createRequestAgents(
loadToolsForActor?.(actor.id),
signRunForActor?.(actor.id),
computerGuidance,
loadVendors,
);
};
}
Expand Down Expand Up @@ -571,6 +597,7 @@ export function mountCopilotRuntime(
loadToolsForActor?: (actorId: string) => LoadToolsForBot,
signRunForActor?: (actorId: string) => SignRun,
basePath = "/api/copilotkit",
loadVendors?: () => Promise<readonly string[]>,
) {
const { intelligence } = config.runtime;

Expand Down Expand Up @@ -609,6 +636,7 @@ export function mountCopilotRuntime(
* as impossible. Absent computer, absent guidance: a Bot is not told about hands it has not got.
*/
config.computer ? COMPUTER_GUIDANCE : undefined,
loadVendors,
) as never,
});

Expand Down
23 changes: 21 additions & 2 deletions server/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import { createIntentRouter } from "./routing/classify";
import { createModelCompleter } from "./routing/model";
import { serve } from "bun";
import { mintRunAssertion } from "./agents/callback-token";
import { createAgentProfileStore } from "./agents/profile-store";
Expand Down Expand Up @@ -48,6 +46,8 @@ import { createDatabase } from "./db/client";
import { createPeopleStore } from "./people/store";
import { createPluginStore } from "./plugins/store";
import { grantedTools } from "./plugins/tools";
import { createIntentRouter } from "./routing/classify";
import { createModelCompleter } from "./routing/model";
import {
createPackageStatusReader,
loadTenantPackage,
Expand Down Expand Up @@ -416,6 +416,25 @@ const app = createApp(
*/
(actorId) => (botId, runId) =>
mintRunAssertion({ botId, actorId, runId }, config.keyEncryptionKey),
undefined,
/*
* Which vendors this deployment connects to, held by a Bot or not.
*
* A Bot holding no grants used to be told nothing about connectors at all, so it treated a
* connected vendor as an ordinary website and browsed to it: a Bot with no Drive grant opened
* Google's sign-in page and asked a person to sign in to an account the deployment had already
* connected. Naming them lets it say which one it has not been granted instead.
*
* Read per request rather than held, because a connector added a minute ago has to count, and
* failing is the same as having none: a Bot that cannot be told loses a sentence, not a run.
*/
async () => {
try {
return (await pluginStore.listServers()).map((server) => server.id);
} catch {
return [];
}
},
),
// The only path to an acting call.
computerGateway,
Expand Down
65 changes: 54 additions & 11 deletions server/src/plugins/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,19 @@ export function parametersFor(inputSchema: Record<string, unknown>): z.ZodType {
*
* Empty when the Bot holds nothing, so a deployment with no connectors says nothing about them.
*/
export function grantedToolGuidance(tools: GrantedTool[]): string {
if (tools.length === 0) return "";
export function grantedToolGuidance(
tools: GrantedTool[],
/**
* Systems this deployment connects to that this Bot holds nothing for.
*
* Without these a Bot holding no grants is told nothing at all, so it treats a connected vendor as
* an ordinary website and browses to it. That is how a Bot with no Drive grant ended up on Google's
* sign-in page asking a person to sign in to an account the deployment had already connected: the
* connector existed, the Bot simply was not on it, and nothing said so.
*/
connectedButNotHeld: readonly string[] = [],
): string {
if (tools.length === 0 && connectedButNotHeld.length === 0) return "";

const bySystem = new Map<string, string[]>();
for (const tool of tools) {
Expand All @@ -78,19 +89,51 @@ export function grantedToolGuidance(tools: GrantedTool[]): string {
bySystem.set(system, [...(bySystem.get(system) ?? []), rest]);
}

const held = [...bySystem.keys()];
const missing = connectedButNotHeld.filter(
(system) => !held.includes(system),
);

return [
"You can reach these systems directly, as the person asking, with their own access:",
...(tools.length > 0
? [
"You can reach these systems directly, as the person asking, with their own access:",
]
: []),
...[...bySystem.entries()].map(
([system, names]) => `- ${system}: ${names.join(", ")}`,
),
"Use them for anything about those systems. Do NOT browse to one of their websites instead: your",
"browser is signed in as nobody, so it sees less than these tools do and will meet a sign-in wall",
"that connecting an account has already solved.",
"If one of these systems is involved and no tool above covers the part you need, that is a",
"missing grant and not something to work around. Say so plainly, name the capability you would",
"need, and say an administrator can grant it on that connector. Do not reach for the browser, do",
"not ask the person to sign in, and do not ask them to fetch it for you: they already have the",
"access, and the thing that is missing is yours, not theirs.",
...(tools.length > 0
? [
"Use them for anything about those systems. Do NOT browse to one of their websites instead: your",
"browser is signed in as nobody, so it sees less than these tools do and will meet a sign-in wall",
"that connecting an account has already solved.",
"If one of these systems is involved and no tool above covers the part you need, that is a",
"missing grant and not something to work around. Say so plainly, name the capability you would",
"need, and say an administrator can grant it on that connector. Do not reach for the browser, do",
"not ask the person to sign in, and do not ask them to fetch it for you: they already have the",
"access, and the thing that is missing is yours, not theirs.",
]
: []),
/*
* The vendors this deployment connects to and this Bot does not hold.
*
* Named so the Bot can say which one, because "I have not been granted it" is only actionable if
* the person is told what "it" is. The browser is refused for these by the same reasoning as
* above and for a sharper reason: a connector exists precisely so the vendor is reached as the
* person asking, and the container's browser is signed in as nobody, so browsing there abandons
* the per-person path and lands on a login wall by construction.
*/
...(missing.length > 0
? [
...(tools.length > 0 ? [""] : []),
`This deployment also connects to: ${missing.join(", ")}. You hold none of their tools.`,
"If a question needs one of them, say plainly that you have not been granted it and that an",
"administrator can grant it on that connector. Do NOT browse to its website: that is not the",
"same thing, your browser is signed in as nobody, and it will meet a sign-in wall that the",
"connector exists to avoid. Do not ask the person to sign in there either.",
]
: []),
].join("\n");
}

Expand Down
Loading