Skip to content

Tell a Bot what it holds, not only hand it the tools - #143

Merged
davidmckayv merged 1 commit into
mainfrom
fix/present-the-tools-a-bot-holds
Aug 22, 2026
Merged

Tell a Bot what it holds, not only hand it the tools#143
davidmckayv merged 1 commit into
mainfrom
fix/present-the-tools-a-bot-holds

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

Fixes the behaviour behind #141.

The problem

You connect Google Drive. You grant its tools to a Bot. You ask that Bot what is in a document. It opens drive.google.com in its own browser, meets Google's sign-in page, and asks you to sign in — to a vendor you connected ten minutes earlier.

The tools were loaded the whole time.

Why

A tool array tells a model a tool exists. It does not tell it the tool is the right way to reach that system. And it is competing with COMPUTER_GUIDANCE: a page of emphatic prose about the browser, given to every Bot whether or not it holds a single connector, containing lines like

Never claim you cannot browse: opening a page is something you can actually do.

and mentioning connectors nowhere.

Read on its own, that text answers "how do I browse" and never "should I". So the browser wins, because it is the most capable tool and nothing says otherwise.

The remote path had it worst: a framework Bot gets COMPUTER_GUIDANCE as its entire prompt, so it had the least reason of anything to reach for a granted tool. That is the Bot that browsed to Drive.

The fix

Say the grants out loud, in the Bot's instructions, before the browser prose:

You can reach these systems directly, as the person asking, with their own access:
- google-drive: search_files, read_file_content, list_recent_files, get_file_metadata
Use them for anything about those systems. Do NOT browse to one of their websites instead: your
browser is signed in as nobody, so it sees less than these tools do and will meet a sign-in wall
that connecting an account has already solved. If one of these systems is involved but no tool
above covers the part you need, say which part is missing rather than going around it.

Generated from the grants, not written down. An administrator switching a connector on, or granting one more of its tools, changes what the Bot is told on its next run with nothing to remember and nothing to keep in step. A Bot holding nothing is told nothing, so a deployment with no connectors is not lectured about connectors.

Applied to both paths, because they are built by different functions and a fix to one is the bug again in the other.

Order matters as much as presence, and there is a test for it: the grants land before COMPUTER_GUIDANCE, because a Bot that read the browser prose last reached for the browser.

Proof, checked against the document rather than the answer

A plausible answer is not evidence, so the ground truth came out of the file first, programmatically:

$ read_file_content(1ejGAr…)  →  12,581 chars
"… Weekly active users inside CopilotKit. Kai reached 83% company-wide; that is the bar. …"

Then the same question in the chat:

Q: In my OpenBot PRD in Google Drive, what percentage did Kai reach company-wide, and which metric is that the bar for?

A: Kai reached 83% company-wide. That is the bar for the trailing metric: weekly active users inside CopilotKit.

Exact match, including the metric it belongs to.

The audit trail shows the contrast inside one three-minute window:

time Bot what it did
03:21 Knowledge, no Drive grants computer_navigatecomputer.help_requested → sign-in wall
03:23 Risk Analyst, holds them search_files, search_files, read_file_contentno navigate at all

All three tool calls reachedAs the asking user.

Before this change, the second row looked like the first.

result
bun run test 1149 pass, 8 skip, 0 fail, 1157 across 102 files
bun run typecheck clean, all four packages
bun run format:check clean
bunx biome lint . 1 info, identical to main

Tests

Four. That the guidance names the system and its tools; that it carries the do-not-browse instruction, which is the line that does the work; that a Bot holding nothing gets nothing; and that the grants land before the browser prose.

Not in this PR

  • Routing is capability-blind. The same question routed to knowledge, which holds no Drive grants, because routing reads role descriptions and not what a Bot can actually reach. A question about Drive should prefer a Bot that can reach Drive. Its own change, and closer to K3 — search is a tool call: shipped, one wiring gap left #119's territory.
  • The browser as a general research tool. A Bot with no relevant grant still browses to look things up, which is how one ended up looping on a government 404. Worth deciding separately: the fix here only covers vendors a Bot has a tool for.

A tool array tells a model a tool exists. It does not tell it the tool is
the right way to reach that system, and it competes with COMPUTER_GUIDANCE
— a page of emphatic prose about the browser that every Bot receives
whether or not it has a single connector, and that mentions connectors
nowhere. "Never claim you cannot browse" is in there. Nothing is.

The browser prose won. A Bot holding four Google Drive tools browsed to
drive.google.com, met a sign-in page its container can never satisfy, and
asked its person to sign in to a vendor that person had already connected.
The tools were loaded the whole time.

So the grants are now said out loud, in the Bot's own instructions, and
before the browser prose rather than after it. The text is generated from
the grants rather than written anywhere: switching a connector on, or
granting one more of its tools, changes what the Bot is told on its next
run with nothing to keep in step. A Bot holding nothing is told nothing.

Both paths, because they are built by different functions and the remote
one is where it failed: a framework Bot gets COMPUTER_GUIDANCE as its
entire prompt, so it had the least reason of any to reach for a tool.

Verified against the document rather than the answer. Read a PRD's text
through the API first, took a line out of it — "Kai reached 83%
company-wide" against weekly active users — then asked in the chat. The
Bot answered 83% and named the metric, using search_files, search_files,
read_file_content, and never touching the browser. Two minutes earlier,
on the same trail, a Bot without those grants met the sign-in wall.
@davidmckayv
davidmckayv merged commit 1213cfe into main Aug 22, 2026
8 checks passed
@davidmckayv
davidmckayv deleted the fix/present-the-tools-a-bot-holds branch August 22, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant