Skip to content

Run every Bot on a current model - #168

Merged
davidmckayv merged 2 commits into
mainfrom
chore/every-bot-runs-a-current-model
Aug 22, 2026
Merged

Run every Bot on a current model#168
davidmckayv merged 2 commits into
mainfrom
chore/every-bot-runs-a-current-model

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

The example package shipped gpt-4.1 as the default model for every built-in Bot.

That is not a tuning choice. It is the cause of a bug I spent the afternoon attributing to the prompt.

What it was doing

Asked, in a fresh channel, a neutral question — "Open drive.google.com and tell me the name of the first file listed in my Drive" — the built-in Bots answered:

"…Let me know if you would like me to prompt you to sign in."

and called nothing. Three times out of three. No computer_request_help, so requested stayed false, no banner, no button.

COMPUTER_GUIDANCE forbids that exact sentence in as many words, so I went looking for a prompt lever. There wasn't one. On gpt-5.6-terra the same question produces the tool call first try:

{"holder":"bot","requested":true,
 "reason":"Google Drive is showing a sign-in page; sign in to your account so I can read the first listed file."}

The remote Bot had been getting this right all along — because it runs a newer model.

Which model

The 5.6 family is three tiers, and there is no plain gpt-5.6: sol (flagship, most reasoning), terra (balanced, 5.5-competitive at half the cost), luna (fastest, low-reasoning).

terra is the default here: it is the like-for-like replacement for today's 5.5 rather than an upgrade in spend, and BOT_MODEL picks sol for a deployment that wants it. Luna would be the wrong default for Bots whose job includes deciding when to call a tool.

The Responses API is now inferred

gpt-5.6-* rejects function tools on /v1/chat/completions:

Function tools with reasoning_effort are not supported for gpt-5.6-terra in
/v1/chat/completions. To use function tools, use /v1/responses or set
reasoning_effort to 'none'.

So a deployment that set BOT_MODEL to a 5.6 model and did not also know about BOT_RESPONSES_API got a Bot that started, looked healthy, and failed on its first tool call. It is now inferred from the model in agent-langgraph and both example Bots; the switch is still honoured for a model this build has not heard of.

agent-bot stays on 5.5, and says why

It speaks /v1/chat/completions by hand. The endpoint offers exactly two ways to use 5.6 there: the Responses API, which is the streaming-loop rewrite this file exists to avoid, or reasoning_effort: 'none'.

Turning reasoning off on the Bot whose whole job today is deciding when to ask a person for help would be trading the fix for the bug. Verified both: all three 5.6 tiers do call tools on chat-completions with reasoning_effort: 'none', and 5.5 calls them with reasoning intact.

Swept

Where Was Now
examples/fintech/model.yaml gpt-4.1 gpt-5.6-terra
docker-compose.yml ×2 gpt-5.5 gpt-5.6-terra
agent-langgraph default gpt-5.5 gpt-5.6-terra + inferred Responses API
examples/langgraph-bot gpt-5.5 gpt-5.6-terra + inferred Responses API
examples/mastra-bot gpt-5.5 gpt-5.6-terra + openai.responses()
.env.example gpt-4o, gpt-5.5 current, with the tiers named
docs/configuration.md gpt-4o ×2, gpt-4.1 current
test fixtures gpt-4.1 current
agent-bot gpt-5.5 unchanged, with the reason written down

Model ids checked against the live /v1/models list rather than assumed.

Lint, format, typecheck clean; 52 model-related tests pass.

The example package shipped `gpt-4.1` as the default model for every
built-in Bot. That is not a tuning choice, it is the cause of a bug I
spent the afternoon attributing to the prompt.

Asked to open a page behind a sign-in, the built-in Bots answered "would
you like me to prompt you to sign in?" and called nothing, three times out
of three. `COMPUTER_GUIDANCE` forbids that sentence in as many words, so I
went looking for a prompt lever, and there was none to find: on
gpt-5.6-terra the same question produces `computer_request_help`, with
`requested: true` and the reason on the row, first try. The remote Bot had
been getting this right all along because it runs a newer model.

So: gpt-5.6-terra everywhere it can go. Of the three 5.6 tiers, terra is
the like-for-like replacement for 5.5 rather than an upgrade in spend, and
sol is there for a deployment that wants it.

The Responses API is now inferred from the model rather than left to a
separate switch. gpt-5.6-* rejects function tools on chat completions, so
a deployment that set BOT_MODEL and did not also know about
BOT_RESPONSES_API got a Bot that started, looked healthy, and failed on
its first tool call. The switch is still honoured for a model this build
has not heard of.

agent-bot stays on gpt-5.5 and says why. It speaks /v1/chat/completions by
hand, and the endpoint offers exactly two ways to use 5.6 there: the
Responses API, which is the streaming-loop rewrite this file exists to
avoid, or reasoning_effort 'none'. Turning reasoning off on the Bot whose
whole job today is deciding when to ask a person for help would be trading
the fix for the bug.

Also swept: two `gpt-4o` references in .env.example and the configuration
docs that predate all of this, and the gpt-4.1 fixtures in the tests.
@davidmckayv
davidmckayv merged commit f74d116 into main Aug 22, 2026
@davidmckayv
davidmckayv deleted the chore/every-bot-runs-a-current-model branch August 22, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant