Skip to content

The read boundary says when it fires - #188

Merged
tobert merged 3 commits into
mainfrom
containment-telemetry
Sep 10, 2026
Merged

The read boundary says when it fires#188
tobert merged 3 commits into
mainfrom
containment-telemetry

Conversation

@tobert

@tobert tobert commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Two run_kaish calls through MCP yesterday — one served, one refused at the boundary — produced exactly one span and zero log lines about the path outside the allowed set. kaibo logged 329 info + 12 warn that day and none of them was the refusal.

For a product whose whole claim is read-only containment, the boundary firing is the most interesting event there is, and it left no trace at all. An operator watching a fleet could not tell a boundary that never fired from one that fired a hundred times — and #184 shipped three fixes to exactly that boundary with no way to see it exercised in the field.

Observability, not correctness. Nothing here changes what kaibo refuses.

The funnel

Every containment refusal already routes through one function, Resolver::containment_error. resolve_root, resolve_attachments, and read_contained_file all call it, and they are reached from five MCP tools and four CLI subcommands. One warn there covers the whole class, and a fourth surface added later is reported the moment it uses the shared check.

Where the path goes, and why

The split between the field and the message is kaibo's existing export policy, not a new one:

  • outcome = "refused" is a field, and outcome is already on SAFE_ATTRIBUTES — so that a refusal happened always exports, with no allowlist change.
  • The paths live in the message body, which CONTENT_ATTRIBUTES holds back from traces unless the operator sets capture_content.

A caller-named path is content by the same rule that keeps gen_ai.tool.arguments off the safe list — the comment there says "which names paths" in as many words. The alternative was adding a path to SAFE_ATTRIBUTES; Amy chose this one, because it needs no new policy. The logs signal carries the line whole, which is what that signal is for.

run_kaish opens its span before the check

It used to build the span after resolve_root, so a refused call was the one outcome of the tool that produced no span at all. tool_span.rs had already settled the same question for the inner tools — a malformed-args refusal is reported as outcome = error rather than never appearing, "the honest reading of did this call work" — and this is the MCP surface catching up. The span now brackets the whole call, so its duration includes the check and the worker spawn; nothing asserted the old bracket.

The review found a real bug in the first cut

A cross-family pass (kaibo cast crusoe — explorer DeepSeek-V4-Flash, synth GLM-5.3) caught that the first cut recorded outcome = "refused" on every resolve_root error. It refuses four ways — no default root, a path that does not resolve, a path that is not a directory, and the boundary — and only the last is the boundary firing. A typo'd path made the span claim a boundary hit the logs knew nothing about, so an operator alerting on refused would have been counting typos.

Fixed by putting each writer where the knowledge is: containment_error records refused itself; Span::record is a no-op for a field the current span never declared, so it reaches exactly the spans built to receive it. The handler pre-sets error and lets that refinement land after it — which also means a future early return closes error rather than closing with nothing.

Every citation in that review was verified against source before I acted on any of it. Three prose corrections came from the same pass, and one drive-by: resolve_attachments' doc comment was sitting on read_contained_file, fused ahead of that function's own doc, leaving resolve_attachments undocumented.

What still doesn't reach traces

The other four handlers refuse before they have a phase to open a span around, so for them the logs signal is the only road. That's named in telemetry.rs's module doc rather than left for the next reader to rediscover; giving them handler-level spans is a larger change than this one. The sibling refusal for an attachment outside the session root stays quiet on purpose — that file is inside the allowed set, and the refusal is about which project the call mounts, not about the boundary.

Tests

Seven, all run against the pre-fix code and failing there: three refusal surfaces (zero events captured), the run_kaish pair (no span, no outcome field), and the vocabulary test that reproduces the reviewed bug. Two negative controls, both sabotage-checked rather than assumed:

  • an allowed path stays quiet — moved the warn to fire on every resolve, went red.
  • a nonexistent path reads error — restored the first cut's recording, went red.

New reusable capture helpers in test_support.rs, riding the existing serialized_capture guard so they inherit the callsite-interest protection that module already documents.

Suite: 1348 passed, 0 failed. Clippy clean.

🤖 Generated with Claude Code

tobert and others added 3 commits September 10, 2026 08:25
Two run_kaish calls through MCP, one served and one refused at the boundary,
produced exactly one span and zero log lines outside the allowed set. For a
product whose whole claim is read-only containment, the boundary firing is the
most interesting event there is, and it left no trace at all — an operator
watching a fleet could not tell a boundary that never fired from one that fired
a hundred times. We shipped three fixes to exactly that boundary in #184 with no
way to see it exercised in the field.

Every containment refusal funnels through `containment_error`, so one `warn`
there covers the whole class: `resolve_root`, `resolve_attachments`, and
`read_contained_file`, reached from five MCP tools and four CLI subcommands
alike. A fourth surface added later is reported the moment it uses the shared
check.

The split between the field and the message is kaibo's existing export policy,
not a new one. `outcome` is already on SAFE_ATTRIBUTES, so *that* a refusal
happened always exports; the paths live in the message body, which
CONTENT_ATTRIBUTES holds back from traces unless the operator opts in. A
caller-named path is content by the same rule that keeps `gen_ai.tool.arguments`
off the safe list — the comment there says "which names paths" in as many words.
The logs signal carries the line whole, which is what it is for. Amy chose this
over adding a path to the safe set: it needs no new policy.

`run_kaish` additionally opens its span *before* the containment check rather
than after, so a refused call closes a `run_kaish` span tagged
`outcome = "refused"` instead of producing no span at all. `tool_span.rs` settled
the same question for the inner tools — a malformed-args refusal is reported as
`outcome = error` rather than never appearing, "the honest reading of did this
call work" — and this is the MCP surface catching up to it. The span now brackets
the whole call, so its duration includes the check and the worker spawn.

The other four handlers refuse before they have a phase to open a span around,
so for them the logs signal is the only road. Named in telemetry.rs's module doc
rather than left for the next reader to rediscover; giving them handler-level
spans is a larger change than this one.

Five tests, all run against the pre-fix code and failing there: three refusal
surfaces with zero events captured, and the run_kaish pair with no span and no
outcome field. The negative control — an allowed path stays quiet — was
sabotage-checked by moving the warn to fire on every resolve, and went red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cross-family review of the first cut (kaibo cast `crusoe`) found that
`run_kaish` recorded `outcome = "refused"` on every `resolve_root` error, not
just the containment one. `resolve_root` refuses four ways — no default root, a
path that does not resolve, a path that is not a directory, and the boundary
check — and only the last is the boundary firing. So a typo'd path made the span
claim a boundary hit that the logs, which only `containment_error` writes, knew
nothing about. An operator alerting on `refused` would have been counting typos.

The fix puts each writer where the knowledge is. `containment_error` records
`refused` itself, from the one place that knows the boundary fired; `Span::record`
is a no-op for a field the current span never declared, so it reaches exactly the
spans built to receive it and costs nothing in the four handlers that resolve
before they have a span at all. The handler pre-sets `error` and lets that
refinement land after it, which also means a future early return closes `error`
rather than closing with no outcome.

A seventh test pins it, and it reproduces the reviewed bug when the first cut's
recording is restored.

Three prose corrections from the same review, all verified against source before
acting on them: "the one place kaibo records" understated the span as a second
record and is now "logs"; telemetry.rs's "four of the five MCP tools" read as
exhaustive when attachment refusals are logs-only roads too; and the funnel doc
now says which boundary it means, since a consult attachment outside the session
root is a different class and stays quiet.

Drive-by in the same file: `resolve_attachments`' doc comment was sitting on
`read_contained_file`, fused ahead of that function's own doc and leaving
`resolve_attachments` undocumented. Moved to the function it describes.

Reviewed-by: kaibo cast `crusoe` (explorer DeepSeek-V4-Flash, synth GLM-5.3)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tobert
tobert merged commit e9fe324 into main Sep 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant