Skip to content

fix: gate tool execution on raw stream completion, not just JSON validity - #3434

Open
canblmz1 wants to merge 1 commit into
apache:mainfrom
canblmz1:fix/tool-execution-integrity
Open

fix: gate tool execution on raw stream completion, not just JSON validity#3434
canblmz1 wants to merge 1 commit into
apache:mainfrom
canblmz1:fix/tool-execution-integrity

Conversation

@canblmz1

Copy link
Copy Markdown

Problem

Maka deliberately keeps tool execution outside the Vercel AI SDK and settles
returned tool calls through its own ToolRuntime after each provider step.

Before this change, the final execution gate relied on the provider step being
classified as completed.

However, settleModelStepOutcome() also classifies
finishReason: "length" as completed.

That means a mutating tool call can have syntactically complete arguments,
reach Maka's returned-tool settlement path, and still belong to a provider
generation that was cut off by a token limit.

A complete tool-call payload is not, by itself, proof that the surrounding
provider step terminated safely.

There is a second integrity boundary as well: the AI SDK's final tool-call
input is already parsed/post-processed data. Where providers expose raw
tool-input-delta chunks, those raw bytes are stronger evidence of whether the
arguments were actually streamed to structural completion.

Fix

Add a narrow tool-execution safety layer around each physical provider request.

For incrementally streamed tool arguments:

  • observe the raw AI SDK stream before Maka translates it;
  • track tool-input-start, tool-input-delta, tool-input-end, and terminal
    stream events;
  • use prefix-safe-json@0.0.1-alpha.4's
    createAiSdkExecutionGuard() to derive a per-tool-call execution verdict;
  • require a positive verdict before the call may reach ToolRuntime.

For providers that deliver tool arguments atomically and expose no raw argument
deltas, no raw-JSON completeness claim is made.

Those calls instead require the provider step itself to finish with an
explicitly execution-safe reason:

  • stop
  • tool-calls

Other terminal states, including length, are not execution-safe.

This intentionally leaves settleModelStepOutcome() unchanged because its
length -> completed behavior has broader continuation/bookkeeping semantics.
The stricter rule exists only at the irreversible tool-execution boundary.

Atomic provider delivery

Some real provider paths do not stream argument bytes incrementally. They may
emit:

tool-input-start
tool-input-end
tool-call

with the actual parsed arguments appearing only in the final tool-call
event.

The integration does not fabricate raw-byte evidence for that case.

"No raw delta evidence" means exactly that: raw argument completeness is
unknown. It does not mean the call is safe.

Execution is therefore allowed only when the provider step also has a
positively safe terminal reason.

Scope

This change is intentionally limited to the existing tool-settlement path.

Unchanged:

  • ToolRuntime.settleToolCall()
  • tool implementation behavior
  • settleModelStepOutcome()
  • Maka's provider retry / continuation semantics
  • the existing invalid-tool rejection/result path

Rejected tool calls continue through Maka's existing settlement/result
mechanism rather than introducing a new placeholder transcript state.

Concurrency

The execution guard is scoped to one physical provider request.

There is no process-global state keyed only by toolCallId, so concurrent
provider requests may safely reuse identifiers such as call_1 without
cross-resolving each other's decisions.

Production-path coverage includes concurrent calls sharing the same tool-call
id with different safety outcomes.

Dependency

Adds the exact dependency:

"prefix-safe-json": "0.0.1-alpha.4"

The package is used only where raw AI SDK stream evidence is available and does
not replace Maka's existing tool/runtime architecture.

The version is pinned exactly rather than using a prerelease range.

Tests

Added focused guard tests and end-to-end production-path tests through:

AiSdkBackend
  -> ModelAdapter
  -> stream safety decision
  -> ToolRuntime settlement boundary

Covered cases include:

  • incremental arguments + stop -> executes
  • incremental arguments + tool-calls -> executes
  • incremental arguments + length -> withheld
  • atomic arguments + stop -> executes
  • atomic arguments + tool-calls -> executes
  • atomic arguments + length -> withheld
  • truncated raw JSON -> withheld
  • provider error -> withheld
  • content filter -> withheld
  • abort -> withheld
  • unknown terminal state -> withheld
  • missing terminal event -> withheld
  • concurrent requests reusing the same tool-call id remain isolated

Validation on the final branch:

  • tool-call-execution-guard.test.ts: 22/22
  • production-path execution-integrity tests: 20/20
  • relevant runtime suites: 284/284
  • Biome lint on changed source/test files: clean

The repository runtime typecheck still reports the same 68 pre-existing errors
observed on pristine upstream; the error set was unchanged by this patch.

…dity

A tool call whose own arguments streamed to genuine completion is not
proof the surrounding provider step actually finished safely.
settleModelStepOutcome (model-adapter.ts) classifies finishReason:
"length" as { kind: 'completed' } - the same branch "stop" and
"tool-calls" take - and ai-sdk-backend.ts's only gate before settling
returnedToolCalls was providerOutcome.kind !== 'completed'. A step that
streams a structurally complete tool call for a mutating tool (Write,
Bash, apply_patch, etc.) and is then cut off by a token limit while
producing more content still reaches ToolRuntime and its real side
effect. translateChunk also never re-verifies a 'tool-call' chunk's
input against the raw tool-input-delta bytes that produced it - it
trusts whatever the SDK decided, including anything repairToolCall (or
the SDK itself) may have coerced.

tool-call-execution-guard.ts wraps prefix-safe-json's
createAiSdkExecutionGuard() around one physical provider request (one
ModelAdapter.startStream call), observing the raw tool-input-start /
tool-input-delta / tool-input-end / tool-call / finish / error chunks
verbatim so a call's completeness is proven from its own raw byte
stream. Not every real provider integration streams arguments
incrementally though - this project's own Anthropic-compatible wire
protocol emits tool-input-start immediately followed by tool-input-end
with zero delta chunks, carrying the arguments only in the trailing
tool-call chunk. That is a legitimate atomic-delivery pattern, not a
truncation, so a call the guard never saw real delta bytes for has no
raw-byte evidence either way.

For that no-raw-delta case, execution is allowed only when the step
itself positively finished with an execution-safe reason -
isSafeToolExecutionStepOutcome() checks specifically for "stop" or
"tool-calls", deliberately narrower than settleModelStepOutcome's own
kind === 'completed' (which also covers "length", correct for that
function's own continuation/retry bookkeeping but not an
execution-safe outcome). "No raw-byte evidence" must never become
"safe to execute" on its own; a call the guard did observe streaming
still gets the full, strict raw-byte-verified verdict regardless of
how the step ended.

Scope is intentionally narrow: only the tool-settlement call site in
ai-sdk-backend.ts is changed. ToolRuntime.settleToolCall and
settleModelStepOutcome itself are untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant