Skip to content

perf(streaming): stream tool input up to 24x faster by skipping re-parses inside open strings - #2022

Open
deadstrobe5 wants to merge 1 commit into
anthropics:mainfrom
deadstrobe5:perf-stream-tool-input
Open

deadstrobe5 wants to merge 1 commit into
anthropics:mainfrom
deadstrobe5:perf-stream-tool-input

Conversation

@deadstrobe5

Copy link
Copy Markdown

accumulate_event re-parses the whole input_json_delta buffer with jiter.from_json(partial_mode=True) on every delta, so streaming a tool call costs O(n²) in the size of its input. A model writing a 74 KB file through a tool argument spends about 175 ms of CPU just re-parsing.

partial_mode drops a trailing unclosed string, so a delta that only extends an open string cannot change the parsed value. This tracks string and escape state across deltas, including \uXXXX and surrogate pairs, and skips the parse in that case. Anything that is not valid inside a JSON string still goes through jiter, so errors are raised at the same event with the same message. The beta fine-grained-tool-streaming trailing-strings mode is unchanged.

The TypeScript SDK fixed the same cost by parsing lazily (65a0106). That does not carry over here, because every InputJsonEvent carries the snapshot, so skipping the parse is what helps.

10-byte deltas, CPython 3.14, median of 7 runs:

input before after
file content, 4 KB 1.2 ms 0.5 ms
file content, 18 KB 12.7 ms 1.8 ms
file content, 74 KB 174.0 ms 7.0 ms
many small fields, 64 KB 470 ms 474 ms

A differential test against main fed random nested inputs, escapes, unicode and corrupted inputs split into random deltas. It found no difference in content.input or in the errors raised over 2 million+ events. Tests in tests/lib/streaming pass, and ruff and pyright are clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant