feat: add parallel tool calling support - #214
Open
maralbahari wants to merge 4 commits into
Open
Conversation
Forward parallel_tool_calls to upstream inference and execute gateway built-in tool calls concurrently with bounded, per-tool safety controls. Align behavior with OpenAI reference recordings by: - validating missing function and custom tool call outputs - supporting batched web-search queries - preserving typed MCP and web-search output lifecycles - emitting mcp_list_tools discovery only once per stored chain - retaining discovery history without exposing it to model input - adding streaming and non-streaming cassette parity coverage Signed-off-by: maral <maralbahari.98@gmail.com>
maralbahari
marked this pull request as ready for review
August 28, 2026 03:07
maralbahari
requested review from
bbrowning,
franciscojavierarceo,
jiahuei,
leseb,
noobHappylife,
qandrew and
tjtanaa
as code owners
August 28, 2026 03:07
…-call Signed-off-by: maral <maralbahari.98@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds end-to-end parallel tool calling support to the typed Responses API (#181) and aligns gateway behavior
with the OpenAI reference behavior captured in matched cassettes.
Previously, the gateway forced
parallel_tool_calls: falseupstream and executed gateway-executed built-in toolcalls serially. This change:
parallel_tool_callsmodel-generation preference for function, custom, built-in, and mixedtool declarations, while preserving the existing
falsedefault when the field is omitted;window (
tools.max_concurrent_gateway_calls, default5), while preserving model call order in public output;unrelated tools in the same round; MCP and web search explicitly support same-tool parallel execution;
calls to complete; and
lifecycle events around concurrent execution.
The tool registry and execution path were also reworked to make this behavior explicit.
ToolOwnershipnow separatesclient-executed tools from gateway-executed built-in tools, and
GatewayBindingcarries the executor and its same-toolconcurrency policy.
GatewayExecutorowns the started and completed public-output projections for its tool type, andGatewayRoundowns bounded dispatch, per-call timeouts, result ordering, and output shaping. For MCP, typed handlermetadata replaces the duplicated registry side table that could drift out of sync with the main tool entries.
OpenAI parity and correctness fixes found during cassette comparison
The parallel-call recordings exposed several independent bugs in addition to the main feature:
Missing function call outputs were accepted. A continuation could omit the
function_call_outputorcustom_tool_call_outputfor a pending client-executed call and the gateway wouldcontinue with a
200. Rehydration now detects unresolved call IDs in both stored-response and conversationhistory and rejects the continuation with a
400 invalid_request_error, matching the reference behavior.One web-search call could not batch multiple queries. The normalized web-search schema only accepted a
singular
query, despite the publicweb_search_call.actionalready supportingqueries. The schema now acceptseither form, and one call can fan out multiple independent provider requests concurrently and aggregate their
results into one web-search output item.
Mixed built-in rounds could expose an MCP call as a raw
function_call. This occurred when MCP and anothergateway-executed built-in tool ran in the same round. Handler-owned public-output projection and explicit registry
ownership now ensure MCP and web-search calls retain their correct public item types in both streaming and
non-streaming responses.
mcp_list_toolsdiscovery was repeated on every turn. The matched cassettes established an otherwiseundocumented OpenAI behavior: an MCP server's discovery item and lifecycle are exposed on the first response in a
stored chain, and later
previous_response_idturns behave as though that discovery is cached rather than emittingit again. The gateway now retains
mcp_list_toolsin item history and lets the request-scoped registry group currentand historical discovery records by
server_label. Discovery is emitted only when that server has no prior listrecord, and the streaming path consumes the lifecycle after the first inference round.
mcp_list_toolsis history metadata, not model context:InOutItem::into_input_itemspreserves it for registrylifecycle decisions, while
ResponsesInput::model_inputand compaction remove it before inference. Likewise, theoriginal model-facing
function_callandfunction_call_outputitems are the single persisted source for executedweb-search and MCP calls; public
web_search_callandmcp_callitems are no longer lossily reconstructed into asecond copy of model input during rehydration.
Test Plan
OpenAI and gateway cassette coverage
Each scenario was recorded against both the OpenAI reference endpoint and the gateway in streaming and non-streaming
modes with
parallel_tool_calls: true.parallel-builtin-onlymcp_list_toolsis emitted only on the first turn.parallel-mixedparallel-clientparallel-failuresAutomated integration and unit coverage
parallel_tool_calls_testloads the OpenAI and gatewayparallel-builtin-onlycassettes for both response modes andcompares the three-turn request chain, normalized query/argument contracts, two-call grouping per turn, public
output types and order, terminal statuses, SSE lifecycle counts, contiguous sequence numbers, and first-turn-only
MCP discovery.
parallel_tool_calls: true,false, and omitted behavior for client-executed,built-in, and mixed tool declarations, including an HTTP gateway-to-upstream integration check.
opt into same-tool parallelism, stable model call ordering, per-call timeout isolation, and missing-provider error
isolation.
previous_response_idand conversation continuations.compaction input, is ignored for servers absent from the active request, and is emitted only once for blocking and
streaming continuations.
Commands used for focused verification:
cargo build --release cargo test -p agentic-server-core --test parallel_tool_calls_test cargo fmt --all -- --check