Skip to content

feat: add parallel tool calling support - #214

Open
maralbahari wants to merge 4 commits into
vllm-project:mainfrom
EmbeddedLLM:gateway-parallel-tool-call
Open

feat: add parallel tool calling support#214
maralbahari wants to merge 4 commits into
vllm-project:mainfrom
EmbeddedLLM:gateway-parallel-tool-call

Conversation

@maralbahari

Copy link
Copy Markdown
Collaborator

Summary

This PR adds end-to-end parallel tool calling support to the typed Responses API (#181) and aligns gateway behavior
with the OpenAI reference behavior captured in matched cassettes.

Previously, the gateway forced parallel_tool_calls: false upstream and executed gateway-executed built-in tool
calls serially. This change:

  • forwards the client's parallel_tool_calls model-generation preference for function, custom, built-in, and mixed
    tool declarations, while preserving the existing false default when the field is omitted;
  • executes gateway-executed built-in tool calls concurrently within an inference round using a bounded sliding
    window (tools.max_concurrent_gateway_calls, default 5), while preserving model call order in public output;
  • scopes parallel-safety to each model-visible tool name, so a handler can serialize calls to itself without blocking
    unrelated tools in the same round; MCP and web search explicitly support same-tool parallel execution;
  • isolates timeouts and individual execution failures as tool call outputs, allowing concurrently executing sibling
    calls to complete; and
  • preserves streaming event order and contiguous sequence numbers while synthesizing the correct web-search and MCP
    lifecycle events around concurrent execution.

The tool registry and execution path were also reworked to make this behavior explicit. ToolOwnership now separates
client-executed tools from gateway-executed built-in tools, and GatewayBinding carries the executor and its same-tool
concurrency policy. GatewayExecutor owns the started and completed public-output projections for its tool type, and
GatewayRound owns bounded dispatch, per-call timeouts, result ordering, and output shaping. For MCP, typed handler
metadata replaces the duplicated registry side table that could drift out of sync with the main tool entries.

OpenAI parity and correctness fixes found during cassette comparison

The parallel-call recordings exposed several independent bugs in addition to the main feature:

  1. Missing function call outputs were accepted. A continuation could omit the
    function_call_output or custom_tool_call_output for a pending client-executed call and the gateway would
    continue with a 200. Rehydration now detects unresolved call IDs in both stored-response and conversation
    history and rejects the continuation with a 400 invalid_request_error, matching the reference behavior.

  2. One web-search call could not batch multiple queries. The normalized web-search schema only accepted a
    singular query, despite the public web_search_call.action already supporting queries. The schema now accepts
    either form, and one call can fan out multiple independent provider requests concurrently and aggregate their
    results into one web-search output item.

  3. Mixed built-in rounds could expose an MCP call as a raw function_call. This occurred when MCP and another
    gateway-executed built-in tool ran in the same round. Handler-owned public-output projection and explicit registry
    ownership now ensure MCP and web-search calls retain their correct public item types in both streaming and
    non-streaming responses.

  4. mcp_list_tools discovery was repeated on every turn. The matched cassettes established an otherwise
    undocumented OpenAI behavior: an MCP server's discovery item and lifecycle are exposed on the first response in a
    stored chain, and later previous_response_id turns behave as though that discovery is cached rather than emitting
    it again. The gateway now retains mcp_list_tools in item history and lets the request-scoped registry group current
    and historical discovery records by server_label. Discovery is emitted only when that server has no prior list
    record, and the streaming path consumes the lifecycle after the first inference round.

    mcp_list_tools is history metadata, not model context: InOutItem::into_input_items preserves it for registry
    lifecycle decisions, while ResponsesInput::model_input and compaction remove it before inference. Likewise, the
    original model-facing function_call and function_call_output items are the single persisted source for executed
    web-search and MCP calls; public web_search_call and mcp_call items are no longer lossily reconstructed into a
    second copy of model input during rehydration.

Test Plan

OpenAI and gateway cassette coverage

Each scenario was recorded against both the OpenAI reference endpoint and the gateway in streaming and non-streaming
modes with parallel_tool_calls: true.

Cassette suite Cases covered
parallel-builtin-only Three-turn stored-response chain: two parallel web-search calls, with two queries batched inside each call; two calls to the same MCP tool with different arguments; and a mixed web-search + MCP round. Also verifies that mcp_list_tools is emitted only on the first turn.
parallel-mixed Three-way round containing a gateway-executed web search, a client-executed function call, and a custom tool call, followed by a turn that depends on the submitted custom-tool output.
parallel-client Three parallel client-executed calls across function and custom tool types; correlation and submission of every call output on the continuation; two calls to the same function with different arguments; and a final multi-result follow-up.
parallel-failures A failed function call alongside a successful sibling; a continuation that omits one of two required function call outputs; and a mixed built-in + client-executed round whose client call output is omitted. The omission cases verify rejection rather than silently continuing with a dangling call ID.

Automated integration and unit coverage

  • parallel_tool_calls_test loads the OpenAI and gateway parallel-builtin-only cassettes for both response modes and
    compares the three-turn request chain, normalized query/argument contracts, two-call grouping per turn, public
    output types and order, terminal statuses, SSE lifecycle counts, contiguous sequence numbers, and first-turn-only
    MCP discovery.
  • Responses request tests cover parallel_tool_calls: true, false, and omitted behavior for client-executed,
    built-in, and mixed tool declarations, including an HTTP gateway-to-upstream integration check.
  • Gateway-round tests verify real overlap for parallel-safe same-tool calls, serialization for handlers that do not
    opt into same-tool parallelism, stable model call ordering, per-call timeout isolation, and missing-provider error
    isolation.
  • Web-search tests verify singular-query compatibility and multi-query fan-out/aggregation.
  • Pending-call and rehydration tests cover resolved and unresolved function/custom calls for both
    previous_response_id and conversation continuations.
  • MCP history tests verify that list-tools metadata survives storage and rehydration, is excluded from model and
    compaction input, is ignored for servers absent from the active request, and is emitted only once for blocking and
    streaming continuations.
  • MCP/web-search lifecycle tests verify the public output item types and completed/failed streaming event sequences.

Commands used for focused verification:

cargo build --release
cargo test -p agentic-server-core --test parallel_tool_calls_test
cargo fmt --all -- --check

  Forward parallel_tool_calls to upstream inference and execute gateway
  built-in tool calls concurrently with bounded, per-tool safety controls.

  Align behavior with OpenAI reference recordings by:
  - validating missing function and custom tool call outputs
  - supporting batched web-search queries
  - preserving typed MCP and web-search output lifecycles
  - emitting mcp_list_tools discovery only once per stored chain
  - retaining discovery history without exposing it to model input
  - adding streaming and non-streaming cassette parity coverage

Signed-off-by: maral <maralbahari.98@gmail.com>
Signed-off-by: maral <maralbahari.98@gmail.com>
…-call

Signed-off-by: maral <maralbahari.98@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant