Skip to content

[Feature]: Batched call_tool so independent upstream calls run in parallel inside the sandbox #987

Description

@algis-dumbris

Problem / motivation

call_tool() in the JS sandbox is synchronous and strictly serial: N independent upstream calls
cost the sum of their latencies. A measured run made 31 upstream calls in 20.6s — nearly all of
it waiting on I/O that has no ordering dependency. Fan-out patterns (fetch 20 PRs, read 15 issues,
poll 8 servers) are exactly what the code-execution feature exists for, and they are the ones that
pay the most.

There is no way to work around this in userland: the sandbox deliberately ships no timers
(setTimeout/setInterval removed) and an ES5.1-only stdlib, so there is no Promise or event-loop
story to build concurrency on. The primitive has to come from the host.

Proposed solution

Add a host-provided batch primitive rather than JS async:

var results = call_tools([
  {server: "github", tool: "get_pull_request", args: {owner: "o", repo: "r", pullNumber: 1}},
  {server: "github", tool: "get_pull_request", args: {owner: "o", repo: "r", pullNumber: 2}}
]);
// results[i] is the same {ok, result} | {ok, error} envelope call_tool returns,
// in input order, one entry per request, never short-circuited.

Details worth pinning down in review:

  • Error semantics: one failing call must not fail the batch — each slot carries its own
    {ok:false, error}. This is the main reason to prefer an explicit batch API over implicit
    parallelism.
  • Concurrency bound: a max_parallel option (config default, per-call override), and the batch
    must respect whatever per-server request queueing exists — see closed [Feature]: request queueing / concurrency limit for multi-user deployments #955 on request queueing /
    concurrency limits, which is the counterweight this design has to honour rather than bypass.
  • Accounting: each element counts against max_tool_calls individually.
  • Timeout interaction: the batch is bounded by the overall execution timeout; note that the CLI
    daemon path currently caps at ~2s regardless of --timeout (filed separately), which would mask
    most of the benefit until fixed.

This keeps the sandbox ES5.1 and timer-free while turning sum(latency) into max(latency) for the
common fan-out case.

Which edition?

Both.

Alternatives considered

  • Promises / async-await in the sandbox. Requires an event loop and timers in goja — much larger
    change, and the timers were removed deliberately.

  • Callers issuing N separate code_execution calls in parallel from the client. Loses the entire
    point of the sandbox (a single round trip with local aggregation) and multiplies script-transfer
    cost by N.

  • Implicit auto-parallelisation of consecutive call_tool() statements. Not statically safe;
    ordering and side-effect dependencies are undecidable at that level.

  • I searched existing issues and discussions and did not find a duplicate

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions