Skip to content

perf(sdk): batch invoke-path lookups with DataLoader semantics - #1244

Closed
RhysSullivan wants to merge 1 commit into
mainfrom
sdk-invoke-batching
Closed

perf(sdk): batch invoke-path lookups with DataLoader semantics#1244
RhysSullivan wants to merge 1 commit into
mainfrom
sdk-invoke-batching

Conversation

@RhysSullivan

Copy link
Copy Markdown
Collaborator

What

Concurrent execute() calls in the same microtask window now share one storage query per table (tool, connection, integration) plus one policy-rule-set snapshot, instead of four point queries per call. Naive fan-out code — Promise.all over tool calls, code-mode loops, parallel MCP tool calls — no longer produces N+1 query storms.

Built on Effect Request/RequestResolver: each hot lookup on the invoke path is a request keyed by plain data, and a resolver serves the collected batch with a single OR-of-tuples query (keys deduped before hitting storage).

Behavior preserved

  • No cross-batch caching. Every batch window re-reads storage, so read-your-writes and policy invalidation are unchanged. A batch of one is exactly the point query it replaced, so sequential callers see identical query patterns.
  • Transactions bypass the batch window. Batches execute on a fiber forked with the first caller's context; inside fuma.transaction the handle rides on activeFumaDbRef, so a shared batch would read through the wrong connection. Transactional reads run the loader directly with exact isolation.
  • Public API and error types unchanged; a missing row still surfaces as the same typed not-found error per caller.

Proof

invoke-batching.test.ts counts real queries through a spying FumaDb wrapper:

  • 25 concurrent executes → at most 2 queries per table (was ~25).
  • 2 sequential executes → exactly 2 point reads per table, no regression.
  • A missing tool inside a batch fails alone with ToolNotFoundError; peers succeed.

Gates

format:check, lint, typecheck, and the full test suite (36 packages) all pass.

Concurrent execute calls in the same microtask window now share one
query per table (tool, connection, integration) plus one policy
rule-set snapshot instead of four point queries per call, killing the
N+1 pattern under code-mode fan-out and parallel MCP tool calls.

- invoke-batching.ts: Request types + resolvers keyed by plain data;
  batch loaders serve mixed tuples with a single OR-of-tuples query and
  dedupe equal keys before hitting storage. No cross-batch caching, so
  read-your-writes behavior is unchanged.
- Transactional callers bypass the batch window (a shared batch fiber
  would read through the wrong activeFumaDbRef) and keep exact
  per-transaction semantics.
- Proof test counts real queries through a spying FumaDb wrapper: 25
  concurrent executes touch each table at most twice; sequential calls
  still issue the same per-call point reads as before.
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant