🐢 fix: Back Off Waiting Completion Wake-ups and Keep Their Trace Out of Requests

[danny-avila]

Summary

Background tool and subagent completion wake-ups re-read the database every five seconds for as long as they wait, and on a busy deployment they wait a long time. Since #15350 (Wake Agents on Background Tool Completion), every backgrounded call pre-registers its completion delivery at dispatch. The delivery engine claims it, finds the result not ready or the parent generation still busy, defers it, and repeats. Each cycle is a claim, a lease check, a generation-state read and a defer, every five seconds, for the whole time a tool runs, a turn continues, or an approval pause waits for a person. On the demo deployment, one user's waiting wake-ups sustained up to 32 delivery operations per second: 801,915 AgentTriggerDelivery operations for 1,091 deliveries in 40 hours, about 735 per delivery. v0.8.7 has none of this, so it is a regression in the v0.8.8 release line.

The resolvers already asked for a one-second retryAfter on these deferrals, but the engine never read it. A readiness deferral (deferWithoutAttempt) always waited the fixed five-second default, so the waiting cadence could not be tuned from the producer side at all.

This PR makes waiting cheap. A waiting delivery backs off by a tenth of its age, between the existing five-second floor and a configurable cap (60 s by default). The engine honors a readiness retryAfter whenever it asks for longer than that floor, and never shorter, so queued turns and every other deferral keep their cadence.

It also stops the delivery loop from hijacking request traces. The engine starts at boot but is woken from inside request handlers, and claim passes and poll timers inherited that request's async context. Every later tick belonged to that request forever: one chat request's trace on the demo carried 207,147 delivery operations over 6.6 hours. Claim passes and their timers now run under the root context.

Delivering a waiting result the moment it becomes ready (expediting on a durable result or a settled turn) is left to #16339, which builds on this. Until then, a result can wait up to one backoff interval after its turn ends.

How it works

dispatch      pre-register delivery
claim pass    PARENT_NOT_READY / RESULT_NOT_READY / CHILD_NOT_READY
                retryAfter = clamp(age / 10, 5 s, completionWaitMaxIntervalMs)   # was: ignored, fixed 5 s
engine        readiness deferral waits max(5 s, retryAfter)
              claim passes and poll timers run under ROOT_CONTEXT

A deferral whose parent has settled and is only finishing terminal persistence keeps the short re-check, since that clears within moments. The cap is endpoints.agents.eventDriven.idlePolling.completionWaitMaxIntervalMs (default 60000, 5000–300000), next to the existing idle-polling caps. A new non-sparse { user, status, availableAt } index on agenttriggerdeliveries serves user-scoped reads of waiting deliveries. The collection's only existing user index is sparse on actorActionAdmittedAt and can't.

Type of change

• Bug fix
• Performance improvement

Testing

Verified against a local LibreChat build under concurrent load, measured from delivery rows in MongoDB and per-span delivery operations in ClickStack (ClickHouse Cloud), with the same run on dev and on this branch. Eight conversations each dispatch four background tasks (20–40 s) through a deterministic MCP fixture, then run a 5-minute foreground task, so 32 finished results wait behind busy turns:

	dev	This branch
Results delivered	32 / 32	32 / 32
Delivery attempts per result	—	1 (no requeues, no dead letters)
AgentTriggerDelivery operations, whole run	12,016	7,616
Operations per minute while all 32 wait (minutes 2–5)	1,848	770
Readiness defers per minute (same window)	341	104
Largest single trace	11,410 operations (the dispatching request)	43
Claimed after the busy turn ended (p50 / max)	11 s / 26 s	33 s / 63 s

The saving grows with waiting time, because the interval is a tenth of a delivery's age. In a separate run where four results waited behind a 12-minute turn, dev held about 295 delivery operations a minute for the whole wait, while this branch fell to about 30 (mostly the engine's idle ticks) and its largest trace was 51 operations instead of 4,112. On the demo, waits ran 16 minutes at the median and up to 5.8 hours, where the fixed five-second cycle cost the most. The longer claim delay after a busy turn is the trade this PR makes on its own; #16339 removes it by expediting waiting results the moment they become ready.

Both runs logged JobPredecessorMismatchError (12 on dev, 29 here). This is an existing race when several wake-ups for one conversation start their turns at the same moment. The losing dispatch is deferred without consuming an attempt, and every result still delivered on its first attempt.

Tested environments/configuration:

• Node 24.16.0, MongoDB 8.2.6 (single-node replica set), Redis (USE_REDIS=true)
• Agents endpoint with eventDriven.completionWakeups, durableReceipts, coalescing, actorMailbox
• OpenRouter openai/gpt-6-luna; deterministic MCP fixture slow_task(seconds, label) over streamable HTTP
• OpenTelemetry → ClickStack collector → ClickHouse Cloud, for per-span delivery operations

Automated tests:

• backoff.spec.ts; waiting-age backoff and the configured cap for running, paused and persisting parents in backgroundCompletionWakeup.spec.ts and subagentCompletionWakeup.spec.ts
• engine.spec.ts: longer readiness requests are honored, shorter ones floored at the default; claim passes and poll timers run under the root context
• service.delivery.spec.ts: the configured cap is exposed; config.spec.ts: defaults and bounds
• triggerDelivery.spec.ts (mongodb-memory-server): the new index exists and is not sparse
• packages/api src/agents and src/stream suites; npx tsc --noEmit in packages/api, packages/data-schemas and packages/data-provider

Screenshots / recordings

No user-facing change.

Risk / compatibility

Waiting deliveries re-check less often: a result that becomes ready while its delivery is backed off waits up to one interval (at most the cap, 60 s by default) instead of at most 5 s. Operators can lower completionWaitMaxIntervalMs to trade database load for latency. Readiness deferrals that ask for less than five seconds, including every queued-turn deferral, keep exactly the cadence they have today. The new index is created at startup by ensureAgentTriggerDeliveryIndexes, like the others. No stored-data migration.

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published
• Required documentation PR: N/A