Skip to content

⚡ fix: Deliver Waiting Completion Wake-ups the Moment They Are Ready - #16339

Merged
danny-avila merged 8 commits into
devfrom
danny-avila/bg-wakeup-backoff
Sep 25, 2026
Merged

danny-avila merged 8 commits into
devfrom
danny-avila/bg-wakeup-backoff

Conversation

@danny-avila

@danny-avila danny-avila commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Builds on #16349 (merged), which shipped the backoff, the engine honoring readiness retryAfter, root trace context, the configurable wait cap and the { user, status, availableAt } index. This PR adds what keeps delivery prompt once waiting is cheap: expediting waiting results the moment their result is durable or their turn settles, including held deliveries (wake marker), approval expiry, unexposed-job terminalization and subagent children. Under the same 8-conversation stress run, that brings delivery after a busy turn to a 4 s median (17 s max), against 33 s (63 s) with #16349 alone and 11 s (26 s) on the earlier dev, with the same database load as #16349.

Background tool and subagent completion wake-ups re-read the database every five seconds for as long as they wait, and on a busy deployment they wait a long time. Since #15350 (Wake Agents on Background Tool Completion), every backgrounded call pre-registers its completion delivery at dispatch. The delivery engine claims it within a second, finds the result not ready or the parent generation still busy, defers it, and repeats. Each cycle is a claim, a lease check, generation state and a defer. That repeats every five seconds for the whole time a tool runs, a turn continues, or an approval pause waits for a person. On the demo deployment one user's paused wake-ups held results for up to 5.8 hours, and the delivery collection sustained up to 32 operations per second from that one user: 801,915 AgentTriggerDelivery operations for 1,091 deliveries in 40 hours, about 735 per delivery. v0.8.7 has none of this, so it is a regression in the v0.8.8 release line.

The resolvers already asked for a one-second retryAfter on these deferrals, but the engine never read it: a readiness deferral (deferWithoutAttempt) always waited the fixed five-second default. So the waiting cadence could not be tuned from the producer side at all.

This PR makes waiting cheap without making delivery slower. A waiting delivery now backs off by how long it has waited, a tenth of its age between the existing five-second floor and a minute, and the engine honors a readiness retryAfter whenever it asks for longer than that floor (never shorter, so queued turns and every other deferral keep their cadence). In exchange, the two events a waiting delivery is actually waiting for now pull it forward immediately: when a background result becomes durable, its delivery is expedited, and when a generation settles, waiting completion deliveries in that conversation are expedited. After the busy turn ends, results arrive together within seconds instead of trickling out over half a minute.

It also stops the delivery loop from hijacking request traces. The engine is started at boot but woken from inside request handlers, and each claim pass rescheduled its timer from within that request's async context, so every later tick belonged to that request forever: one chat request's trace on the demo carried 207,147 delivery operations over 6.6 hours. Claim passes now run under the root context.

How it works

A waiting delivery's re-check interval grows with its age and is honored by the engine; producers expedite it the moment its condition changes:

dispatch        pre-register delivery (availableAt = now + 250ms)
claim pass      PARENT_NOT_READY / RESULT_NOT_READY / CHILD_NOT_READY
                  retryAfter = clamp(age / 10, 5s, cap)     # was: ignored, fixed 5s; cap defaults to 60s
result stored   persistBackgroundToolResult → expedite(deliveryKey)
projection only background tool persisted its row but not its receipt → admission.expedite()
turn settles    GenerationJobManager.onGenerationSettled → expedite(user, conversation)    # complete, error, abort, approval expiry (local or store-won)
held when any   expedite marks it (wakeRequestedAt) → fenced readiness deferral or ordering release makes it due
child persisted SubagentThreadTaskStore onTaskSettled → expedite(user, parent, taskIds) # includes a repaired predecessor

expediteAgentTriggerDeliveries runs two classic-operator updates, so it stays DocumentDB-compatible. One moves deferred rows no worker holds to now. The other stamps rows a worker holds with wakeRequestedAt, because that worker may be about to defer on readiness it read before the change. Readiness deferral and ordering release consume the marker and make the row due in one fenced, classic-operator write. Unmarked rows retain their requested deadline; only readiness deferrals restore an attempt. No unfenced follow-up update is needed. Every expedite is followed by a claim pass here, which also catches a matching row that was already due. A deferral whose parent has settled and is only finishing terminal persistence keeps the short re-check, since that clears within moments.

The cap is endpoints.agents.eventDriven.idlePolling.completionWaitMaxIntervalMs (default 60000, 5000–300000), alongside the existing idle-polling caps.

Ordinary generation settlement is announced after terminal cleanup. Direct approval expiry, store-won expiry relay, and unexposed-job terminalization announce separately. An ambiguous terminal CAS also announces when its probe confirms the same generation reached error, but not when the stream was replaced. Subagent results announce after their terminal message is durable; recovered and replayed results announce after wakeup registration, using the canonical task identity and any repaired predecessor. Poll-only tasks and explicitly declined admissions do not announce.

Type of change

  • Bug fix
  • Performance improvement

Testing

Reproduced and verified end to end against a local LibreChat build exporting OpenTelemetry to a ClickStack (ClickHouse Cloud) service, with the same scenario run on dev and on this branch. The scenario uses a deterministic MCP fixture (slow_task(seconds, label)) with the durable wake-up flags the demo runs (completionWakeups, durableReceipts, coalescing, actorMailbox): four background tasks finish while the conversation is busy with a long foreground task, so every result has to wait and then arrive as a wake-up.

Long wait (four 30–45 s tasks behind a 12-minute foreground task, about 13½ minutes of waiting each), measured from ClickStack spans and the delivery rows:

dev This branch
AgentTriggerDelivery operations 4,142 1,373
Claims (findOneAndUpdate) 1,632 454
Defers and other updates (updateOne) 620 190
Operations per minute while waiting ~295, flat for the whole wait 330 in the first minute, falling to ~30
Largest single trace 4,112 operations (the dispatching chat request) 51
All four results delivered after the busy turn ended within 20 s within 18 s

On dev the cost is linear in waiting time, about 74 operations a minute for each waiting result, indefinitely. On this branch each waiting result settles to about one re-check a minute, and the remaining steady ~30 operations a minute are mostly the engine's idle claim ticks, which exist without any waiting delivery. Delivery latency is unchanged, because the busy turn settling expedites every waiting result instead of waiting for its next re-check.

Short wait (four 45–75 s tasks behind a 2-minute foreground task): results were delivered within ~5 s of the busy turn ending on this branch, against a 33 s spread on dev, and the largest single trace fell from 928 delivery operations to 59.

Tested environments/configuration:

  • Node 24.16.0, MongoDB 8.2.6 (single-node replica set), Redis (USE_REDIS=true)
  • Agents endpoint with eventDriven.completionWakeups, durableReceipts, coalescing, actorMailbox
  • OpenRouter openai/gpt-6-luna; MCP fixture over streamable HTTP
  • OpenTelemetry → ClickStack collector → ClickHouse Cloud

Automated tests:

  • packages/api: src/agents and src/stream suites (5,341 tests) plus the new cases below
  • packages/data-schemas: triggerDelivery.spec.ts (99 tests) with mongodb-memory-server
  • New: backoff.spec.ts; waiting-age backoff for running, paused and persisting parents in backgroundCompletionWakeup.spec.ts and subagentCompletionWakeup.spec.ts; the engine honoring longer readiness delays, flooring shorter ones, and running claim passes under the root context in engine.spec.ts; expedite on a durable result and on a settled generation in service.delivery.spec.ts; generationSettled.spec.ts for complete, error, abort, a throwing listener and unsubscribe; and expediteAgentTriggerDeliveries against a real MongoDB (claimable after expedite, scoping, held rows untouched, unbounded input refused), plus the non-sparse { user, status, availableAt } index
  • Readiness signals: approval expiry announces settlement once (generationSettled.spec.ts); a settled subagent child is announced only after its terminal message is durable (subagentThreads.spec.ts); a projection-only persist expedites its delivery (handlers.background.spec.ts); a held delivery is marked, and its deferral returns expedited at now then clears the marker (real MongoDB, plus documentdb.spec.ts for update portability); the engine re-dispatches an expedited deferral at once (engine.spec.ts); every expedite runs a claim pass (service.delivery.spec.ts); the configured cap bounds both resolvers; and the completionWaitMaxIntervalMs defaults and bounds are covered (config.spec.ts)
  • npx tsc --noEmit in packages/api, packages/data-schemas and packages/data-provider

Readiness-race regression verification

  • Focused API suites: 355 passing tests across generation settlement/idempotency, delivery engine/service, background completion/handlers, and subagent resolver/thread lifecycle.
  • Real Mongo delivery methods and the DocumentDB static compatibility guard: 179 passing tests. Covers ordinary, legacy capability, and shielded leases; stale claim rejection; expedite racing conditional release writes; exact repaired-task targeting.
  • CJS subagent-store composition: 5 passing tests.
  • npx tsc --noEmit in packages/api and packages/data-schemas; scoped ESLint, Prettier, import sorting, manifest validation, and circular-dependency checks passed.
  • Tests used an isolated lockfile install and MongoDB 8.2.1 with workspace-local temporary files and --nounixsocket for the sandbox. No environment workaround is shipped.
  • npm run lighthouse built the frontend/backend and reached server readiness, but Playwright could not launch because its Chromium executable is not installed. No local Lighthouse score is claimed.
  • Redis cross-replica integration, live DocumentDB integration, full suites, and the author’s model-driven load measurements above were not rerun for these readiness-race repairs.

Screenshots / recordings

No user-facing change.

Risk / compatibility

A settled parent generation expedites waiting completion deliveries only in that user's conversation. A child's settlement narrows the selection to its own task id, plus the original abandoned task when recovery persisted a replacement terminal row. Unfinished siblings retain their backoff. Delivery ownership, source, user, and conversation fences remain in force. Expediting is best effort; a missed expedite (a crash between persistence and the update) only means the delivery re-checks at its backoff, at most a minute, instead of immediately. The settle-time expedite adds two indexed updateMany calls and one local claim pass per settled generation, served by a new { user, status, availableAt } index on agenttriggerdeliveries. The collection's only existing user index is sparse on actorActionAdmittedAt and cannot serve it, so without the new index every turn end would fetch every deferred delivery deployment-wide. The index is created at startup by ensureAgentTriggerDeliveryIndexes like the others. An expedite for one result uses the unique deliveryKey index. Readiness deferrals that ask for less than five seconds, including every queued-turn deferral, keep exactly the cadence they have today.

Checklist

  • I reviewed my own changes
  • Relevant tests have been added or updated
  • Existing relevant tests pass
  • The change does not introduce new warnings or errors
  • User-facing or complex behavior is documented where necessary
  • Required dependency changes have been merged/published
  • Required documentation PR: N/A

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T00:08:13.727248Z 9773ead Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d58e792d57

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

/** How long a deferred delivery may wait before re-checking a condition nobody has
* announced a change to. Matches the delivery engine's floor for readiness deferrals. */
export const WAITING_RETRY_FLOOR_MS = 5_000;
export const WAITING_RETRY_CAP_MS = 60_000;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Expose the retry cap through configSchema

This introduces a new operator-facing 60-second retry limit as a hard-coded constant, so deployments cannot tune the latency/database-load tradeoff without patching the package. Add it to configSchema with 60 seconds as the compatibility default, as required for new limits and timeouts.

AGENTS.md reference: AGENTS.md:L77-L79

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c94bd62: the cap is now endpoints.agents.eventDriven.idlePolling.completionWaitMaxIntervalMs (default 60000, range 5000–300000), next to the other idle-polling caps. The service exposes it via getCompletionWaitMaxIntervalMs, and both resolvers read it through getWaitMaxIntervalMs.

Comment on lines +2325 to +2326
availableAt: { $gt: input.now },
leaseBy: { $exists: false },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve wake-ups that race an active lease

When a result becomes durable or a generation settles while its delivery is being processed, this predicate skips the leased row. If the resolver already read the old readiness state, it subsequently defers the row using the age-based delay, and no later event remains to expedite it; an otherwise-ready completion can therefore wait the full 60-second cap. Persist a wake marker for held rows or recheck/fence readiness when applying the defer so the racing wake-up is not lost.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c94bd62: the expedite is now one pipeline update over every undelivered row in the selection. It moves only deferred, unheld rows, and returns { matched, expedited }. When a matching row was held (or already due), the service wakes the engine and re-expedites exactly once 2s later, after the holder has released or deferred it, so a resolver that deferred on stale readiness is pulled forward.

metricStatus = 'error';
}
recordGenerationJob(this.storeLabel, metricStatus);
this.notifyGenerationSettled(claim);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Notify waiters when an approval expires

A parent can also become terminal through expireApproval: ApprovalLifecycle.expireWithIdentity performs the direct requires_action -> aborted transition, and that path never creates a TerminalJobClaim or reaches finishTerminalJobInternal. Consequently this notification is absent when an approval times out, leaving completion deliveries backed off for up to 60 seconds after the parent is already terminal; emit the settled event from the approval-expiry winner/relay path as well.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c94bd62: expireApproval announces settlement (status aborted) after its host hooks, since that transition never builds a terminal claim. Covered in generationSettled.spec.ts, which checks it fires once.

Comment on lines +655 to +656
unsubscribeGenerationSettled ??= deps.subscribeGenerationSettled?.(({ userId }) =>
expediteCompletions({ user: userId }),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Expedite only after the subagent result is durable

For detached subagents, the child generation can settle before its task result is durable: SubagentThreadTaskStore.start awaits request.run and only afterward writes the terminal task message in persistResult (subagentThreads.ts lines 1357-1376). This callback can therefore expedite and claim the delivery while the result is still absent; it then receives CHILD_NOT_READY and backs off, while the later task-message write emits no second expedite. Signal from the successful result/failure/cancellation persistence boundary as well so completed children do not wait up to the 60-second cap.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c94bd62: SubagentThreadTaskStore takes an onTaskSettled hook, called from touchAfterMessage after the terminal child message (completed, failed, cancelled or abandoned) is durable. The host wires it to expediteCompletionWakeups({ user }). The test asserts the terminal message was saved before the announcement.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardened in ebd6ebc: recovered and terminal-replayed tasks announce after wakeup registration, including the original abandoned task whose ordered delivery may still be waiting. The host scopes the signal to those task ids in the owning parent conversation, leaving siblings backed off. subagentThreads.spec.ts covers durable completion, terminal replay, abandoned recovery and replay of that recovery; poll-only and declined admissions do not signal.

Comment on lines +774 to +776
const persisted = persist == null ? false : await persist(input);
if (persisted) {
expediteCompletions({ deliveryKeys: [input.deliveryKey], sourceIds: [input.sourceId] });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wake deliveries after projection-only persistence

This only expedites after the independent background receipt is stored. If that receipt write throws but backgroundToolCompletion.persist successfully writes the parent-message projection, handlers.ts deliberately keeps the delivery because deliveryReady is true, yet the projection persistence path emits no expedite. The resolver can consume that projection, but it remains unavailable until its age-based delay expires; arrange for the successful projection fallback to expedite the delivery too.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c94bd62: the admission carries a best-effort expedite() bound to its delivery key. The handler calls it when the projection persisted but the independent receipt did not.

Comment on lines +571 to +574
.then((expedited) => {
if (expedited > 0) {
deliveryEngine?.wake();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wake the engine when a matching delivery is already due

A readiness event can correspond to a matching row whose availableAt is already due but which this replica has not claimed yet, especially when another replica created or deferred it while this engine is in its idle backoff. expediteAgentTriggerDeliveries excludes such rows with availableAt: { $gt: now }, returns modifiedCount === 0, and this branch consequently suppresses the wake, so the ready delivery waits for the next idle poll (up to 15 seconds by default). Distinguish “already due” from “no matching delivery,” or wake for the exact-delivery readiness event even when no timestamp changed.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c94bd62: the same { matched, expedited } result covers this. Any match wakes the engine, including already-due rows that moved nothing.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c94bd62864

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +2356 to +2359
[
{
$set: {
availableAt: { $cond: [deferredAndUnheld, input.now, '$availableAt'] },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace the pipeline-form expedite update

On supported Amazon DocumentDB deployments, every readiness notification reaches this aggregation-pipeline updateMany, which DocumentDB 5.0 rejects; the repository's packages/data-schemas/src/methods/documentdb.spec.ts explicitly guards against this update form. Because expediteCompletions catches the rejection as best effort, completed background and subagent work is not woken and may wait for the full completion backoff. Split the match/count operation from a classic-operator update instead.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e188eef: the expedite is two classic-operator updateMany calls in parallel, one moving deferred unheld rows and one stamping held rows with wakeRequestedAt. No pipeline form, no $$REMOVE, and documentdb.spec.ts passes.

Comment on lines +587 to +588
if (matched > expedited && !followUp) {
setTimeout(() => expediteCompletions(input, true), EXPEDITE_FOLLOW_UP_MS).unref?.();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep wake intent until the held delivery is released

When readiness changes after a resolver's stale read but releasing/defering the held row takes more than two seconds, both the initial expedite and this sole follow-up observe the lease and modify nothing; !followUp then prevents another check, so once the row is finally deferred it waits for the full completion backoff. The fresh evidence in the current fix is that the retry is hard-stopped after one fixed-delay attempt rather than persisted or fenced to the lease release.

AGENTS.md reference: AGENTS.md:L38-L40

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e188eef by persisting the intent instead of a timed follow-up. The expedite stamps held rows with wakeRequestedAt. deferAgentTriggerDeliveryAttempt unsets it in the same fenced write, reads the pre-image, and if a marker was set, moves the row to now and returns expedited. The engine's post-processing claim pass then picks it up. This works however long the holder takes, and on any replica. Covered against real Mongo (held → marked → defer returns expedited and clears the marker → the next defer honors its requested time) and in engine.spec.ts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardened in ebd6ebc: both readiness deferral and ordering release consume wakeRequestedAt and make the row due in the same claim-fenced write. This also removes the unfenced second update. Real-Mongo coverage exercises all three lease formats, stale-token rejection, and an expedite arriving between conditional release writes. The engine test verifies immediate pickup after ordering release.

Comment on lines +8921 to +8925
this.notifyGenerationSettled({
streamId,
conversationId: expiredJob.conversationId,
userId: expiredJob.userId,
status: 'aborted',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Notify settlement from the store-won expiry relay

In a multi-replica Redis deployment, RedisJobStore.cleanupRequiresActionIndex can win the direct requires_action -> aborted transition before the owning manager calls expireApproval; that manager later handles the terminal row in the relay branch around lines 9121-9128, which runs the host hook and releases ownership without calling notifyGenerationSettled. The fresh evidence is that this added notification covers only the local expireApproval winner, so store-won approval expiry still leaves completion deliveries backed off until their next poll.

AGENTS.md reference: AGENTS.md:L38-L40

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e188eef: the relay branch announces settlement (aborted) when it releases this replica's ownership of the store-expired job, so only the owning replica announces it.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e188eef0f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

/** Classic operators only: aggregation-pipeline updates are not portable. A
* held row may be deferred on readiness its worker read before the change,
* so it keeps a marker that its deferral honors instead of moving now. */
const [moved, held] = await Promise.all([

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Serialize the held-marker and unheld-move updates

When a worker releases a delivery between these parallel updates, the move operation can inspect it while it is still leased and the held-marker operation can inspect it after it becomes pending; both updates then miss the row. The readiness notification is lost, leaving the now-ready completion deferred until its configured backoff expires. Fresh evidence in the current fix is that the marker and move queries run concurrently rather than establishing an order across lease release; run the held-marker update before the unheld-move update, or otherwise make the transition atomic.

AGENTS.md reference: AGENTS.md:L38-L40

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d9f1ab5: the held-marker update now runs first and the move second, sequentially. A row released after the marker read it as held is seen unheld by the move, so no release between the two escapes both.

Comment on lines +722 to +723
activeClaim = context
.with(ROOT_CONTEXT, () => runAsSystem(runClaimPass))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Detach the timer chain from request context

When wake() is called by an HTTP request, it invokes schedule() before reaching this root-context wrapper, so schedule() creates its setTimeout while the request context is still active. That timer callback invokes schedule() again, perpetuating the same request trace indefinitely even though the database claim itself runs under ROOT_CONTEXT; construct or bind the timer chain under the root context as well.

AGENTS.md reference: AGENTS.md:L38-L40

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d9f1ab5: schedule() also creates its setTimeout under ROOT_CONTEXT, so neither the claim pass nor the timer chain keeps the waking request's context. Covered in engine.spec.ts.

metricStatus = 'error';
}
recordGenerationJob(this.storeLabel, metricStatus);
this.notifyGenerationSettled(claim);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Notify waiters after unexposed-job terminalization

When an ambiguous Redis create is recovered but its idempotency or membership proof cannot be confirmed, terminalizeUnexposedGeneration directly transitions the visible job from running to error without creating a TerminalJobClaim, so this notification never runs. A completion resolver that observed that transient running job can therefore remain deferred until the completion-wait cap despite the parent already being terminal; carry the job's user identity into that direct terminalization path and announce its successful transition.

AGENTS.md reference: AGENTS.md:L38-L40

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d9f1ab5: terminalizeUnexposedGeneration takes the job's userId and announces settlement (error) when its direct running -> error CAS succeeds. All callers already pass full job data.

logger.error(`[subagentThreads] Failed to refresh ${outcome} child thread`, error);
}
try {
this.onTaskSettled?.(scope.userId);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate child-settlement wakeups on completion delivery

When backgroundTasks.completionWakeups is disabled, buildSubagentThreadTaskConfig uses the raw store and registerTaskWakeup deliberately creates no delivery, but every completed, failed, or cancelled child still reaches this unconditional hook. In production that issues user-scoped expedite updates and wakes the delivery engine even though no subagent completion delivery can exist, adding avoidable database and claim traffic for every poll-only child; retain whether a completion delivery was registered and invoke the settled hook only for those tasks.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d9f1ab5: the store records task ids whose completion wake-up was registered, and only those announce settlement. Poll-only children issue no expedite. Covered by a poll-only test.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

💡 Codex Review

this.notifyGenerationSettled({

P2 Badge Notify after recovering an ambiguously committed terminal CAS

When Redis applies the running -> error transition but loses the reply—a case the following catch explicitly anticipates—the same-epoch probe at lines 2321–2327 observes the terminal job and returns without reaching this notification. A completion delivery that observed the transient running state therefore receives no settlement signal and can wait until its configured recheck cap. Fresh evidence beyond the earlier fix is this probe-confirmed commit path; notify when the probe confirms the same generation reached error.


this.wakeupTaskIds.add(task.taskId);

P2 Badge Remove replayed tasks from the wakeup set

When prepareThread returns a terminal replay, registerTaskWakeup adds its task ID here, but the caller returns or throws at lines 1334–1338 without persisting another terminal message, so touchAfterMessage is never called and the only delete for this set never runs. Replaying distinct completed tasks therefore grows this process-lifetime set without bound; avoid tracking terminal replays or remove their IDs on the replay exit path. CLAUDE.mdL180-L186


onTaskSettled: (userId, conversationId) =>
expediteCompletionWakeups({ user: userId, conversationId }),

P2 Badge Target the settled subagent's delivery

When several completion-enabled subagents run under one parent conversation, every child settlement expedites all subagent and background completion deliveries for that conversation rather than the delivery identified by the settling task. Each completion therefore pulls still-running siblings out of backoff and wakes the engine to repeat their generation/message reads, producing quadratic readiness traffic across a batch of children and undermining the polling reduction this change implements; carry the task's delivery key or task ID through this callback and expedite only that delivery.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9773ead148

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +2306 to +2311
this.notifyGenerationSettled({
streamId,
conversationId: job.conversationId,
userId: job.userId,
status: 'error',
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Notify after confirming an ambiguous terminalization

When transitionStatus commits the running -> error CAS but its response is lost, the catch path probes getJob, sees this generation is terminal, and returns true without reaching this notification. Because this direct terminalization creates no TerminalJobClaim, waiting completion deliveries receive no other settlement signal and remain deferred for up to the configured completion-wait cap. Fresh evidence in this head is that the notification was added only to the non-throwing CAS-success branch; also notify when the probe confirms that the same generation reached the intended terminal state.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ebd6ebc. The terminal probe now notifies only when the same createdAt generation is confirmed in error. idempotencyClaim.spec.ts exercises the real public create/recovery path with a lost terminal CAS reply and rejects notifications for a replacement generation; the focused API run passed 355 tests.

@danny-avila danny-avila changed the title 🐢 fix: Back Off Waiting Completion Wake-ups and Deliver Them When Ready ⚡ fix: Deliver Waiting Completion Wake-ups the Moment They Are Ready Sep 25, 2026
danny-avila and others added 3 commits September 24, 2026 21:38
…backoff

# Conflicts:
#	packages/api/src/agents/triggers/README.md
#	packages/api/src/agents/triggers/engine.spec.ts
#	packages/api/src/agents/triggers/service.delivery.spec.ts
#	packages/api/src/agents/triggers/service.ts
#	packages/data-schemas/src/methods/triggerDelivery.spec.ts
#	packages/data-schemas/src/schema/triggerDelivery.ts
The e2e fake model reported input tokens for the chat messages alone, because the SDK hands a test override model the pruned messages without the systemRunnable pipe. Once #16301 gave every memory-enabled chat a system prompt, the calibrated context snapshot put used tokens below the instructions and the gauge dropped its Messages row. Count input over the complete prompt, as a real provider bills it.
@lia-by-librechat

Copy link
Copy Markdown
Contributor

Implementation review handoff for exact remote head ebd6ebc0882e54b2a279ae3a3edaabb96a7b03c2.

This head fixes the remaining in-scope readiness gaps:

  • Probe-confirmed unexposed terminalization announces only the matching generation, including a committed CAS with a lost reply.
  • Held wake markers are consumed atomically with both ordering release and readiness deferral. Claim-token fencing, attempt accounting, and classic DocumentDB-compatible operators are preserved.
  • Recovered/replayed child results announce after registration. Signals target the canonical task and its repaired predecessor, not unrelated siblings; declined admissions and poll-only tasks stay quiet.

Subsystem self-review covered terminal writers, admission and replay readers, lease/expedite interleavings, failure and retry paths, ownership and user/conversation/source scoping, cleanup, and mixed-version marker fallback.

Verified locally: 355 API tests, 179 Mongo/DocumentDB-guard tests, 5 CJS wiring tests; no-emit typechecks in packages/api and packages/data-schemas; scoped static checks and production frontend/backend builds. Lighthouse was attempted, but its browser setup is blocked by missing Playwright Chromium. Redis cross-replica integration, live DocumentDB, full suites, and model-driven load tests were not rerun.

A maintainer needs to trigger the next Codex review for this exact SHA. A Lia GitHub App comment cannot trigger it. CI and an independent review of this head are not yet claimed green.

@danny-avila
danny-avila merged commit 7238be3 into dev Sep 25, 2026
31 checks passed
@danny-avila
danny-avila deleted the danny-avila/bg-wakeup-backoff branch September 25, 2026 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants