Skip to content

📮 fix: Count Undelivered Background Results as Outstanding - #16343

Merged
danny-avila merged 8 commits into
devfrom
danny-avila/bg-delivery-semantics
Sep 25, 2026
Merged

danny-avila merged 8 commits into
devfrom
danny-avila/bg-delivery-semantics

Conversation

@danny-avila

@danny-avila danny-avila commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

check_background_task reports when a background tool stopped running. It does not report when the tool's result actually reached the conversation, and with automatic completion delivery (#15350) those are different moments. A finished task whose wake-up is still waiting reads status: "completed" exactly like one whose result the agent already consumed. So an agent that lists its tasks concludes the work is done, tells the user so, and is then resumed by a result it has already written off. On the demo deployment an agent reported three tasks as finished and cancelled while their results were still queued, and they arrived as new turns minutes later.

The listing is also blind outside one process. It reads only the in-process task registry, which holds tasks this replica dispatched in the last hour. A result dispatched in an earlier turn on another replica, before a restart, or more than an hour ago is invisible, even though its durable delivery row will still resume the agent. Cancelling such a task returns not_found, so an agent has no way to stop a result it no longer wants.

This PR makes delivery, not execution, the definition of done. Every background tool task carries delivery: "pending" | "delivered" when automatic delivery applies. The listing adds the conversation's undelivered completions from the durable delivery store and returns an outstanding count that includes finished-but-undelivered results. When any of those exist, it also returns guidance telling the agent not to report them as finished. Cancelling a finished, undelivered task that this process does not hold retires its pending delivery so it never arrives, and reports honestly when that is not possible: the task is still running elsewhere, or a wake-up already owns the result. The tool description states the contract.

How it works

A listing merges the process-local registry with the durable view, deduplicated by task id. outstanding counts work that will still resume the agent:

check_background_task (list)
  registry.list(user, conversation)                       # this replica, last hour
  pending.list(user, conversation)                        # durable, any replica / restart / age
    agenttriggerdeliveries: source = background-tool-completion,
                            target.conversationId, status ∉ settled, not capability-dead, ≤ 200
                            projects receipt status/settledAt/claim only
  local finished task absent from a complete durable listing → delivery = "delivered"
  outstanding = running + (delivery == "pending"), subagents included

Cancelling a task this process does not hold goes to the durable store, which retires the delivery only while no wake-up has claimed it:

Durable state Result
no undelivered completion ordinary lookup (not_found, subagent controls)
tool still running (no receipt) unavailable — cannot be stopped from this replica; still pending
settled, claimed by a wake-up delivery_scheduled
settled, unclaimed retired onlyIfUnclaimed → cancelled; a lost race reports delivery_scheduled

Tasks the process still holds keep their existing poll path. Cancelling a finished local task polls it, which retires the pending delivery and returns the result. It is no longer refused under the default ordinaryToolCancellation: false policy, because there is no execution left to stop. The retirement is an exact transition: a wake-up that claims and finishes the delivery between the lookup and the retire reports delivery_scheduled, never cancelled.

The dispatching replica's registry never learns that a wake-up on any replica delivered its task's result. So when the durable listing is complete, a finished local task that is absent from it reports delivered. A local task whose automatic delivery dead-lettered reports delivery: "failed" instead. It still counts as outstanding, with guidance to poll for the result, which is the only copy left to recover. Finished detached subagents count as outstanding with delivery: "pending" only while their own completion delivery is undelivered in the durable store, so the status follows how each task was admitted, not the current setting.

Type of change

  • Bug fix

Testing

Verified end to end on two replicas of a local LibreChat build sharing MongoDB and Redis, exporting OpenTelemetry to a ClickStack (ClickHouse Cloud) service, with the durable wake-up flags the demo runs and a deterministic MCP fixture (slow_task(seconds, label)). Two background tasks are dispatched on replica 1, and every later turn runs on replica 2, whose registry never held them:

Turn (replica 2) check_background_task result
List while both still run on replica 1 both running, delivery: "pending", outstanding: 2
List after both settled, inside a busy turn that holds their wake-ups both completed, delivery: "pending", outstanding: 2, plus the "not delivered yet" guidance
Cancel one finished, undelivered task cancelled: "The finished result was discarded and will not arrive as a new turn."
List again on replica 1, which dispatched both, after delivery both completed, delivery: "delivered", outstanding: 0 (its registry was reconciled against the durable store)

After the cancel turn ended, exactly one wake-up arrived, for the other task. The discarded task's delivery row settled with reason background result discarded by its owner, and no warnings or errors were logged on either replica. On dev the listing reads only the answering replica's own registry, so replica 2 would list no tasks, and the cancel would fall through to not_found.

Tested environments/configuration:

  • Node 24.16.0, MongoDB 8.2.6 (single-node replica set), Redis (USE_REDIS=true), two API processes
  • Agents endpoint with eventDriven.completionWakeups, durableReceipts, coalescing, actorMailbox
  • OpenRouter openai/gpt-6-luna; MCP fixture over streamable HTTP

Automated tests:

  • background.spec.ts: a finished task is outstanding until delivered (pending → delivered on claim); running work has no delivery field and adds no guidance; durable completions from earlier turns and other replicas are listed without result content and deduplicated against local tasks; a failing durable view degrades to a partial listing; cancel outcomes discarded, running and delivering map to cancelled, unavailable and delivery_scheduled, and not_pending falls through to the ordinary lookup; the description states the delivery contract
  • backgroundCompletionWakeup.spec.ts: createPendingBackgroundCompletions lists without delivery internals, discards only unclaimed deliveries, and never retires running or claimed ones
  • triggerDelivery.spec.ts (mongodb-memory-server): the listing returns running and settled completions without output, excludes delivered rows and other users, conversations and sources, and refuses malformed lookups
  • packages/api src/agents suites (747 tests) and api server/services/Endpoints/agents (101 tests)
  • npx tsc --noEmit in packages/api and packages/data-schemas

Screenshots / recordings

No user-facing change.

Risk / compatibility

The listing adds one read per check_background_task list call when completion wake-ups are enabled. It filters on user and the undelivered statuses, which the { user, status, availableAt } index from #16339 serves; before that index exists it is answered from the existing status indexes over in-flight rows. It is bounded to 200 rows, which matches the per-conversation task limit. It projects only receipt metadata, never result content, and skips capability-dead rows, which no worker will deliver. A truncated listing says so with partial: true and doesn't reconcile local tasks, and a failed read falls back to the local listing the same way. Cancelling uses a task-specific lookup, so it doesn't depend on the listing limit. The response gains outstanding and, per task, delivery; existing fields are unchanged. Discarding a finished result through the durable path does not require ordinaryToolCancellation, because it stops no execution: it only retires a delivery that has not started, and only for the owning user and conversation. Legacy (V1) completion rows keep their results only on the parent message, so they are left out of the listing rather than misreported as running, and they drain on their existing path. A manual poll that claims a task this process doesn't hold retires that task's pending delivery, as the local poll path already did. The pending controls stay attached when an operator disables completion wake-ups, so deliveries admitted before the change remain visible while they drain. Deployments that never enabled completion wake-ups see no change.

Checklist

  • I reviewed my own changes
  • Relevant tests have been added or updated
  • Existing relevant tests pass
  • The change does not introduce new warnings or errors
  • User-facing or complex behavior is documented where necessary
  • Required dependency changes have been merged/published
  • Required documentation PR: N/A

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T00:11:29.708120Z bb03823 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd96945139

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1916 to +1919
if (task.resultClaim != null || task.completionWakeupRetired === true) {
return { delivery: 'delivered' };
}
return { delivery: 'pending' };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reconcile local tasks after automatic delivery

When createBackgroundToolCompletionWakeupResolver delivers a result, it claims the message/trigger receipt but never updates backgroundTaskRegistry.resultClaim or completionWakeupRetired. Consequently, a completed task retained on the dispatching replica keeps returning delivery: "pending" for its one-hour TTL even after the durable delivery succeeds and disappears from the pending query, so subsequent listings incorrectly count it as outstanding and tell the agent not to report it as done. Reconcile local entries with the durable delivery state rather than treating the process-local claim as the only evidence of delivery.

AGENTS.md reference: AGENTS.md:L42-L45

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 352e8ce: when the durable listing is complete, a finished local task that is absent from it (so delivered by a wake-up on any replica, or retired) reports delivery: "delivered" and stops counting as outstanding. An incomplete listing proves nothing, so the local view stands and the response is marked partial.

'background result discarded by its owner',
{ onlyIfUnclaimed: true },
);
return retired ? 'discarded' : 'delivering';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Distinguish a lost discard race from successful retirement

If a wake-up wins between read() and this retirement, retireAgentTriggerDelivery(...onlyIfUnclaimed) cannot update the now-leased row; however, once that wake-up completes, the method's existing idempotency fallback returns true for the resulting succeeded row. This ternary then reports cancelled and promises the result will not arrive even though it was already delivered. The retirement API needs to distinguish “this call retired it” from “it was already succeeded,” or the caller must re-read the outcome after the failed CAS.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 352e8ce: retireAgentTriggerDelivery takes requireTransition, which skips the idempotent already-succeeded fallback. Discard retires with { onlyIfUnclaimed: true, requireTransition: true }, so a wake-up that wins and finishes in between reports delivery_scheduled, never cancelled. Covered against real Mongo.

Comment on lines +1853 to +1854
const PENDING_DELIVERY_GUIDANCE =
'Some finished tasks have not been delivered yet (delivery: "pending"); each will arrive as a new turn. Poll one to collect its result now, or cancel it so it does not arrive. Do not report these tasks as finished or cancelled until then.';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor cancellation guidance with the default policy

With the default ordinaryToolCancellation: false, a finished task that is still in the local registry is rejected by the pre-existing cancellation gate before the new durable discard path is reached. The listing now explicitly tells the model to cancel such a pending result, but following that instruction returns status: "invalid"; only the same task after a restart or on another replica can be discarded. Handle already-settled local tasks as result-discard operations before applying the live-execution cancellation policy.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 352e8ce: the ordinaryToolCancellation gate now applies only to a running local task. Cancelling a finished one falls through to the poll path, which retires its pending delivery and returns the result, under the default policy too.

Comment on lines +47 to +48
/** A conversation's undelivered completions are few; this bounds a pathological listing. */
const MAX_PENDING_BACKGROUND_COMPLETIONS = 50;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Make the pending-completion limit configurable

This hard cap is shared by both listing and discard: once a conversation has more than 50 undelivered durable completions, newer tasks are absent from outstanding, and direct cancellation of one outside the oldest 50 returns not_pending/not_found even though its result will still arrive. The in-process conversation limit is 200 and durable rows survive restarts, so this is reachable; provide a task-specific lookup for cancellation and expose the listing limit through configSchema instead of fixing it at 50.

AGENTS.md reference: AGENTS.md:L77-L79

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 352e8ce, without a new config field: this isn't a new operator lever. The listing cap now reuses the registry's existing per-conversation task limit (200), a truncated listing is reported (partial: true plus a warning, with no reconciliation), and cancel uses a task-specific lookup (taskId filter), so discard no longer depends on the listing window.

Comment thread packages/api/src/agents/background.ts Outdated
Comment on lines +2717 to +2719
const outstanding =
ordinaryTasks.filter((task) => task.status === 'running' || task.delivery === 'pending')
.length + subagentTasks.filter((task) => task.status === 'running').length;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Count undelivered subagent results as outstanding

When automatic wake-ups are enabled, a detached subagent can be terminal with result_available: true and result_claimed absent while its completion is still waiting to resume the parent. This counter includes only running subagents, so that undelivered terminal result contributes zero to outstanding and receives none of the pending-delivery guidance, recreating the premature “all work is done” signal this change is intended to prevent. Count terminal, unclaimed subagent results while completion delivery is enabled.

AGENTS.md reference: AGENTS.md:L42-L45

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 352e8ce: with completion wake-ups enabled, a terminal subagent with result_available and no result_claimed gets delivery: "pending", counts toward outstanding, and triggers the not-delivered guidance.

Comment on lines +2344 to +2349
.find({
user: input.user,
'envelope.event.source.type': 'internal',
'envelope.event.source.id': input.sourceId,
'envelope.target.conversationId': input.conversationId,
status: { $in: UNDELIVERED_STATUSES },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude capability-dead rows from pending completions

During rolling-version compatibility, the delivery code treats rows such as status: "leased", capabilityStatus: "dead" as dead letters, but this query checks only the public status and therefore lists those terminal rows as undelivered. Such a row will not resume the conversation without an operator requeue, yet the model sees it as delivery: "pending" and outstanding indefinitely; cancellation also reports it as running or already delivering. Apply the existing capability-dead predicate when selecting pending completions.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 352e8ce: the listing excludes capabilityStatus: 'dead' rows. Covered with a leased/dead row against real Mongo.

'envelope.target.conversationId': input.conversationId,
status: { $in: UNDELIVERED_STATUSES },
})
.select('+backgroundToolResult deliveryKey createdAt envelope.event.payload')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Project metadata instead of loading every result body

Selecting +backgroundToolResult fetches the entire hidden subdocument, including up to 64 KiB of output per row, even though this method returns only status, settlement time, and claim presence. A normal 50-row listing or cancellation can therefore read and materialize more than 3 MiB of tool output on every invocation, contrary to the method's stated content-free contract. Select only the required nested receipt fields so status checks do not repeatedly load potentially sensitive, large result bodies.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 352e8ce: the projection selects only backgroundToolResult.status, .settledAt and .resultClaim, so no result body is read.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 352e8cede2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +2411 to +2414
...(receipt != null && {
result: { status: receipt.status, settledAt: receipt.settledAt },
}),
claimedByWakeup: receipt?.resultClaim != null,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle V1 rows before treating missing receipts as running

During a rolling upgrade, retained background_tool_completion_v1 deliveries also match this query, but those workers persisted terminal results only on the parent message and never populated backgroundToolResult. Consequently, a finished V1 task reaches this projection with receipt == null, is serialized as running, and discard returns running instead of retiring its pending delivery; cross-replica cancellation therefore fails until the automatic wake-up runs. Consult the legacy message receipt or otherwise distinguish V1 rows rather than interpreting every missing V2 receipt as live execution.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecb5d3e: the listing (and so discard) selects only rows admitted with the receipt-capable worker capability. Legacy V1 rows keep their result only on the parent message, so a missing receipt can't tell running from finished; they drain through their existing path and are left out rather than misreported. Covered against real Mongo.

Comment on lines +2701 to +2704
const durable = await params.pendingCompletions.list({ userId, conversationId });
pendingCompletions = durable.completions.filter(
(completion) => !localTaskIds.has(completion.taskId),
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retire remote deliveries when a manual poll claims them

When a task is absent from the local registry, a targeted manual poll claims its result only on the parent message; it does not retire or settle the durable trigger row. While that same generation remains active, the automatic resolver must defer, so a subsequent list in the turn re-adds the row here as delivery: "pending", counts it as outstanding, and tells the agent not to finish even though the result was just returned; another poll has a new invocation ID and can only report that the result is already claimed. Retire the pending delivery when the manual claim succeeds, or reconcile the durable listing with the parent-message claim.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecb5d3e: when a manual poll acquires the durable result of a task this process doesn't hold, it retires that task's pending delivery (settleClaimed, onlyIfUnclaimed, reason completion claimed by manual poll), matching the local same-generation path. Later listings in the turn no longer re-add it.

Comment on lines +518 to +522
...(completionWakeupsEnabled
? {
preregister: preregisterBackgroundToolCompletion,
pending: pendingBackgroundToolCompletions,
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep pending controls enabled while old wakeups drain

If an operator disables backgroundTasks.completionWakeups while previously admitted deliveries still exist, this conditional removes both new admission and the pending-completion controls. The trigger service and resolver still advertise both background-completion worker capabilities and therefore continue delivering those retained rows, but new turns cannot list, count, or discard them and receive the manual-delivery tool description instead. Gate only preregister on the current setting, while retaining pending controls until old durable work has drained.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecb5d3e: only preregister is gated on completionWakeups. The pending controls stay attached, so deliveries admitted before an operator disabled wake-ups are still listed, counted and discardable while they drain.

Comment on lines +1853 to +1854
const PENDING_DELIVERY_GUIDANCE =
'Some finished tasks have not been delivered yet (delivery: "pending"); each will arrive as a new turn. Poll one to collect its result now, or cancel it so it does not arrive. Do not report these tasks as finished or cancelled until then.';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retire pending subagent wakeups on terminal cancellation

This guidance also applies to the newly marked terminal subagent entries, but cancelling a settled subagent routes to controlTask, which returns not_running; it neither claims the result nor retires the pre-registered subagent-completion trigger. Thus, when an agent follows the instruction to cancel a finished subagent result, that result still arrives as a new turn. Add a terminal-result discard path for subagent wakeups or restrict the cancellation guidance to task types that actually support it.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecb5d3e: finished, unclaimed subagents get their own guidance, which tells the agent to poll to collect the result and says nothing about cancelling. The cancel guidance now appears only for ordinary tool tasks, where discard is supported.

Comment thread packages/api/src/agents/background.ts Outdated
Comment on lines +2471 to +2478
outcome = await params.pendingCompletions.discard({ userId, conversationId, taskId });
} catch (error) {
logger.warn(`[background] Failed to discard pending completion ${taskId}:`, error);
return JSON.stringify({
status: 'unavailable',
background_task_id: taskId,
message:
'The pending result could not be discarded right now. It may still arrive as a new turn; retry the cancel shortly.',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Let subagent cancellation proceed when completion lookup fails

For every nonlocal cancel, the ordinary-completion discard lookup now runs before subagent resolution, and an error returns immediately here. If the trigger-delivery collection is temporarily unavailable while the task is a live subagent owned locally or reachable through its routing transport, the unrelated read failure prevents its cancellation entirely; the poll path below already resolves subagents first specifically to avoid this masking. Resolve the subagent identity/control path before treating an ordinary-completion lookup failure as the final result.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ecb5d3e: a failed discard lookup no longer returns early. Cancellation falls through to subagent resolution and control, and unavailable is reported only if nothing else claims the task id.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ecb5d3e594

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/api/src/agents/background.ts Outdated
Comment on lines +2769 to +2772
if (completionWakeups) {
subagentTasks = subagentTasks.map((task) =>
task.status !== 'running' && task.result_available === true && task.result_claimed !== true
? { ...task, delivery: 'pending' as const }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Track subagent delivery per admitted task

When completionWakeups is toggled while a subagent task is still live, this uses the current request's configuration rather than the delivery mode recorded when that task started. Disabling wake-ups makes a previously admitted terminal result stop counting as outstanding even though its durable wake-up will still resume the conversation; enabling them makes an older poll-only result appear pending even though no wake-up exists. The fresh evidence beyond the earlier ordinary-tool toggle fix is this separate subagent-only gate over restored snapshots; preserve admission mode per task or reconcile against durable subagent deliveries instead.

AGENTS.md reference: AGENTS.md:L42-L45

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8887c46: subagent pending status no longer depends on the current request's setting. A finished, unclaimed subagent is marked delivery: "pending" only when its subagent-completion delivery is still undelivered in the durable store (new listUndeliveredAgentTriggerTaskIds). That follows the policy the task was admitted under, through toggles in either direction. A poll-only child with no delivery is not counted.

) {
return task;
}
return { ...task, delivery: 'delivered' };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Distinguish dead deliveries before reconciling them

When a finished local task's wake-up has dead-lettered, the durable query intentionally omits that row, but this absence-only reconciliation then changes the task from delivery: "pending" to "delivered". The result was never delivered and remains available for the existing manual dead-claim recovery path, so the model sees outstanding: 0 and is discouraged from polling the only recoverable copy. The fresh evidence after the earlier capability-dead exclusion is this downstream interpretation of every omitted row as successful delivery; return a dead/failed marker or otherwise require positive settlement evidence before reporting delivered.

AGENTS.md reference: AGENTS.md:L42-L45

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8887c46: the listing reads dead-lettered rows too (dead, capability_dead, or capabilityStatus: 'dead') and returns their task ids separately. A local finished task whose delivery dead-lettered now reports delivery: "failed", still counts as outstanding, and gets guidance to poll for its result. Only absence from a complete, non-dead listing reports delivered.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8887c467ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +2593 to +2595
/** The result reaches the agent here, so its automatic delivery is redundant. */
await params.pendingCompletions
?.settleClaimed({ userId, conversationId, taskId })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retire local deliveries after durable manual claims

When a retained local task is polled after its parent-message receipt exists, the earlier branch receives durableClaim.status === 'acquired' and returns the result without calling settleClaimed; this retirement is performed only for the nonlocal path here. A subsequent list in the same generation therefore still finds the durable row, reports the completed local task as delivery: "pending", and counts it as outstanding until the generation ends and the resolver drains it. Fresh evidence beyond the earlier remote-task finding is that the local acquired-claim branch remains untreated; apply the same retirement there.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bb03823: when a poll of a task this process holds acquires the durable result, it retires the task's pending wake-up through the registry (onlyIfUnclaimed, completion claimed by manual poll). That also marks it delivered locally, so a later list in the same generation reports delivered and outstanding: 0. Covered by a test.

Comment on lines +2416 to +2420
if (isDeadDelivery(row)) {
if (typeof payload?.taskId === 'string') {
deadTaskIds.push(payload.taskId);
}
return [];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve dead tasks in restored listings

When a completion dead-letters after the process-local registry is lost through a restart, eviction, or another replica, this branch records only its ID and drops the completion metadata. Since runCheckBackgroundTask uses deadTaskIds solely to reconcile local tasks, the restored dead task is absent from tasks, contributes nothing to outstanding, and never triggers the new failure guidance even though its result remains recoverable by polling. Fresh evidence beyond the earlier local dead-letter finding is this no-local-registry path; return enough dead-task metadata to serialize it as delivery: "failed".

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bb03823: the listing returns dead-lettered completions with their metadata (dead) instead of bare ids. A dead letter this process no longer holds is serialized from the durable store as delivery: "failed", counts as outstanding, and gets the poll-to-recover guidance.

Comment on lines +2385 to +2389
.find({
user: input.user,
'envelope.event.source.type': 'internal',
'envelope.event.source.id': input.sourceId,
'envelope.target.conversationId': input.conversationId,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Index pending-completion lookups by conversation

On installations with many active or retained dead trigger deliveries, this new per-status-check query has no matching index beginning with user, source, and conversation—the schema only indexes delivery status/lease fields and an unrelated sparse user field. MongoDB must therefore examine the global undelivered/dead population and perform the createdAt sort before returning at most 201 rows for one conversation; the analogous subagent lookup has the same shape. Add compound indexes matching these lookup predicates so routine background-task listings do not become globally proportional.

Useful? React with 👍 / 👎.

Comment on lines +2791 to +2795
const waiting = new Set(wakeups.taskIds);
subagentTasks = subagentTasks.map((task) =>
task.status !== 'running' &&
task.result_claimed !== true &&
waiting.has(task.background_task_id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor truncated subagent wakeup listings

When a conversation has more than 200 undelivered subagent wakeups, listSubagentWakeups returns complete: false, but this caller ignores it and treats every absent ID as proof that delivery is not pending. This is especially damaging because the subagent snapshot store retains the newest 200 tasks while the delivery query returns the oldest 200, so visible terminal tasks can all lose delivery: "pending", disappear from outstanding, and produce no partial warning despite still being scheduled to resume the conversation. Fresh evidence beyond the earlier ordinary-completion cap finding is that this separate subagent path discards its truncation signal; paginate or query the visible task IDs, or conservatively preserve unknown delivery state.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: bb0382355d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…y-semantics

# Conflicts:
#	packages/data-schemas/src/methods/triggerDelivery.spec.ts
#	packages/data-schemas/src/methods/triggerDelivery.ts
@danny-avila
danny-avila merged commit 34938eb into dev Sep 25, 2026
32 checks passed
@danny-avila
danny-avila deleted the danny-avila/bg-delivery-semantics branch September 25, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant