Dead-letter gap: an outage >~45min permanently discards queued agent work, silently

[lilyshen0722]

An outage longer than ~45 minutes permanently discards queued agent work, and nothing says so

Found 2026-08-18 while diagnosing a fleet-wide quota outage. @sprint-review noticed the word unacked in its own failure lines and counted the gap; the code confirms it.

What happens

agentEventService requeues delivered-but-unacked events, capped at AGENT_EVENT_REQUEUE_MAX_ATTEMPTS (default 3). Events that hit the cap are retired to a terminal failed status. From the code comment:

Events that hit the requeue cap and were retired to the terminal 'failed' state. Without this pass they would sit in 'delivered' — invisible to list(), ineligible for requeue — until the 168h retention delete.

The spacing is longer than the 10-minute default suggests, and the code says so:

Effective redelivery latency is NOT 10 minutes: schedulerService runs this job on */10, so an event delivered just after a pass waits for the one after next. Period P and threshold T give [T, T+P) — uniform over 10-20 min here, mean ~15.

3 attempts × ~15 min mean ≈ 45 minutes. After that the event is gone: not retried, not surfaced, deleted at 168h.

Observed

During the 4am quota exhaustion, sprint-review lost three events — 6a842b0d…, 6a842f2a…, 6a842fd6… — with a measured 46-minute gap between its last failed spawn and its next activity. pod-architect was in the same state with four consecutive failures.

The seat was not at fault and this is worth stating because it is the natural first suspicion: it kept polling throughout and processed a human mention at 04:10. It asked; the kernel had stopped serving.

Why this is worth fixing

The requeue exists precisely to prevent silent loss. Its own comment cites the incident that motivated it:

Saw this 2026-05-18: Cody's old pod marked 2 chat.mentions delivered then died on a stale config; the new pod never re-fetched them.

The attempt cap reintroduces that exact failure on a 45-minute timer. And the triggering condition is not exotic — BYO seats share the operator's Claude subscription, so quota exhaustion is routine and hits every seat at once. Tonight it hit the operator's own session in the same minutes. Any such outage reliably exceeds 45 minutes.

What I am NOT proposing

Do not simply raise the cap. It is a reasonable poison-event guard — an event that crashes its handler should not retry forever. The defect is not the cap, it is that hitting it is invisible.

Suggested direction (for whoever picks this up to challenge)

1. Surface the terminal state. failed events are currently invisible to list(). At minimum they should be countable and queryable per (agentName, instanceId).
2. Signal a human. A seat that has events retired to failed is a seat that silently lost work. That is the same class of thing the silence alert (feat(onboarding): alert when a newcomer types and nothing answers (W4 item 2) #954) exists for.
3. Distinguish poison from outage. Repeated failure of one event with different errors is a poison event. All events for a seat failing with the same error is an outage — and an outage should not consume the poison budget. This distinction is probably the real fix.

Related

• spawnRetryPolicy has an explicit QUOTA failure class that opens the circuit at the 15-minute ceiling, but classifySpawnFailure did not match "You've hit your session limit" — the failures logged as runtime. Worth fixing separately; note it does not change this outcome, since at either delay the event still dies on the requeue cap.
• ADR-023 (hosted runtime): the shared-subscription ceiling is what makes the triggering outage routine.

[lilyshen0722]

Live outage 2026-08-18 04:55–09:00 produced ZERO retirements — the model in this issue needs correcting

A second quota outage ran ~4 hours, far past the ~45-minute cap this issue describes. I expected mass dead-lettering. It did not happen: grep over 5h of backend logs finds no [agent-event] retired N … line (agentEventService:687), and only two requeue passes fired (18 and 14 events) where ~30 passes ran on the */10 schedule.

Why, and it inverts the risk model. The requeue and expiry passes only act on events in delivered status. When the circuit opened, the seats stopped fetching — so the events stayed pending, were never re-delivered, and never incremented attempts. The circuit breaker accidentally protected them.

So the dead-letter gap does not bite during long clean outages, which is the case I originally framed it around. It bites when a seat keeps fetching and failing — churning pending → delivered → stale → requeue and burning an attempt each cycle. That is what cost sprint-review three events in the earlier 46-minute window: it kept fetching.

This makes the fix more clearly about classification, not about the cap. An attempt should count as a poison-signal only when the failure is attributable to the event — not when a seat is failing every event with the same provider error. The suggested direction in the issue body (distinguish poison from outage) is now the primary fix rather than one option among three; raising the cap would have changed nothing here.

Also worth noting for whoever picks this up: the invisibility claim in the issue body is partly wrong. Retirement is logged at :687 as a console.warn. What is missing is a queryable surface (list() still excludes failed) and any signal that reaches a human — a log line in a pod nobody tails is not a signal. The 'silence alert' framing still holds; 'unlogged' does not.

Corrected by measurement rather than reasoning — I had asserted the loss would recur and it did not.

[lilyshen0722]

Taking this (TASK-027). Two corrections first, because the framing currently attached to it would aim the design at the wrong failure.

Correction 1 — the "long clean outages are safe" reading is falsified

The observation "a 4-hour outage retired ZERO events" is true. The inference "so the gap bites on short churning failures, not long clean ones" is not.

Measured on the backend's own GC log across that same window (2026-08-18):

13:30:01  pending=1
13:50:00  requeued 18   pending=4
14:10:01  requeued 14   pending=14
14:20:00  pending=1
15:30:00  pending=1

21 pending events destroyed during the outage. Zero retirements is the signature of that failure, not evidence against it: retirement requires attempts >= 3, attempts only increments at the claim site, and a seat with an open circuit never fetches — so nothing ever reached delivered, nothing accumulated attempts, and everything died on the stale-pending sweep, which is the one path that emits no per-event line. Checking for the observable half and finding none is exactly what the silent half looks like from outside.

Correction 2 — #1001 changed the landscape again

That was pre-#1001. Since 13f68032 deployed (07:59Z today, verified live: two GC passes at 08:10 and 08:20 both pending=0), the 30-minute sweep is gone. So the same outage now preserves those events as pending indefinitely, up to the 168h retention.

Which splits the problem cleanly, and only one half is now about the cap:

outage shape	what happens post-#1001	visible?
fetch-and-fail (poll works, spawn dies)	attempts climbs, retire fires ~45 min, failed row with an error	yes
can't-fetch (circuit open, seat down, network)	attempts frozen at 0, event sits pending for 168h	no

So the design target is not poison-vs-outage classification

That framing addresses the retire path, which is the half that already surfaces. The gap is the other one: nothing records a live consumer holding a queue it has never managed to fetch.

A seat with an active AgentInstallation, a growing pending backlog, and zero deliveries is a distinct and detectable state — and it is precisely the state yesterday's outage produced for four hours while every existing signal read normal. attempts: 0 plus age plus a live installation is the fingerprint; nothing currently looks for it.

This is the same requirement ADR-021 D3 states as "the fault must survive its own sweep", extended one case further: the fault must also survive never having been delivered at all. I filed that half separately as #1007; this issue is where the detection design belongs.

Not yet verified: whether an accumulating-pending-with-live-install state can be distinguished from a legitimately idle seat without false positives. That is the design question, and it is the one worth spending the effort on rather than the cap.