Rescue should drain a large abandoned-claim backlog in bounded time #627

Problem

Stale-heartbeat and deadline rescue on queue storage handle at most RECEIPT_RESCUE_BATCH_LIMIT (500) claims per pass per rescue kind, oldest claim slot first. That is plenty while rescue only mops up the odd lost worker, but it can't keep up when one runtime holding tens of thousands of claims dies at once.

Seen in a production deployment on 0.6.12: a high-throughput runtime (about 1.7–2M claims an hour across five queues) was repeatedly OOM-killed. Each kill abandoned 25k–70k open claims. The maintenance leader rescued at the cap, about 1,000 claims a minute per kind, on every tick, so it fell hours behind. Rescues were working through claims made 3.7 hours earlier. Because a claim slot can't be reused while it holds open claims, the claim ring stopped rotating for about 8 hours. The abandoned jobs waited hours before running again.

The cooldown change in #621 doesn't help here: rescue was already running every tick, and the limit is the batch size.

Proposal

Make rescue drain a large backlog in bounded time instead of a fixed row count per pass:

• Time-budgeted sub-batches. Keep each statement at the current size (500 rows), but let a pass keep taking sub-batches, each in its own transaction, while the previous one was full and a per-pass time budget remains (e.g. 2 s, configurable). A backlog then drains at whatever rate the database sustains, while each statement stays as small as today.
• Report the backlog. Export the number of rescue-eligible claims (or the age of the oldest) per ring slot, so operators can see a backlog forming instead of inferring it from a stalled ring.
• Optionally, a dead-runtime fast path. When a runtime's snapshot has gone stale (it is gone, not just slow), its claims can be rescued without waiting for each deadline or heartbeat cutoff. That needs claims to record their owning runtime, which they may not today.

Risks to design for

• False rescues scale with the batch. Rescue can't tell a dead worker from a slow heartbeat (database stall, GC pause, network partition). A bigger batch rescues more live jobs in one go, which means more duplicate executions. Growth should be gated on evidence that the owner is gone (runtime snapshot staleness), or kept to deadline rescue, where the deadline has definitely passed.
• Re-enqueue surge. Rescued jobs go straight back to ready. Draining 50k in seconds makes them all available at once, which spikes claim load and contends with producers on the enqueue lane locks. Sub-batches should be lane-ordered, as enqueue batches now are (fix(insert): take enqueue lane locks in one order across batch inserts (0.6) #626), and the budget may need a row cap as well as a time cap.
• Lock and transaction length. Each sub-batch must stay a short transaction. One long statement would hold the slot cursor's row lock and many claims' completion advisory locks, which delays completions and holds back the xmin horizon for vacuum.
• WAL, replicas and CDC. A fast drain writes closures, deltas and ready rows in a burst, which shows up as replica or CDC lag.
• The PostgreSQL 18 constraint. Keep the two-statement shape from fix(queue-storage): avoid PostgreSQL 18's freed pruning state after a claim recheck (0.6) #604 and fix(queue-storage): take receipt rescue locks before deciding (0.6) #614: lock first, then decide in a statement that row-locks only the slot's own claim partition. More statements per pass means more rechecks, not riskier ones, as long as that shape holds.
• Cooldown interaction. A time-budgeted pass should finish within its branch interval, or the overrun hysteresis will back it off. Tie the budget to the interval.
• Poison and DLQ load. With rescue bounded by max_attempts (Backport to 0.6: bound poison jobs, rescues and RetryAfter by max_attempts #616), a large drain can also dead-letter in bulk. That's fine, but it should be visible in metrics and not surprise anyone watching DLQ alerts.

Done when

A test that abandons 50k claims across several slots (killing a client mid-run) sees rescue clear them and the claim ring resume rotation within a configurable bound (say minutes), without duplicate execution of jobs whose owner is still alive. A metric shows the rescue backlog while it drains.

Resolution — 2026-11-11

The bounded backlog-drain work shipped in v0.6.16 through #647, with acceptance-test hardening in #706. Rescue uses separately committed bounded batches, configurable pass time and row thresholds, indexed cursor progress, and accelerated follow-up passes. Accelerated stale-heartbeat rescue also requires settled runtime-snapshot evidence and an older heartbeat cutoff; deadline rescue still requires expired deadlines.

The released 50,000-claim acceptance test abruptly stops a client runtime, spreads abandoned claims across several slots, requires bounded drainage and resumed claim-ring rotation, and checks exactly-once recovery without rescues or duplicate execution of healthy live jobs. The release tag contains both the implementation and that coverage; this reconciliation does not claim a fresh test run.

awa.maintenance.rescue.pass_size and awa.maintenance.rescue.oldest_attempt_age expose drain progress. The age is the oldest attempt rescued in the latest pass, by rescue kind; it is not an exact per-slot pending-backlog gauge. Runtime snapshots are a heuristic because claims do not record their owning runtime identity: this is not an unconditional guarantee against rescuing a still-running process whose heartbeats expire. Pass limits bound work, not absolute transaction latency or universal drain time.

Closing the implemented backlog-drain scope as released. The remaining ambiguous receipt-claim cursor starvation is tracked separately in #693, which remains open. Exact per-slot backlog telemetry and a runtime-owner fast path were not added by this work.