Reduce recurring queue and claim prune lock-acquisition failures (0.6) #703

Under a modest mixed workload on PostgreSQL 18, queue and claim prune repeatedly time out acquiring their partition locks. Rolling back and backing off keeps this correct, but every queued ACCESS EXCLUSIVE request can briefly stall other traffic on those partitions, and failed attempts defer reclamation. The goal is less interference while the rings keep recycling, not just a lower error count.

Observed behavior

Two sequential 15-minute runs (then drain and shutdown) on 9 November 2026 against one PostgreSQL 18.3 instance limited to 1.2 CPU and 3 GiB and shared with unrelated workloads. The client runtimes used released 0.6.14 and 0.6.15 respectively; the maintenance leader ran 0.6.15 in both runs, so this is not a comparison of two leader versions.

Workload: two client runtimes, 96 total worker concurrency, at most 22 pooled connections, three queues (fast completions, slow heartbeating jobs, flaky jobs with retries and automatic DLQ routing), sampled cancellations and a live drain. Nominal offered rate at most 60 jobs/s; acknowledged 47.2 and 46.0 jobs/s. The database was not reset between runs.

Server-side 55P03 errors from the maintenance leader, classified by the failing statement:

Over windows of 920 and 922 seconds that is 17.4 and 15.5 prune lock-acquisition failures per minute. None occurred at COMMIT, TRUNCATE or a later prune statement.

Queue prune locks the eight children of one slot (ready_entries, ready_claim_attempt_batches, done_entries, ready_tombstones, ready_segments, receipt_completion_batches, receipt_completion_tombstones, queue_terminal_count_deltas); claim prune locks four (lease_claims, lease_claim_batches, lease_claim_closures, lease_claim_closure_batches).

Both runs finished with zero enqueue errors, duplicate attempt starts, server restarts or OOM kills, and no available or running test jobs at the end. Exact completion accounting was inconclusive because completed history rotated out before the final check, so this is not a loss-free proof. Blocker identities, actual lock-wait durations, per-ring prune success counts and maintenance branch overruns were not captured, so the effect on application latency is unmeasured.

Existing behavior and related work

• 0.6.15 prune settlement rolls a timed-out attempt back and returns Blocked; PostgreSQL still logs an ERROR for the timed-out statement. These records alone don't indicate an escaping maintenance error or a regression in fix(queue-storage): report Blocked when any prune statement times out on a lock (0.6) #641.
• Existing backoff skips up to 32 ticks after repeated Blocked or SkippedActive outcomes, resets after a successful prune, and is scoped to a leadership term. Empty-slot guards and pre-lock liveness checks already avoid some doomed requests.
• feat(worker): configurable queue-storage prune lock timeout and prune outcome metrics (0.6) #630 added the configurable prune lock timeout and outcome metrics. The default is 10 ms per lock acquisition, not per prune; queue prune acquires up to eight.
• main changed rotation cadence in Reduce ring rotation catalog churn with observed fill cadence #519 and perf(queue-storage): seal ring slots on a row floor or an age ceiling, not on age alone #534. Evaluate them for 0.6; the locking protocol differs, so not a blind cherry-pick.
• maintenance: split rescue + promote into a separate tokio task (v0.7) #303 tracks separating rescue/promotion scheduling from other maintenance, and Long-jobs WAL: heartbeat is under 7% of steady-state WAL; rotation, per-job cost and maintenance tick locks dominate #525 tracks maintenance and rotation WAL cost. Neither covers measuring and reducing this 0.6 prune contention.
• fix(0.6): add durable operator prune controls for 0.6.16 #675 (merged for 0.6.16) adds a durable maintenance.prune halt: an operator escape hatch that leaves cadence and contention policy unchanged.
• Related but separate: a ring can also stop recycling because prune correctly skips a slot (SkippedActive, e.g. queue_unclosed_claim_refs when a killed runtime leaves a large backlog of abandoned claims without closures). That is a rescue throughput problem, addressed by fix(queue-storage): drain a large abandoned-claim backlog in bounded time (0.6) #647, not lock contention.

The promotion failures are a separate population. 0.6.15 promotion already uses a 50 ms fast-path lock budget and dedicated lane waiters; read awa_maintenance_promote_lane_contention_total by outcome before deciding whether those timeouts point to a remaining scheduling problem.

Proposed work

1. Attribute the waits before changing lock policy. Correlate awa_maintenance_prune_attempts_total by ring and outcome, successful prunes, ring generation progress, slot sizes and branch overruns with the server errors. Capture blockers during a bounded diagnostic window (one-second sampling can miss a 10 ms wait), and separate initial partition-lock failures from later statements and COMMIT. Hypothesis, unproven: reads of the parent partitioned tables (admin and cancellation queries, background polling) block otherwise reclaimable children.
2. Measure cadence first. Compare the default one-second rotation with a longer queue interval, then the claim interval in a separate experiment, keeping the 10 ms lock timeout and everything else fixed. If that reduces strong-lock requests and application latency without hurting reclamation, implement a 0.6 cadence policy based on a minimum useful slot fill and a maximum age. Longer intervals put more rows in each slot and coarsen retained history; report those costs. At ~46 new jobs/s, a five-second queue slot receives about 230 initial ready rows instead of 46, before retries or recycling stalls.
3. Shorten the confirmed blocking reads. Restrict them to the child partitions and transaction scope they need where correctness allows, keeping multi-statement read consistency and prune's liveness checks. Don't remove locks or add fail-fast acquisition everywhere without testing continuous-reader starvation. If cadence and reader changes still leave repeated ineffective attempts, evaluate a contention-based cooldown against the existing backoff rather than adding a second retry mechanism.

Raising prune_lock_timeout can help a ring that can't recycle, but each exclusive request then blocks new accesses for longer; measure that tradeoff separately. See PostgreSQL's per-acquisition lock_timeout and table-lock conflicts.