Repository navigation
fix(queue-storage): report Blocked when any prune statement times out on a lock (0.6) - #641
Merged
Merged
Conversation
…(0.6) Ring prune and terminal-delta rollup take their exclusive locks under a 10 ms lock_timeout that could only be changed through a hidden QueueStorage method. When short readers of the ring partitions hold a conflicting lock at that instant, prune reports blocked every tick and the rings stop recycling. Add ClientBuilder::prune_lock_timeout and the Python queue_storage_prune_lock_timeout_ms start() argument. The default stays 10 ms and zero is rejected, as with_prune_lock_timeout does. Make QueueStorage::with_prune_lock_timeout public and add set_prune_lock_timeout / prune_lock_timeout. awa.maintenance.prune.attempts already counts pruned, noop, blocked and skipped_active per ring; add backed_off (prune skipped by the post-skipped_active backoff) and error (prune failed with a database error) so a ring that has stopped recycling is visible for every cause.
A lane's enqueue and claim sequences were created by whichever role first touched the lane, through SECURITY INVOKER helpers, so only that role and the roles its default privileges named could use them. Another runtime role then failed with "permission denied for sequence" on enqueue, claim or promotion. Each queue-storage schema now has two SECURITY DEFINER functions owned by the schema owner, provision_lane and provision_lane_sequences. The lane helpers create a missing sequence through them, and each new sequence gets the privileges of the head table it backs. ensure_lane_inserts calls provision_lane when it exists and is executable, inside a savepoint, and otherwise writes the lane rows as before. The definer_lane_provisioning schema patch refreshes the helpers and repairs existing lane sequences one at a time, reassigning and granting only what the applying role may and reporting the rest. awa.lane_sequence_problems and the new `awa storage lane-sequences` command list what remains with the statements to run; awa.repair_lane_sequences and `--repair` rerun the repair.
…vision_lane Queue storage takes any queue name, including an empty one, and any smallint priority, so the provisioning functions only reject a null argument or a shard outside 0-63. The released-binary rehearsal now expects a 0.6.9 producer without CREATE to create a new lane, owned by the schema owner.
…and hand busy lanes to a waiter (0.6) Backport of #636. Promotion re-enqueues due scheduled and retryable rows into their lane, which takes the lane's reservation lock that producers hold until they commit. One promotion batch was a single transaction across every queue it touched, and the maintenance leader runs its branches one after another, so a lane kept busy by producers delayed due jobs of unrelated queues in the same batch and started rescue, rotation and prune late. QueueStorage::promote_due_lanes selects a batch without locking it and promotes each (queue, priority) lane's share in its own transaction, under an optional lock_timeout; a lane that hits it rolls back and is reported as contended. promote_due_lane promotes the oldest due rows of one lane. promote_due keeps its signature and promotes lane by lane with no timeout. Within a lane, promoted rows take lane sequence numbers in due order. The leader's promoter runs each tick under a 50 ms lane lock timeout, skips contended lanes for the rest of the tick, and hands each to a dedicated waiter task (at most four at once) that promotes the lane under a 30 s lock timeout. PostgreSQL grants a lock to its waiters in arrival order, so the waiter is served after the producers queued ahead of it however busy the lane stays. awa.maintenance.promote.lane_contention counts handed_off, waiter_limit and waiter_timed_out by state and queue. The overrunning-promotion test slowed the queue-storage DELETE, which no longer runs when nothing is due; it now slows the canonical-plane promotion DELETE that every body runs.
… on a lock (0.6) Prune and terminal-count rollup set a transaction-local lock_timeout before taking ACCESS EXCLUSIVE on a slot's partitions, but only the LOCK TABLE and TRUNCATE steps mapped a timeout to Blocked. The timeout also applies to the statements after them: the post-lock liveness checks over the lease and claim partitions, the failed-row carry into the current slot's done_entries and terminal-count deltas, the rollup fold and the live-counter cleanup. A wait there surfaced as a 55P03 database error, was logged as a prune error and skipped the prune backoff. Run everything after the timeout is set as one block and settle it in one place: commit on success, and on a lock timeout from any statement roll back and report Blocked. Lease and claim prune use the same helper so the rule holds for every ring. The new tests hold a SHARE lock on each relation the queue prune writes after its child locks, and on queue_terminal_live_counts for the rollup; both fail with 55P03 before this change. The retry-failed outcome test now retries a Blocked prune as the maintenance loop does, since a background session can legitimately block it.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Once a handler returned, the executor removed the attempt from the heartbeat set and finalized it in a detached task that Client::shutdown did not wait for. When the last handler returned as the drain ended, shutdown cancelled the completion batcher before that finalize reached it, and the direct-finalize fallback raced the process exit. The attempt stayed open with no heartbeat, so stale-heartbeat rescue ran the finished job again after heartbeat_staleness. Track finalizers in a TaskTracker and wait for them, within the drain timeout, before stopping the completion batcher, heartbeat and maintenance services. This is the shutdown part of #521 on main.
…ut or statement_timeout
…ound lane waiters by pool size The batch selection was a plain SELECT ... LIMIT and only the per-lane DELETE skipped locked rows. When the oldest due rows were locked by another transaction, such as a lane waiter left running by a former leader, every batch of a tick reselected them and promoted nothing until the lock went away. The selection now takes FOR UPDATE OF deferred SKIP LOCKED. It runs as its own statement, so the row locks end with it, and deferred_jobs is not partitioned, so no run-time pruning meets them. Each lane waiter holds a pool connection while it waits. The leader now runs one waiter per eight pool connections, at least one and at most four, and the configuration docs say so.
…ocked slot (0.6) The remaining sealed slots would most likely wait out the same lock timeout, and with a raised prune timeout that holds the maintenance leader for one timeout per slot. The rollup test now seals two slots with pending deltas and expects one blocked slot, then both rolled once the lock is released.
…ded lock wait The definer_lane_provisioning patch reassigned lane sequences inside the patch transaction. ALTER SEQUENCE ... OWNER takes the sequence's ACCESS EXCLUSIVE lock until commit, with no lock_timeout, so an open enqueue on one lane could stall the patch and every repaired lane waited for it to commit. The patch now only refreshes the helpers. After it commits, awa migrate repairs each lane sequence with its own statement through awa.repair_lane_sequence(schema, sequence), which has lock_timeout = 300ms; on 55P03 the sequence is left unchanged and reported as busy. The results are in MigrationReport::lane_sequences. `awa storage lane-sequences --repair` uses the same path, and the set-returning awa.repair_lane_sequences(schema) is gone. Reassigning needs the current owner's privileges, which the schema owner lacks for sequences the migration role or an app role created. A migration role connected with role set to the owner therefore could not reassign its own sequences. The repair now retries such a sequence as the login role, lane_sequence_problems reports repair_roles (the login roles that can repair each sequence), and leftovers are reported with the next step to take. The docs state which role to run the repair as.
hardbyte
added
the
full-ci
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Queue, lease and claim prune, and the terminal-count rollup, set a transaction-local
lock_timeout(10 ms by default) before they takeACCESS EXCLUSIVEon a slot's partitions. Only theLOCK TABLEand the finalTRUNCATEturned a timeout intoPruneOutcome::Blocked. The timeout applies to every later statement in the transaction as well, and in the queue prune several of them touch relations the prune has not locked:leasesandlease_claims*partitions;done_entrieschild andqueue_terminal_count_deltaschild;queue_terminal_rollupsand deletes the slot'squeue_terminal_live_countsrows.The rollup's post-lock checks and its
queue_terminal_live_countsfold have the same exposure. When one of these statements waited past the timeout, prune returnedErr(Database(55P03 lock_not_available)): the maintenance loop logged it as a prune failure and did not apply the prune backoff, and the rollup pass stopped at that slot. Anything that holds a conflicting lock on one of those relations for longer than the timeout triggers it. Examples are autovacuum's heap truncation (ACCESS EXCLUSIVE, released only when it notices a waiter, every 20 ms), a hosted Postgres's own background workers, and operator DDL.This is the CI failure seen once in "Rust prune tests (AlloyDB Omni)" on #630:
test_queue_storage_retry_failed_outcome_surfaces_pruned_rowsgot55P03from the carry'sINSERT INTO ... queue_terminal_count_deltas. The server log shows the statement that was cancelled.Change
Everything after
set_prune_lock_timeout_txnow runs as one block, and a sharedsettle_prune_txcommits it on success. On a lock timeout from any statement, it rolls the transaction back and returns the caller'sBlockedvalue. The per-statement handling of theLOCK TABLEandTRUNCATEerrors goes away. Lease and claim prune use the same helper. None of their post-lock statements can wait today, but this way the rule holds for every ring.A rollup pass now stops at its first blocked slot instead of trying each remaining slot. Those slots would most likely wait out the same timeout, and with a raised prune timeout (#629/#630) the leader would otherwise wait once per slot. The rollup test seals two slots and expects one blocked slot.
The whole attempt is one transaction, so a
Blockedprune leaves nothing behind. The carried rows, deltas, rollup increments and counter deletes all roll back, and the slot is pruned intact on a later tick. A test checks this.Interaction with #630
#630 adds prune outcome metrics. With this change, the late timeouts count as
blocked(and enter the prune backoff) instead oferror. After both PRs land,awa.maintenance.prune.errorsshould only show genuine failures.#630 also makes the timeout configurable. On 0.6 the queue prune locks its children in one statement, ready rows first, while claims read segments before ready rows. When a claimer holds a segment read, that ordering forms a short prune/claim lock cycle. At the default 10 ms, prune's
lock_timeoutfires long beforedeadlock_timeout(1 s), so prune reportsBlockedand the claimer waits at most 10 ms. If the timeout is configured abovedeadlock_timeout, the deadlock detector runs first. It aborts prune with40P01, which is still reported as an error, and the claimer waits up todeadlock_timeout.mainremoved the cycle in #519 by locking the segment child first and taking the restNOWAIT. #630 could cap the setting belowdeadlock_timeoutor document the limit.Tests
test_queue_storage_prune_lock_timeout_after_child_locks_reports_blockedtakes aSHARElock in turn on the currentdone_entrieschild, the currentqueue_terminal_count_deltaschild,queue_terminal_rollupsandqueue_terminal_live_counts, while the prune carries a failed row. Each attempt must reportBlocked { slot: 0 }with no rows carried or folded and the terminal-count invariant intact. Once the locks are released the prune must succeed.test_queue_terminal_count_delta_rollup_lock_timeout_after_child_lock_reports_blockeddoes the same for the rollup's live-counter fold, across two sealed slots, and checks that the pass stops after the first.55P03on PostgreSQL 17 and 18.test_queue_storage_retry_failed_outcome_surfaces_pruned_rowsnow retries aBlockedprune for up to 5 s, as the maintenance loop does. A background session can legitimately block it.mainalready does this in the same test.Validation
cargo fmt --all --check,SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings.queue_storage_runtime_test, all 149 tests, on privatepostgres:17-alpineandpostgres:18-alpine. One PG18 run timed outtest_queue_storage_runtime_drains_scheduled_backlog_due_at_once(a 10k-job throughput bound) while both suites shared a loaded host; it passed when run alone.Same fix on
main: #642.Independent review and validation
Includes the small #651 point-read fix. After a priority or queue move crossed a ready-ring wrap, retained tombstoned source rows could make
get_jobreturn old fields even after the batch succeeded. The available-candidate query now excludes full-identity tombstones (a 13-line SQL change), with priority and queue ring-wrap regressions. Both regressions fail before the fix and pass after it on PostgreSQL 17 and 18; existing aged-read and batch cases also pass. Required formatting, Clippy across all targets/features, and workspace build checks pass for the prepared source.Full CI passed all 29 jobs for
5c08b7df79820c0226c490ffad8bd236b12880ab.