You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(queue-storage): report Blocked when any prune statement times out on a lock - #642
Queue, lease and claim prune, and the terminal-count rollup, set a transaction-local lock_timeout (10 ms by default) before they take ACCESS EXCLUSIVE on a slot's partitions. Only the LOCK TABLE statements and the final TRUNCATE turned a timeout into PruneOutcome::Blocked. The timeout applies to every later statement in the transaction as well, and in the queue prune several of them touch relations the prune has not locked:
the post-lock liveness checks read the leases and lease_claims* partitions;
the failed-row carry inserts into the current slot's done_entries child and queue_terminal_count_deltas child;
the fold appends queue_terminal_rollup_deltas and deletes the slot's queue_terminal_live_counts rows.
The rollup's post-lock checks and its queue_terminal_live_counts fold have the same exposure. When one of these statements waited past the timeout, prune returned Err(Database(55P03 lock_not_available)): the maintenance loop logged it as a prune failure and did not apply the prune backoff, and the rollup pass stopped at that slot. Anything that holds a conflicting lock on one of those relations for longer than the timeout triggers it. Examples are autovacuum's heap truncation (ACCESS EXCLUSIVE, released only when it notices a waiter, every 20 ms), a hosted Postgres's own background workers, and operator DDL.
It was seen in CI on the 0.6 line. On AlloyDB Omni, test_queue_storage_retry_failed_outcome_surfaces_pruned_rows got 55P03 from the carry's INSERT INTO ... queue_terminal_count_deltas. On main, that test already retries a Blocked prune, but this error came back as Err and failed the test the same way.
Change
Everything after set_prune_lock_timeout_tx now runs as one block, and a shared settle_prune_tx commits it on success. On a lock timeout from any statement, it rolls the transaction back and returns the caller's Blocked value. The per-statement handling of the lock and TRUNCATE errors goes away. Lease and claim prune use the same helper. None of their post-lock statements can wait today, but this way the rule holds for every ring. PruneDurations::commit is still measured around the commit.
A rollup pass now stops at its first blocked slot instead of trying each remaining slot. Those slots would most likely wait out the same timeout, and with a raised prune timeout (#629/#630) the leader would otherwise wait once per slot. The rollup test seals two slots and expects one blocked slot.
The whole attempt is one transaction, so a Blocked prune leaves nothing behind. The carried rows, deltas, rollup deltas and counter deletes all roll back, and the slot is pruned intact on a later tick. A test checks this.
#629 adds prune outcome metrics. With this change, the late timeouts count as blocked (and enter the prune backoff) instead of error.
Tests
test_queue_storage_prune_lock_timeout_after_child_locks_reports_blocked takes a SHARE lock in turn on the current done_entries child, the current queue_terminal_count_deltas child, queue_terminal_rollup_deltas and queue_terminal_live_counts, while the prune carries a failed row. Each attempt must report Blocked { slot: 0 } with no rows carried or folded and the terminal-count invariant intact. Once the locks are released the prune must succeed.
test_queue_terminal_count_delta_rollup_lock_timeout_after_child_lock_reports_blocked does the same for the rollup's live-counter fold, across two sealed slots, and checks that the pass stops after the first.
Both tests fail before the change with 55P03 on PostgreSQL 17 and 18.
… on a lock
Prune and terminal-count rollup set a transaction-local lock_timeout before
taking ACCESS EXCLUSIVE on a slot's partitions, but only the LOCK TABLE and
TRUNCATE steps mapped a timeout to Blocked. The timeout also applies to the
statements after them: the post-lock liveness checks over the lease and claim
partitions, the failed-row carry into the current slot's done_entries and
terminal-count deltas, the rollup delta append and the live-counter cleanup.
A wait there surfaced as a 55P03 database error, was logged as a prune error
and skipped the prune backoff.
Run everything after the timeout is set as one block and settle it in one
place: commit on success, and on a lock timeout from any statement roll back
and report Blocked. Lease and claim prune use the same helper so the rule
holds for every ring. The commit duration in PruneDurations is still measured
around the commit.
The new tests hold a SHARE lock on each relation the queue prune writes after
its child locks, and on queue_terminal_live_counts for the rollup; both fail
with 55P03 before this change.
No actionable comments were generated in the recent review.
Walkthrough
Queue, lease, and claim pruning and terminal-delta rollup now roll back transactions when lock contention occurs and report blocked outcomes. PostgreSQL regression tests check rollback and successful retries after locks are released.
Changes: Maintenance transaction lock handling
Layer / File(s)
Summary
Transaction settlement and terminal rollup awa-model/src/queue_storage.rs, awa/tests/queue_storage_runtime_test.rs, CHANGELOG.md
A shared transaction settlement path rolls back errors and maps lock contention to a blocked outcome. Terminal-delta rollup uses this path. A regression test checks the blocked-slot result and unchanged counters until the lock is released.
Queue, lease, and claim pruning use shared transaction settlement. A queue-prune regression test checks that lock contention leaves rows and terminal counts unchanged, and that pruning succeeds after locks are released.
hardbyte/awa#629: This PR also changes prune lock-timeout handling and outcomes.
hardbyte/awa#534: This PR changes terminal-count folding, which this PR updates to handle lock timeouts.
Suggested labels:full-ci
Suggested reviewers:claude
Merge Risk: Minimal · up to 0c4ba
Lock timeouts now produce retryable blocked outcomes across maintenance transactions without an identified behavioral regression. Mergeable subject to normal checks.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title clearly describes the main change: reporting Blocked for lock timeouts during pruning.
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
A rabbit checks the queue at dawn, A lock holds fast; the work rolls back. The blocked result marks where to wait, Then freed locks let the rows proceed. The counters settle, safe and neat.
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: fdf1eaa5-4aaf-4925-bef7-a2ad22db9f28
Commits
Reviewing files that changed from the base of the PR and between 695acaa and 0c4ba0b.
Files selected for processing: 3
CHANGELOG.md
awa-model/src/queue_storage.rs
awa/tests/queue_storage_runtime_test.rs
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
…ocked slot
The remaining sealed slots would most likely wait out the same lock timeout,
and with a raised prune timeout that holds the maintenance leader for one
timeout per slot. The rollup test now seals two slots with pending deltas and
expects one blocked slot, then both rolled once the lock is released.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
full-ciRun the full CI matrix (Python build+test, E2E) on this PR
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
coderabbitai
Bot
added
the
full-ci
Problem
Queue, lease and claim prune, and the terminal-count rollup, set a transaction-local
lock_timeout(10 ms by default) before they takeACCESS EXCLUSIVEon a slot's partitions. Only theLOCK TABLEstatements and the finalTRUNCATEturned a timeout intoPruneOutcome::Blocked. The timeout applies to every later statement in the transaction as well, and in the queue prune several of them touch relations the prune has not locked:leasesandlease_claims*partitions;done_entrieschild andqueue_terminal_count_deltaschild;queue_terminal_rollup_deltasand deletes the slot'squeue_terminal_live_countsrows.The rollup's post-lock checks and its
queue_terminal_live_countsfold have the same exposure. When one of these statements waited past the timeout, prune returnedErr(Database(55P03 lock_not_available)): the maintenance loop logged it as a prune failure and did not apply the prune backoff, and the rollup pass stopped at that slot. Anything that holds a conflicting lock on one of those relations for longer than the timeout triggers it. Examples are autovacuum's heap truncation (ACCESS EXCLUSIVE, released only when it notices a waiter, every 20 ms), a hosted Postgres's own background workers, and operator DDL.It was seen in CI on the 0.6 line. On AlloyDB Omni,
test_queue_storage_retry_failed_outcome_surfaces_pruned_rowsgot55P03from the carry'sINSERT INTO ... queue_terminal_count_deltas. Onmain, that test already retries aBlockedprune, but this error came back asErrand failed the test the same way.Change
Everything after
set_prune_lock_timeout_txnow runs as one block, and a sharedsettle_prune_txcommits it on success. On a lock timeout from any statement, it rolls the transaction back and returns the caller'sBlockedvalue. The per-statement handling of the lock andTRUNCATEerrors goes away. Lease and claim prune use the same helper. None of their post-lock statements can wait today, but this way the rule holds for every ring.PruneDurations::commitis still measured around the commit.A rollup pass now stops at its first blocked slot instead of trying each remaining slot. Those slots would most likely wait out the same timeout, and with a raised prune timeout (#629/#630) the leader would otherwise wait once per slot. The rollup test seals two slots and expects one blocked slot.
The whole attempt is one transaction, so a
Blockedprune leaves nothing behind. The carried rows, deltas, rollup deltas and counter deletes all roll back, and the slot is pruned intact on a later tick. A test checks this.#629 adds prune outcome metrics. With this change, the late timeouts count as
blocked(and enter the prune backoff) instead oferror.Tests
test_queue_storage_prune_lock_timeout_after_child_locks_reports_blockedtakes aSHARElock in turn on the currentdone_entrieschild, the currentqueue_terminal_count_deltaschild,queue_terminal_rollup_deltasandqueue_terminal_live_counts, while the prune carries a failed row. Each attempt must reportBlocked { slot: 0 }with no rows carried or folded and the terminal-count invariant intact. Once the locks are released the prune must succeed.test_queue_terminal_count_delta_rollup_lock_timeout_after_child_lock_reports_blockeddoes the same for the rollup's live-counter fold, across two sealed slots, and checks that the pass stops after the first.55P03on PostgreSQL 17 and 18.Validation
cargo fmt --all --check,SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings.queue_storage_runtime_test, all 214 tests, on privatepostgres:17-alpineandpostgres:18-alpine.0.6 backport: #641.