Repository navigation
fix(queue-storage): drain a large abandoned-claim backlog in bounded time (0.6) - #647
Merged
Merged
Conversation
…time (0.6) Backport of the main change. Rescue passes keep taking 500-attempt batches, each in its own transaction, within a time budget and a row cap, and run again a second later while a backlog remains. Stale-heartbeat passes take more than one batch only once runtime snapshots show the claims' owners gone, and then only claims heartbeated before the most recent gone runtime stopped reporting. Batches visit claim slots oldest first with a periodic others-first batch, walk only the claims near each slot's cursor, and settle rescued attempts in one statement per outcome. Refs: #627
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc168e1df8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The executor removed an attempt from the heartbeat set and released its worker slot as soon as the handler returned, then queued its finalize on the completion batcher. When finalization was slower than claiming, the dispatcher kept claiming and the queue of finished, unheartbeated attempts grew without bound. Those that waited past heartbeat_staleness were rescued and run again. Admit at most the completion pipeline's depth (2 x batch size x shards, and at least one per worker) of attempts to finalization. Past that, a finished attempt keeps its worker slot and heartbeat until there is room, which stops the dispatcher claiming. This is the bound main already has.
The OTel-to-Prometheus name check reads each name constant from a single line.
…terministically The doomed runtime now claims the backlog in chunks, and the test checks the open claims span several slots directly rather than inferring it from ring rotations, which a fast claim phase could finish within one slot.
Closed or freshly heartbeated history can exhaust a scan without rescuing anything. Report scan progress independently of rescued counts, preserve complete sweep boundaries across early passes, and revisit finished slots at the ordinary cadence. Keep first-batch heartbeat rescue on its original cadence unless settled owner evidence allows acceleration. Remove the retry pacing horizon because normal backoffs can cluster due times; document the actual rescue write bound. Validation: cargo fmt --all; SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings; SQLX_OFFLINE=true cargo build --workspace; seven maintenance rescue tests on PostgreSQL 17/18; sparse-history and 501-attempt rescue regressions on PostgreSQL 17. Broader rescue tests continue on both engines.
hardbyte
added
the
full-ci
ChatGPT Codex Connector
Address compact claims by their batch ID and item index, validating the job, attempt and receipt before closing. Avoid expanding every retained compact batch for each metadata or retry completion flush, and avoid locking immutable shared batch rows. Preserve per-attempt advisory serialization, row-local prelocking and the fresh deciding snapshot. A 50,000-claim recovery rehearsal with heartbeat protection timed out waiting 120 seconds for retries on PostgreSQL 17 and 18 before this change. With keyed completion, the same full rehearsal passes in 116.35s and 123.30s total. A new regression rejects a wrong item index, completes while the immutable batch row is locked elsewhere, and checks retained tags and metadata. Validation: cargo fmt --all; SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings; SQLX_OFFLINE=true cargo build --workspace; eight focused metadata, stale-completion, cancellation, idempotence and finalization-bound tests plus the shutdown-waits test on PostgreSQL 17 and 18.
Preserve the approved read, promotion, pruning, lane ownership and graceful shutdown fixes alongside bounded rescue and heartbeat-until-finalized behavior. Resolve shared exports, serde scan metadata, and the Python prune/rescue keyword signatures together. Document the actual soft rescue cap and preserve all release notes. Combined source passes workspace formatting, all-target/all-feature Clippy with warnings denied, the workspace build, and strict bulk acceptance test compilation. Full combined PostgreSQL and Python binding validation follows.
Wait for committed completed terminal facts, require every recovered job to run exactly once on attempt 2 with its rescue recorded, and reject additional rescues of either live or recovered attempts. Keep the existing timeouts. The strict test compiles and all mandatory Rust workspace prechecks pass; final combined PostgreSQL acceptance is pending.
Use the existing follow-up scan window from the first heartbeat batch when confirmed-owner evidence and both acceleration controls are enabled. Keep ordinary, unconfirmed and disabled paths on the wider initial scan. Exercise sparse closed-prefix progress, exact-once disposition, fresh protection and disabled/unconfirmed wide coverage through real maintenance passes. SQL, cutoffs, lock ordering and timing bounds are unchanged.
This was referenced Nov 7, 2026
Merged
Brings in 0.6.15 (bounded finalization, rescue reselection past ambiguous attempts, the claim-batch job index and batch insert), the prune controls and the load_job retry. Each rescue batch now runs its selection inside a savepoint attempt that is retried when hydration finds busy or ambiguous retained bodies, as the release rescue does. The receipt accumulator, its cursor progress and idle slots are created inside each attempt, so a rolled-back attempt contributes nothing. The exclusion and reselection sets reach the cursor scans of both receipt rescue kinds, and the lease-side statements bind their row limit explicitly. The leader loop takes the release's observer and admission structure. Rescue branches keep their tick origin and backlog re-arm, still pass through admission, and are not gated by a prune halt. Finalization keeps the release's heartbeat guard and adds the finalizing mark that stops progress flushes once the handler has returned. The duplicate metadata-completion test is dropped and the release's stalled-finalization tests replace this branch's copy, keeping the heartbeat-through-stalled-finalize test.
Rescued retries keep their ordinary backoff, and nothing sets the floor any more, so the option and its plumbing through settlement go. The callback batch keeps its options parameter for symmetry with the other kinds; none of them apply to callback waits.
A deadline pass that may take several batches now starts with the 2,000-claim window that follow-up batches and confirmed stale-heartbeat passes already use, instead of 10,000. Every claim in the window pays its closure checks, so on a dense backlog the wide first window spent most of a batch evaluating claims it could not take: a 10,000-claim deadline batch of row-local claims took about 680 ms of statement time against about 220 ms for a 2,000-claim one. The pass keeps taking batches while windows come back full, so long closed prefixes are still walked; a pass limited to one batch keeps the wide window. Against 200,000 abandoned row-local claims with default settings the drain went from about 820 to about 1,020 claims a second, and the slowest deadline statement from 605 ms to 221 ms.
A killed runtime that held claims with deadlines leaves row-local lease_claims rows, not compact batches. The new test builds that shape directly: a backlog over two claim slots that all reference one queue slot, claimed, heartbeated and past their deadlines hours ago, with the dead runtime's snapshot gone. Until the claims close, queue prune skips that queue slot and claim prune skips both claim slots. With default rescue intervals the test requires every claim to close, both rings to rotate past the pinned slots and every rescued job to complete on its second attempt. It runs 20,000 claims by default. Environment variables scale the backlog, override the pass controls, or stop after a fixed time and report the drain rate, which is how the larger measurements were taken.
The scheduler registration check counted only arms written as `_ = timer.tick()`, so the three rescue arms, which bind the tick origin, dropped out of the count after the merge with the prune controls.
The heartbeat and deadline receipt scans compared each claim with the slot's cursor through a cross join on the locked cursor row, so the comparison ran as a join filter. Every scan walked the cursor index from its start and discarded every claim the sweep had already passed: with 790,000 claims in a slot, each batch read and dropped them all twice, once per statement. A drain slowed as it progressed, from about 1,000 claims a second to about 440, and 1.6 million abandoned row-local claims took 48 minutes. The cursor values now come from scalar subqueries on the same locked row, which the planner evaluates once and uses as index conditions, so a scan seeks straight to the cursor. On a slot with 294,000 claims behind the cursor, the deadline candidate scan dropped from 190 ms to 33 ms. The cursor row is still locked by the same statements and an empty cursor row still yields no candidates.
Queue prune removes completed rows once the ring wraps past them, which a drain of a million claims does long before every recovery run finishes, so the completed-row count never reached the backlog. The handler now counts second-attempt runs and any other attempt, and the test requires exactly one second-attempt run per abandoned job and no others.
The heartbeat and deadline cursor scans are estimated far above their cost, enough to cross PostgreSQL's JIT thresholds. With a 10,000-claim window the stale-heartbeat lock statement executed in about 130 ms but spent 1.6 s compiling, and its closing statement about 3.5 s, so each batch outside owner-confirmed bulk rescue held its cursor row and attempt locks for 4 to 10 seconds against a 200,000 to 1.6 million claim backlog. Each rescue batch transaction now sets jit off locally, as queue drains already do.
A long-lived backlog has settled statistics. Without them, rescue's ready-body lookups of the freshly built backlog chose the primary key over the job index and took seconds per batch until autovacuum analyzed the partition.
After 1.6 million claims drain, promotion admits their retries for a while and the rings took about two minutes to move past the pinned slots, beyond the fixed two-minute wait.
The unreleased changelog gains the drain, its controls and metrics, and the heartbeat fixes this branch carries; the entries it held for fixes since released in 0.6.14 and 0.6.15 are gone with the merge. The configuration guide gives measured rates and costs for a row-local backlog, the troubleshooting entry names the queue-ring symptom and the budget cap, and the 0.6.15 to 0.6.16 upgrade guide has a self-contained section on what changes for a fleet with a backlog.
…unded-rescue-0.6 # Conflicts: # CHANGELOG.md
…unded-rescue-0.6 # Conflicts: # CHANGELOG.md
This was referenced Nov 9, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #627.
Summary
A killed runtime can leave hundreds of thousands of claims open with no closure. Queue and claim prune can't reclaim a slot while claims pointing at it are unclosed, so both rings stop rotating until rescue closes them. On 0.6.15 rescue looks at one 10k-claim window per slot per pass, rescues at most 500 per kind per pass, and its heartbeat statement materialises the whole claim partition before sorting, so a pass slows as the partition grows. A backlog of a million-plus abandoned claims takes days.
Rescue now drains a backlog in bounded passes:
(claimed_at, job_id, run_lease)index from the cursor with aLIMIT(the cursor bound is a scalar subquery the index can use), so each batch costs the same however large the partition is. Previously the bound ran as a join filter and every batch re-read every claim already swept.rescue_pass_budget(default 1 s, capped at half the rescue interval) orrescue_pass_max_rows(default 5,000) is reached, then re-runs a second later while a backlog remains. Deadline passes start with a 2,000-claim window. Stale-heartbeat rescue takes extra batches only once runtime snapshots show the owners gone for a full staleness window.Rescue keeps the release branch's savepoint retry with busy and ambiguous attempts set aside; each attempt's batch accumulator and cursor progress are created inside the attempt, so a rolled-back attempt adds nothing. Rescue is not gated by a
maintenance.prunehalt.Testing
rescue_drains_abandoned_row_local_claims_and_unpins_both_ringsbuilds the production shape — row-local claims over two claim slots, all referencing one queue slot, backdated, owners gone, both rings pinned — and requires every claim to close, both rings to rotate past the pinned slots and every job to run exactly once on attempt 2. On PG17 with default settings, 1.6M such claims drained in about 17 minutes (~1,600/s) against ~13 hours at the release tip's rate; the rings rotated about two minutes later. Per 500-claim batch: deadline 22–25 ms lock + 72–83 ms close, heartbeat 27–30 ms + 46–53 ms; at most 500 advisory locks held by one backend. fmt and clippy clean;awa-workerunit tests and the rescue/prune/heartbeat/finalize subset ofqueue_storage_runtime_testpass on PG18, fullqueue_storage_runtime_teston PG17 (two backlog-timing tests pass when run alone). The 50k compact-claim bulk test is occasionally flaky on PG18, as it was at this PR's original head.Raising
rescue_pass_budgetandrescue_pass_max_rowsdrains faster at the cost of holding the maintenance leader for longer per pass, delaying promotion, rotation and prune by the same amount.