Repository navigation
perf(queue-storage): drain a queue in bulk chunks (0.6) - #599
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 549d9b4ab5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (cancelled as i64) < DRAIN_BATCH_SIZE { | ||
| break; | ||
| } |
There was a problem hiding this comment.
Keep draining after a short skipped-lock batch
When an eligible row is concurrently locked, SKIP LOCKED can make a chunk return fewer than DRAIN_BATCH_SIZE even though work remains, but this condition treats that short batch as exhaustion and exits. In the available query, the limit is applied in candidates before the locking CTE, so even one locked row among those candidates can leave thousands of later unlocked jobs untouched; the deferred pass has the same problem when fewer than 1,000 unlocked rows remain. Consequently, draining an active queue can report success while scheduled, retryable, or available jobs are still present; termination needs an explicit exhaustion check or a retry strategy for skipped rows.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in this PR before it merged, backported from #598. Available rows are locked before the limit applies. The deferred pass already had LIMIT ... FOR UPDATE SKIP LOCKED at one level, which counts only locked rows, so a short deferred chunk did mean fewer than 1,000 unlocked rows remained. It now also walks job_id in order, so each chunk resumes where the last one stopped.
549d9b4 to
d1efc7e
Compare
Backport of #598. admin::drain_queue on queue storage listed the queue's jobs through the current-jobs view, which reads every retained ready row, then cancelled them one by one inside a single transaction. Scheduled, retryable and available jobs are now cancelled 1,000 at a time, one transaction per chunk, and a striped queue drains every persisted stripe.
Backport of the follow-up on #598. The drain reads every lane's enqueue head once and job ids below the next id at the start, so work enqueued while it runs is left for a later call. Each chunk locks rows before applying its limit and the next chunk resumes after the last cancelled row. Stripes are matched by name in every pass, which covers stripes holding only scheduled jobs. Cancel, move and reprioritise skip ready rows a claim has recorded but the claim cursor has not yet passed, so they no longer cancel or re-enqueue a job that is already running.
…rsor past it (0.6) A noisy-neighbour benchmark (a 200/s workload on one queue while another is drained, including a drain of a queue with live workers and a producer) found three problems in the bulk drain: - A claim records its attempt in the transaction that holds its lane's claim head, so a chunk read while that claim was in flight saw the row as available and cancelled a job a worker then ran. Each chunk now locks the claim heads of its lanes first, and cancel_job / move / cancel_jobs_by_ids lock the heads of the jobs' lanes, in the byte order cursor advances use. - The drain advanced a lane's cursor only if it still sat at the start of the cancelled run. With workers claiming, it rarely did, and claims then crossed the tombstoned range one claim batch per poll: about 64k jobs enqueued during a 100k drain were still unclaimed three minutes later. After each chunk the drain now moves the cursor over tombstoned rows and committed claims, under the claim-head lock. - Chunk statements were JIT-compiled (about 3 s each against milliseconds of execution), and the tombstone NOT EXISTS was planned as an anti join that re-read the lane's tombstones for every segment. Drain transactions turn JIT off, and the tombstone check is kept a per-row probe. At 100k available plus 100k scheduled the drain now takes 11 s quiet and 11 s with live workers (previously 340 s and 321 s), with no running job cancelled, no job left stranded and the neighbouring queue's latency at baseline. Backport of the follow-up on #598.
eaa0d2d to
a1531ab
Compare
Backport of #598 to
release/0.6.6for 0.6.12. It is stacked on #597 because both edit the 0.6.12 changelog section, so merge #597 first. This PR then rebases to the single drain commit.Problem
admin::drain_queueon queue storage, behindawa queue drainand the Pythondrain_queue, listed the queue's jobs through thecurrent_jobsview. That view reads every retained ready row. It then cancelled each job throughcancel_job_tx, several statements per job, inside one transaction. Draining 4,000 jobs over a network connection took 13.7 s, and the time grows linearly with the backlog while one transaction holds every cancelled row's lock. It also matched only the logical queue name, so a striped queue drained nothing.Change
The same as #598. Scheduled and retryable jobs are deleted from
deferred_jobs, and unclaimed ready rows are tombstoned, 1,000 at a time with one transaction per chunk. Ready rows are found from each lane's claim cursor throughready_segments. Each run of consecutive cancelled positions gets one claim-cursor advance. Waiting jobs still go throughcancel_jobs_by_ids, andadmin::drain_queueexpands the queue to its persisted stripes.0.6 differences from main:
sqlx::query(&format!(..))in place ofaudited_sql, no claim-evidence field on cursor advances, and nooutputcolumn on terminal rows. No migration and no public API change.Tests
awa/tests/queue_storage_runtime_test.rs:cargo clippy --workspace --all-targets --all-features -D warnings,awa-model --lib(81),queue_storage_runtime_test(133),integration_test,external_wait_test(which includes draining a queue with waiting jobs) andvalidation_test.