Skip to content

perf(queue-storage): drain a queue in bulk chunks (0.6) - #599

Merged
hardbyte merged 3 commits into
release/0.6.6from
brian/bulk-drain-0.6
Nov 4, 2026
Merged

hardbyte merged 3 commits into
release/0.6.6from
brian/bulk-drain-0.6

Conversation

@hardbyte

@hardbyte hardbyte commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Backport of #598 to release/0.6.6 for 0.6.12. It is stacked on #597 because both edit the 0.6.12 changelog section, so merge #597 first. This PR then rebases to the single drain commit.

Problem

admin::drain_queue on queue storage, behind awa queue drain and the Python drain_queue, listed the queue's jobs through the current_jobs view. That view reads every retained ready row. It then cancelled each job through cancel_job_tx, several statements per job, inside one transaction. Draining 4,000 jobs over a network connection took 13.7 s, and the time grows linearly with the backlog while one transaction holds every cancelled row's lock. It also matched only the logical queue name, so a striped queue drained nothing.

Change

The same as #598. Scheduled and retryable jobs are deleted from deferred_jobs, and unclaimed ready rows are tombstoned, 1,000 at a time with one transaction per chunk. Ready rows are found from each lane's claim cursor through ready_segments. Each run of consecutive cancelled positions gets one claim-cursor advance. Waiting jobs still go through cancel_jobs_by_ids, and admin::drain_queue expands the queue to its persisted stripes.

0.6 differences from main: sqlx::query(&format!(..)) in place of audited_sql, no claim-evidence field on cursor advances, and no output column on terminal rows. No migration and no public API change.

Tests

  • The same two tests as perf(queue-storage): drain a queue in bulk chunks #598, in awa/tests/queue_storage_runtime_test.rs:
    • a drain of a 5,006-job mixed backlog larger than one chunk;
    • a drain of a two-stripe queue.
  • Passing locally: cargo clippy --workspace --all-targets --all-features -D warnings, awa-model --lib (81), queue_storage_runtime_test (133), integration_test, external_wait_test (which includes draining a queue with waiting jobs) and validation_test.

@hardbyte hardbyte added the full-ci Run the full CI matrix (Python build+test, E2E) on this PR label Nov 4, 2026
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2582b87c-c942-4439-951a-c34eb1ccb0b0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T08:21:27.208531Z 549d9b4 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 549d9b4ab5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +8440 to +8442
if (cancelled as i64) < DRAIN_BATCH_SIZE {
break;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep draining after a short skipped-lock batch

When an eligible row is concurrently locked, SKIP LOCKED can make a chunk return fewer than DRAIN_BATCH_SIZE even though work remains, but this condition treats that short batch as exhaustion and exits. In the available query, the limit is applied in candidates before the locking CTE, so even one locked row among those candidates can leave thousands of later unlocked jobs untouched; the deferred pass has the same problem when fewer than 1,000 unlocked rows remain. Consequently, draining an active queue can report success while scheduled, retryable, or available jobs are still present; termination needs an explicit exhaustion check or a retry strategy for skipped rows.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in this PR before it merged, backported from #598. Available rows are locked before the limit applies. The deferred pass already had LIMIT ... FOR UPDATE SKIP LOCKED at one level, which counts only locked rows, so a short deferred chunk did mean fewer than 1,000 unlocked rows remained. It now also walks job_id in order, so each chunk resumes where the last one stopped.

@hardbyte
hardbyte force-pushed the brian/bulk-drain-0.6 branch from 549d9b4 to d1efc7e Compare November 4, 2026 14:22
Backport of #598. admin::drain_queue on queue storage listed the
queue's jobs through the current-jobs view, which reads every retained
ready row, then cancelled them one by one inside a single transaction.
Scheduled, retryable and available jobs are now cancelled 1,000 at a
time, one transaction per chunk, and a striped queue drains every
persisted stripe.
Backport of the follow-up on #598. The drain reads every lane's enqueue
head once and job ids below the next id at the start, so work enqueued
while it runs is left for a later call. Each chunk locks rows before
applying its limit and the next chunk resumes after the last cancelled row.
Stripes are matched by name in every pass, which covers stripes holding only
scheduled jobs.

Cancel, move and reprioritise skip ready rows a claim has recorded but the
claim cursor has not yet passed, so they no longer cancel or re-enqueue a
job that is already running.
…rsor past it (0.6)

A noisy-neighbour benchmark (a 200/s workload on one queue while another is
drained, including a drain of a queue with live workers and a producer)
found three problems in the bulk drain:

- A claim records its attempt in the transaction that holds its lane's
  claim head, so a chunk read while that claim was in flight saw the row as
  available and cancelled a job a worker then ran. Each chunk now locks the
  claim heads of its lanes first, and cancel_job / move / cancel_jobs_by_ids
  lock the heads of the jobs' lanes, in the byte order cursor advances use.
- The drain advanced a lane's cursor only if it still sat at the start of
  the cancelled run. With workers claiming, it rarely did, and claims then
  crossed the tombstoned range one claim batch per poll: about 64k jobs
  enqueued during a 100k drain were still unclaimed three minutes later.
  After each chunk the drain now moves the cursor over tombstoned rows and
  committed claims, under the claim-head lock.
- Chunk statements were JIT-compiled (about 3 s each against milliseconds of
  execution), and the tombstone NOT EXISTS was planned as an anti join that
  re-read the lane's tombstones for every segment. Drain transactions turn
  JIT off, and the tombstone check is kept a per-row probe.

At 100k available plus 100k scheduled the drain now takes 11 s quiet and
11 s with live workers (previously 340 s and 321 s), with no running job
cancelled, no job left stranded and the neighbouring queue's latency at
baseline.

Backport of the follow-up on #598.
@hardbyte
hardbyte force-pushed the brian/bulk-drain-0.6 branch from eaa0d2d to a1531ab Compare November 4, 2026 21:10
@hardbyte
hardbyte merged commit 08ac81b into release/0.6.6 Nov 4, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run the full CI matrix (Python build+test, E2E) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant