Repository navigation
perf(queue-storage): asynchronous claim commits and claimer-lease reuse (0.6) - #722
Merged
Merged
Conversation
* perf(queue-storage): commit claim transactions asynchronously
A claim transaction holds its lane's queue_claim_heads row lock until it
commits, and the post-commit cursor advance takes the same row. Both
committed synchronously, so every claim spent a WAL fsync inside the lock.
Under load each freed worker permit claims one job, which made claiming one
fsync-bounded, lane-serialised transaction per job.
The claim, claim_batch and cursor-advance transactions now begin with
`SET LOCAL synchronous_commit = off`, sent in the same round trip as BEGIN.
A claim lost to a server crash leaves its job claimable again, which
at-least-once delivery already allows. A synchronous commit flushes all
earlier WAL, so a completion or enqueue can never be durable while the
claim it depends on is not. Enqueues, completions and every other
transaction stay synchronous.
Job-queue bench, 2 replicas x 200 jobs/s offered, 8 workers each,
PostgreSQL 18.6 on a SATA SSD (~7 ms fsync), stock sqlx 0.9.0:
completed/s claim p50 claim p99 backlog
before 320 6468 ms 9761 ms 3494
after 383 10 ms 67 ms 3
LWLock:WALWrite samples fell from 137 to 38 over the same window.
* perf(queue-storage): write enqueue batches with a single unnest INSERT
The INSERT writers for ready_entries, ready_segments and deferred_jobs built
multi-row VALUES statements with one bind per column per row, so every batch
size produced new SQL text to parse and plan. They now send each column as
one array through a fixed-text INSERT ... SELECT FROM unnest(...).
The COPY entry points (QueueStorage::enqueue_params_copy, insert_many_copy
and the Python *_copy methods) route batches below COPY_MIN_ROWS (2048) to
that INSERT. Below the threshold the INSERT is faster than COPY, and with
sqlx-core 0.9.0 every COPY also pays a ~40 ms Nagle/delayed-ACK stall: the
driver no longer sets TCP_NODELAY and sends CopyData and CopyDone as
separate small writes (fixed upstream in transact-rs/sqlx#4336, not yet
released). Large batches keep COPY.
Measured on PostgreSQL 18.6, one connection, stock sqlx-core 0.9.0
(ms per transaction):
rows COPY unnest (with TCP_NODELAY: COPY / unnest)
5 41.4 0.55 0.46 / 0.36
128 41.8 0.93 1.01 / 0.89
512 2.3 1.77 1.87 / 1.36
2048 19.6 5.07 2.87 / 3.32
8192 28.7 12.96 8.13 / 11.73
Producer call p50 in the job-queue bench (2 replicas x 200 jobs/s) drops
from 86 ms to ~6 ms on stock sqlx.
The existing COPY tests now exercise the INSERT path; new tests at the
threshold keep the CSV encoder covered for both the canonical and
queue-storage COPY writers.
* fix(queue-storage): advance a claim cursor only past recorded claims
The post-claim cursor advance ran in its own transaction after the claim
committed, and moved the lane's claim sequence unconditionally. If the
primary failed after acknowledging a claim but before that claim reached a
standby, the advance could retry through the pool against the promoted
standby and move its cursor past a ready row whose claim never happened
there; the job would then be skipped permanently. Asynchronous claim commits
widen that window for synchronous-replication setups; with asynchronous
replication it existed already.
Advances derived from a claim now carry the claimed row's ready slot,
generation and lane position, and only apply while a
ready_claim_attempt_batches row covering that position is visible (an
index probe on one partition). A lagging cursor is already safe: the next
claim's spent-prefix path repairs it.
* docs: keep the SQLx TCP_NODELAY release gate explicit
* fix(queue-storage): guard legacy cursor replay and model async failover
* perf(queue-storage): commit claim transactions asynchronously
A claim transaction holds its lane's queue_claim_heads row lock until it
commits, and the post-commit cursor advance takes the same row. Both
committed synchronously, so every claim spent a WAL fsync inside the lock.
Under load each freed worker permit claims one job, which made claiming one
fsync-bounded, lane-serialised transaction per job.
The claim, claim_batch and cursor-advance transactions now begin with
`SET LOCAL synchronous_commit = off`, sent in the same round trip as BEGIN.
A claim lost to a server crash leaves its job claimable again, which
at-least-once delivery already allows. A synchronous commit flushes all
earlier WAL, so a completion or enqueue can never be durable while the
claim it depends on is not. Enqueues, completions and every other
transaction stay synchronous.
Job-queue bench, 2 replicas x 200 jobs/s offered, 8 workers each,
PostgreSQL 18.6 on a SATA SSD (~7 ms fsync), stock sqlx 0.9.0:
completed/s claim p50 claim p99 backlog
before 320 6468 ms 9761 ms 3494
after 383 10 ms 67 ms 3
LWLock:WALWrite samples fell from 137 to 38 over the same window.
* perf(queue-storage): write enqueue batches with a single unnest INSERT
The INSERT writers for ready_entries, ready_segments and deferred_jobs built
multi-row VALUES statements with one bind per column per row, so every batch
size produced new SQL text to parse and plan. They now send each column as
one array through a fixed-text INSERT ... SELECT FROM unnest(...).
The COPY entry points (QueueStorage::enqueue_params_copy, insert_many_copy
and the Python *_copy methods) route batches below COPY_MIN_ROWS (2048) to
that INSERT. Below the threshold the INSERT is faster than COPY, and with
sqlx-core 0.9.0 every COPY also pays a ~40 ms Nagle/delayed-ACK stall: the
driver no longer sets TCP_NODELAY and sends CopyData and CopyDone as
separate small writes (fixed upstream in transact-rs/sqlx#4336, not yet
released). Large batches keep COPY.
Measured on PostgreSQL 18.6, one connection, stock sqlx-core 0.9.0
(ms per transaction):
rows COPY unnest (with TCP_NODELAY: COPY / unnest)
5 41.4 0.55 0.46 / 0.36
128 41.8 0.93 1.01 / 0.89
512 2.3 1.77 1.87 / 1.36
2048 19.6 5.07 2.87 / 3.32
8192 28.7 12.96 8.13 / 11.73
Producer call p50 in the job-queue bench (2 replicas x 200 jobs/s) drops
from 86 ms to ~6 ms on stock sqlx.
The existing COPY tests now exercise the INSERT path; new tests at the
threshold keep the CSV encoder covered for both the canonical and
queue-storage COPY writers.
* fix(queue-storage): advance a claim cursor only past recorded claims
The post-claim cursor advance ran in its own transaction after the claim
committed, and moved the lane's claim sequence unconditionally. If the
primary failed after acknowledging a claim but before that claim reached a
standby, the advance could retry through the pool against the promoted
standby and move its cursor past a ready row whose claim never happened
there; the job would then be skipped permanently. Asynchronous claim commits
widen that window for synchronous-replication setups; with asynchronous
replication it existed already.
Advances derived from a claim now carry the claimed row's ready slot,
generation and lane position, and only apply while a
ready_claim_attempt_batches row covering that position is visible (an
index probe on one partition). A lagging cursor is already safe: the next
claim's spent-prefix path repairs it.
* perf(queue-storage): reuse the claimer lease across claims
Every claim_runtime_batch_with_aging_for_instance call read the queue's
claimer target and this instance's claimer lease before claiming: two extra
round trips and two transactions per claim. With few free worker permits
each claim returns about one job, so that was two transactions per job.
The lease row is now cached per (queue, instance) for 250 ms and never used
within a second of its expiry. A successful heartbeat refresh updates the
cached row, and a lost lease (refresh matches no row, or acquisition finds
no free slot) drops it. The adaptive claimer target is a 500 ms control
signal, so reading it at most every 250 ms changes nothing material.
Depth-target bench, W=4, 1 ms jobs, PG 18.6: transactions per completed
job 2.35 -> 1.39, throughput 358 -> 392 jobs/s.
* fix(queue-storage): forget cached claimer leases on reset
reset() truncates queue_claimer_leases; a claim within the cache TTL could
otherwise reuse a lease that no longer exists and bypass the bounded-claimer
gate.
* docs: keep the SQLx TCP_NODELAY release gate explicit
* fix(queue-storage): guard legacy cursor replay and model async failover
* test(queue-storage): group claimer cache coverage for stacked merge safety
* docs: clarify COPY fallback and async claim durability boundaries
…ch on 0.6 The cursor-advance guard from #507 took a claim with no claim batch id to be a legacy claim and looked for its leases row. On main every receipt claim is compact, so that holds. On 0.6 a receipt claim with a deadline, the default, is written to lease_claims with no batch id and has no leases row, so its advance never applied and the cursor moved only through spent-prefix recovery. The guard now keys on the receipt id, and the regression's lost-claim fixture also removes lease_claims rows.
A claim lost before its WAL flush leaves the job claimable again, and the next claim reuses the lost claim's run lease. The worker still running the lost claim can therefore have its heartbeat or completion accepted against the new attempt. Asynchronous claim commits make this reachable after a primary crash, not only after a failover with asynchronous replication.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
hardbyte
added
the
full-ci
Claude
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0.6 port of #507 and #508, fix-first on the 0.6 line per the branch policy in #383. No schema, configuration or protocol change.
Why
On 0.6.16 a claim holds its lane's
queue_claim_headsrow lock until it commits, and the post-commit cursor advance takes the same row. Both committed synchronously, so every claim spent a WAL flush inside the lane lock and claims queued behind each other while neither PostgreSQL nor the worker was saturated.What changes
claim_runtime_batch,claim_batchand the claim-cursor advance begin withBEGIN; SET LOCAL synchronous_commit = off. Enqueues, completions, heartbeats and every other transaction stay synchronous. The setting is local to those transactions.AwaAsyncClaimCursorTLA+ model and its unguarded counterexample come with it.This is not a mechanical port
The claim path on 0.6 differs from
mainwhere #507 touches it, because compact deadline claim batches (#410) are not on 0.6.leasesrow. Onmainevery receipt claim is compact, so that holds. On 0.6 a receipt claim with a deadline, the default, is written tolease_claimswith no batch id and has noleasesrow. Its advance never applied, and the cursor moved only through spent-prefix recovery. Four tests inqueue_storage_runtime_testfailed on the straight port. The guard now keys on the receipt id: both receipt variants on 0.6 carry one and both write aready_claim_attempt_batchesrow in the claim transaction.maincommits also carried perf(queue-storage): unnest INSERT for enqueue batches; skip COPY below 2048 rows #506's enqueue rewrite and its 2,048-row COPY cutoff, and sqlx 0.9'saudited_sqlin the tests. Neither is in this PR.Why it is safe on 0.6
setval, written to WAL after the claim's records. Recovery and streaming replication both apply a WAL prefix, so a surviving advance implies a surviving claim.ready_claim_attempt_batchesrows are insert-only on 0.6 and are removed only by prune or reset.A behaviour change to know about
A worker still running a claim that was lost before its WAL flush is not fenced. The job's next claim reuses the lost claim's run lease, so that worker's heartbeat or completion can be accepted against the new attempt. Delivery stays at-least-once. Before this change it needed a failover with asynchronous replication; a primary crash can now cause it, until the next synchronous commit or WAL writer flush.
docs/architecture.mdand the changelog say so.With synchronous replication, an acknowledged claim is no longer guaranteed to have reached a standby.
Measurements
test_throughput_rust_workers_queue_storagein release mode: 20,000 no-op jobs, 16 workers, one queue, PostgreSQL 18 in a container on a 4-core host.baseis0515f339;backportis this branch before the rebase onto #720. Two runs per cell, end-to-end jobs/s.commit_delay = 5000andcommit_siblings = 0, which makes every synchronous commit wait 5 ms before flushing. It stands in for a slow disk; it is not one.Tests
queue_storage_runtime_test: 268 pass on PostgreSQL 18, includingtest_queue_storage_cursor_advance_requires_claim_evidencefor receipt and legacy claims and the claimer-cache tests. The receipt-claim case fails on the straight port.awa-modellib tests pass;cargo fmtand clippy (awa-model,awa,awa-worker, all targets and features) are clean.Docs
docs/architecture.md,docs/configuration.md,correctness/storage/MAPPING.mdandREADME.md, and the changelog.Generated by Claude Code