Repository navigation
Conversation
…and hand busy lanes to a waiter Promotion re-enqueues due scheduled and retryable rows into their lane, which takes the lane's reservation lock that producers hold until they commit. One promotion batch was a single transaction across every queue it touched, and the maintenance leader runs its branches one after another, so a lane kept busy by producers delayed due jobs of unrelated queues in the same batch and started rescue, rotation and prune late. QueueStorage::promote_due_lanes selects a batch without locking it and promotes each (queue, priority) lane's share in its own transaction, under an optional lock_timeout; a lane that hits it rolls back and is reported as contended. promote_due_lane promotes the oldest due rows of one lane. promote_due keeps its signature and promotes lane by lane with no timeout. Within a lane, promoted rows take lane sequence numbers in due order. The leader's promoter runs each tick under a 50 ms lane lock timeout, skips contended lanes for the rest of the tick, and hands each to a dedicated waiter task (at most four at once) that promotes the lane under a 30 s lock timeout. PostgreSQL grants a lock to its waiters in arrival order, so the waiter is served after the producers queued ahead of it however busy the lane stays. awa.maintenance.promote.lane_contention counts handed_off, waiter_limit and waiter_timed_out by state and queue. The overrunning-promotion test slowed the queue-storage DELETE, which no longer runs when nothing is due; it now slows the canonical-plane promotion DELETE that every body runs.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37cf4f3c39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const PROMOTE_LANE_WAITER_LOCK_TIMEOUT: Duration = Duration::from_secs(30); | ||
| /// Most lanes with a dedicated waiter at once. A waiter holds a pool | ||
| /// connection while it waits. | ||
| const PROMOTE_MAX_LANE_WAITERS: usize = 4; |
There was a problem hiding this comment.
When four lanes contend, this fixed limit allows four waiter transactions to occupy shared pool connections for up to 30 seconds each. With the documented pool sizing (docs/deployment.md:46-57), the queue listeners, cancel listener, and leader connection can leave four or fewer transient connections, so these waiters can exhaust the pool and make rescue, heartbeat, completion, and the other maintenance branches block on pool acquisition—the leader returns quickly from promotion but still cannot perform the work this change is intended to unblock. Limit waiters according to reserved pool headroom or run them on separately budgeted connections.
Useful? React with 👍 / 👎.
|
…ound lane waiters by pool size The batch selection was a plain SELECT ... LIMIT and only the per-lane DELETE skipped locked rows. When the oldest due rows were locked by another transaction, such as a lane waiter left running by a former leader, every batch of a tick reselected them and promoted nothing until the lock went away. The selection now takes FOR UPDATE OF deferred SKIP LOCKED. It runs as its own statement, so the row locks end with it, and deferred_jobs is not partitioned, so no run-time pruning meets them. Each lane waiter holds a pool connection while it waits. The leader now runs one waiter per eight pool connections, at least one and at most four, and the configuration docs say so.
|
Parking this Generated by Claude Code |
hardbyte
changed the title
A busy enqueue lane can block promotion for unrelated queues and delay other maintenance. Promote each selected lane in its own bounded transaction, skip locked deferred rows, and hand contended lanes to a limited set of dedicated waiters.
Each physical queue/priority lane has one waiter shared by scheduled and retryable rows. The leader skips that lane in both states while its waiter alternates states within a total 32-batch budget. Waiters retain the existing pool-based concurrency limit and 30-second transaction lock timeout; ordinary promotion retains its 50-millisecond timeout. A timeout releases the waiter's queue position and leaves work deferred for a later retry, so sustained contention can still prevent progress. Dropping the promoter aborts its waiters and releases lane bookkeeping; unexpected task exits are logged.
Validation on head
7f74dc89:Database rehearsals used isolated test database names. Fresh CI is pending. Mixed-workload throughput and lock-pressure acceptance remain separate; this PR does not narrow the transaction-wide timeout scope.