Repository navigation
fix(admin): bound queue-storage available counts and keep tombstone probes correlated - #720
Conversation
…robes correlated The admin queue views counted every unclaimed ready row and checked each against ready_tombstones through a correlated NOT EXISTS that Postgres could flatten into a merge anti join over the whole tombstone table. On a large backlog with many tombstones both costs exceeded the statement timeout. Fence every ready and receipt-completion tombstone probe in the admin SQL with OFFSET 0 so cost follows the rows checked. Walk at most AVAILABLE_LANE_EXACT_COUNT_CAP (5,000) unclaimed rows per lane for the per-queue available count: a lane within the cap is exact, a deeper lane contributes next_seq - claim_seq, as the worker queue-depth gauge does, and the queue is flagged with available_is_estimate on QueueOverview, the Python QueueStat, the CLI and the admin UI. The state timeseries reads each lane from its newest rows with the same cap.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Title check |  |
This assessment has not completed. The check will be retried on the next pre-merge check run. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12ab961156
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
state_counts now returns StateCounts { counts, available_is_estimate } and
/api/stats serves the same shape, so the dashboard's available tile marks an
estimated count from the same read instead of inferring it from the per-queue
overviews. The canonical-engine path always reports an exact count.
hardbyte
added
the
full-ci
Claude
While a queue's available count is an estimate its lag is a lower bound, and the state timeseries undercounts a lane that gained more available jobs than the cap inside the window. Neither was stated.
The admin queue views time out (30 s statement timeout) on a database with a large available backlog and many ready tombstones. Two costs combine:
NOT EXISTSin the admin lane walk can be flattened into a merge anti join that seq-scans and sorts the wholeready_tombstonestable, then re-walks it for every lane. With ~430k tombstones that was 300–400 ms per lane.availablecount costs ~13 µs per unclaimed row, so a backlog of millions exceeds the timeout regardless of plan.Change
ready_tombstonesandreceipt_completion_tombstones) carriesOFFSET 0, so Postgres keeps it a correlated index probe and cost follows the rows checked. This mirrors the claim-path fix in fix(storage): keep claim tombstone checks correlated #713.availablecount walks at mostAVAILABLE_LANE_EXACT_COUNT_CAP(5,000) unclaimed rows per lane. Within the cap the count is exact. A deeper lane contributes its head-sequence depth (next_seq - claim_seq, what the worker queue-depth gauge publishes) and the queue is flaggedavailable_is_estimateonQueueOverview, PythonQueueStatandawa queue stats(~N). The UI shows≈Nwith a tooltip.state_countsreturnsStateCounts { counts, available_is_estimate }, and/api/statsreturns{ "counts": {...}, "available_is_estimate": bool }instead of a bare state map, so the dashboard's available tile marks an estimate from the same pass that produced it. This is a breaking change for direct consumers ofstate_countsor/api/stats.Kind counts and
list_jobsget the probe fix only. No schema change.Validation
Measured against a production-sized database (one ready slot with ~55M rows, ~430k tombstones, one queue with >1M available across 32 lanes whose segments hold ~1 row each):
The capped walk plans as Incremental Sort → Limit and stops at the cap.
Tests, on PostgreSQL 17 and 18:
available_count_probes_tombstones_per_rowasserts no sequential scan reads tombstone rows; it fails withOFFSET 0removed.admin_available_count_test: exact at the cap with tombstones, estimate past it, mixed lanes and neighbouring queues,state_countsconsistency, and recent rows counted behind an older backlog larger than the cap.admin_api_testchecks the new/api/statsshape and flag.cargo fmt, clippy (-D warnings) andtsc --noEmitare clean.admin_read_cancel_testhas two timeouts (cancelled_job_list_begin_…,cancelled_kind_overviews_begin_…) that fail identically on the base branch.The history-backed legs (running from open receipt claims, completed and failed totals) still scale with retained ring contents; moving them onto the counter tables is separate work.