fix(admin): bound queue-storage available counts and keep tombstone probes correlated

CHANGELOG.md

@@ -22,6 +22,7 @@ Notable changes between releases. Detailed migration notes for storage transitio
 - The embedded admin UI now works when its axum router is nested under a path such as `/awa` (#502). Assets, API requests, navigation and direct deep links automatically use the mount path, including outer nested routers, without rebuilding the frontend or changing existing router calls.
 - `admin::list_jobs` and `admin::job_kind_overviews` no longer omit a failed job, or leave it out of its kind's count, while queue prune moves its row out of the slot being pruned. Both read several queries from one snapshot and release partition locks between them, so the queries after a prune committed saw neither the pruned slot's rows nor the moved ones. These reads and the point reads such as `admin::get_job` now compare the relfilenode of every ring partition before their snapshot and after it ends, and read again in a new snapshot when prune or the terminal-count rollup truncated one in between, where 0.6.16 read a single job again only when it found nothing. After three reads that each overlapped a truncation, the fourth locks the ring partitions before taking its snapshot and holds them until it commits, so a read never returns a result a truncation could have changed; prune skips its pass if it cannot get its locks within its lock timeout. The comparison adds two catalog lookups to each read and takes no partition locks, so prune is blocked no more often than before.
 - Claiming from a ready slot that holds tombstones no longer scans the slot's whole tombstone partition on every claim (#713, reported and fixed by @Wallace99). Each claim checks its candidate rows against `ready_tombstones`, and for a small batch PostgreSQL could plan that check as a hashed subplan over the partition instead of one index probe per row: a 32-row claim read 10,000 tombstone rows in the regression test. The check now carries `OFFSET 0`, which keeps it correlated. Tombstones are left by cancelled, retried and moved jobs, so the cost grew with churn in the slot. Applied by the `correlated_claim_tombstones` schema patch; no configuration or protocol change.
+- The admin queue views no longer time out on a large backlog with many ready tombstones. Tombstone checks in the admin SQL are per-row index probes, and the per-queue `available` count reads at most 5,000 unclaimed rows per lane (`admin::AVAILABLE_LANE_EXACT_COUNT_CAP`). A deeper lane reports its head-sequence depth, as the worker queue-depth gauge does, and sets `QueueOverview::available_is_estimate`; the UI shows `≈N` and `awa queue stats` shows `~N`. While a queue's count is an estimate its `lag_seconds` is a lower bound. The state timeseries reads each lane's newest 5,000 available rows, so a lane that gained more than that inside the window is undercounted there. No schema change. `admin::state_counts` now returns `StateCounts { counts, available_is_estimate }` and `/api/stats` returns `{ "counts": {...}, "available_is_estimate": bool }` instead of a bare state map.
 - Cleanup, cron sync, batch operations, metadata refresh, rollup and ring rotation no longer hold back promotion and rescue (#303). The maintenance leader ran every branch from one loop, so while one branch's statement waited on a lock or ran a long scan, scheduled and retryable jobs were not promoted and stale, overdue and timed-out attempts were not rescued. With one `awa.cron_jobs` row locked by another session, the leader's cron sync stalled both for as long as the lock was held, and stopped cron from firing. Promotion and the three rescues now run in their own loop within the leader term, and the cron evaluator starts before the startup cron sync. A rescue pass still delays promotion, and the other branches still delay one another: a stuck cleanup or cron sync still holds back ring rotation and prune. Leadership and the shutdown grace are unchanged, and no configuration or schema change is needed. A leader holds one more pool connection at a time. While a rescue pass runs, lease prune may report more `blocked` outcomes and claim prune may wait out a rescue batch. See [ADR-051](docs/adr/051-maintenance-recovery-loop.md).
 - One completion waiting on another transaction's lock no longer stalls every other completion in the runtime. A transaction that changes a running attempt, such as `admin::cancel_tx` inside an application's transaction, holds the attempt's completion lock until it commits. If the attempt's handler returned `Completed` meanwhile, the completion batch waited on that lock, so no other job's completion committed. Once the in-flight completion allowance filled, the runtime stopped claiming in every queue until the transaction ended. With receipts on (the default), a completion batch now leaves an attempt whose lock is held out of the batch, and the worker retries that attempt with backoff for up to two minutes without holding a pool connection, then leaves it to stale-heartbeat rescue as before. Non-success outcomes and kinds with follow-up enqueues already finalized one job at a time and still wait on the lock. With receipts off, a batch still waits. No configuration or schema change. See [a completion waits on an application's transaction](docs/troubleshooting.md#a-completion-waits-on-an-applications-transaction).
 - Python handlers receive an asyncio cancellation request when the Rust executor abandons their attempt after its deadline grace or shutdown exceeds its worker timeout (#697). Previously, dropping the Rust result receiver left the Python coroutine running, so repeated abandoned attempts leaked live handlers and could produce late effects. Shutdown also waits up to one additional second for that client's Python handlers to finish cancellation cleanup, then warns if any remain. Cancellable handlers receive `asyncio.CancelledError` and can release resources in `finally`; cancellation is cooperative and cannot stop blocking code, spawned threads, child tasks or handlers that suppress cancellation. No schema change.

awa-cli/src/main.rs

@@ -1439,7 +1439,11 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
                                 println!(
                                     "{:<15} {:<10} {:<10} {:<10} {:<15} {:<10} {:<8}",
                                     stat.queue,
-                                    stat.available,
+                                    if stat.available_is_estimate {
+                                        format!("~{}", stat.available)
+                                    } else {
+                                        stat.available.to_string()
+                                    },
                                     stat.running,
                                     stat.failed,
                                     stat.completed_last_hour,