diff --git a/CHANGELOG.md b/CHANGELOG.md index aa70b13e5..de37a8d7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,9 @@ Notable changes between releases. Detailed migration notes for storage transitions live in [`docs/upgrade-0.5-to-0.6.md`](docs/upgrade-0.5-to-0.6.md). -## [Unreleased] +## [0.6.17] — 2026-11-12 + +Recommended for everyone on 0.6.16 using queue storage. Claims commit asynchronously and reuse the runtime's claimer lease, which raised throughput by 19 to 21% at 4, 16 and 64 workers and cut end-to-end p99 at 800 jobs/s from 31 ms to 24 ms in a three-round comparison against 0.6.16 on PostgreSQL 18 with `synchronous_commit = on`. Promotion and rescue no longer wait behind the leader's other maintenance work, a runtime can opt out of maintenance leadership, and several admin reads and the admin UI are fixed. See the [upgrade guide](docs/upgrade-0.6.16-to-0.6.17.md). ### Upgrading @@ -17,12 +19,11 @@ Notable changes between releases. Detailed migration notes for storage transitio - Syncing a periodic job declaration that matches the stored schedule no longer rewrites its `awa.cron_jobs` row or moves `updated_at`, so runtimes that sync on a timer stop generating a row version per schedule per interval. - **Claim transactions commit asynchronously (0.6 port of #507).** Claims and the claim-cursor advance that follows them run with `synchronous_commit = off`, so a lane's claim head is no longer locked across a WAL fsync. A claim lost to a server crash leaves the job claimable again, consistent with at-least-once delivery; enqueues, completions and every other transaction stay synchronous. The setting is `SET LOCAL` in those transactions only. A post-claim cursor advance now applies only while the claim it follows is visible on the server it runs against, so a retry that lands on a promoted standby cannot move the cursor past a claim that never reached it. A worker still running a claim lost this way is not fenced by its run lease, because the job's next claim reuses it; see [architecture](docs/architecture.md). No schema or configuration change. - - **Claims reuse the instance's claimer lease (0.6 port of #508).** A queue-storage claim no longer re-reads the queue's claimer target and its own claimer lease on every call; the last read or refreshed lease is reused for up to 250 ms (never within a second of expiry), removing two round trips and two transactions from each claim. On `main` at 4 workers this cut transactions per completed job from 2.35 to 1.39. ### Fixed -- Queue and claim ring prunes no longer starve under sustained claim load (#721). Prune requested `ACCESS EXCLUSIVE` on the sealed slot's partitions for 10 ms, so with claims, completions and heartbeats always overlapping it never got the lock. That request now waits the prune lock timeout, lets waiting readers through, and retries with twice the wait up to 320 ms (`QueueStorage::prune_lock_wait_ceiling`) before reporting `blocked`. Queue prune also ran its safety checks under that lock, so when it did get it every reader of those partitions waited for the checks, which take seconds on a large slot. It now takes `SHARE ROW EXCLUSIVE` first, which blocks writes to the sealed slot and no reads, runs the checks, the failed-row carry and the rollup scan under it, and requests `ACCESS EXCLUSIVE` only for the `TRUNCATE`; after the upgrade it re-checks active leases and pending ready rows, which is what catches a lease committed while it waited. Queue prune requests the segment partition first because claims lock it first; a statement that holds a later partition while it waits for an earlier one still blocks that attempt until its timeout. Claim prune still runs its open-claim check under `ACCESS EXCLUSIVE`, because heartbeat, retry, fail and snooze read `lease_claims` and then update it in one transaction and would deadlock with a lock upgrade. A deadlock that PostgreSQL resolves by cancelling a prune is now counted as `blocked` rather than `error`. Readers of a sealed slot can now wait up to 320 ms behind a contended prune instead of 10 ms. The startup warning compares the 320 ms ceiling, not the base timeout, with `deadlock_timeout`, and eight times the ceiling with `statement_timeout`, because the lock timeout applies to each of the eight partitions one lock statement takes. +- Queue prune no longer runs its safety checks under `ACCESS EXCLUSIVE`, and neither ring prune holds a lock-order cycle with rescue, heartbeats or claims while it waits (#721, partial). Queue prune takes `SHARE ROW EXCLUSIVE` on the sealed slot first, which blocks writes to the slot and no reads, runs the checks, the failed-row carry and the rollup scan under it, and requests `ACCESS EXCLUSIVE` only for the `TRUNCATE`; after the upgrade it re-checks active leases and pending ready rows. On 0.6.16 a prune that got the lock made every reader of those partitions wait for the checks, which take seconds on a large slot. For both rings the `ACCESS EXCLUSIVE` request now waits the prune lock timeout, lets waiting readers through, and retries with twice the wait up to 320 ms (`QueueStorage::prune_lock_wait_ceiling`) before reporting `blocked`, so a reader of a sealed slot can wait up to 320 ms behind a contended prune instead of 10 ms. Queue prune requests the segment partition first because claims lock it first, and claim prune locks its `claim_ring_slots` row only after the slot's partitions because rescue takes them in that order. A deadlock that PostgreSQL resolves by cancelling a prune is counted as `blocked` rather than `error`. The startup warning compares the 320 ms ceiling with `deadlock_timeout`, and eight times the ceiling with `statement_timeout`. This does not end the starvation the issue describes: with a large backlog of abandoned claims prune is still blocked in most attempts, as on 0.6.16, because claims, completions and heartbeats still lock sealed slots. - The embedded admin UI now works when its axum router is nested under a path such as `/awa` (#502). Assets, API requests, navigation and direct deep links automatically use the mount path, including outer nested routers, without rebuilding the frontend or changing existing router calls. - `admin::list_jobs` and `admin::job_kind_overviews` no longer omit a failed job, or leave it out of its kind's count, while queue prune moves its row out of the slot being pruned. Both read several queries from one snapshot and release partition locks between them, so the queries after a prune committed saw neither the pruned slot's rows nor the moved ones. These reads and the point reads such as `admin::get_job` now compare the relfilenode of every ring partition before their snapshot and after it ends, and read again in a new snapshot when prune or the terminal-count rollup truncated one in between, where 0.6.16 read a single job again only when it found nothing. After three reads that each overlapped a truncation, the fourth locks the ring partitions before taking its snapshot and holds them until it commits, so a read never returns a result a truncation could have changed; prune skips its pass if it cannot get its locks within its lock timeout. The comparison adds two catalog lookups to each read and takes no partition locks, so prune is blocked no more often than before. - Claiming from a ready slot that holds tombstones no longer scans the slot's whole tombstone partition on every claim (#713, reported and fixed by @Wallace99). Each claim checks its candidate rows against `ready_tombstones`, and for a small batch PostgreSQL could plan that check as a hashed subplan over the partition instead of one index probe per row: a 32-row claim read 10,000 tombstone rows in the regression test. The check now carries `OFFSET 0`, which keeps it correlated. Tombstones are left by cancelled, retried and moved jobs, so the cost grew with churn in the slot. Applied by the `correlated_claim_tombstones` schema patch; no configuration or protocol change. diff --git a/Cargo.lock b/Cargo.lock index f21e8fc5f..51247be5d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -383,7 +383,7 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "awa" -version = "0.6.16" +version = "0.6.17" dependencies = [ "async-trait", "awa-macros", @@ -409,7 +409,7 @@ dependencies = [ [[package]] name = "awa-cli" -version = "0.6.16" +version = "0.6.17" dependencies = [ "assert_cmd", "awa-model", @@ -431,7 +431,7 @@ dependencies = [ [[package]] name = "awa-macros" -version = "0.6.16" +version = "0.6.17" dependencies = [ "proc-macro-crate", "proc-macro2", @@ -441,7 +441,7 @@ dependencies = [ [[package]] name = "awa-metrics" -version = "0.6.16" +version = "0.6.17" dependencies = [ "awa-model", "opentelemetry", @@ -450,7 +450,7 @@ dependencies = [ [[package]] name = "awa-model" -version = "0.6.16" +version = "0.6.17" dependencies = [ "awa-macros", "blake3", @@ -471,7 +471,7 @@ dependencies = [ [[package]] name = "awa-seaorm" -version = "0.6.16" +version = "0.6.17" dependencies = [ "awa", "awa-testing", @@ -484,7 +484,7 @@ dependencies = [ [[package]] name = "awa-testing" -version = "0.6.16" +version = "0.6.17" dependencies = [ "async-trait", "awa-model", @@ -499,7 +499,7 @@ dependencies = [ [[package]] name = "awa-ui" -version = "0.6.16" +version = "0.6.17" dependencies = [ "awa-metrics", "awa-model", @@ -522,7 +522,7 @@ dependencies = [ [[package]] name = "awa-worker" -version = "0.6.16" +version = "0.6.17" dependencies = [ "anyhow", "async-trait", diff --git a/Cargo.toml b/Cargo.toml index e0423e97c..d4244fcd7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,7 +14,7 @@ members = [ exclude = ["awa-python", "spike"] [workspace.package] -version = "0.6.16" +version = "0.6.17" edition = "2021" license = "MIT OR Apache-2.0" description = "Postgres-native background job queue — transactional enqueue, heartbeat crash recovery, SKIP LOCKED dispatch" @@ -25,11 +25,11 @@ categories = ["database", "asynchronous", "web-programming"] [workspace.dependencies] # Internal crates -awa-model = { path = "awa-model", version = "0.6.16" } -awa-macros = { path = "awa-macros", version = "0.6.16" } -awa-metrics = { path = "awa-metrics", version = "0.6.16" } -awa-worker = { path = "awa-worker", version = "0.6.16" } -awa = { path = "awa", version = "0.6.16" } +awa-model = { path = "awa-model", version = "0.6.17" } +awa-macros = { path = "awa-macros", version = "0.6.17" } +awa-metrics = { path = "awa-metrics", version = "0.6.17" } +awa-worker = { path = "awa-worker", version = "0.6.17" } +awa = { path = "awa", version = "0.6.17" } # Database sea-orm = { version = "=2.0.0-rc.38", default-features = false, features = ["sqlx-postgres", "runtime-tokio-rustls", "with-chrono", "with-json"] } diff --git a/README.md b/README.md index d8d637d62..ce4512593 100644 --- a/README.md +++ b/README.md @@ -391,6 +391,7 @@ All coordination through Postgres. The Rust runtime owns dispatch, leases, heart | [0.6.9 → 0.6.10 upgrade](docs/upgrade-0.6.9-to-0.6.10.md) | Rolling patch upgrade, rollback, and optional runtime privilege tightening | | [0.6.14 → 0.6.15 upgrade](docs/upgrade-0.6.14-to-0.6.15.md) | Rolling patch upgrade: bounded worker finalization, the claim batch index and the batch compatibility insert | | [0.6.15 → 0.6.16 upgrade](docs/upgrade-0.6.15-to-0.6.16.md) | Rolling patch upgrade: bounded rescue drains of abandoned claims, batch compatibility inserts that give way to per-row producers, new-lane provisioning order, the operator prune halt and its grants, and claim batch index fixes | +| [0.6.16 → 0.6.17 upgrade](docs/upgrade-0.6.16-to-0.6.17.md) | Rolling patch upgrade: asynchronous claim commits, the maintenance eligibility table, the claim tombstone patch, the changed `/api/stats` and `state_counts` shape, and prune lock waits | | [Configuration reference](docs/configuration.md) | `QueueConfig`, `ClientBuilder`, Python `start()`, env vars | | [HTTP workers and callbacks](docs/http-callbacks.md) | Serverless dispatch, callback receiver endpoints, signature verification | | [Lifecycle hooks](docs/lifecycle-hooks.md) | React to job start/finish/retry/callback events for metrics, alerting, cleanup | diff --git a/awa-cli/Cargo.toml b/awa-cli/Cargo.toml index 75ca22fec..e77b378e0 100644 --- a/awa-cli/Cargo.toml +++ b/awa-cli/Cargo.toml @@ -14,7 +14,7 @@ path = "src/main.rs" [dependencies] awa-model.workspace = true awa-worker.workspace = true -awa-ui = { path = "../awa-ui", version = "0.6.16" } +awa-ui = { path = "../awa-ui", version = "0.6.17" } axum.workspace = true sqlx.workspace = true tokio.workspace = true @@ -28,5 +28,5 @@ hex.workspace = true uuid.workspace = true [dev-dependencies] -awa-testing = { path = "../awa-testing", version = "0.6.16" } +awa-testing = { path = "../awa-testing", version = "0.6.17" } assert_cmd = "2" diff --git a/awa-cli/pyproject.toml b/awa-cli/pyproject.toml index 8bc2d2be4..5d954b5bf 100644 --- a/awa-cli/pyproject.toml +++ b/awa-cli/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "maturin" [project] name = "awa-cli" -version = "0.6.16" +version = "0.6.17" description = "CLI for the Awa Postgres-native job queue (migrations, admin, serve)" readme = "README.md" requires-python = ">=3.10" diff --git a/awa-python/Cargo.lock b/awa-python/Cargo.lock index 4946bf44a..468cf42a0 100644 --- a/awa-python/Cargo.lock +++ b/awa-python/Cargo.lock @@ -95,7 +95,7 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "awa-macros" -version = "0.6.16" +version = "0.6.17" dependencies = [ "proc-macro-crate", "proc-macro2", @@ -105,7 +105,7 @@ dependencies = [ [[package]] name = "awa-metrics" -version = "0.6.16" +version = "0.6.17" dependencies = [ "awa-model", "opentelemetry", @@ -113,7 +113,7 @@ dependencies = [ [[package]] name = "awa-model" -version = "0.6.16" +version = "0.6.17" dependencies = [ "awa-macros", "blake3", @@ -133,7 +133,7 @@ dependencies = [ [[package]] name = "awa-python" -version = "0.6.16" +version = "0.6.17" dependencies = [ "async-trait", "awa-model", @@ -156,7 +156,7 @@ dependencies = [ [[package]] name = "awa-worker" -version = "0.6.16" +version = "0.6.17" dependencies = [ "async-trait", "awa-macros", diff --git a/awa-python/Cargo.toml b/awa-python/Cargo.toml index 363843d5f..769bc09d4 100644 --- a/awa-python/Cargo.toml +++ b/awa-python/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "awa-python" -version = "0.6.16" +version = "0.6.17" edition = "2021" [lib] @@ -12,8 +12,8 @@ default = ["cel"] cel = ["awa-model/cel"] [dependencies] -awa-model = { path = "../awa-model", version = "0.6.16" } -awa-worker = { path = "../awa-worker", version = "0.6.16", features = ["__python-bridge"] } +awa-model = { path = "../awa-model", version = "0.6.17" } +awa-worker = { path = "../awa-worker", version = "0.6.17", features = ["__python-bridge"] } pyo3 = { version = "0.29", features = ["macros", "abi3-py310", "chrono"] } pyo3-async-runtimes = { version = "0.29", features = ["tokio-runtime"] } sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "chrono", "json"] } diff --git a/awa-python/pyproject.toml b/awa-python/pyproject.toml index d6a7b0dc6..c97b71677 100644 --- a/awa-python/pyproject.toml +++ b/awa-python/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "maturin" [project] name = "awa-pg" requires-python = ">=3.10" -version = "0.6.16" +version = "0.6.17" description = "Postgres-native background job queue — Python SDK with async/sync workers, transactional enqueue, and progress tracking. Install awa-pg[ui] to add the bundled web dashboard." readme = "README.md" license = {text = "MIT OR Apache-2.0"} @@ -34,7 +34,7 @@ classifiers = [ # # Kept opt-in so the default `awa-pg` install stays small — workers and # producers don't need the ~10 MB UI bundle. -ui = ["awa-cli==0.6.16"] +ui = ["awa-cli==0.6.17"] [project.urls] Homepage = "https://github.com/hardbyte/awa" diff --git a/awa-python/uv.lock b/awa-python/uv.lock index 2470233a8..4f76b23b5 100644 --- a/awa-python/uv.lock +++ b/awa-python/uv.lock @@ -88,12 +88,12 @@ wheels = [ [[package]] name = "awa-cli" -version = "0.6.16" +version = "0.6.17" source = { directory = "../awa-cli" } [[package]] name = "awa-pg" -version = "0.6.16" +version = "0.6.17" source = { editable = "." } [package.optional-dependencies] diff --git a/awa-seaorm/Cargo.toml b/awa-seaorm/Cargo.toml index 7ed3fe6c1..da4514cdc 100644 --- a/awa-seaorm/Cargo.toml +++ b/awa-seaorm/Cargo.toml @@ -14,6 +14,6 @@ serde_json.workspace = true sqlx.workspace = true [dev-dependencies] -awa-testing = { path = "../awa-testing", version = "0.6.16" } +awa-testing = { path = "../awa-testing", version = "0.6.17" } serde = { workspace = true, features = ["derive"] } tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } diff --git a/awa/Cargo.toml b/awa/Cargo.toml index 1c2a3c1e3..e9a3386c4 100644 --- a/awa/Cargo.toml +++ b/awa/Cargo.toml @@ -20,7 +20,7 @@ awa-macros.workspace = true awa-worker.workspace = true [dev-dependencies] -awa-testing = { path = "../awa-testing", version = "0.6.16" } +awa-testing = { path = "../awa-testing", version = "0.6.17" } sqlx.workspace = true serde.workspace = true serde_json.workspace = true diff --git a/docs/test-plan.md b/docs/test-plan.md index 36b0747fd..30e96fc3a 100644 --- a/docs/test-plan.md +++ b/docs/test-plan.md @@ -15,6 +15,7 @@ Tests run against real Postgres 15+ (not managed services) using a dedicated tes | T5 | Deadline rescue (hung job) | ✓ | | | T5a | Poison jobs (handler panic, crash loop, deadline loop) stop at `max_attempts` on both engines | ✓ | | | T5b | Rescue drains a backlog of abandoned claims in bounded passes without rescuing live claims, and the queue and claim rings then rotate past the pinned slots | ✓ | | +| T5c | Ring prune wins against overlapping short readers, sees a lease committed while it waits to truncate, and does not deadlock with rescue or a read-then-write transaction on the claim ring | ✓ | | | T6 | Transactional atomicity | ✓ | | | T7 | Transactional atomicity (Python) | | ✓ | | T8 | Uniqueness under contention | ✓ | ✓ | @@ -247,6 +248,7 @@ Concurrent lifecycle benchmark (1 queue × 128 workers, 20K jobs): | TLA12 | AwaStorageLockOrder | Postgres lock ordering across enqueue, new-lane provisioning, claim, complete, cancel, rescue, rotate, and prune | | TLA13 | AwaStorageTransition | Storage-transition prepare, mixed-entry, finalize, and abort gates | | TLA14 | AwaDeadTupleContract | Hot-table reclaim-kind and partition-truncate contract | +| TLA15 | AwaQueuePruneTwoPhase | Queue prune's writer lock, exclusive upgrade and recheck against a claim that commits during the upgrade | The formal suite includes passing configs and expected-counterexample configs. The expected-counterexample configs keep historical bugs executable: old dispatch claim, old view trigger, naive segment race, old storage-transition gate, shard-ignorant prune, and deliberate lock-order cycles. Trace configs use a `TraceIncomplete` invariant as a positive witness: a valid trace violates that invariant after TLC consumes every event. diff --git a/docs/upgrade-0.6.16-to-0.6.17.md b/docs/upgrade-0.6.16-to-0.6.17.md new file mode 100644 index 000000000..8b0beb39f --- /dev/null +++ b/docs/upgrade-0.6.16-to-0.6.17.md @@ -0,0 +1,82 @@ +# Upgrading from 0.6.16 to 0.6.17 + +0.6.17 is a rolling patch upgrade: the schema stays at v40 and traffic keeps running throughout. Claims commit asynchronously and reuse the runtime's claimer lease, promotion and rescue run in their own maintenance loop, a runtime can opt out of maintenance leadership, and queue prune no longer runs its checks under an exclusive lock. + +Binaries and the migration can roll in either order. One response shape changes for callers of the admin API and `admin::state_counts`. + +## Upgrade + +1. Install the 0.6.17 CLI. +2. Run `awa migrate` once with the schema owner as the effective role, while workers and producers keep running: + + ```bash + PGOPTIONS='-c role=awa_owner' \ + awa --database-url "$AWA_MIGRATOR_DATABASE_URL" migrate + ``` + + On a 0.6.16 database the output reports `Schema patch runtime_maintenance_eligibility: applied` and `Schema patch correlated_claim_tombstones: applied`. Each patch commits in its own transaction and neither takes a lock on an existing `awa` or queue-storage table. + +3. Grant runtime roles `SELECT, INSERT, DELETE` on `awa.runtime_maintenance_eligibility`. The default privileges in the [security guide](security.md) already cover it for roles set up that way. +4. Roll workers, producers and admin processes to 0.6.17, in any order. +5. Update callers of `/api/stats` and `admin::state_counts` for the [new shape](#state-counts-report-whether-available-is-an-estimate). + +0.6.16 binaries ignore the new table and run unchanged against the patched claim function. + +### External migration runners + +`awa migrate --sql --pending` prints `R__runtime_maintenance_eligibility` and `R__correlated_claim_tombstones` on a 0.6.16 database. Apply them as repeatable scripts after `V40`; `R__correlated_claim_tombstones.sql` must run as the owner of the claim functions. Extract the full set with: + +```bash +awa --database-url "$AWA_MIGRATOR_DATABASE_URL" migrate --extract-to ./sql/awa --from 40 +``` + +Replace all `R__` files together with that set and keep previously applied `V` files unchanged. + +## Claims commit asynchronously + +This part needs no migration and applies to each queue-storage runtime as it rolls. Queue-storage claim transactions and the claim-cursor advance that follows them run with `synchronous_commit = off`, set with `SET LOCAL` in those transactions only. Enqueues, completions and every other transaction stay synchronous, and so do claims on the canonical backend, including during a storage transition. + +What changes: + +- A claim acknowledged to a worker can be lost if the server crashes before its WAL is flushed, or if a standby is promoted before it received it. The job is then claimable again, which is at-least-once delivery as before. +- A worker still running a claim lost this way is not fenced by its run lease, because the job's next claim reuses it. See [architecture](architecture.md#enqueue-and-claim). +- Throughput rises where WAL flushes were the limit. Against 0.6.16 on PostgreSQL 18 with `synchronous_commit = on`, three interleaved rounds gave 19 to 21% more jobs per second at 4, 16 and 64 workers, and end-to-end p99 at 800 jobs/s fell from 31 ms to 24 ms. PostgreSQL used about a quarter more CPU for the extra work. + +A queue-storage claim also reuses the runtime's claimer lease for up to 250 ms instead of reading it on every call. No configuration changes either behaviour. + +## Promotion and rescue have their own loop + +The maintenance leader runs promotion and the three rescues in a loop of their own, so a slow cleanup, cron sync, rollup or rotation no longer delays them. A leader holds one more pool connection at a time. While a rescue pass runs, lease prune may report more `blocked` outcomes and claim prune may wait out a rescue batch. See [ADR-051](adr/051-maintenance-recovery-loop.md). + +## Opting a runtime out of maintenance leadership + +`ClientBuilder::maintenance_leader(false)` (Python `client.start(..., maintenance_leader=False)`) keeps a runtime from contending for leadership. It needs the `runtime_maintenance_eligibility` patch to report itself. Register `Rescued` hooks and enqueue specs on a runtime that may lead, because only the leader's registrations fire. See [leader election](configuration.md#leader-election). + +## Ring prune lock waits + +Queue prune takes `SHARE ROW EXCLUSIVE` on the sealed slot's partitions, runs its checks under it, and requests `ACCESS EXCLUSIVE` only for the `TRUNCATE`. For the queue and claim rings that request waits `prune_lock_timeout`, then retries with twice the wait up to 320 ms before the attempt counts as `blocked`. + +What changes for operators: + +- A reader of a sealed slot's partitions can wait up to 320 ms behind a contended prune, where it waited 10 ms. +- A prune that gets the lock on the queue ring no longer stalls readers for the duration of its checks. +- A deadlock that PostgreSQL resolves by cancelling a prune counts as `blocked` in `awa.maintenance.prune.attempts`, not `error`. +- Startup warns when 320 ms (or a longer `prune_lock_timeout`) reaches `deadlock_timeout`, or when eight times that reaches `statement_timeout`. Keep `deadlock_timeout` at its 1 s default or above. + +With a large backlog of abandoned claims, prune is still blocked in most attempts, as on 0.6.16; rescue has to drain the backlog first. + +## State counts report whether `available` is an estimate + +The per-queue `available` count reads at most 5,000 unclaimed rows per lane. A deeper lane reports its head-sequence depth and marks the count as an estimate. + +- `admin::state_counts` returns `StateCounts { counts, available_is_estimate }`, where it returned the state map. +- `/api/stats` returns `{ "counts": {...}, "available_is_estimate": bool }`, where it returned the bare state map. +- `QueueOverview` gains `available_is_estimate`; the UI shows `≈N` and `awa queue stats` shows `~N`. While a queue's count is an estimate, its `lag_seconds` is a lower bound. + +## Python handlers are cancelled when their attempt is abandoned + +A Python handler receives `asyncio.CancelledError` when the executor abandons its attempt after the deadline grace, or when shutdown exceeds the worker timeout. Release resources in `finally`. Cancellation is cooperative: it does not stop blocking code, threads or handlers that suppress it. Shutdown waits up to one more second for handlers to finish cancelling. + +## Rollback + +Roll binaries back to 0.6.16 in any order. Leave both patches in place: 0.6.16 ignores `awa.runtime_maintenance_eligibility` and runs against the patched `claim_ready_runtime`. Runtimes started with `maintenance_leader(false)` contend for leadership again once they run 0.6.16.