diff --git a/CHANGELOG.md b/CHANGELOG.md index 976ff54f3..afaaa314f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,201 +4,67 @@ Notable changes between releases. Detailed migration notes for storage transitio ## [Unreleased] -### Added +## [0.6.14-alpha.1] — 2026-11-07 -- **The queue-storage prune `lock_timeout` is configurable, and prune - attempts are counted by outcome for every ring.** Prune (queue, lease and - claim rings) and terminal-delta rollup take their exclusive locks under a - 10 ms `lock_timeout` that could only be changed through the hidden - `QueueStorage::with_prune_lock_timeout`. When short readers of the ring - partitions held a conflicting lock at that instant, prune reported - `blocked` every tick and the rings stopped recycling. Set it with - `ClientBuilder::prune_lock_timeout` (Rust) or - `queue_storage_prune_lock_timeout_ms` on `client.start()` (Python); the - timeout applies separately to each lock acquisition, not to the whole prune; - queue prune can wait on up to eight partition locks. The default stays 10 ms and zero or a value above PostgreSQL's `lock_timeout` - maximum is rejected. Start logs a warning when the value is at or above the - connection's `deadlock_timeout` (prune waiting that long can make deadlock - detection cancel a claim instead) or `statement_timeout`. `with_prune_lock_timeout` is now - public, with `set_prune_lock_timeout` and `prune_lock_timeout()`. - `awa_maintenance_prune_attempts_total{awa_ring,awa_ring_outcome,awa_ring_reason}` - gains `backed_off` (prune skipped by the backoff that follows consecutive - `skipped_active` or `blocked` results) and - `error` (prune failed with a database error) alongside the existing - `pruned`, `noop`, `blocked` and `skipped_active`. See - [Configuration](docs/configuration.md#what-the-knobs-mean). +Prerelease of the 0.6.14 queue-storage fixes. The schema remains v40. Apply the +repeatable `definer_lane_provisioning` patch once, then roll processes in any +order; binaries can roll back to 0.6.13 with the patch retained. See the +[upgrade guide](docs/upgrade-0.6.13-to-0.6.14.md). ### Fixed -- Reduce accumulated partition locks during job-detail reads, receipt-backed - lists and kind counts by releasing completed read phases within the same - snapshot. Job-detail reads retain the located ready body until they finish; - receipt lists and kind counts process retained claim slots separately. Kind - counts capture lane cursors before processing receipts. -- Preserve the original job body when manually retrying a priority-aged attempt - that exhausted its retries during rescue. Retained failed reads and failed-row - carry through pruning now use the physical ready-lane identity. - Hydration excludes tombstoned source lanes and verifies the prior run lease. - Rust terminal retry and failed-row carry also recover retained bodies written - by older workers with an effective priority in the terminal key. Manual - terminal retry restarts aging from the physical lane; automatic retryable - rescue and automatic DLQ routing retain the effective priority. Manual - terminal-to-DLQ moves preserve the recovered body and physical priority for - later DLQ retry, while decrementing the original terminal counter key. The installed `terminal_jobs` - SQL view still cannot hydrate those older mismatched-priority rows before - carry widens them; this change does not update that view or repair bodies - whose ready backing has already been pruned. - -- Admin job lists and kind counts include running receipt claims before - they materialize into leases, including compact and legacy row-local claims. -- Exclude tombstoned source rows when hydrating moved running jobs, including - compact receipts. Point reads retain the [concurrent-prune limitation](docs/troubleshooting.md#transient-admin-reads-during-partition-pruning). -- Close read connections if their transaction start is cancelled or fails before - SQLx acknowledges it, so later reads cannot inherit an untracked transaction. - -- Avoid repeated terminal-ledger probes when counting open deadline-backed - receipts, while preserving all closure and materialization exclusions. -- Reduce running-count query work by keeping compact closure membership - correlated and limiting row-local closure lookups to their claim partition. -- Hydrate completed job-kind counts through each member's full ready-row key, - avoiding repeated scans of other retained rows in the same enqueue lane. - -- **Ring prune and terminal-count rollup report `Blocked` when any of their - statements times out on a lock.** Prune sets a short `lock_timeout` (10 ms - by default) before it takes `ACCESS EXCLUSIVE` on a slot's partitions, but - only that `LOCK TABLE` and the final `TRUNCATE` turned a timeout into - `PruneOutcome::Blocked`. The timeout covers every later statement in the - transaction too: the post-lock liveness checks read the lease and claim - partitions, the failed-row carry inserts into the current slot's - `done_entries` and terminal-count deltas, and the fold writes - `queue_terminal_rollups` and deletes `queue_terminal_live_counts` rows. When - one of those waited out the timeout, for example behind autovacuum - truncating a relation or a hosted Postgres's own background workers, prune - failed with a `lock_not_available` (`55P03`) database error, was logged as - an error and skipped the prune backoff. A lock timeout anywhere in a queue, - lease or claim prune, or in a terminal-count rollup slot, now rolls the - whole transaction back and reports `Blocked` (`blocked_slots` for rollup), - so maintenance backs off and retries; nothing the attempt wrote survives - the rollback. A rollup pass stops at its first blocked slot rather than - waiting out the timeout again on each remaining one. -- **On queue storage, a queue whose enqueue lane is busy no longer holds - back promotion of other queues, nor the maintenance leader's rescue, - rotation and prune.** Promotion re-enqueues due scheduled and retryable - rows into their lane, which takes the lane's reservation lock that - producers hold until they commit. A promotion batch was one transaction - across every queue it touched, and the leader runs its branches one after - another, so a lane kept busy by producers delayed due jobs of unrelated - queues in the same batch, and rescue started seconds late. Promotion now - gives each `(queue, priority)` lane its own transaction under a 50 ms - `lock_timeout`. A lane that times out keeps its rows deferred and is - handed to a dedicated task that waits up to 30 s for the lock, queued - behind the producers already waiting, so the lane is still promoted in - turn however busy it stays; the leader skips it meanwhile. Each waiter - holds a pool connection, so the leader runs one per eight pool - connections, at least one and at most four; a further contended lane is - retried on the next tick. Batch selection skips rows another transaction - has locked, so locked due rows cannot fill every batch of a tick. `awa.maintenance.promote.lane_contention` - (`awa_maintenance_promote_lane_contention_total{awa_job_state,awa_job_queue,awa_promote_outcome}`) - counts `handed_off`, `waiter_limit` and `waiter_timed_out`. Promoted rows - now take lane sequence numbers in due order within their lane. - `QueueStorage::promote_due` keeps its signature and promotes lane by lane - without a lock timeout; `promote_due_lanes` and `promote_due_lane` expose - the per-lane form. -- **A queue lane is usable by every runtime role, whichever role creates it - first.** A lane's enqueue and claim sequences belonged to the role that - first enqueued into, claimed from or promoted into it, so with several - runtime roles sharing a group role, any role other than the creator whose - default privileges did not cover it failed with `permission denied for - sequence`. That included the maintenance leader promoting into a shard - created after `enqueue_shards` was raised. The new schema patch - `definer_lane_provisioning` creates lanes through two `SECURITY DEFINER` - functions per queue-storage schema, `provision_lane` and - `provision_lane_sequences`, owned by the schema owner. Each new sequence gets - the privileges of the head table it backs, explicitly rather than through - default privileges. A runtime without schema `CREATE` can therefore create - lanes itself, and `awa storage prepare-queue` is optional and may run as any - runtime role. Once the patch commits, `awa migrate` gives existing lane - sequences to the schema owner and adds their missing grants, one sequence - per transaction with a 300 ms lock wait, and only where it is allowed to, - trying the effective role and then the login role. A sequence an open - transaction holds is skipped and reported as busy; one it may not change is - reported with the roles that can repair it. Neither fails the run. The new read-only `awa storage lane-sequences` (or - `awa.lane_sequence_problems(schema)`) lists what remains, with the - statements to run and the roles to run them as, and exits 2 while any - remain; `--repair` (or `awa.repair_lane_sequence(schema, sequence)` per - sequence) repairs what the connected role may. `awa migrate` prints a summary of the leftovers. A binary - running before the patch detects that `provision_lane` is missing and - creates lanes as before, and older binaries create correctly owned lanes once - the patch has refreshed the helpers they call. `awa migrate` applies the - patch on any v040 database; external runners apply the exported - `R__definer_lane_provisioning.sql` after `V40`. See - [`docs/upgrade-0.6.13-to-0.6.14.md`](docs/upgrade-0.6.13-to-0.6.14.md). -- **A graceful shutdown no longer leaves a finished attempt for stale-heartbeat - rescue to run again.** Once a handler returned, its finalize (completion, - retry, cancel or failure, and any ADR-029 follow-up inserts) ran in a task - that `Client::shutdown` did not wait for. When the last handler returned - just as the drain ended, shutdown stopped the completion batcher before - that finalize reached it, and the direct fallback raced the process exit. - The attempt had already left the heartbeat set, so it went stale and was - rescued and executed again `heartbeat_staleness` later; a deploy could - re-run a few jobs on every rolling restart. Shutdown now waits for - pending finalization, within the same drain timeout, before it stops the - completion batcher, heartbeat and maintenance services. A finalize that - outlives the timeout is still left to rescue, as before. -- **Point reads show the live queue and priority after a lane move.** Retained - ready rows from before a priority or queue change could win a read's tie - after the ring wrapped, returning the old fields even though the batch - operation succeeded. Point reads now exclude tombstoned source rows. - -- **Callback completion, `get_job`, lifecycle events and the admin reads - find running jobs that priority aging promoted on queue storage** (backport - of #600). Claims and leases record a job's aged effective priority while its - ready row keeps the lane priority, and eight reads joined the two on - priority. With aging on (the default, every 60 s), a job that waited past a - minute before it was claimed could not be completed by callback with an - expected kind, failed the mixed-transition callback routing check, read back - from `get_job` in the wrong state, was missing from the job list, the kinds - page and the dashboard's recent-jobs counts, and was not found by - `cancel_by_unique_key`. They now match the attempt (`job_id` and `run_lease`). No - migration. -- **`get_job` no longer misses a job that moves while it reads.** It read - ready, deferred, leased, claimed, completed and DLQ rows in separate - statements, so a job promoted, claimed or completed between two of them - could read as missing. It now reads all of them from one read-only snapshot. -- **`admin::job_kind_overviews` on queue storage no longer depends on the - planner's row estimates.** Behind the admin UI's kinds page, it counted - every job through the `current_jobs` view. Two nodes of that view are - estimated at one row whatever the statistics say: the join from completion - batches to `ready_entries`, and the `UNION ALL` over every place a job can - live. Postgres therefore chose nested loops that visit each retained row, - and the join on `queue_claim_heads` was only as good as that table's - estimate, which autovacuum never refreshes. It now aggregates each place to - `(kind, queue)` on its own and combines the results: available jobs are - found per lane from the claim cursor, and a completion batch's members are - unnested and probed along the ready primary key. The rows are identical, - and a test compares them with the former definition across every job - state. With 2.6 million retained ready rows (1.8 million of them available) - and 860,000 retained completions it takes about 6 to 8 s instead of 20 s on - PostgreSQL 17 and 18. It still reads each available job and each retained - completion's ready row, because a job's kind lives there. JIT is off for - the read. -- **`admin::list_jobs` on queue storage reads only the places that can hold - the requested state.** It selected candidate ids from the `current_jobs` - view, so every listing, whatever its filter, planned around the same - single-row estimates and joined each retained completion to a scan of the - ready rows. On PostgreSQL 18 a kind filter ran past a four-minute statement - timeout. Each place a job can live is now read and limited on its own, - only those that can hold the state asked for, with a predicate for each - filter that is set rather than `$n IS NULL OR` terms, which a prepared - statement's generic plan can't drop. Completion batches are ordered by job - id before their ready rows are probed, so a page probes only as many as it - returns. The ids are identical, and a test compares them with the former - definition for every state, kind, queue, limit and page. On the dataset - above, `state=failed&limit=10` drops from 12 s to 0.7 s, - `state=completed&limit=100` from 20 s to 6 s, and an unfiltered or - kind-filtered page of 100 from 23 s to 11 s (a kind filter on PostgreSQL 18 - from more than four minutes). Available jobs and completions are still read - in full when the state allows them. JIT is off for the read. +- New queue lanes and their sequences belong to the schema owner and receive + the head table's grants, regardless of which runtime creates them. Migration + also repairs existing lanes where permitted and reports busy or unauthorized + sequences for a later repair. +- Promotion isolates contended queue/priority lanes so they cannot stall other + queues or maintenance. Lock timeouts anywhere in prune or terminal rollup + roll back the attempt and report `Blocked`. +- Graceful shutdown waits for pending finalization before stopping heartbeat + and maintenance, subject to the existing drain timeout. +- Reads and callbacks identify priority-aged and moved attempts by job ID and + run lease, excluding tombstoned source rows. Admin lists and kind counts also + include running receipt claims that have not materialized into leases. +- Failed terminal writes preserve the live ready body's arguments, attempt limit and + physical lane. Rust retry and retention carry recover older narrow terminal + bodies while their backing remains retained. Stored terminal metadata wins + over ready defaults, and counters use the correct stored and replacement + priorities. +- Manual terminal retry restarts aging from the physical lane. Manual + terminal-to-DLQ moves preserve that lane for a later DLQ retry. Automatic + retryable rescue and automatic DLQ routing retain effective aged priority. +- Read connections close if BEGIN is cancelled before acknowledgement, so later + callers cannot inherit an open transaction and its partition locks. +- Job details, receipt-backed lists and kind counts release completed read + phases within one snapshot, reducing retained partition locks. Job details + retain the located ready body until the read finishes. +- Admin list and kind queries reduce repeated work when reading retained history and retain + their state, queue, kind and pagination semantics. + +### Added + +- Configurable prune lock timeout in Rust and Python, with prune outcome + counters. The timeout applies to each acquisition; queue prune can wait on + up to eight partition locks. The default remains 10 ms. +- Multi-runtime soak coverage for killed workers, shard changes, mixed Rust and + Python workloads, durable recovery evidence and ring-lock diagnostics. + +### Changed + +- Python CI tests 3.14 and 3.15. Older interpreter versions remain best-effort; + package classifiers list 3.14 and 3.15. +- Operator guidance explains that longer rotation intervals reduce churn while + increasing rows and index size per slot and extending retained history. + +### Known limitations + +- Prune can truncate a ready partition before a job-detail read locates its + body, or truncate a lease/claim partition before that read phase reaches it. + A completed job can briefly appear available or missing; a fresh read can + recover the current result. +- The installed `terminal_jobs` SQL view and creation-time bucket joins still + use the stored priority for older mismatched rows. Retention carry can widen + surviving failed rows, but already-pruned backing bodies cannot be rebuilt. ## [0.6.13] — 2026-11-06 diff --git a/Cargo.lock b/Cargo.lock index 4386517ed..bc51013c4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -383,7 +383,7 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "awa" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "async-trait", "awa-macros", @@ -409,7 +409,7 @@ dependencies = [ [[package]] name = "awa-cli" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "assert_cmd", "awa-model", @@ -431,7 +431,7 @@ dependencies = [ [[package]] name = "awa-macros" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "proc-macro-crate", "proc-macro2", @@ -441,7 +441,7 @@ dependencies = [ [[package]] name = "awa-metrics" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "awa-model", "opentelemetry", @@ -450,7 +450,7 @@ dependencies = [ [[package]] name = "awa-model" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "awa-macros", "blake3", @@ -471,7 +471,7 @@ dependencies = [ [[package]] name = "awa-seaorm" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "awa", "awa-testing", @@ -484,7 +484,7 @@ dependencies = [ [[package]] name = "awa-testing" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "async-trait", "awa-model", @@ -499,7 +499,7 @@ dependencies = [ [[package]] name = "awa-ui" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "awa-metrics", "awa-model", @@ -522,7 +522,7 @@ dependencies = [ [[package]] name = "awa-worker" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "anyhow", "async-trait", diff --git a/Cargo.toml b/Cargo.toml index eba7da12b..8119bcf96 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,7 +14,7 @@ members = [ exclude = ["awa-python", "spike"] [workspace.package] -version = "0.6.13" +version = "0.6.14-alpha.1" edition = "2021" license = "MIT OR Apache-2.0" description = "Postgres-native background job queue — transactional enqueue, heartbeat crash recovery, SKIP LOCKED dispatch" @@ -25,11 +25,11 @@ categories = ["database", "asynchronous", "web-programming"] [workspace.dependencies] # Internal crates -awa-model = { path = "awa-model", version = "0.6.13" } -awa-macros = { path = "awa-macros", version = "0.6.13" } -awa-metrics = { path = "awa-metrics", version = "0.6.13" } -awa-worker = { path = "awa-worker", version = "0.6.13" } -awa = { path = "awa", version = "0.6.13" } +awa-model = { path = "awa-model", version = "0.6.14-alpha.1" } +awa-macros = { path = "awa-macros", version = "0.6.14-alpha.1" } +awa-metrics = { path = "awa-metrics", version = "0.6.14-alpha.1" } +awa-worker = { path = "awa-worker", version = "0.6.14-alpha.1" } +awa = { path = "awa", version = "0.6.14-alpha.1" } # Database sea-orm = { version = "=2.0.0-rc.38", default-features = false, features = ["sqlx-postgres", "runtime-tokio-rustls", "with-chrono", "with-json"] } diff --git a/awa-cli/Cargo.toml b/awa-cli/Cargo.toml index 14f235f7b..fb822a8a5 100644 --- a/awa-cli/Cargo.toml +++ b/awa-cli/Cargo.toml @@ -14,7 +14,7 @@ path = "src/main.rs" [dependencies] awa-model.workspace = true awa-worker.workspace = true -awa-ui = { path = "../awa-ui", version = "0.6.13" } +awa-ui = { path = "../awa-ui", version = "0.6.14-alpha.1" } axum.workspace = true sqlx.workspace = true tokio.workspace = true @@ -28,5 +28,5 @@ hex.workspace = true uuid.workspace = true [dev-dependencies] -awa-testing = { path = "../awa-testing", version = "0.6.13" } +awa-testing = { path = "../awa-testing", version = "0.6.14-alpha.1" } assert_cmd = "2" diff --git a/awa-cli/pyproject.toml b/awa-cli/pyproject.toml index 7e4f95bbf..595032fd6 100644 --- a/awa-cli/pyproject.toml +++ b/awa-cli/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "maturin" [project] name = "awa-cli" -version = "0.6.13" +version = "0.6.14-alpha.1" description = "CLI for the Awa Postgres-native job queue (migrations, admin, serve)" readme = "README.md" requires-python = ">=3.10" diff --git a/awa-python/Cargo.lock b/awa-python/Cargo.lock index 8922f4d9a..668ebb099 100644 --- a/awa-python/Cargo.lock +++ b/awa-python/Cargo.lock @@ -95,7 +95,7 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "awa-macros" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "proc-macro-crate", "proc-macro2", @@ -105,7 +105,7 @@ dependencies = [ [[package]] name = "awa-metrics" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "awa-model", "opentelemetry", @@ -113,7 +113,7 @@ dependencies = [ [[package]] name = "awa-model" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "awa-macros", "blake3", @@ -133,7 +133,7 @@ dependencies = [ [[package]] name = "awa-python" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "async-trait", "awa-model", @@ -156,7 +156,7 @@ dependencies = [ [[package]] name = "awa-worker" -version = "0.6.13" +version = "0.6.14-alpha.1" dependencies = [ "async-trait", "awa-macros", diff --git a/awa-python/Cargo.toml b/awa-python/Cargo.toml index 373a48e8e..f929fae01 100644 --- a/awa-python/Cargo.toml +++ b/awa-python/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "awa-python" -version = "0.6.13" +version = "0.6.14-alpha.1" edition = "2021" [lib] @@ -12,8 +12,8 @@ default = ["cel"] cel = ["awa-model/cel"] [dependencies] -awa-model = { path = "../awa-model", version = "0.6.13" } -awa-worker = { path = "../awa-worker", version = "0.6.13", features = ["__python-bridge"] } +awa-model = { path = "../awa-model", version = "0.6.14-alpha.1" } +awa-worker = { path = "../awa-worker", version = "0.6.14-alpha.1", features = ["__python-bridge"] } pyo3 = { version = "0.29", features = ["macros", "abi3-py310", "chrono"] } pyo3-async-runtimes = { version = "0.29", features = ["tokio-runtime"] } sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "chrono", "json"] } diff --git a/awa-python/pyproject.toml b/awa-python/pyproject.toml index 01ed5cf11..8914445c0 100644 --- a/awa-python/pyproject.toml +++ b/awa-python/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "maturin" [project] name = "awa-pg" requires-python = ">=3.10" -version = "0.6.13" +version = "0.6.14-alpha.1" description = "Postgres-native background job queue — Python SDK with async/sync workers, transactional enqueue, and progress tracking. Install awa-pg[ui] to add the bundled web dashboard." readme = "README.md" license = {text = "MIT OR Apache-2.0"} @@ -34,7 +34,7 @@ classifiers = [ # # Kept opt-in so the default `awa-pg` install stays small — workers and # producers don't need the ~10 MB UI bundle. -ui = ["awa-cli==0.6.13"] +ui = ["awa-cli==0.6.14-alpha.1"] [project.urls] Homepage = "https://github.com/hardbyte/awa" diff --git a/awa-python/uv.lock b/awa-python/uv.lock index 8b3cbff7a..240f52bd5 100644 --- a/awa-python/uv.lock +++ b/awa-python/uv.lock @@ -88,12 +88,12 @@ wheels = [ [[package]] name = "awa-cli" -version = "0.6.13" +version = "0.6.14a1" source = { directory = "../awa-cli" } [[package]] name = "awa-pg" -version = "0.6.13" +version = "0.6.14a1" source = { editable = "." } [package.optional-dependencies] diff --git a/awa-seaorm/Cargo.toml b/awa-seaorm/Cargo.toml index 781c95ab9..86a6f6688 100644 --- a/awa-seaorm/Cargo.toml +++ b/awa-seaorm/Cargo.toml @@ -14,6 +14,6 @@ serde_json.workspace = true sqlx.workspace = true [dev-dependencies] -awa-testing = { path = "../awa-testing", version = "0.6.13" } +awa-testing = { path = "../awa-testing", version = "0.6.14-alpha.1" } serde = { workspace = true, features = ["derive"] } tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } diff --git a/awa/Cargo.toml b/awa/Cargo.toml index 23ba05b62..e2971be51 100644 --- a/awa/Cargo.toml +++ b/awa/Cargo.toml @@ -20,7 +20,7 @@ awa-macros.workspace = true awa-worker.workspace = true [dev-dependencies] -awa-testing = { path = "../awa-testing", version = "0.6.13" } +awa-testing = { path = "../awa-testing", version = "0.6.14-alpha.1" } sqlx.workspace = true serde.workspace = true serde_json.workspace = true diff --git a/docs/configuration.md b/docs/configuration.md index e59a1e665..a93285501 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -585,6 +585,19 @@ Use the defaults unless you have a reason not to: - Increase rotation intervals to reduce partition churn and metadata activity. - Decrease rotation intervals to tighten dead-tuple bounds at the cost of more frequent rotate/prune work. +Longer rotation intervals accumulate more work in each slot. At a steady enqueue +rate, ready rows per queue slot are roughly the enqueue rate multiplied by +`queue_rotate_interval`; retries and blocked recycling can increase the retained +footprint. Budget for larger partitions and indexes. Queue-ring history becomes +coarser and generally spans about `queue_slot_count × queue_rotate_interval`; +active or blocked slots can retain it longer. The independent +`claim_rotate_interval` changes claim-evidence retention, not completed-job +history in the ready/terminal ring. + +A longer interval does not remove the locks required by each prune. +`prune_lock_timeout` still applies separately to each acquisition; queue prune +can lock up to eight partitions before its later reads and writes. + ### Sharding the enqueue head per queue Each queue's `awa.queue_meta.enqueue_shards` (default `1`, range `1..=64`) governs how many independent enqueue head rows the queue has. With the default value, the queue uses a single head row and the ordering contract is strict FIFO per `(queue, priority)` — identical to v016. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 083ffa69b..bdb9a1d3d 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -96,6 +96,21 @@ appear available or missing. Retry with a fresh read before treating that result as missing durable work. Filtering tombstoned source rows after a queue or priority move does not close these separate windows. +### Failed Jobs Rescued After Priority Aging + +0.6.14 preserves the physical ready-lane identity when rescue writes a failed +terminal row, so manual retry, failed-row carry and `terminal_jobs` retain its +original arguments and retry limit. Rust reads, manual retry and failed-row +carry also recover older narrow terminal rows by attempt identity while their +ready backing remains retained. These paths preserve terminal metadata and +fill missing defaults from the live ready row. The installed `terminal_jobs` +SQL view and creation-time bucket joins remain unchanged for older rows whose +stored priority differs from their ready lane. Already-pruned backing data +cannot be recovered by this fix. Manual terminal retry restarts aging from the +physical lane. Manual terminal-to-DLQ moves retain that lane for later DLQ +retry; automatic retryable rescue and automatic DLQ routing retain effective +priority. + ### Inspect Runtime Health ```sql