Repository navigation
chore: prepare 0.6.10 release - #582
Merged
Merged
Conversation
…sequences (0.6 schema patch) Backport of #501 to the 0.6 line. The lane helpers and head-sync triggers ran CREATE SEQUENCE IF NOT EXISTS even when the sequence existed, which requires schema CREATE and defeats the documented migrator/runtime role split. They now skip the DDL when the sequence exists and raise SQLSTATE 42501 with a provisioning hint when it does not and the role lacks schema CREATE. 0.6 cannot add migration versions, so the change ships as the idempotent schema patch prepared_lane_sequences: it redefines awa.install_queue_storage_substrate and re-runs it for every installed queue-storage schema with its existing slot counts and receipt mode (lease_claims or lease_claim_batches in claim_ready_runtime). Fresh installs pick up the helper through v023. v023 is split in two files so the patch embeds only the helper definition: the default-schema installer, legacy fixups and schema_version row stay in v023_install_default_queue_storage_substrate.sql, which V23_UP still appends. The patch lint now skips the installer body, whose guarded ALTER/DROP TABLE statements run only when the function is called. Add QueueStorage::prepare_queue and `awa storage prepare-queue` to provision all priorities, shards and stripes as the migrator.
…etained history Backport of #580 and #581 to the 0.6 line. On queue storage, admin::queue_overviews, state_counts and state_timeseries built every current job through queue_storage_current_jobs_cte, whose terminal_jobs branch expands each completion batch and joins it back to ready_entries, so their cost grew with retained history. One production deployment saw queue_overviews take 22 s, polled by an open admin dashboard. queue_state_counts is not maintained once queue storage is active, so it can't stand in. - queue_overviews reads the live-work tables the worker gauges read, counts available rows exactly per lane from its claim cursor, and takes completed_last_hour from completion-batch sizes (less tombstoned members) and completed done_entries rows. - state_counts counts completions from batch sizes less tombstones, and available rows per lane. - state_timeseries bounds every source by the window before any join; terminal rows come only from done rows and batches finalised within it. - JIT is off for all three; compiling cost more than running. Running keeps 0.6's leases-only definition.
hardbyte
added
the
full-ci
Claude
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7ca816325
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Lane cursors need UPDATE on their sequences for setval; the SQL grant block already said so, the pgroles manifest did not.
Rust 1.99's clippy flags the must_use that async-trait 0.1.89 puts on the boxed future of Worker::perform. main already locks 0.1.92, whose expansion passes; the 0.6 lockfiles follow it.
ubuntu-latest now resolves to 26.04, where the pinned Playwright refuses to install Chromium. main pins the job the same way.
…ctions Backport of #583. When a client gives up on an admin request, the server drops it but Postgres keeps running its query, and sqlx holds the connection until the query ends. If the query then ended in an error, sqlx's release check closed the connection with a hard close, which hangs on a TLS connection, so the pool lost that slot for good (sqlx 0.8.6 here, as 0.9 on main). awa serve now builds its pool with awa_ui::admin_pool_options and admin_connect_options: a 60 s statement timeout as a startup option (--statement-timeout / AWA_STATEMENT_TIMEOUT), a release hook that reads an abandoned query's result and keeps the connection, and client_connection_check_interval = 5s on Postgres 14+.
…e patch The 0.6.10 prepared_lane_sequences patch re-ran awa.install_queue_storage_substrate for every installed queue-storage schema. The installer takes exclusive locks on hot tables such as leases and deferred_jobs, rebuilds the head-sync triggers and scans retained ready entries for backfills, all inside the migration transaction. Under live load on a v0.6.9 database it stalled every worker and producer for 23 s and deadlocked one enqueue. The three lane helpers (ensure_lane_sequences and the two head-sync trigger functions) now live in awa.refresh_lane_sequence_helpers(schema), as on main. The installer calls it, so new schemas get the fixed helpers. On existing databases the patch calls only the refresh, under the per-schema install advisory lock. It takes no table locks and reads no queue rows: the same live rehearsal applied it in 22 ms with no completion gap and no errors. The applied probe now checks all three helpers and the refresh function. A new test asserts the patch holds no relation lock on queue-storage objects, and that the installer does.
…backport of #584) A review of the #580/#581 backport found the last-hour completed count scanning completion batches once per recent tombstone (5.4 s on a production-sized fixture), and queue lag reading each lane's first row, which hides an older job a queue move or priority aging appends behind newer ones. - Available counts walk each lane's segments from its claim cursor and probe the ready key per segment; a range without the key's leading ring slot scanned every retained ready row. They also yield the oldest available run_at, which is the queue lag again, as in 0.6.9. - The tombstone check expands each recent batch in a tombstoned lane once and matches its members by hash. - The timeseries reads its window's terminal rows with ready-key probes instead of through terminal_jobs, which joined them to every ready row. - DLQ retry, purge and move invalidate the dashboard caches, as job mutations already do (from #579). admin_read_cost_test serves every polled endpoint against ten times more retained history and fails if a read does more work; on the previous head it fails for /api/queues and the timeseries.
…ed ready rows (backport of #586) On queue storage, cancel_by_unique_key and cancel_by_unique_key_tx built every available ready row before filtering by key, reading every retained ready row, and missed jobs running on receipt claims not yet materialised into leases, so those couldn't be cancelled by key (#585). The key is now filtered where each source is read: available rows are walked lane by lane from the claim cursor through ready_segments (only the given queue's lanes when a queue is passed), scheduled and retryable rows use the deferred unique-key index, and running jobs come from leases and open receipt claims. open_receipt_running_claims_sql is ported from main, with the closed-claim-batch skip. The queue-storage tests from #586 cover each live state, oldest-first among jobs sharing a key, the tx variant's rollback and commit-time notice, and rows read staying flat as history and other queues' backlogs grow. Against the previous lookup they fail: the running job isn't found, and a scoped missing-key lookup reads 720 rows, then 5,206.
Port of the #586 follow-up: enter the callback wait so the test cancels a job in waiting_external, and note that the open-receipt-claim leg costs claim-ring retention.
…has it The prepared_lane_sequences probe checked the three helpers in awa only. A custom queue-storage schema restored or prepared with the 0.6.9 helpers passed once awa did, and a database whose only active substrate was a custom schema could never pass. The probe now checks the refresh function, the installer and the helpers of every installed substrate, from the catalogs so a schema the migrating role cannot use is reported. migrations::schema_patch_status names what each missing patch still has to repair, and migrations::run_with_report returns the schema versions, the migrations and patches applied, what each patch repaired or still lacks, and storage routing before and after.
- Refuse --from, --to and --version when applying; they only select a range to render, and the apply path always migrates fully. - Wrap --sql output in one transaction that takes the migration advisory lock, with ON_ERROR_STOP guidance in the header; --no-transaction prints it bare for runners that own the transaction. - Report the result: schema version, each patch and what it updated, and storage routing before and after. Exit 1 when a patch is still missing on some schema, naming it. Tests cover the refused flags, a patch interrupted mid-run (nothing applied, retry completes it) and exported SQL that fails under psql with ON_ERROR_STOP and then applies cleanly.
A runtime role with schema CREATE owns the lane sequences it creates. Provisioning such a lane as the owner (awa storage prepare-queue, the first step of revoking runtime CREATE) failed with permission denied for sequence: the head tables' BEFORE INSERT triggers run even for a row ON CONFLICT DO NOTHING then skips, and they read and set the lane's sequence. Lane inserts now skip rows that exist, under the same per-lane advisory lock.
Released 0.6.9 artifacts (the awa-cli and awa-pg wheels) install the schema as an owner role and run a separate runtime role in a custom queue-storage schema with the grants a working 0.6.9 deployment needs. Over a banked backlog with retries, scheduled jobs and slow jobs in flight, the candidate awa migrate patches under traffic, with no lock above RowExclusive on any awa or queue-storage relation (one owner transaction audit plus sampling during the real run), idle lanes at their exact positions, live lanes only advancing, and routing unchanged. Candidate and released workers then run together through killed and restarted workers; the fleet rolls back to 0.6.9 (including its awa migrate, which must leave the patch in place); privileges are tightened as the owner and the released workers restart. Every job must complete, retried jobs on a later attempt, with per-queue counts matching what was enqueued. CI runs it on PostgreSQL 17 and 18 alongside the other full-CI jobs; release.yml already requires this workflow to pass on the tagged SHA. Evidence is uploaded as JSON.
One path: install the 0.6.10 CLI, run awa migrate once as the owner under traffic, roll processes. States what does not need doing, what the report means, how external runners apply the patch, how to roll back (leave the patch installed), and the optional procedure for revoking runtime CREATE, with a per-sequence grant because GRANT ON ALL SEQUENCES fails on sequences a 0.6.9 runtime created.
cargo runs the rehearsal from awa/, so AWA_CANDIDATE_CLI=target/debug/awa pointed at awa/target. Resolve it, and a relative CARGO_TARGET_DIR, from the workspace root as the released-artifact paths already are.
The 0.6.10 lookup read every open receipt claim before filtering by key. A long-running job keeps the claim ring from pruning, so on a database with 120,000 retained row-local claims each lookup took about 9.7 s, worse than 0.6.9's 2.8 s on the same data. Given a queue, the lookup now reads that queue's ready rows holding the key through the lane index and checks only those jobs: available past the claim cursor and not tombstoned, in leases, or on an open receipt claim found by job_id. With no such row only the deferred_jobs key index can hold the key, so that query alone answers, which also skips planning the full lookup across partitioned tables (about 10 ms). Measured on 180,200 retained ready rows and 120,300 claims in other queues, through cancel_by_unique_key_tx rolled back: 1.7 ms for a missing key, 11 ms and 20 ms to cancel a scheduled and an available job (p50 of 100); 0.6.9's lookup took 2.8 s. The cost test now also retains row-local claims; against the previous lookup it reads 434 rows, then 32,912.
Review follow-ups on the queue-scoped unique-key lookup: - Match leases by job_id, as open receipt claims already are. A job moved to another queue while available is leased on its new lane, so matching by its original ready row's lane missed it once it ran or waited there; 0.6.9 found it. - Read the ready-row check and the deferred_jobs lookup in one statement, so a job promoted from deferred_jobs between two reads is not missed by both. - Say what the compact-batch leg of the job-restricted open-claims query reads: every retained batch header, unnesting only batches that hold one of the jobs. New test: two jobs moved to another queue, one running on a receipt claim and one waiting on a callback, are both found by a lookup naming their original queue.
The runtime page showed the card on every cluster running queue storage, long after the cutover finished. It now appears only while an engine is prepared or a mixed transition is in flight.
This was referenced Nov 3, 2026
test_failure_path_metrics_reach_prometheus timed out waiting for its three completed jobs: with the default one-second ring rotation, maintenance can prune completed rows before the test counts them. The failure-path and dashboard tests, which both count completed jobs, now build their clients with a one-hour queue and lease rotation, as #588 did for the job output tests on main.
Owner
Author
|
Generated by Claude Code |
… its worker The rehearsal failed on PostgreSQL 17 with "fail_once job 331 never retried". A worker killed after claiming a fail_once job but before its handler returns leaves no Retry event; heartbeat rescue then runs attempt 2, which completes as designed. The accounting now requires a Retry event only when attempt 1 started on a worker that survived, and still requires every fail_once job to complete on attempt 2 or later.
Owner
Author
|
A worker can be killed after claiming a
Generated by Claude Code |
…key cancel The admin store is rebuilt from the active schema name with the default config, so a queue-scoped unique-key lookup bound only the logical queue name and missed available, running and waiting jobs in a striped queue's queue#N lanes. The physical queues now come from the schema's queue_lanes. A job that finishes between the lookup and the cancel made the queue-storage branch return JobNotFound; it now returns None, as the canonical engine does and the docs say. Both predate 0.6.10. Port of the same fixes on main (#586); the new striped-lanes test fails on the previous lookup.
…k, release date A runner that tracks only schema versions sees nothing new in 0.6.10, and the exported patch skips queue-storage schemas the running role cannot use without failing. The upgrade guide now says to add the R__ script, run it as the owner and confirm with awa migrate --sql --pending. The CHANGELOG links the guide absolutely so the link works in the GitHub release body.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepare the 0.6 maintenance line for v0.6.10.
admin::queue_overviews,state_countsandstate_timeseriesbuilt every current job throughterminal_jobs, which expands each completion batch and joins it back toready_entries. Their cost therefore grew with retained history. One production deployment saw the queue overview take 22 s, re-run by an open dashboard.terminal_jobs.admin_read_cost_testserves every endpoint the UI polls against ten times more retained history and fails if a read does more work. On the head before the Keep admin dashboard reads off retained history, with a guard test #584 port it fails for/api/queuesand the timeseries.runningkeeps 0.6's leases-only definition.cancel_by_unique_key/cancel_by_unique_key_txon queue storage, backported from Find unique-key cancel candidates without scanning retained ready rows #586 for Avoid queue-storage scans in cancel_by_unique_key candidate lookup #585.The candidate lookup built every available ready row before filtering by key, so it read every retained ready row. It also missed jobs running on receipt claims not yet materialised into
leases, which therefore couldn't be cancelled by key.Given a queue, it now starts from that queue's ready rows holding the key (lane index) and checks only those jobs: available past the claim cursor and not tombstoned, in
leases, or on an open receipt claim found byjob_id. With no such row, only thedeferred_jobskey index is read.Without a queue it walks available rows lane by lane from the claim cursor and reads open receipt claims in full (Queue storage: cancel_by_unique_key without a queue still reads every retained receipt claim #587).
Benchmark (
d87459ac). Setup: 180,200 retained ready rows and 120,300 retained row-local claims in other queues, plus 300 long-running open claims; timed throughcancel_by_unique_key_tx, rolled back, p50 of 100.The cost test now retains row-local claims; the first 0.6.10 lookup fails it (434 → 32,912 rows read).
A job moved to another queue while available isn't found by a lookup naming its original queue; cancel it by id.
main, andpool_limits_testfails here without the fix.awa servenow builds its pool withawa_ui::admin_pool_options/admin_connect_options:--statement-timeout,AWA_STATEMENT_TIMEOUT);client_connection_check_intervalon Postgres 14+.chore/release-0.6.10on 2026-10-06 and never PR'd.prepared_lane_sequences, plusawa storage prepare-queue/QueueStorage::prepare_queue.UPDATEgrant.awa.install_queue_storage_substratefor every queue-storage schema. That holds exclusive locks on hot tables (includingleasesanddeferred_jobs) until commit, rebuilds the head-sync triggers and scans retained ready entries. The helpers now live inawa.refresh_lane_sequence_helpers(schema), as onmain(v049).prepared_lane_sequencesprobe looked at the three helpers inawaonly. A custom queue-storage schema restored or prepared with the 0.6.9 helpers was missed onceawapassed, and a database whose only active substrate was a custom schema could never pass. The probe now checks the installer,awa.refresh_lane_sequence_helpersand every substrate's helpers, from the catalogs, so a schema the migrating role cannot use is still reported.migrations::schema_patch_statusnames what each missing patch still has to repair. Regression test:awapatched, a custom schema with the old helpers, and a third schema the migrator cannot use.awa migratefixes.--from/--to/--versionwhen applying; it used to ignore them and migrate fully.--sqloutput is one transaction holding the migration advisory lock, withON_ERROR_STOPguidance;--no-transactionprints it bare.migrations::run_with_report), and exits 1 naming any schema a patch could not reach.psql -v ON_ERROR_STOP=1leaves nothing applied and then applies cleanly.awa storage prepare-queueworks on lanes a 0.6.9 runtime created. Found by the rehearsal. A runtime with schemaCREATEowns the lane sequences it creates. Provisioning such a lane as the owner failed withpermission denied for sequence, because the head tables'BEFORE INSERTtriggers run even for a rowON CONFLICT DO NOTHINGthen skips. Lane inserts now skip rows that exist, under the same per-lane advisory lock. Regression test included.patch_upgrade_rehearsal_test, a new CI job on PostgreSQL 17 and 18.release.ymlalready requiresci.ymlto pass on the tagged SHA. See "Upgrade rehearsal" below.docs/upgrade-0.6.9-to-0.6.10.md:CREATE, with a per-sequence grant, sinceGRANT ... ON ALL SEQUENCESrun by the owner fails on runtime-owned sequences.async-traitgoes to 0.1.92: Rust 1.99's clippy rejects 0.1.89's expansion withdouble_must_use. (ci(release): take release notes from the CHANGELOG.md section for the tag (0.6) #496 carried the same bump ontorelease/0.6.6.)ubuntu-24.04, because Chromium won't install on 26.04.main).main.Left out on purpose:
feat(migrations): record schema patches in the awa.schema_patches ledger(54f0bc6aon that branch). Its message says 0.7 creates the same table from a numbered migration, butmainhas noschema_patchestable. A 0.6.10 database would therefore carry a table that 0.7 knows nothing about. It can ship oncemainhas a matching migration.No new migration version. The admin fixes need no migration: only the process serving the admin UI, or calling the admin API, has to run 0.6.10 to get them. The one API addition is the new
awa_ui::poolhelpers.Measurements
Admin reads
The data is 50,000 compact completions plus 50,000 queued jobs on this branch:
queue_overviews(awa queue stats)state_countsstate_timeseries(60 min, every completion in the window)The v0.6.9 figure comes from running the v0.6.9
awabinary'squeue statsagainst the same database. It was killed at 300 s, and its server-side query was still running ten minutes later. These timings predate the #584 port.admin_read_cost_test, tuples read per request with small and then ten times more retained history:/api/queues/api/stats/timeseries/api/statsA queue-scoped
cancel_by_unique_keylookup of a missing key read 720 rows, then 5,206 with ten times the retained completions. It now stays flat in both that case and beside another queue's 5,000-job backlog.awa migrateunder live loadSetup:
pg_locksevery 50 ms.awa migrate)After the run:
awa migratereports nothing pending.R__prepared_lane_sequences.sqlapplied twice with psql to a fresh v0.6.9 database is also clean, andawa migratethen finds nothing pending.Upgrade rehearsal
awa/tests/patch_upgrade_rehearsal_test.rsstarts from released artifacts only, theawa-cliandawa-pg0.6.9 wheels from PyPI.prepare-queue-storage-schema,prepare,enter-mixed-transition --quiesced,finalize).CREATEincluded). They bank a backlog on one queue and run retries, jobs scheduled 45 s out, and 1.5 s jobs.awa migrate --sql --pendingruns in an owner transaction, which must hold no relation lock on anyawaor queue-storage table.awa migrateruns while a 0.6.9 producer enqueues and jobs are in flight. A sampler checks that the owner holds no lock aboveRowExclusive. The banked lanes must keep their exact sequence positions, the other lanes may only advance, and the report must say v40, helpers updated inawaand the custom schema, and routing unchanged.awa migrateruns and the patch must stay applied, and 0.6.9 workers restart and serve old queues and one created after the rollback. The 0.6.9awa queue statsreads as the runtime role.prepare-queuefor every queue, the per-sequence grant and its verification, then revokeCREATE. The 0.6.9 workers restart and serve provisioned queues; an unprovisioned queue fails with the provisioning hint.available / running / terminalmust equal0 / 0 / enqueued.Three consecutive local runs (PostgreSQL 16) passed in about 100 s each, for example: 1,120 jobs, 4 workers killed,
awa migrate160 ms, 8 enqueues and 24 jobs in flight during it, and no strong locks sampled. Before theprepare-queuefix, the run failed at the tightening step withpermission denied for sequence.The PG 17/18 runs happen in CI. This container can't run them: apt.postgresql.org is blocked, and I didn't run third-party Postgres binaries here.
Validation
cargo fmt --all -- --checkandSQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warningsare clean.awa-pythonworkspace,cargo check --all-targets --all-featuresanduv lock --checkpass.GITHUB_REF_NAME=v0.6.10, reports every surface at 0.6.10.awa-model: lib 80 passed, plus every integration suite. The newprepared_lane_patch_replaces_helpers_without_table_locksasserts that the patch holds no relation lock on queue-storage objects and that the installer holdsAccessExclusiveLockonleases.least_privilege_migration_testpasses;dirty_marks_wait_free_test: 6 passed;migration_test: 44 passed;queue_storage_runtime_test: 120 passed. This includes:awa-workerlib: 46 passed;awa-uisuites pass, includingpool_limits_testandadmin_read_cost_test;storage-transition.test.ts: 17 passed;help_cli_testpasses;migrate_patch_cli_test: 3 passed (refused range flags, interrupted patch and retry, exported SQL underON_ERROR_STOP);least_privilege_runtime_test: 4 passed, including the stale custom schema and the runtime-created lane;patch_upgrade_rehearsal_test: passed 3 consecutive runs.Release gate
After merge, wait for the automatically triggered full
CIbranch-push run on the exactrelease/0.6.6merge commit. Tag that exact commitv0.6.10only after it passes; the release workflow enforces this exact-SHA gate. #496 is merged, so the v0.6.10 release takes its notes from this PR'sCHANGELOG.mdsection.