diff --git a/CHANGELOG.md b/CHANGELOG.md index dd033804d..cca63fe19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -430,8 +430,9 @@ for each step. runtime role keeps working; only the migration is refused. Before upgrading, remove `TRIGGER` from runtime-role grants, which `GRANT ALL ON ALL TABLES` includes, remove runtime roles' `CREATE` on the schema, and remove triggers - of your own from awa tables. Revoking a running 0.6 fleet's `CREATE` needs - the order in the upgrade guide. + of your own from awa tables. A running 0.6 fleet keeps running if + `awa migrate` from a 0.6.x CLI, 0.6.14 or later, has applied its lane schema patches + before the `CREATE` revoke; see the upgrade guide. - **External migration runners.** Generate the pending SQL with an upgraded binary, for example `awa migrate --pending --extract-to ./pending`, apply complete bundles, and keep the files already applied unchanged. Apply diff --git a/awa/tests/queue_storage_runtime_test.rs b/awa/tests/queue_storage_runtime_test.rs index 66fb9c5b6..7a9e19b5e 100644 --- a/awa/tests/queue_storage_runtime_test.rs +++ b/awa/tests/queue_storage_runtime_test.rs @@ -455,6 +455,34 @@ async fn create_store_with_config(pool: &sqlx::PgPool, config: QueueStorageConfi store } +/// Prune `slot`, retrying while it reports `Blocked` for that slot. Prune takes +/// its locks NOWAIT, so a background session on the slot (autovacuum, or a +/// hosted Postgres's own workers) blocks it; the maintenance loop retries the +/// same way. Any other outcome, or `Blocked` past the deadline, is returned for +/// the caller to assert on. +async fn prune_retrying_blocked( + store: &QueueStorage, + pool: &sqlx::PgPool, + retention: Duration, + slot: i32, +) -> PruneOutcome { + let deadline = Instant::now() + scaled_timeout(Duration::from_secs(5)); + loop { + let outcome = store + .prune_oldest(pool, retention) + .await + .expect("prune_oldest"); + match outcome { + PruneOutcome::Blocked { slot: blocked } + if blocked == slot && Instant::now() < deadline => + { + tokio::time::sleep(Duration::from_millis(50)).await; + } + other => return other, + } + } +} + async fn create_store(pool: &sqlx::PgPool, schema: &str) -> QueueStorage { // Tests that go through this helper exercise the legacy // (non-receipts) lease-materialization path. The receipts mode @@ -15711,11 +15739,7 @@ async fn test_queue_storage_prune_re_carries_failed_row_exactly_once() { RotateOutcome::Rotated { slot: 1, .. } => {} other => panic!("expected Rotated {{ slot: 1 }}, got {other:?}"), } - match store - .prune_oldest(&pool, Duration::from_secs(3600)) - .await - .expect("prune_oldest") - { + match prune_retrying_blocked(&store, &pool, Duration::from_secs(3600), 0).await { PruneOutcome::Pruned { slot: 0, carried_failed_rows: 1, @@ -15759,11 +15783,7 @@ async fn test_queue_storage_prune_re_carries_failed_row_exactly_once() { // Second prune: current slot is 0, so the oldest sealed non-current // slot is slot 1 (the carried row). It is carried again into slot 0. - match store - .prune_oldest(&pool, Duration::from_secs(3600)) - .await - .expect("prune_oldest re-carry") - { + match prune_retrying_blocked(&store, &pool, Duration::from_secs(3600), 1).await { PruneOutcome::Pruned { slot: 1, carried_failed_rows: 1, @@ -18048,10 +18068,7 @@ async fn test_queue_storage_prune_oldest_blocks_on_reader_lock() { .await .expect("Failed to release reader lock"); - let pruned = store - .prune_oldest(&pool, Duration::ZERO) - .await - .expect("Failed to prune after reader lock release"); + let pruned = prune_retrying_blocked(&store, &pool, Duration::ZERO, 0).await; assert!( matches!(pruned, PruneOutcome::Pruned { slot: 0, .. }), "unexpected prune outcome after reader lock release: {pruned:?}" diff --git a/docs/upgrade-0.6-to-0.7.md b/docs/upgrade-0.6-to-0.7.md index f7a5efa9e..ade299b59 100644 --- a/docs/upgrade-0.6-to-0.7.md +++ b/docs/upgrade-0.6-to-0.7.md @@ -104,39 +104,18 @@ schema owner: REVOKE TRIGGER, REFERENCES ON TABLES FROM awa_runtime; ``` -- **A 0.6 split-role install whose runtime holds `CREATE` on the schema.** 0.6 - runtimes created lane sequences themselves, so they needed `CREATE`. - Sequences they own are accepted; the `CREATE` grant is not, because it lets - the runtime add objects while a migration runs. Until the migration refreshes - them, the 0.6 lane helpers and enqueue-head triggers run - `CREATE SEQUENCE IF NOT EXISTS` on every reservation, and that checks - `CREATE` even for a sequence that exists, so revoking the grant makes running - 0.6 producers and workers fail with `42501` until the migration commits. - Provisioning lanes in advance does not avoid this. Take a short pause: - - 1. Stop 0.6 producers and workers, or hold them at a maintenance gate, so no - enqueue or claim runs. - 2. Revoke the grant and commit: +- **A 0.6 split-role install whose runtime holds `CREATE` on the schema.** Before the 0.6.14 schema patch, 0.6 runtimes created lane sequences themselves, so they needed `CREATE`. Sequences they own are accepted; the `CREATE` grant is not, because it lets the runtime add objects while a migration runs. Remove it while 0.6 producers and workers keep running: + + 1. Run `awa migrate` with a 0.6.x CLI, 0.6.14 or later, with the schema owner as the effective role. The schema stays at v40. A 0.7 CLI cannot do this step: it applies the versioned 0.7 migrations, whose schema check refuses the runtime role's `CREATE` grant. Its `prepared_lane_sequences` patch makes the lane helpers and head-table triggers skip `CREATE SEQUENCE` when the lane's sequence exists, and its `definer_lane_provisioning` patch creates a missing lane's sequences through the schema owner's `SECURITY DEFINER` functions, so no runtime role needs `CREATE`. The patches only replace functions, in `awa` and every other queue-storage schema, and earlier 0.6 binaries call the patched helpers with the same arguments. External migration runners apply the `R__` patches as the [0.6.13 to 0.6.14 upgrade guide](https://github.com/hardbyte/awa/blob/v0.6.14/docs/upgrade-0.6.13-to-0.6.14.md#external-migration-runners) describes. + 2. Revoke the grant in each queue-storage schema: ```sql REVOKE CREATE ON SCHEMA awa FROM awa_runtime; ``` - 3. Run `awa migrate` as the migrator. The bundle refreshes the lane helpers - to use provisioned sequences without `CREATE` (v049). - 4. Provision any lanes the 0.7 runtimes will need, as the migrator: - - ```bash - PGOPTIONS='-c role=awa_owner' \ - awa --database-url "$AWA_MIGRATOR_DATABASE_URL" storage prepare-queue \ - --queue --enqueue-shards - ``` - - 5. Start the 0.7 runtimes. + 3. Continue with the 0.7 migration. - An uninterrupted path would need the revoke and the helper refresh to become - visible to every session at once; that has not been verified, so do not - combine them into one transaction as a substitute for the pause. + Revoking before the patches makes running 0.6 producers and workers fail with `42501`: the unpatched helpers run `CREATE SEQUENCE IF NOT EXISTS` on every reservation, which checks `CREATE` even for a sequence that exists. The 0.6 release branch's `awa/tests/patch_upgrade_rehearsal_test.rs` applies the patches under a released 0.6.9 or 0.6.15 fleet, provisions its queues, grants sequence privileges and revokes `CREATE` from its runtime role, then checks that restarted released workers complete every job and a released producer creates a new queue's lanes through the schema owner's functions. - **A custom schema prepared by a runtime or another role.** A worker that auto-prepares a schema, or a DBA who ran the preparation as another user,