Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,8 +430,9 @@ for each step.
runtime role keeps working; only the migration is refused. Before upgrading,
remove `TRIGGER` from runtime-role grants, which `GRANT ALL ON ALL TABLES`
includes, remove runtime roles' `CREATE` on the schema, and remove triggers
of your own from awa tables. Revoking a running 0.6 fleet's `CREATE` needs
the order in the upgrade guide.
of your own from awa tables. A running 0.6 fleet keeps running if
`awa migrate` from a 0.6.x CLI, 0.6.14 or later, has applied its lane schema patches
before the `CREATE` revoke; see the upgrade guide.
- **External migration runners.** Generate the pending SQL with an upgraded
binary, for example `awa migrate --pending --extract-to ./pending`, apply
complete bundles, and keep the files already applied unchanged. Apply
Expand Down
45 changes: 31 additions & 14 deletions awa/tests/queue_storage_runtime_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -455,6 +455,34 @@ async fn create_store_with_config(pool: &sqlx::PgPool, config: QueueStorageConfi
store
}

/// Prune `slot`, retrying while it reports `Blocked` for that slot. Prune takes
/// its locks NOWAIT, so a background session on the slot (autovacuum, or a
/// hosted Postgres's own workers) blocks it; the maintenance loop retries the
/// same way. Any other outcome, or `Blocked` past the deadline, is returned for
/// the caller to assert on.
async fn prune_retrying_blocked(
store: &QueueStorage,
pool: &sqlx::PgPool,
retention: Duration,
slot: i32,
) -> PruneOutcome {
let deadline = Instant::now() + scaled_timeout(Duration::from_secs(5));
loop {
let outcome = store
.prune_oldest(pool, retention)
.await
.expect("prune_oldest");
match outcome {
PruneOutcome::Blocked { slot: blocked }
if blocked == slot && Instant::now() < deadline =>
{
tokio::time::sleep(Duration::from_millis(50)).await;
}
other => return other,
}
}
}

async fn create_store(pool: &sqlx::PgPool, schema: &str) -> QueueStorage {
// Tests that go through this helper exercise the legacy
// (non-receipts) lease-materialization path. The receipts mode
Expand Down Expand Up @@ -15711,11 +15739,7 @@ async fn test_queue_storage_prune_re_carries_failed_row_exactly_once() {
RotateOutcome::Rotated { slot: 1, .. } => {}
other => panic!("expected Rotated {{ slot: 1 }}, got {other:?}"),
}
match store
.prune_oldest(&pool, Duration::from_secs(3600))
.await
.expect("prune_oldest")
{
match prune_retrying_blocked(&store, &pool, Duration::from_secs(3600), 0).await {
PruneOutcome::Pruned {
slot: 0,
carried_failed_rows: 1,
Expand Down Expand Up @@ -15759,11 +15783,7 @@ async fn test_queue_storage_prune_re_carries_failed_row_exactly_once() {

// Second prune: current slot is 0, so the oldest sealed non-current
// slot is slot 1 (the carried row). It is carried again into slot 0.
match store
.prune_oldest(&pool, Duration::from_secs(3600))
.await
.expect("prune_oldest re-carry")
{
match prune_retrying_blocked(&store, &pool, Duration::from_secs(3600), 1).await {
PruneOutcome::Pruned {
slot: 1,
carried_failed_rows: 1,
Expand Down Expand Up @@ -18048,10 +18068,7 @@ async fn test_queue_storage_prune_oldest_blocks_on_reader_lock() {
.await
.expect("Failed to release reader lock");

let pruned = store
.prune_oldest(&pool, Duration::ZERO)
.await
.expect("Failed to prune after reader lock release");
let pruned = prune_retrying_blocked(&store, &pool, Duration::ZERO, 0).await;
assert!(
matches!(pruned, PruneOutcome::Pruned { slot: 0, .. }),
"unexpected prune outcome after reader lock release: {pruned:?}"
Expand Down
33 changes: 6 additions & 27 deletions docs/upgrade-0.6-to-0.7.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,39 +104,18 @@ schema owner:
REVOKE TRIGGER, REFERENCES ON TABLES FROM awa_runtime;
```

- **A 0.6 split-role install whose runtime holds `CREATE` on the schema.** 0.6
runtimes created lane sequences themselves, so they needed `CREATE`.
Sequences they own are accepted; the `CREATE` grant is not, because it lets
the runtime add objects while a migration runs. Until the migration refreshes
them, the 0.6 lane helpers and enqueue-head triggers run
`CREATE SEQUENCE IF NOT EXISTS` on every reservation, and that checks
`CREATE` even for a sequence that exists, so revoking the grant makes running
0.6 producers and workers fail with `42501` until the migration commits.
Provisioning lanes in advance does not avoid this. Take a short pause:

1. Stop 0.6 producers and workers, or hold them at a maintenance gate, so no
enqueue or claim runs.
2. Revoke the grant and commit:
- **A 0.6 split-role install whose runtime holds `CREATE` on the schema.** Before the 0.6.14 schema patch, 0.6 runtimes created lane sequences themselves, so they needed `CREATE`. Sequences they own are accepted; the `CREATE` grant is not, because it lets the runtime add objects while a migration runs. Remove it while 0.6 producers and workers keep running:

1. Run `awa migrate` with a 0.6.x CLI, 0.6.14 or later, with the schema owner as the effective role. The schema stays at v40. A 0.7 CLI cannot do this step: it applies the versioned 0.7 migrations, whose schema check refuses the runtime role's `CREATE` grant. Its `prepared_lane_sequences` patch makes the lane helpers and head-table triggers skip `CREATE SEQUENCE` when the lane's sequence exists, and its `definer_lane_provisioning` patch creates a missing lane's sequences through the schema owner's `SECURITY DEFINER` functions, so no runtime role needs `CREATE`. The patches only replace functions, in `awa` and every other queue-storage schema, and earlier 0.6 binaries call the patched helpers with the same arguments. External migration runners apply the `R__` patches as the [0.6.13 to 0.6.14 upgrade guide](https://github.com/hardbyte/awa/blob/v0.6.14/docs/upgrade-0.6.13-to-0.6.14.md#external-migration-runners) describes.
2. Revoke the grant in each queue-storage schema:

```sql
REVOKE CREATE ON SCHEMA awa FROM awa_runtime;
```

3. Run `awa migrate` as the migrator. The bundle refreshes the lane helpers
to use provisioned sequences without `CREATE` (v049).
4. Provision any lanes the 0.7 runtimes will need, as the migrator:

```bash
PGOPTIONS='-c role=awa_owner' \
awa --database-url "$AWA_MIGRATOR_DATABASE_URL" storage prepare-queue \
--queue <queue> --enqueue-shards <n>
```

5. Start the 0.7 runtimes.
3. Continue with the 0.7 migration.

An uninterrupted path would need the revoke and the helper refresh to become
visible to every session at once; that has not been verified, so do not
combine them into one transaction as a substitute for the pause.
Revoking before the patches makes running 0.6 producers and workers fail with `42501`: the unpatched helpers run `CREATE SEQUENCE IF NOT EXISTS` on every reservation, which checks `CREATE` even for a sequence that exists. The 0.6 release branch's `awa/tests/patch_upgrade_rehearsal_test.rs` applies the patches under a released 0.6.9 or 0.6.15 fleet, provisions its queues, grants sequence privileges and revokes `CREATE` from its runtime role, then checks that restarted released workers complete every job and a released producer creates a new queue's lanes through the schema owner's functions.

- **A custom schema prepared by a runtime or another role.** A worker that
auto-prepares a schema, or a DBA who ran the preparation as another user,
Expand Down
Loading