diff --git a/CHANGELOG.md b/CHANGELOG.md index ad43fac2f..d3079d3fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,17 @@ Notable changes between releases. Detailed migration notes for storage transitio ## [Unreleased] +### Upgrading + +- The schema stays at v40. `awa migrate` applies one new repeatable patch, `replica_identity`. Unlike the earlier 0.6 patches it takes table locks: `ACCESS EXCLUSIVE` on each table it changes (`awa.admin_dirty_queue_marks`, `awa.admin_dirty_kind_marks`, `awa.job_unique_claims` and each `queue_terminal_count_deltas` table and partition), plus `SHARE` on `idx_awa_jobs_unique`, held until the patch commits, which is milliseconds once granted. Each lock wait is bounded by the session's `lock_timeout`, or 300 ms when none is set; a timeout fails the patch with `55P03` and changes nothing, so run `awa migrate` again. A re-run after success takes no lock. External migration runners apply `R__replica_identity.sql` after `V40` and after the other `R__` scripts, as the owner of the awa and queue-storage tables. 0.6.17 binaries run unchanged on the patched schema, so binaries and the patch can roll in either order. See the [upgrade guide](docs/upgrade-0.6.17-to-0.6.18.md). + +### Changed + +- Ring prune truncates only the children of the sealed slot that hold storage (0.6 port of #512). Queue, lease and claim prune take the same locks on every child of the slot, in the same order, as in 0.6.17, then `TRUNCATE` only the children with heap, TOAST, free-space-map or visibility-map pages, or index pages beyond an empty btree, GiST or BRIN index. A child that never held a row keeps its relfilenode, so a prune no longer swaps relfilenodes and writes WAL and catalog rows for empty partitions, such as the receipt children of a schema without receipts. A child emptied by `DELETE` and `VACUUM` that kept expanded indexes or TOAST is still truncated. The empty-slot fast path is unchanged. No configuration or schema change. + ### Fixed +- **awa keeps working when its schema is in a logical-replication publication (0.6 port of #509).** With a `CREATE PUBLICATION ... FOR ALL TABLES` (Debezium's default) or `FOR TABLES IN SCHEMA awa`, PostgreSQL rejected every `UPDATE` and `DELETE` on the 19 awa tables without a primary key, so the admin dirty-mark drain failed on every maintenance tick with `cannot delete from table "admin_dirty_queue_marks" because it does not have a replica identity and publishes deletes`, and unique-claim release failed the same way. The `replica_identity` patch gives `awa.job_unique_claims` its uniqueness index as replica identity and sets `REPLICA IDENTITY FULL` on the two mark tables and on `queue_terminal_count_deltas` and each of its partitions, which adds no WAL on their insert-and-truncate path. PostgreSQL does not copy the setting to partitions, so the queue-storage installer sets it on every partition of a schema prepared later. Keeping the awa schema out of CDC publications remains the recommendation; see [logical replication and CDC](docs/deploying-on-managed-postgres.md#logical-replication-and-cdc). - `JobContext::flush_progress` (Python `ctx.flush_progress()`) on a queue-storage attempt with receipts, the default, no longer returns success without writing the progress. A heartbeat leaves out an attempt whose lock another transaction holds, and since 0.6.16 an explicit flush shared that path, so a flush that overlapped a heartbeat or a completion of the same attempt was dropped and the worker treated the progress as persisted. An explicit flush now waits for the attempt's lock; heartbeats still skip. ## [0.6.17] — 2026-11-12 diff --git a/README.md b/README.md index ce4512593..ec57237c4 100644 --- a/README.md +++ b/README.md @@ -392,6 +392,7 @@ All coordination through Postgres. The Rust runtime owns dispatch, leases, heart | [0.6.14 → 0.6.15 upgrade](docs/upgrade-0.6.14-to-0.6.15.md) | Rolling patch upgrade: bounded worker finalization, the claim batch index and the batch compatibility insert | | [0.6.15 → 0.6.16 upgrade](docs/upgrade-0.6.15-to-0.6.16.md) | Rolling patch upgrade: bounded rescue drains of abandoned claims, batch compatibility inserts that give way to per-row producers, new-lane provisioning order, the operator prune halt and its grants, and claim batch index fixes | | [0.6.16 → 0.6.17 upgrade](docs/upgrade-0.6.16-to-0.6.17.md) | Rolling patch upgrade: asynchronous claim commits, the maintenance eligibility table, the claim tombstone patch, the changed `/api/stats` and `state_counts` shape, and prune lock waits | +| [0.6.17 → 0.6.18 upgrade](docs/upgrade-0.6.17-to-0.6.18.md) | Rolling patch upgrade: the replica identity patch and the table locks it takes, and prune truncating only partitions that hold rows | | [Configuration reference](docs/configuration.md) | `QueueConfig`, `ClientBuilder`, Python `start()`, env vars | | [HTTP workers and callbacks](docs/http-callbacks.md) | Serverless dispatch, callback receiver endpoints, signature verification | | [Lifecycle hooks](docs/lifecycle-hooks.md) | React to job start/finish/retry/callback events for metrics, alerting, cleanup | diff --git a/awa-model/migrations/patches/replica_identity.sql b/awa-model/migrations/patches/replica_identity.sql new file mode 100644 index 000000000..1f8982e78 --- /dev/null +++ b/awa-model/migrations/patches/replica_identity.sql @@ -0,0 +1,91 @@ +-- Schema patch replica_identity: give every awa table a replica identity +-- (awa 0.7 migration v047, minus the schema_version row). +-- +-- PostgreSQL rejects UPDATE and DELETE on a table that is in a logical +-- replication publication and has no primary key, replica-identity index or +-- FULL identity. Tables without one on 0.6: +-- * awa.admin_dirty_queue_marks, awa.admin_dirty_kind_marks: FULL. ADR-045 +-- forbids any index or constraint on them. +-- * awa.job_unique_claims: USING INDEX idx_awa_jobs_unique. +-- * .queue_terminal_count_deltas and each of its partitions, in +-- every queue-storage schema: FULL. Partitions do not inherit the +-- parent's setting. +-- Tracking: https://github.com/hardbyte/awa/pull/509 +-- +-- Applied after the v023 definitions, whose installer sets the delta tables' +-- identity in schemas prepared later. +-- +-- Live load: each ALTER takes ACCESS EXCLUSIVE on its table (and SHARE on +-- idx_awa_jobs_unique), held until the patch's transaction commits. Each lock +-- wait is bounded by the session's lock_timeout, or 300ms when none is set; +-- a timeout aborts the patch and leaves nothing changed. A table that already +-- has its identity is not altered, so a re-run locks nothing. +-- +-- 0.6 delivery: the 0.6 series cannot add a migration version, so this ships as +-- an idempotent schema patch. `awa migrate` applies it when the schema is +-- exactly at v040 and the probe in migrations.rs reports it missing; +-- `awa migrate --sql` prints it as a repeatable R__ script for external +-- runners. Replica identity only changes what the WAL records for UPDATE and +-- DELETE, so released 0.6 binaries run the same against a patched or an +-- unpatched schema. Run it as the owner of the awa and queue-storage tables. + +DO $$ +DECLARE + v_lock_timeout TEXT := current_setting('lock_timeout'); + v_target RECORD; +BEGIN + IF v_lock_timeout = '0' THEN + PERFORM set_config('lock_timeout', '300ms', true); + END IF; + + FOR v_target IN + SELECT c.oid::regclass AS relation, 'FULL' AS identity, 0 AS ord, + n.nspname, c.relkind, c.relname + FROM pg_class AS c + JOIN pg_namespace AS n ON n.oid = c.relnamespace + WHERE n.nspname = 'awa' + AND c.relname IN ('admin_dirty_queue_marks', 'admin_dirty_kind_marks') + AND c.relkind = 'r' + AND c.relreplident <> 'f' + UNION ALL + SELECT c.oid::regclass, 'USING INDEX idx_awa_jobs_unique', 1, + n.nspname, c.relkind, c.relname + FROM pg_class AS c + JOIN pg_namespace AS n ON n.oid = c.relnamespace + WHERE n.nspname = 'awa' + AND c.relname = 'job_unique_claims' + AND to_regclass('awa.idx_awa_jobs_unique') IS NOT NULL + AND NOT EXISTS ( + SELECT 1 FROM pg_index AS i + WHERE i.indrelid = c.oid AND i.indisreplident + ) + UNION ALL + SELECT c.oid::regclass, 'FULL', 2, n.nspname, c.relkind, c.relname + FROM pg_class AS c + JOIN pg_namespace AS n ON n.oid = c.relnamespace + WHERE has_schema_privilege(current_user, n.oid, 'USAGE') + AND EXISTS ( + SELECT 1 FROM pg_proc AS awa_p + WHERE awa_p.pronamespace = n.oid + AND awa_p.proname = 'claim_ready_runtime' + AND oidvectortypes(awa_p.proargtypes) + = 'text, bigint, double precision, double precision' + ) + AND c.relkind IN ('r', 'p') + AND c.relreplident <> 'f' + AND c.relname ~ '^queue_terminal_count_deltas(_[0-9]+)?$' + ORDER BY ord, nspname, relkind, relname + LOOP + BEGIN + EXECUTE format('ALTER TABLE %s REPLICA IDENTITY %s', v_target.relation, v_target.identity); + EXCEPTION WHEN lock_not_available THEN + RAISE EXCEPTION 'replica_identity: could not lock % within lock_timeout %', + v_target.relation, current_setting('lock_timeout') + USING ERRCODE = 'lock_not_available', + HINT = 'Nothing was changed. Run awa migrate again, or raise lock_timeout for the migrating session.'; + END; + END LOOP; + + PERFORM set_config('lock_timeout', v_lock_timeout, true); +END; +$$; diff --git a/awa-model/migrations/patches/wait_free_dirty_marks.sql b/awa-model/migrations/patches/wait_free_dirty_marks.sql index 480aacc44..71b96bc93 100644 --- a/awa-model/migrations/patches/wait_free_dirty_marks.sql +++ b/awa-model/migrations/patches/wait_free_dirty_marks.sql @@ -29,6 +29,37 @@ CREATE TABLE IF NOT EXISTS awa.admin_dirty_kind_marks ( touched_at TIMESTAMPTZ NOT NULL DEFAULT now() ); +-- A mark table created by this transaction gets REPLICA IDENTITY FULL here, +-- as the replica_identity patch would give it, whichever order a migration +-- runner applies the two in. Only a table this session already holds in +-- ACCESS EXCLUSIVE mode is altered, so the change never waits for a lock. +DO $$ +DECLARE + v_table REGCLASS; +BEGIN + FOR v_table IN + SELECT c.oid::regclass + FROM pg_class AS c + WHERE c.oid IN ( + to_regclass('awa.admin_dirty_queue_marks'), + to_regclass('awa.admin_dirty_kind_marks') + ) + AND c.relreplident <> 'f' + AND EXISTS ( + SELECT 1 FROM pg_locks AS l + WHERE l.locktype = 'relation' + AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database()) + AND l.relation = c.oid + AND l.pid = pg_backend_pid() + AND l.mode = 'AccessExclusiveLock' + AND l.granted + ) + LOOP + EXECUTE format('ALTER TABLE %s REPLICA IDENTITY %s', v_table, 'FULL'); + END LOOP; +END; +$$; + -- Carry pending dirtiness across the switch so nothing waits for the next -- full refresh. Marks that released trigger bodies commit into the old -- tables after this copy are reconciled by refresh_admin_metadata(). diff --git a/awa-model/migrations/v023_install_queue_storage_substrate.sql b/awa-model/migrations/v023_install_queue_storage_substrate.sql index aa906d39d..d572b5035 100644 --- a/awa-model/migrations/v023_install_queue_storage_substrate.sql +++ b/awa-model/migrations/v023_install_queue_storage_substrate.sql @@ -869,6 +869,7 @@ DECLARE v_initial_gen BIGINT; v_claimed_cte TEXT; v_claim_runtime REGPROCEDURE; + v_identity_rel REGCLASS; BEGIN IF p_schema IS NULL OR p_schema !~ '^[a-z_][a-z0-9_]*$' THEN RAISE EXCEPTION 'install_queue_storage_substrate: schema name must match ^[a-z_][a-z0-9_]*$; got %', @@ -4343,6 +4344,25 @@ BEGIN 'Queue-storage claim allocator. Claims committed ready lanes, skips spent/tombstoned lanes, and leaves claim-cursor advancement safe under transaction rollback.' ); + -------------------------------------------------------------------- + -- REPLICA IDENTITY FULL on the terminal-count delta parent and each + -- of its partitions, which have no primary key and do not inherit + -- the parent's setting. Guarded: a re-run takes no lock. + -------------------------------------------------------------------- + + FOR v_identity_rel IN + SELECT c.oid::regclass + FROM pg_class AS c + JOIN pg_namespace AS n ON n.oid = c.relnamespace + WHERE n.nspname = p_schema + AND c.relkind IN ('r', 'p') + AND c.relreplident <> 'f' + AND c.relname ~ '^queue_terminal_count_deltas(_[0-9]+)?$' + ORDER BY c.relkind, c.relname + LOOP + EXECUTE format('ALTER TABLE %s REPLICA IDENTITY FULL', v_identity_rel); + END LOOP; + -------------------------------------------------------------------- -- Seed ring-slot generation rows. Slot 0 starts at generation 0; -- the rest start at -1 (sentinel for "never rotated through"). diff --git a/awa-model/src/lib.rs b/awa-model/src/lib.rs index 1e43841e5..b893da120 100644 --- a/awa-model/src/lib.rs +++ b/awa-model/src/lib.rs @@ -16,6 +16,7 @@ mod read_transaction; pub mod reschedule; pub mod runtime_control; pub mod storage; +mod storage_footprint_sql; pub mod unique; // Re-exports for ergonomics diff --git a/awa-model/src/migrations.rs b/awa-model/src/migrations.rs index 506e256a9..a80dd40ff 100644 --- a/awa-model/src/migrations.rs +++ b/awa-model/src/migrations.rs @@ -423,6 +423,47 @@ macro_rules! correlated_claim_tombstones_pending_sql { }; } +/// Rows naming what still lacks the `replica_identity` patch: the installer, +/// the awa tables without a primary key, and each `queue_terminal_count_deltas` +/// table or partition of an installed queue-storage substrate (found as for +/// `correlated_claim_tombstones`) whose replica identity is not FULL. A missing +/// mark table counts, so the patch runs after `wait_free_dirty_marks` creates +/// it. +macro_rules! replica_identity_pending_sql { + () => { + "SELECT 'awa.install_queue_storage_substrate'::text AS target \ + WHERE EXISTS ( \ + SELECT 1 FROM pg_proc AS p \ + WHERE p.pronamespace = 'awa'::regnamespace \ + AND p.proname = 'install_queue_storage_substrate' \ + AND p.prosrc NOT LIKE '%REPLICA IDENTITY FULL%') \ + UNION ALL \ + SELECT 'replica identity of awa.' || marks.relname \ + FROM (VALUES ('admin_dirty_kind_marks'), ('admin_dirty_queue_marks')) AS marks(relname) \ + WHERE NOT EXISTS (SELECT 1 FROM pg_class AS c \ + WHERE c.oid = to_regclass('awa.' || marks.relname) \ + AND c.relreplident = 'f') \ + UNION ALL \ + SELECT 'replica identity of awa.job_unique_claims' \ + WHERE to_regclass('awa.job_unique_claims') IS NOT NULL \ + AND NOT EXISTS (SELECT 1 FROM pg_index AS i \ + WHERE i.indrelid = to_regclass('awa.job_unique_claims') \ + AND i.indisreplident) \ + UNION ALL \ + SELECT format('replica identity of %I.%I', n.nspname, c.relname) \ + FROM pg_class AS c \ + JOIN pg_namespace AS n ON n.oid = c.relnamespace \ + WHERE c.relkind IN ('r', 'p') \ + AND c.relreplident <> 'f' \ + AND c.relname ~ '^queue_terminal_count_deltas(_[0-9]+)?$' \ + AND EXISTS (SELECT 1 FROM pg_proc AS p \ + WHERE p.pronamespace = n.oid \ + AND p.proname = 'claim_ready_runtime' \ + AND oidvectortypes(p.proargtypes) \ + = 'text, bigint, double precision, double precision')" + }; +} + /// SQL returning one `text` row per object or schema that still needs the /// named patch, for patches that touch more than one place. fn patch_targets_sql(name: &str) -> Option<&'static str> { @@ -442,6 +483,7 @@ fn patch_targets_sql(name: &str) -> Option<&'static str> { correlated_claim_tombstones_pending_sql!(), " ORDER BY 1" )), + "replica_identity" => Some(concat!(replica_identity_pending_sql!(), " ORDER BY 1")), _ => None, } } @@ -565,6 +607,19 @@ pub const SCHEMA_PATCHES: &[SchemaPatch] = &[ ")" ), }, + SchemaPatch { + name: "replica_identity", + description: "Replica identity for every awa table (awa 0.7 migration v047)", + // The v023 definitions first, so the installer sets the identity in + // schemas prepared later; the patch then sets it on existing tables. + // Keep last: it holds table locks until its transaction commits. + sql: concat!( + include_str!("../migrations/v023_install_queue_storage_substrate.sql"), + "\n", + include_str!("../migrations/patches/replica_identity.sql"), + ), + applied_probe: concat!("SELECT NOT EXISTS (", replica_identity_pending_sql!(), ")"), + }, ]; /// Schema patches a database still needs to reach this binary's schema shape. @@ -1175,6 +1230,18 @@ mod tests { _ => patch.sql.to_string(), }; let upper = top_level.to_uppercase(); + // The only ALTER TABLE a patch may run is the guarded replica + // identity change. + let guarded_identity = "ALTER TABLE %S REPLICA IDENTITY %S"; + if upper.contains(guarded_identity) { + assert!( + upper.contains("RELREPLIDENT <> 'F'") + && (!upper.contains("USING INDEX") || upper.contains("INDISREPLIDENT")), + "patch {} must skip tables that already have their identity", + patch.name + ); + } + let upper = upper.replace(guarded_identity, ""); assert!( !upper.contains("INSERT INTO AWA.SCHEMA_VERSION"), "patch {} must not record a schema version", diff --git a/awa-model/src/queue_storage.rs b/awa-model/src/queue_storage.rs index ed1289303..82957d393 100644 --- a/awa-model/src/queue_storage.rs +++ b/awa-model/src/queue_storage.rs @@ -2,6 +2,7 @@ use crate::admin::{CallbackConfig, CallbackPollResult}; use crate::dlq::{ListDlqFilter, RetryFromDlqOpts}; use crate::error::AwaError; use crate::insert::prepare_row_raw; +use crate::storage_footprint_sql::brin_fsm_empty_pages_sql; use crate::{InsertParams, JobRow, JobState}; use chrono::TimeDelta; use chrono::{DateTime, Utc}; @@ -42,6 +43,42 @@ const RECEIPT_RESCUE_CURSOR_SCAN_LIMIT: i64 = 10_000; const RECEIPT_DEADLINE_RESCUE_CURSOR_SCAN_LIMIT: i64 = 10_000; const DEFAULT_PRUNE_LOCK_TIMEOUT: Duration = Duration::from_millis(10); +// Whether a ring child holds storage beyond the metadata an empty table keeps: +// any heap, TOAST, free-space-map or visibility-map page, or index pages beyond +// one btree/GiST page and three BRIN main-fork pages plus the BRIN FSM. Other +// index methods count from their first page. +const RECLAIMABLE_CHILD_STORAGE_SQL: &str = concat!( + r#" +WITH heaps AS ( + SELECT oid, reltoastrelid FROM pg_class + WHERE oid = $1::regclass AND relkind = 'r' +), heap_relations AS ( + SELECT oid FROM heaps + UNION ALL + SELECT reltoastrelid FROM heaps WHERE reltoastrelid <> 0 +), storage_relations AS ( + SELECT oid, NULL::name AS amname FROM heap_relations + UNION ALL + SELECT i.indexrelid, am.amname FROM pg_index i + JOIN pg_class c ON c.oid = i.indexrelid + JOIN pg_am am ON am.oid = c.relam + WHERE i.indrelid IN (SELECT oid FROM heap_relations) +) +SELECT EXISTS ( + SELECT 1 FROM storage_relations + CROSS JOIN (SELECT current_setting('block_size')::bigint AS block_size) settings + WHERE pg_relation_size(oid, 'fsm') > block_size * + CASE amname WHEN 'brin' THEN "#, + brin_fsm_empty_pages_sql!(), + r#" ELSE 0 END + OR pg_relation_size(oid, 'vm') > 0 + OR pg_relation_size(oid) > block_size * + CASE amname WHEN 'btree' THEN 1 WHEN 'gist' THEN 1 + WHEN 'brin' THEN 3 ELSE 0 END +) +"# +); + /// Portable 64-bit hash over raw ordering-key bytes. /// /// This is intentionally simple enough to implement byte-for-byte in @@ -17436,6 +17473,33 @@ impl QueueStorage { Ok(false) } + /// TRUNCATE those of `children` that hold reclaimable storage. The caller + /// must already hold ACCESS EXCLUSIVE on every child; a child it skips has + /// no heap page, so it holds no row. + async fn truncate_used_children_tx( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + children: &[&str], + ) -> Result<(), AwaError> { + let mut targets: Vec<&str> = Vec::with_capacity(children.len()); + for child in children { + let reclaimable: bool = sqlx::query_scalar(RECLAIMABLE_CHILD_STORAGE_SQL) + .bind(child) + .fetch_one(tx.as_mut()) + .await + .map_err(map_sqlx_error)?; + if reclaimable { + targets.push(child); + } + } + if !targets.is_empty() { + sqlx::query(&format!("TRUNCATE TABLE {}", targets.join(", "))) + .execute(tx.as_mut()) + .await + .map_err(map_sqlx_error)?; + } + Ok(()) + } + #[tracing::instrument(skip(self, pool), name = "queue_storage.rotate")] pub async fn rotate(&self, pool: &PgPool) -> Result { let schema = self.schema(); @@ -18566,10 +18630,20 @@ impl QueueStorage { }); } - sqlx::query(&format!("TRUNCATE TABLE {slot_children}")) - .execute(tx.as_mut()) - .await - .map_err(map_sqlx_error)?; + Self::truncate_used_children_tx( + &mut tx, + &[ + &segment_child, + &done_child, + &ready_child, + &claim_attempt_child, + &tomb_child, + &receipt_batch_child, + &receipt_tomb_child, + &delta_child, + ], + ) + .await?; if !pruned_terminal_counts.is_empty() { self.adjust_terminal_rollups_batch(&mut tx, pruned_terminal_counts.into_iter()) @@ -18719,10 +18793,7 @@ impl QueueStorage { }); } - sqlx::query(&format!("TRUNCATE TABLE {lease_child}")) - .execute(tx.as_mut()) - .await - .map_err(map_sqlx_error)?; + Self::truncate_used_children_tx(&mut tx, &[&lease_child]).await?; Ok(PruneOutcome::Pruned { slot, carried_failed_rows: 0, @@ -19028,10 +19099,16 @@ impl QueueStorage { }); } - sqlx::query(&format!("TRUNCATE TABLE {slot_children}")) - .execute(tx.as_mut()) - .await - .map_err(map_sqlx_error)?; + Self::truncate_used_children_tx( + &mut tx, + &[ + &claim_child, + &claim_batch_child, + &closure_child, + &closure_batch_child, + ], + ) + .await?; self.ensure_claim_batch_job_index_tx(&mut tx, slot).await?; sqlx::query(&format!( r#" diff --git a/awa-model/src/storage_footprint_sql.rs b/awa-model/src/storage_footprint_sql.rs new file mode 100644 index 000000000..03c1c1857 --- /dev/null +++ b/awa-model/src/storage_footprint_sql.rs @@ -0,0 +1,11 @@ +// Pages an empty BRIN index keeps in its free space map: PostgreSQL 17/18 +// FSM_TREE_DEPTH is four levels below a 4096-byte block size, three otherwise. +// https://github.com/postgres/postgres/blob/REL_17_STABLE/src/backend/storage/freespace/freespace.c +// https://github.com/postgres/postgres/blob/REL_18_STABLE/src/backend/storage/freespace/freespace.c +// `block_size` is supplied by the enclosing SQL query. +macro_rules! brin_fsm_empty_pages_sql { + () => { + "CASE WHEN block_size < 4096 THEN 4 ELSE 3 END" + }; +} +pub(crate) use brin_fsm_empty_pages_sql; diff --git a/awa-model/tests/replica_identity_test.rs b/awa-model/tests/replica_identity_test.rs new file mode 100644 index 000000000..f41122a60 --- /dev/null +++ b/awa-model/tests/replica_identity_test.rs @@ -0,0 +1,408 @@ +//! Every awa table must stay writable while it is in a logical-replication +//! publication: PostgreSQL rejects UPDATE and DELETE on a published table +//! without a primary key, replica-identity index or FULL identity. + +use awa_model::{migrations, PruneOutcome, QueueStorage, QueueStorageConfig}; +use sqlx::PgPool; +use std::time::Duration; + +type TestResult = Result<(), Box>; + +const PATCH: &str = "replica_identity"; + +fn patch() -> &'static migrations::SchemaPatch { + migrations::SCHEMA_PATCHES + .iter() + .find(|patch| patch.name == PATCH) + .expect("the patch is registered") +} + +async fn patch_pending(pool: &PgPool) -> Result, Box> { + Ok(migrations::schema_patch_status(pool) + .await? + .into_iter() + .find(|status| status.name == PATCH) + .expect("the patch is registered") + .needs_repair) +} + +fn custom_store(schema: &str, receipts: bool) -> Result> { + Ok(QueueStorage::new(QueueStorageConfig { + schema: schema.into(), + queue_slot_count: 4, + lease_slot_count: 2, + claim_slot_count: 2, + lease_claim_receipts: receipts, + ..Default::default() + })?) +} + +fn is_delta_table(table: &str) -> bool { + table + .strip_prefix("queue_terminal_count_deltas") + .is_some_and(|suffix| { + suffix.is_empty() + || suffix.strip_prefix('_').is_some_and(|slot| { + !slot.is_empty() && slot.bytes().all(|b| b.is_ascii_digit()) + }) + }) +} + +/// Tables without a usable replica identity, and tables that use FULL without +/// being expected to. +async fn identity_problems(pool: &PgPool, schemas: &[&str]) -> Result, sqlx::Error> { + let tables: Vec<(String, String, String, bool, bool)> = sqlx::query_as( + "SELECT n.nspname::text, c.relname::text, c.relreplident::text, + EXISTS (SELECT 1 FROM pg_index i WHERE i.indrelid = c.oid AND i.indisprimary), + EXISTS (SELECT 1 FROM pg_index i WHERE i.indrelid = c.oid AND i.indisreplident) + FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = ANY($1) AND c.relkind IN ('r', 'p') + ORDER BY 1, 2", + ) + .bind(schemas) + .fetch_all(pool) + .await?; + assert!( + tables.len() > 100, + "expected the full schema, found {}", + tables.len() + ); + let mut problems = Vec::new(); + for (schema, table, replident, primary_key, identity_index) in tables { + let usable = match replident.as_str() { + "d" => primary_key, + "i" => identity_index, + "f" => true, + _ => false, + }; + if !usable { + problems.push(format!( + "{schema}.{table}: no usable replica identity ({replident})" + )); + } + let full_expected = is_delta_table(&table) + || (schema == "awa" + && matches!( + table.as_str(), + "admin_dirty_queue_marks" | "admin_dirty_kind_marks" + )); + if replident == "f" && !full_expected { + problems.push(format!( + "{schema}.{table}: unexpected REPLICA IDENTITY FULL" + )); + } + if schema == "awa" && table == "job_unique_claims" && replident != "i" { + problems.push(format!("{schema}.{table}: expected its uniqueness index")); + } + } + Ok(problems) +} + +/// UPDATE and DELETE on every table and partition. PostgreSQL checks the +/// replica identity at executor start, so `WHERE false` is enough. +async fn assert_update_and_delete_accepted(pool: &PgPool, schemas: &[&str]) -> TestResult { + let tables: Vec<(String, String, String)> = sqlx::query_as( + "SELECT n.nspname::text, c.relname::text, a.attname::text + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum = 1 + WHERE n.nspname = ANY($1) AND c.relkind = 'r' + ORDER BY 1, 2", + ) + .bind(schemas) + .fetch_all(pool) + .await?; + assert!( + tables.len() > 100, + "expected the full schema, found {}", + tables.len() + ); + for (schema, table, column) in tables { + let mut tx = pool.begin().await?; + sqlx::raw_sql(&format!( + "UPDATE {schema}.{table} SET {column} = {column} WHERE false; + DELETE FROM {schema}.{table} WHERE false;" + )) + .execute(&mut *tx) + .await + .unwrap_or_else(|err| panic!("{schema}.{table} rejected UPDATE or DELETE: {err}")); + tx.rollback().await?; + } + Ok(()) +} + +/// Puts the 0.6.17 state back: no identity on any patch target. +async fn clear_identities(pool: &PgPool, schemas: &[&str]) -> TestResult { + sqlx::raw_sql( + "ALTER TABLE awa.admin_dirty_queue_marks REPLICA IDENTITY DEFAULT; + ALTER TABLE awa.admin_dirty_kind_marks REPLICA IDENTITY DEFAULT; + ALTER TABLE awa.job_unique_claims REPLICA IDENTITY DEFAULT;", + ) + .execute(pool) + .await?; + let deltas: Vec = sqlx::query_scalar( + "SELECT format('%I.%I', n.nspname, c.relname) + FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = ANY($1) AND c.relname ~ '^queue_terminal_count_deltas(_[0-9]+)?$'", + ) + .bind(schemas) + .fetch_all(pool) + .await?; + assert!(!deltas.is_empty()); + for delta in deltas { + sqlx::raw_sql(&format!("ALTER TABLE {delta} REPLICA IDENTITY DEFAULT")) + .execute(pool) + .await?; + } + Ok(()) +} + +#[sqlx::test(migrations = false)] +async fn every_table_has_a_usable_replica_identity(pool: PgPool) -> TestResult { + migrations::run(&pool).await?; + custom_store("ri_receipts", true)? + .prepare_schema(&pool) + .await?; + custom_store("ri_legacy", false)? + .prepare_schema(&pool) + .await?; + let schemas = ["awa", "ri_receipts", "ri_legacy"]; + + let problems = identity_problems(&pool, &schemas).await?; + assert!(problems.is_empty(), "{}", problems.join("\n")); + assert!(patch_pending(&pool).await?.is_empty()); + Ok(()) +} + +#[sqlx::test(migrations = false)] +async fn replica_identity_patch_repairs_existing_tables_and_replays_without_locks( + pool: PgPool, +) -> TestResult { + migrations::run(&pool).await?; + custom_store("ri_custom", true)? + .prepare_schema(&pool) + .await?; + let schemas = ["awa", "ri_custom"]; + clear_identities(&pool, &schemas).await?; + + let pending = patch_pending(&pool).await?; + for expected in [ + "replica identity of awa.admin_dirty_kind_marks", + "replica identity of awa.admin_dirty_queue_marks", + "replica identity of awa.job_unique_claims", + "replica identity of awa.queue_terminal_count_deltas", + "replica identity of awa.queue_terminal_count_deltas_15", + "replica identity of ri_custom.queue_terminal_count_deltas", + "replica identity of ri_custom.queue_terminal_count_deltas_3", + ] { + assert!( + pending.iter().any(|target| target == expected), + "{expected} missing from {pending:?}" + ); + } + assert_eq!(pending.len(), 3 + 17 + 5, "{pending:?}"); + + // The patch locks exactly the tables it alters, until its transaction ends. + let mut tx = pool.begin().await?; + sqlx::raw_sql(patch().sql).execute(&mut *tx).await?; + let locks: Vec = sqlx::query_scalar( + "SELECT format('%s.%s %s', n.nspname, c.relname, l.mode) + FROM pg_locks l + JOIN pg_class c ON c.oid = l.relation + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE l.pid = pg_backend_pid() AND l.locktype = 'relation' + AND n.nspname = ANY($1) + AND l.mode NOT IN ('AccessShareLock', 'RowShareLock', 'RowExclusiveLock') + ORDER BY 1", + ) + .bind(&schemas[..]) + .fetch_all(&mut *tx) + .await?; + let mut expected: Vec = pending + .iter() + .map(|target| { + format!( + "{} AccessExclusiveLock", + target.trim_start_matches("replica identity of ") + ) + }) + .collect(); + expected.push("awa.idx_awa_jobs_unique ShareLock".to_string()); + expected.sort(); + assert_eq!(locks, expected); + tx.commit().await?; + + assert!(patch_pending(&pool).await?.is_empty()); + let problems = identity_problems(&pool, &schemas).await?; + assert!(problems.is_empty(), "{}", problems.join("\n")); + + let mut tx = pool.begin().await?; + sqlx::raw_sql(patch().sql).execute(&mut *tx).await?; + let replay_locks: i64 = sqlx::query_scalar( + "SELECT count(*) FROM pg_locks l + JOIN pg_class c ON c.oid = l.relation + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE l.pid = pg_backend_pid() AND l.locktype = 'relation' + AND n.nspname = ANY($1)", + ) + .bind(&schemas[..]) + .fetch_one(&mut *tx) + .await?; + assert_eq!(replay_locks, 0, "a replay must lock no table"); + tx.rollback().await?; + Ok(()) +} + +#[sqlx::test(migrations = false)] +async fn replica_identity_patch_gives_up_on_a_lock_wait_and_changes_nothing( + pool: PgPool, +) -> TestResult { + migrations::run(&pool).await?; + clear_identities(&pool, &["awa"]).await?; + + let mut holder = pool.begin().await?; + sqlx::query("INSERT INTO awa.admin_dirty_kind_marks (kind) VALUES ('held')") + .execute(&mut *holder) + .await?; + + let started = std::time::Instant::now(); + let error = sqlx::raw_sql(patch().sql) + .execute(&pool) + .await + .expect_err("the patch cannot lock a table another transaction writes"); + assert!(started.elapsed() < Duration::from_secs(5)); + let db_error = error.as_database_error().expect("a database error"); + assert_eq!(db_error.code().as_deref(), Some("55P03"), "{error}"); + assert!( + db_error.message().contains("admin_dirty_kind_marks"), + "{error}" + ); + holder.rollback().await?; + + let identity: String = sqlx::query_scalar( + "SELECT relreplident::text FROM pg_class WHERE oid = 'awa.admin_dirty_queue_marks'::regclass", + ) + .fetch_one(&pool) + .await?; + assert_eq!(identity, "d", "a failed patch leaves every table unchanged"); + + sqlx::raw_sql(patch().sql).execute(&pool).await?; + assert!(patch_pending(&pool).await?.is_empty()); + Ok(()) +} + +#[sqlx::test(migrations = false)] +async fn all_tables_publication_accepts_update_and_delete(pool: PgPool) -> TestResult { + migrations::run(&pool).await?; + let receipts = custom_store("ri_receipts", true)?; + receipts.prepare_schema(&pool).await?; + sqlx::raw_sql("SET client_min_messages = error; CREATE PUBLICATION awa_ri_all FOR ALL TABLES") + .execute(&pool) + .await?; + + // Partitions created after the patch and the publication. + let legacy = custom_store("ri_legacy", false)?; + legacy.prepare_schema(&pool).await?; + let schemas = ["awa", "ri_receipts", "ri_legacy"]; + assert_update_and_delete_accepted(&pool, &schemas).await?; + + // Unique-claim release and the dirty-mark drain delete rows. + sqlx::raw_sql( + "INSERT INTO awa.job_unique_claims (unique_key, job_id) VALUES ('\\x01', 1); + DELETE FROM awa.job_unique_claims WHERE job_id = 1; + INSERT INTO awa.jobs (kind, queue, args) VALUES ('ri', 'ri_canonical', '{}');", + ) + .execute(&pool) + .await?; + let marks = "SELECT (SELECT count(*) FROM awa.admin_dirty_queue_marks) + + (SELECT count(*) FROM awa.admin_dirty_kind_marks)"; + let before: i64 = sqlx::query_scalar(marks).fetch_one(&pool).await?; + assert!(before > 0, "the canonical insert marked its queue and kind"); + sqlx::raw_sql( + "SELECT awa.recompute_dirty_admin_metadata(100); + SELECT awa.refresh_admin_metadata();", + ) + .execute(&pool) + .await?; + let after: i64 = sqlx::query_scalar(marks).fetch_one(&pool).await?; + assert_eq!(after, 0, "the drain deleted the marks"); + + // Queue-storage traffic and maintenance, on a schema prepared before and + // one prepared after the publication. + for store in [&receipts, &legacy] { + store.enqueue_batch(&pool, "ri", 2, 20).await?; + let claimed = store + .claim_runtime_batch(&pool, "ri", 20, Duration::from_secs(60)) + .await?; + assert_eq!(claimed.len(), 20); + store.complete_runtime_batch(&pool, &claimed).await?; + store.rotate(&pool).await?; + store.rotate_leases(&pool).await?; + store.rotate_claims(&pool).await?; + store.rollup_terminal_count_deltas(&pool, 4).await?; + let pruned = store.prune_oldest(&pool, Duration::ZERO).await?; + assert!(matches!(pruned, PruneOutcome::Pruned { .. }), "{pruned:?}"); + store.prune_oldest_leases(&pool).await?; + store.prune_oldest_claims(&pool).await?; + store.rebuild_terminal_counters(&pool).await?; + } + Ok(()) +} + +async fn mark_identities(pool: &PgPool) -> Result, sqlx::Error> { + sqlx::query_as( + "SELECT relname::text, relreplident::text FROM pg_class + WHERE oid IN (to_regclass('awa.admin_dirty_queue_marks'), + to_regclass('awa.admin_dirty_kind_marks')) + ORDER BY relname", + ) + .fetch_all(pool) + .await +} + +/// External runners apply repeatable scripts in name order, so +/// `R__replica_identity` can run before `R__wait_free_dirty_marks` creates the +/// mark tables. The mark tables still end up with a FULL identity, and a +/// replay of `wait_free_dirty_marks` leaves existing tables alone. +#[sqlx::test(migrations = false)] +async fn mark_tables_created_after_the_replica_identity_patch_get_their_identity( + pool: PgPool, +) -> TestResult { + migrations::run(&pool).await?; + let wait_free = migrations::SCHEMA_PATCHES + .iter() + .find(|patch| patch.name == "wait_free_dirty_marks") + .expect("the patch is registered"); + + sqlx::raw_sql( + "DROP TABLE awa.admin_dirty_queue_marks; + DROP TABLE awa.admin_dirty_kind_marks;", + ) + .execute(&pool) + .await?; + sqlx::raw_sql(patch().sql).execute(&pool).await?; + sqlx::raw_sql(wait_free.sql).execute(&pool).await?; + assert_eq!( + mark_identities(&pool).await?, + vec![ + ("admin_dirty_kind_marks".to_string(), "f".to_string()), + ("admin_dirty_queue_marks".to_string(), "f".to_string()), + ] + ); + + sqlx::raw_sql( + "ALTER TABLE awa.admin_dirty_queue_marks REPLICA IDENTITY DEFAULT; + ALTER TABLE awa.admin_dirty_kind_marks REPLICA IDENTITY DEFAULT;", + ) + .execute(&pool) + .await?; + sqlx::raw_sql(wait_free.sql).execute(&pool).await?; + assert_eq!( + mark_identities(&pool).await?, + vec![ + ("admin_dirty_kind_marks".to_string(), "d".to_string()), + ("admin_dirty_queue_marks".to_string(), "d".to_string()), + ] + ); + Ok(()) +} diff --git a/awa/tests/dirty_marks_wait_free_test.rs b/awa/tests/dirty_marks_wait_free_test.rs index 41e842eee..41391010f 100644 --- a/awa/tests/dirty_marks_wait_free_test.rs +++ b/awa/tests/dirty_marks_wait_free_test.rs @@ -546,8 +546,11 @@ async fn schema_patch_is_applied_once_by_migrate() { let pending = awa_model::migrations::pending_schema_patches(&pool) .await .unwrap(); - assert_eq!(pending.len(), 1); - assert_eq!(pending[0].name, "wait_free_dirty_marks"); + // The recreated mark tables need their replica identity again. + assert_eq!( + pending.iter().map(|patch| patch.name).collect::>(), + ["wait_free_dirty_marks", "replica_identity"] + ); awa_model::migrations::run(&pool).await.unwrap(); assert!(awa_model::migrations::pending_schema_patches(&pool) diff --git a/awa/tests/patch_upgrade_rehearsal_test.rs b/awa/tests/patch_upgrade_rehearsal_test.rs index 57c65385e..810ecb943 100644 --- a/awa/tests/patch_upgrade_rehearsal_test.rs +++ b/awa/tests/patch_upgrade_rehearsal_test.rs @@ -649,6 +649,34 @@ async fn strong_locks_held_by(pool: &PgPool, role: &str, relation_ids: &[i64]) - .unwrap() } +/// The locks the `replica_identity` patch takes on the tables it alters, as +/// formatted by `strong_locks_held_by`. +fn replica_identity_lock(lock: &str) -> bool { + let Some((relation, mode)) = lock.rsplit_once(' ') else { + return false; + }; + if relation == "awa.idx_awa_jobs_unique" { + return mode == "ShareLock"; + } + let Some((schema, table)) = relation.split_once('.') else { + return false; + }; + let delta = table + .strip_prefix("queue_terminal_count_deltas") + .is_some_and(|suffix| { + suffix.is_empty() + || suffix.strip_prefix('_').is_some_and(|slot| { + !slot.is_empty() && slot.bytes().all(|b| b.is_ascii_digit()) + }) + }); + let awa_table = schema == "awa" + && matches!( + table, + "admin_dirty_queue_marks" | "admin_dirty_kind_marks" | "job_unique_claims" + ); + mode == "AccessExclusiveLock" && (delta || awa_table) +} + async fn lane_patch_applied(pool: &PgPool) -> migrations::SchemaPatchStatus { migrations::schema_patch_status(pool) .await @@ -759,11 +787,12 @@ async fn released_patch_upgrade_and_rollback() { "insert_jobs_compat_batch", "prune_controls", "runtime_maintenance_eligibility", - "correlated_claim_tombstones" + "correlated_claim_tombstones", + "replica_identity" ], "a released 0.6.15 deployment needs the claim batch index fix, the bounded-wait \ - batch function, prune controls, the maintenance eligibility table and the \ - correlated tombstone checks" + batch function, prune controls, the maintenance eligibility table, the \ + correlated tombstone checks and replica identities" ); } @@ -832,7 +861,7 @@ async fn released_patch_upgrade_and_rollback() { if released_version == "0.6.9" { assert!(patch_sql.contains("R__prepared_lane_sequences")); } else { - assert_eq!(patch_sql.matches("-- Schema patch R__").count(), 5); + assert_eq!(patch_sql.matches("-- Schema patch R__").count(), 6); assert!(!patch_sql.contains("R__prepared_lane_sequences")); assert!( patch_sql.contains("R__claim_batch_job_index"), @@ -841,6 +870,7 @@ async fn released_patch_upgrade_and_rollback() { assert!(patch_sql.contains("R__insert_jobs_compat_batch")); assert!(patch_sql.contains("R__runtime_maintenance_eligibility")); assert!(patch_sql.contains("R__correlated_claim_tombstones")); + assert!(patch_sql.contains("R__replica_identity")); } assert!( !patch_sql.contains("-- Migration V"), @@ -878,11 +908,19 @@ async fn released_patch_upgrade_and_rollback() { .unwrap(); tx.rollback().await.unwrap(); owner.close().await; + let (identity_locks, other_locks): (Vec, Vec) = locks + .into_iter() + .partition(|lock| replica_identity_lock(lock)); + assert!( + other_locks.is_empty(), + "the patch locked pre-existing awa or queue-storage relations beyond the \ + replica identity targets: {other_locks:?}" + ); assert!( - locks.is_empty(), - "the patch locked pre-existing awa or queue-storage relations: {locks:?}" + identity_locks.contains(&"awa.job_unique_claims AccessExclusiveLock".to_string()), + "the replica_identity patch alters awa.job_unique_claims: {identity_locks:?}" ); - locks + identity_locks }; let positions_before = lane_positions(&fixture.admin).await; @@ -970,10 +1008,16 @@ async fn released_patch_upgrade_and_rollback() { "the patch must run under traffic: {enqueued_during} enqueued while awa migrate ran \ and {in_flight_count} jobs in flight when it started" ); + let unexpected_locks: Vec<&String> = sampled_locks + .iter() + .filter(|lock| !replica_identity_lock(lock)) + .collect(); assert!( - sampled_locks.is_empty(), - "awa migrate strongly locked pre-existing relations: {sampled_locks:?}" + unexpected_locks.is_empty(), + "awa migrate strongly locked pre-existing relations beyond the replica identity \ + targets: {unexpected_locks:?}" ); + let unexpected_lock_count = unexpected_locks.len(); for expected in [ "Schema: v40 (unchanged)".to_string(), format!( @@ -989,6 +1033,7 @@ async fn released_patch_upgrade_and_rollback() { "Schema patch prune_controls: applied".to_string(), "Schema patch runtime_maintenance_eligibility: applied".to_string(), "Schema patch correlated_claim_tombstones: applied".to_string(), + "Schema patch replica_identity: applied".to_string(), format!("Storage routing: unchanged (state active, engine queue_storage, schema {SCHEMA}"), ] { assert!( @@ -1458,7 +1503,7 @@ async fn released_patch_upgrade_and_rollback() { "patch_relation_locks": patch_relation_locks, "migrate_relation_ids": migrate_relation_ids, "sampled_relation_locks": sampled_locks, - "unexpected_relation_locks_sampled": sampled_locks.len(), + "unexpected_relation_locks_sampled": unexpected_lock_count, "enqueued_during_migrate": enqueued_during, "completed_during_migrate": completed_during, "in_flight_at_migrate": in_flight_count, diff --git a/awa/tests/queue_storage_runtime_test.rs b/awa/tests/queue_storage_runtime_test.rs index 48c8c5c4b..1bd53ae3c 100644 --- a/awa/tests/queue_storage_runtime_test.rs +++ b/awa/tests/queue_storage_runtime_test.rs @@ -4,6 +4,8 @@ //! queue_storage backend enabled. mod ci_timing; +#[path = "../../awa-model/src/storage_footprint_sql.rs"] +mod storage_footprint_sql; use awa::model::{ admin, batch_operations, insert, migrations, storage, AwaError, BatchOperationFilter, @@ -23568,3 +23570,275 @@ async fn test_hot_paths_do_not_lock_sealed_slot_children() { "hot paths waited on a sealed slot's children: {all_blocked:?}" ); } + +/// Relfilenode, heap size and total size of each named relation. +async fn child_storage(pool: &sqlx::PgPool, relations: &[String]) -> Vec<(String, i64, i64, i64)> { + let mut rows = Vec::with_capacity(relations.len()); + for relation in relations { + let (filenode, heap, total): (i64, i64, i64) = sqlx::query_as( + "SELECT pg_relation_filenode($1::regclass)::bigint, + pg_relation_size($1::regclass), + pg_total_relation_size($1::regclass)", + ) + .bind(relation) + .fetch_one(pool) + .await + .expect("read child storage"); + rows.push((relation.clone(), filenode, heap, total)); + } + rows +} + +fn queue_slot_children(schema: &str, slot: usize) -> Vec { + [ + "ready_segments", + "done_entries", + "ready_entries", + "ready_claim_attempt_batches", + "ready_tombstones", + "receipt_completion_batches", + "receipt_completion_tombstones", + "queue_terminal_count_deltas", + ] + .iter() + .map(|table| format!("{schema}.{table}_{slot}")) + .collect() +} + +fn claim_slot_children(schema: &str, slot: usize) -> Vec { + [ + "lease_claims", + "lease_claim_batches", + "lease_claim_closures", + "lease_claim_closure_batches", + ] + .iter() + .map(|table| format!("{schema}.{table}_{slot}")) + .collect() +} + +/// Queue and claim prune truncate only the children of the sealed slot that +/// hold storage beyond an empty table's metadata. Every child is empty +/// afterwards, children that never held a row keep their relfilenode, and the +/// slot is reused cleanly once the ring wraps. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_prune_truncates_only_ring_children_that_hold_rows() { + let (_db_guard, pool) = setup_pool(8).await; + let schema = "awa_qs_prune_used_children"; + let queue = "prune_used_children"; + let store = create_store_with_config( + &pool, + QueueStorageConfig { + schema: schema.into(), + queue_slot_count: 4, + lease_slot_count: 2, + claim_slot_count: 2, + lease_claim_receipts: true, + ..Default::default() + }, + ) + .await; + let children: Vec = queue_slot_children(schema, 0) + .into_iter() + .chain(claim_slot_children(schema, 0)) + .collect(); + let fresh = child_storage(&pool, &children).await; + assert!(fresh.iter().all(|(_, _, heap, _)| *heap == 0)); + + store.enqueue_batch(&pool, queue, 2, 40).await.unwrap(); + let claimed = store + .claim_runtime_batch(&pool, queue, 40, Duration::from_secs(60)) + .await + .unwrap(); + assert_eq!(claimed.len(), 40); + assert_eq!( + store + .complete_runtime_batch(&pool, &claimed) + .await + .unwrap() + .len(), + 40 + ); + let used = child_storage(&pool, &children).await; + + assert!(matches!( + store.rotate(&pool).await.unwrap(), + RotateOutcome::Rotated { slot: 1, .. } + )); + assert!(matches!( + store.rotate_claims(&pool).await.unwrap(), + RotateOutcome::Rotated { slot: 1, .. } + )); + let pruned = store.prune_oldest(&pool, Duration::ZERO).await.unwrap(); + assert!( + matches!(pruned, PruneOutcome::Pruned { slot: 0, .. }), + "{pruned:?}" + ); + let pruned = store.prune_oldest_claims(&pool).await.unwrap(); + assert!( + matches!(pruned, PruneOutcome::Pruned { slot: 0, .. }), + "{pruned:?}" + ); + let after = child_storage(&pool, &children).await; + + let mut truncated = 0; + let mut kept = 0; + for ((fresh, used), after) in fresh.iter().zip(&used).zip(&after) { + let relation = &fresh.0; + let rows: i64 = sqlx::query_scalar(&format!("SELECT count(*) FROM {relation}")) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(rows, 0, "{relation} keeps rows after prune"); + assert_eq!(after.2, 0, "{relation} keeps heap pages after prune"); + // Any page of an index method other than btree, GiST or BRIN (the + // GIN index on claim batches) counts as storage to reclaim. + let other_index_method: bool = sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM pg_index i + JOIN pg_class c ON c.oid = i.indexrelid + JOIN pg_am am ON am.oid = c.relam + WHERE i.indrelid = $1::regclass AND am.amname NOT IN ('btree', 'gist', 'brin'))", + ) + .bind(relation) + .fetch_one(&pool) + .await + .unwrap(); + if used.2 > 0 { + assert_ne!( + after.1, used.1, + "{relation} held rows but was not truncated" + ); + truncated += 1; + } else if used.3 == fresh.3 && !other_index_method { + assert_eq!( + after.1, used.1, + "{relation} never held a row but was truncated" + ); + kept += 1; + } + } + assert!( + truncated >= 3, + "expected ready, receipt and claim batch children with rows: {used:?}" + ); + assert!( + kept >= 3, + "expected unused children to be skipped: {used:?}" + ); + + // Wrap both rings back onto slot 0 and use it again. + for expected in [2, 3, 0] { + let rotated = store.rotate(&pool).await.unwrap(); + assert!( + matches!(rotated, RotateOutcome::Rotated { slot, .. } if slot == expected), + "{rotated:?}" + ); + } + assert!(matches!( + store.rotate_claims(&pool).await.unwrap(), + RotateOutcome::Rotated { slot: 0, .. } + )); + store.enqueue_batch(&pool, queue, 2, 7).await.unwrap(); + let claimed = store + .claim_runtime_batch(&pool, queue, 40, Duration::from_secs(60)) + .await + .unwrap(); + assert_eq!(claimed.len(), 7, "only the new jobs are claimable"); + assert_eq!( + store + .complete_runtime_batch(&pool, &claimed) + .await + .unwrap() + .len(), + 7 + ); + let generations: Vec = sqlx::query_scalar(&format!( + "SELECT DISTINCT ready_generation FROM {schema}.ready_entries_0" + )) + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!(generations, vec![4], "slot 0 holds only its new generation"); + let completed: i64 = sqlx::query_scalar(&format!( + "SELECT COALESCE(sum(cardinality(job_ids)), 0)::bigint + FROM {schema}.receipt_completion_batches_0" + )) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(completed, 7); +} + +/// A child whose rows were deleted and vacuumed away can keep a zero-byte heap +/// with expanded indexes; prune still truncates it. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_prune_truncates_a_vacuumed_child_with_expanded_indexes() { + let (_db_guard, pool) = setup_pool(8).await; + let schema = "awa_qs_prune_vacuumed_child"; + let queue = "prune_vacuumed_child"; + let store = create_store(&pool, schema).await; + let done_child = format!("{schema}.done_entries_0"); + let fresh = child_storage(&pool, std::slice::from_ref(&done_child)).await[0].clone(); + + store.enqueue_batch(&pool, queue, 2, 500).await.unwrap(); + let claimed = store + .claim_runtime_batch(&pool, queue, 500, Duration::from_secs(60)) + .await + .unwrap(); + assert_eq!(claimed.len(), 500); + store.complete_runtime_batch(&pool, &claimed).await.unwrap(); + sqlx::raw_sql(&format!("DELETE FROM {done_child}")) + .execute(&pool) + .await + .unwrap(); + sqlx::raw_sql(&format!("VACUUM (TRUNCATE TRUE) {done_child}")) + .execute(&pool) + .await + .unwrap(); + let vacuumed = child_storage(&pool, std::slice::from_ref(&done_child)).await[0].clone(); + assert_eq!(vacuumed.2, 0, "VACUUM returns every heap page"); + assert!(vacuumed.3 > fresh.3, "the indexes keep their pages"); + + assert!(matches!( + store.rotate(&pool).await.unwrap(), + RotateOutcome::Rotated { slot: 1, .. } + )); + let pruned = store.prune_oldest(&pool, Duration::ZERO).await.unwrap(); + assert!( + matches!(pruned, PruneOutcome::Pruned { slot: 0, .. }), + "{pruned:?}" + ); + let reclaimed = child_storage(&pool, std::slice::from_ref(&done_child)).await[0].clone(); + assert_ne!(reclaimed.1, vacuumed.1, "the vacuumed child is truncated"); + assert_eq!(reclaimed.3, fresh.3, "its indexes are back to empty"); +} + +#[tokio::test] +async fn test_prune_brin_fsm_baseline_follows_block_size() { + let (_db_guard, pool) = setup_pool(2).await; + // Equality is empty metadata; one more FSM page is reclaimable. + let rows: Vec<(i64, bool, bool)> = sqlx::query_as(&format!( + "WITH samples(block_size, empty_pages) AS ( + VALUES (1024::bigint, 4), (2048, 4), (4096, 3), (8192, 3), (16384, 3), (32768, 3) + ) + SELECT block_size, + empty_pages * block_size > block_size * ({depth}) AS empty_is_used, + (empty_pages + 1) * block_size > block_size * ({depth}) AS extra_is_used + FROM samples ORDER BY block_size", + depth = storage_footprint_sql::brin_fsm_empty_pages_sql!(), + )) + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!(rows.len(), 6); + for (block_size, empty_is_used, extra_is_used) in rows { + assert!( + !empty_is_used, + "empty BRIN metadata at block_size={block_size}" + ); + assert!( + extra_is_used, + "one extra FSM page at block_size={block_size}" + ); + } +} diff --git a/correctness/storage/MAPPING.md b/correctness/storage/MAPPING.md index 6818e1e78..40292305d 100644 --- a/correctness/storage/MAPPING.md +++ b/correctness/storage/MAPPING.md @@ -361,3 +361,11 @@ Executable scheduler regressions cover saturated-gate cancellation (including a `AwaSegmentedStorage.Claim` is atomic and does not describe acknowledgment before WAL durability or a post-claim cursor retry on another server. `AwaAsyncClaimCursor` supplies that refinement: a claim may be lost on local crash or standby promotion, a committed claim may survive a lost cursor advance, and a queued advance can be replayed after promotion. WAL flush and replica application preserve order. `NoSkippedJob` forbids advancing past an attempt without evidence on the current server. The unguarded config is an expected counterexample, not a passing proof. Receipt advances probe the covering slot/generation/lane range in `ready_claim_attempt_batches`; legacy advances require the exact `(job_id, run_lease)` lease and ready identity. If completion or maintenance has removed that evidence, replay conservatively leaves the cursor behind; the existing spent-prefix recovery recognizes committed dispositions. Conditional cancellation advances are derived from synchronous terminal commits and remain separate. Rust regressions replay both real receipt and legacy identities with evidence present, then remove the claim and rewind the cursor to represent a promoted standby and verify the job is claimable again. This is a deterministic state reconstruction, not a real replication/failover rehearsal. + +## Publication compatibility (0.6 port of #509) + +Replica identity controls the old-row image PostgreSQL writes for UPDATE and DELETE; it adds no lifecycle transition and no lock-order resource on any runtime path. The `replica_identity` schema patch and the installer take ACCESS EXCLUSIVE on the tables they alter, outside the ring protocols the lock-order model covers. `awa-model/tests/replica_identity_test.rs` checks every regular and partitioned relation in `awa` and in custom queue-storage schemas with and without receipts, the patch's exact lock set and its lock-free replay, its bounded lock wait, and, under a `FOR ALL TABLES` publication, UPDATE and DELETE on every table (including partitions of a schema prepared after the patch) plus the dirty-mark drain, unique-claim release, claim, completion, rotation, rollup and prune. It runs on PostgreSQL 18 as a superuser; it is not a logical-subscriber or Debezium delivery rehearsal. + +## Physical reclamation (0.6 port of #512) + +Prune now issues `TRUNCATE` only for the children of the sealed slot that hold storage beyond an empty table's metadata. It is a refinement after the existing proofs: the lock statements, the children they name and their order are unchanged, so `AwaStorageLockOrder` and `AwaQueuePruneTwoPhase` model the same acquisitions. The storage probe runs under the slot's ACCESS EXCLUSIVE locks and reads `pg_relation_size` of each child heap, its TOAST table and their indexes, which takes ACCESS SHARE on those relations; nothing can hold a conflicting lock on them without first locking the child heap, except autovacuum's conditional truncation of a TOAST table, which never waits on prune, and a wait past the prune lock timeout reports `blocked` as before. A skipped child has no heap page and so no tuple of any generation, visible or not, so slot reuse, the failed-row carry (which writes to the current slot) and the rollup fold (to which an empty child contributes nothing) are unaffected, and an admin read still sees the relfilenode change of every child that held rows. The 0.6.17 empty-slot fast path still decides on visible rows, so a lease child emptied by completion `DELETE` is reclaimed by vacuum, not prune, as in 0.6.17. Runtime tests cover a receipt-mode queue and claim slot with used and unused children, reuse of the slot after the ring wraps, and a vacuumed child with expanded indexes. diff --git a/docs/architecture.md b/docs/architecture.md index 9059c3b46..4c027a9d7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -227,7 +227,7 @@ The common safety pattern is: 3. Prove cheap skip gates before the exclusive-lock path. Queue prune checks active leases and pending ready lanes before receipt-closure proof; claim prune proves open-claim closure before child locks. 4. Take child-table `ACCESS EXCLUSIVE` locks with a short transaction-local `lock_timeout` so maintenance gives up promptly under contention. 5. Recheck the skip gates after acquiring the partition locks so rows that committed while the lock waited are visible before truncate. -6. `TRUNCATE` only partitions that are proven inactive. +6. `TRUNCATE` only partitions that are proven inactive, and of those only the ones that hold storage beyond an empty table's metadata (any heap, TOAST, free-space-map or visibility-map page, or index pages beyond an empty btree, GiST or BRIN index). Every child of the slot is still locked; a child with no heap page holds no row, so leaving it keeps its relfilenode without changing what the slot holds when it is reused. This is why queue storage's hot-path reclamation is a rotation-and-prune discipline, not ordinary row-by-row vacuum cleanup. Ordinary retention cleanup still exists for DLQ rows, stale descriptors, stale runtime snapshots, and the canonical compatibility path. diff --git a/docs/deploying-on-managed-postgres.md b/docs/deploying-on-managed-postgres.md index 9a6f35580..b33205714 100644 --- a/docs/deploying-on-managed-postgres.md +++ b/docs/deploying-on-managed-postgres.md @@ -152,6 +152,34 @@ The queue-storage substrate ships aggressive per-table autovacuum storage parame Note that no autovacuum setting helps while a horizon is pinned — vacuum cannot reclaim what an open snapshot can still see. Rules 1–3 prevent the pin; rule 4 makes recovery fast after it releases. +## Logical replication and CDC + +If the queue database is also a change-data-capture source (Debezium, a native `CREATE SUBSCRIPTION`, a managed CDC service), keep the awa schema out of the publication. awa's tables are a queue, not application state: every enqueue, claim, heartbeat and completion is a write, and a slot that decodes all of them retains WAL on the primary and ships traffic no consumer wants. + +**Recommended: publish only the application's tables.** + +```sql +-- Native logical replication: list the tables or schemas you want. +CREATE PUBLICATION app_changes FOR TABLES IN SCHEMA public; +``` + +With Debezium, `publication.autocreate.mode=all_tables` (the default) creates a `FOR ALL TABLES` publication; set `publication.autocreate.mode=filtered` and `schema.exclude.list=awa` (plus any custom queue-storage schema) so the connector creates a publication from its own filters, or create the publication yourself and point the connector at it with `publication.name`. + +**If the awa schema does end up in a publication, awa still works once the `replica_identity` schema patch is applied.** Without it, PostgreSQL rejects `UPDATE` and `DELETE` on the awa tables that have no primary key, so the admin dirty-mark drain fails with `cannot delete from table "admin_dirty_queue_marks" because it does not have a replica identity and publishes deletes` and unique-claim release fails the same way (see [Troubleshooting](troubleshooting.md#cannot-delete-from-table--because-it-does-not-have-a-replica-identity)). Tables with a primary key use it; `awa.job_unique_claims` uses its uniqueness index; the rest use `REPLICA IDENTITY FULL`: + +| Table | Why FULL | Cost | +| --- | --- | --- | +| `awa.admin_dirty_queue_marks`, `awa.admin_dirty_kind_marks` | ADR-045 forbids any index or constraint on them | A ~40-byte old-row image per mark the drain deletes | +| `{schema}.queue_terminal_count_deltas` and its partitions | Append-only; reclaimed by `TRUNCATE`, never updated or deleted | None: identity only affects `UPDATE` and `DELETE` WAL | + +PostgreSQL does not copy a partitioned table's replica identity to its partitions, so the patch and the queue-storage installer set it on the parent and on each partition, including partitions of a schema prepared after the patch. + +What to expect when the queue is published: + +- WAL volume and slot lag follow the queue traffic, not the identity choice. The slot decodes every job transition. Budget WAL retention (`max_slot_wal_keep_size`) for the consumer's worst-case outage and alert on `pg_replication_slots.confirmed_flush_lsn` lag: a stalled consumer retains WAL on the primary. +- Prune is a `TRUNCATE`, and publications publish truncate by default. Ring prune reclaims a whole partition (`ready_entries_N`, `done_entries_N`, `leases_N`, `lease_claims_N`, `queue_terminal_count_deltas_N`, ...) with `TRUNCATE`, so a native subscriber truncates its copy of that partition and a Debezium stream sees a truncate event unless `skipped.operations` includes `t` (the default). Terminal job rows are not deleted row by row, so a consumer that wants per-job delete events will not get them, and a consumer that mirrors the tables must also create the partitions. +- Everything else behaves as without a publication: completion, unique jobs, cron, the DLQ, batch operations and the admin UI are unaffected. + ## Things that broke for us in staging — worth pre-empting These all eventually have fixes or workarounds, but each one cost hours the first time: diff --git a/docs/migrations.md b/docs/migrations.md index 60ff073cd..fdc5c9144 100644 --- a/docs/migrations.md +++ b/docs/migrations.md @@ -251,6 +251,8 @@ That writes one `V__.sql` file per migration, plus one `R__.claim_ready_runtime` in every queue-storage schema from the function's own definition, adding `OFFSET 0` to the two tombstone lookups, and refreshes the installer so new schemas carry the same check. It replaces functions only and takes no lock on any awa or queue-storage table. Function names, arguments and results are unchanged, so released 0.6 binaries run the same on a patched or an unpatched schema. Run it as the owner of the claim functions. +`R__replica_identity.sql` gives every awa table a replica identity, so `UPDATE` and `DELETE` keep working when the awa schema is in a logical-replication publication (`FOR ALL TABLES`, or `FOR TABLES IN SCHEMA awa`). Tables with a primary key already have one. The patch sets `REPLICA IDENTITY FULL` on `awa.admin_dirty_queue_marks`, `awa.admin_dirty_kind_marks`, and `queue_terminal_count_deltas` and each of its partitions in every queue-storage schema, sets `awa.job_unique_claims` to `REPLICA IDENTITY USING INDEX idx_awa_jobs_unique`, and refreshes the installer so schemas prepared later get the same. Unlike the other patches it takes table locks: each `ALTER TABLE ... REPLICA IDENTITY` takes `ACCESS EXCLUSIVE` on its table (and `SHARE` on `idx_awa_jobs_unique`), which blocks every read and write of that table and is held until the patch's transaction commits. The change is catalog-only, so once granted the locks are held for milliseconds; in an `awa migrate --sql` bundle they are held until the bundle commits, so keep `R__replica_identity.sql` last. Each lock wait is bounded by the session's `lock_timeout`, or 300 ms when it has none; while it waits, statements on that table queue behind it. A timeout fails the patch with `55P03`, changes nothing, and can be run again. A table that already has its identity is skipped, so a re-run takes no lock. `R__wait_free_dirty_marks.sql` gives the mark tables the same identity when it creates them, so a runner that applies `R__replica_identity.sql` first still ends with both set. Run it as the owner of the awa and queue-storage tables. See the [0.6.17 to 0.6.18 upgrade guide](upgrade-0.6.17-to-0.6.18.md) and [logical replication and CDC](deploying-on-managed-postgres.md#logical-replication-and-cdc). + `R__claim_batch_job_index.sql` indexes compact claim batches by job id. It installs `.ensure_claim_batch_job_index(slot)` in every queue-storage schema and builds no index itself, so it takes no lock on any awa or queue-storage table and is safe inside a long transaction under live traffic. Claim prune calls that function right after truncating a claim partition, which builds the GIN index on `lease_claim_batches.job_ids` while the partition is empty, so the index covers the ring within one claim-ring cycle. Only a valid, non-partial GIN index on `job_ids` counts; a different index or relation already named `lease_claim_batches__job_ids_idx` is left in place with a warning naming it, and that slot keeps scanning until it is dropped or renamed. Do not build the index yourself on a populated partition: the build holds a `SHARE` lock, which blocks claim inserts into that partition, for about half a minute per 100k batches. Run the script as the owner of the queue-storage tables. Binaries before 0.6.15 neither use nor build the index and are unaffected by it, and binaries that use the index read the same rows, by scan, from a schema or partition without it. `awa migrate` reports the schema version, each patch and what it updated, and storage routing before and after. It exits 1 if a patch is still missing somewhere, such as a queue-storage schema the migrating role cannot use, and names it. `awa::migrations::schema_patch_status` gives the same per-patch view from Rust. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index e3c854d16..e51e82c2c 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -586,6 +586,17 @@ Fix: SELECT awa.active_queue_storage_schema(); ``` +### `cannot delete from table ... because it does not have a replica identity` + +```text +cannot delete from table "admin_dirty_queue_marks" because it does not +have a replica identity and publishes deletes +``` + +The awa schema is in a logical-replication publication (`CREATE PUBLICATION ... FOR ALL TABLES`, which Debezium creates by default, or `FOR TABLES IN SCHEMA awa`) and the database lacks the `replica_identity` schema patch, so some awa tables have no primary key or replica identity. PostgreSQL then rejects every `UPDATE` and `DELETE` on those tables: the admin dirty-mark drain fails on each maintenance tick, so queue and kind counts go stale, and unique-claim release fails. + +Fix: run `awa migrate` with 0.6.18 or later, which applies the patch; it takes a brief `ACCESS EXCLUSIVE` lock on each table it changes (see the [upgrade guide](upgrade-0.6.17-to-0.6.18.md#replica-identity-locks)). Independently, prefer keeping the awa schema out of the publication; see [logical replication and CDC](deploying-on-managed-postgres.md#logical-replication-and-cdc). + ### `register at least one worker before starting the runtime` Cause: diff --git a/docs/upgrade-0.6.17-to-0.6.18.md b/docs/upgrade-0.6.17-to-0.6.18.md new file mode 100644 index 000000000..4e038c535 --- /dev/null +++ b/docs/upgrade-0.6.17-to-0.6.18.md @@ -0,0 +1,60 @@ +# Upgrading from 0.6.17 to 0.6.18 + +0.6.18 is a rolling patch upgrade: the schema stays at v40 and traffic keeps running throughout. Every awa table gets a replica identity, so awa keeps working when its schema is in a logical-replication publication, and ring prune truncates only the partitions that hold rows. + +Binaries and the migration can roll in either order. + +## Upgrade + +1. Install the 0.6.18 CLI. +2. Run `awa migrate` once with the schema owner as the effective role, while workers and producers keep running: + + ```bash + PGOPTIONS='-c role=awa_owner' \ + awa --database-url "$AWA_MIGRATOR_DATABASE_URL" migrate + ``` + + On a 0.6.17 database the output reports `Schema patch replica_identity: applied`. Unlike the earlier 0.6 patches, it takes table locks; see [replica identity locks](#replica-identity-locks). + +3. Roll workers, producers and admin processes to 0.6.18, in any order. + +0.6.17 binaries run unchanged against the patched schema. + +### External migration runners + +`awa migrate --sql --pending` prints `R__replica_identity` on a 0.6.17 database. Apply it as a repeatable script after `V40` and after the other `R__` scripts, as the owner of the awa and queue-storage tables. Extract the full set with: + +```bash +awa --database-url "$AWA_MIGRATOR_DATABASE_URL" migrate --extract-to ./sql/awa --from 40 +``` + +Replace all `R__` files together with that set and keep previously applied `V` files unchanged. + +## Replica identity locks + +PostgreSQL rejects `UPDATE` and `DELETE` on a table that is in a logical-replication publication and has no primary key, replica-identity index or `FULL` identity. On 0.6.17, 19 awa tables have none: `awa.admin_dirty_queue_marks`, `awa.admin_dirty_kind_marks`, `awa.job_unique_claims` and the 16 partitions of `awa.queue_terminal_count_deltas`, plus the partitions of any custom queue-storage schema. With the awa schema in a `FOR ALL TABLES` or `FOR TABLES IN SCHEMA awa` publication, the admin dirty-mark drain fails on every maintenance tick and unique-claim release fails. + +The `replica_identity` patch runs `ALTER TABLE ... REPLICA IDENTITY` on each of those tables that lacks an identity: + +- `REPLICA IDENTITY FULL` on the two mark tables, and on `queue_terminal_count_deltas` and each of its partitions in every queue-storage schema. PostgreSQL 17 and 18 do not copy a partitioned table's identity to existing or new partitions, so the parent and each partition are set, and the queue-storage installer sets them for schemas prepared later. +- `REPLICA IDENTITY USING INDEX idx_awa_jobs_unique` on `awa.job_unique_claims`. + +Each `ALTER` takes `ACCESS EXCLUSIVE` on its table, and the `USING INDEX` form also takes `SHARE` on `idx_awa_jobs_unique`. These locks block every read and write of the table, including the trigger inserts every canonical job transition makes into the mark tables and every unique insert, and they are held until the patch's transaction commits. What to expect: + +- Once granted, the locks last a few milliseconds: the change is catalog-only and rewrites no data. +- The wait is bounded. Each lock wait is limited by the migrating session's `lock_timeout`, or 300 ms when it has none. While one lock waits, statements on that table queue behind it, so a writer can stall for up to that bound, plus the time the patch holds locks it has already taken. +- A lock that cannot be granted in time, for example behind a long transaction that wrote to a mark table, fails the patch with SQLSTATE `55P03` naming the table. Nothing is changed; run `awa migrate` again, or end the long transaction first. +- A table that already has its identity is skipped, so a re-run after success takes no lock. +- With `awa migrate --sql`, all pending patches run in one transaction and the locks are held until it commits; `R__replica_identity` comes last so that is immediately after it. + +Run the patch at a quiet moment if writers cannot tolerate a stall of a few hundred milliseconds. Replica identity changes only what PostgreSQL writes to WAL for `UPDATE` and `DELETE`: the mark tables log an old-row image of about 40 bytes per drained mark, and the delta tables are only inserted into and truncated, so they log nothing extra. + +Keeping the awa schema out of publications remains the recommendation; see [logical replication and CDC](deploying-on-managed-postgres.md#logical-replication-and-cdc). + +## Prune truncates only children that hold rows + +This part needs no migration and applies once the maintenance leader runs 0.6.18. Queue, lease and claim prune lock every child of the sealed slot exactly as 0.6.17 does, then `TRUNCATE` only the children that hold storage; empty children keep their relfilenode. Expect fewer relfilenode changes and less WAL per prune, most visibly on storage engines where each truncate costs I/O. Prune outcomes, metrics and lock waits are unchanged. + +## Rollback + +Roll binaries back to 0.6.17 in any order. Leave the patch in place: 0.6.17 runs unchanged against tables with a replica identity.