diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b182bf77..803430e91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -401,6 +401,23 @@ true until the next version shipped. ### Fixed +- **A concurrent `UPDATE` of the same row now fails with 40001 on a table with a + unique index, as documented, not 23505 (#1421).** Two sessions updating one row of + a table with a PRIMARY KEY or UNIQUE index: the second used to wait on the first's + new key and then fail with `unique_violation`, naming a duplicate the user never + wrote, so an application retrying on `serialization_failure` did not retry. The + same happened to a row two sessions moved between partitions of a parent with a + key. The update path now serializes the row before its new version reaches the + unique check. On such tables each row is serialized as the `UPDATE` reaches it, so + two multi-row `UPDATE`s reaching shared rows in opposite orders can now deadlock + (40P01), as a heap can. Each row's check reads its row group's committed delete + marks. On PostgreSQL 17, an `UPDATE` of a keyed table took about 25% to 50% + longer: 100,000 rows updated where half the table was deleted, and 200,000 where + nothing was (three runs each, by both of us). A table without a unique index is + unchanged. `test/update_conc.sh` and the new + `test/partition_rows.sh`, with `test_partition_rows.py`, which also holds such an + `UPDATE` to bounded memory. + - **The Iceberg wrapper no longer reads server files while planning for a role the scan would refuse.** `EXPLAIN` on an Iceberg foreign table read its `metadata_path` for any role holding SELECT on it. That showed a role without `pg_read_server_files` the diff --git a/docs/limitations.md b/docs/limitations.md index edcaa84cd..7b343fd3b 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -601,7 +601,12 @@ same way as a `CREATE INDEX` that is not concurrent. Turn projection scans off w is lost. This holds at every isolation level and needs no unique index. Retry the transaction on this error, as you retry one at `REPEATABLE READ`. The behavior is stricter than a heap at `READ COMMITTED`. A heap re-applies both - updates transparently there; pgColumnar makes the loser retry. The GUC + updates transparently there; pgColumnar makes the loser retry. The same holds + for a row an `UPDATE` moves to another partition. On a table with a unique index, + an `UPDATE` serializes each row as it reaches it, not at the end of the statement. + Two multi-row `UPDATE`s that reach shared rows in opposite orders can therefore + deadlock. One of them then gets `deadlock_detected` (40P01), which is also + retryable. The GUC `pgcolumnar.enable_row_update_lock` (default on) controls this serialization. The GUC `pgcolumnar.row_lock_buckets` (default 1024) bounds the held locks per storage, so a bulk update cannot exhaust the lock table. Unrelated rows may @@ -609,6 +614,14 @@ same way as a `CREATE INDEX` that is not concurrent. Turn projection scans off w reason, two multi-statement transactions can rarely deadlock, even on disjoint rows. This happens when their bucketed locks collide in opposite statement order. Retry resolves it, as with any serialization failure. +- `SERIALIZABLE` works on columnar tables: a write skew between two serializable + transactions fails one of them with `serialization_failure` (40001), as on a + heap. The access method takes no predicate locks of its own. The protection comes + from PostgreSQL's serializable isolation on pgColumnar's own catalog tables, which + every read of a columnar table consults and every write changes. + `test/serializable.sh` covers two shapes. In one, each transaction updates a row, + in the same row group or in different ones. In the other, each inserts a row after + a scan of the whole table or an index scan. - Concurrent inserts of the same unique key go in sequence. The server therefore always finds the conflict. Before a new row reaches the uniqueness check, the access method takes an advisory lock with the scope of the transaction. The key @@ -657,7 +670,8 @@ other than where the feature is used: `ON CONFLICT DO NOTHING` takes no row lock and works, with or without a conflict target. -`CREATE TABLE` refuses an unlogged columnar table for the same reason. Both cases +`CREATE TABLE` refuses an unlogged columnar table for the same reason, and so does +`ALTER TABLE ... SET UNLOGGED`. Both cases follow one rule. This access method refuses a configuration that it cannot support at the point where you choose it. It does not refuse it at each later use. @@ -783,7 +797,11 @@ therefore still be cheaper as a scan. of a deleted row only while a transaction that began before the delete is still running, as it does on heap. - `CREATE INDEX CONCURRENTLY` (the concurrent validate path) and partial - block-range index builds are not supported. + block-range index builds are not supported. A refused `CREATE INDEX + CONCURRENTLY` leaves its index behind, marked invalid, as PostgreSQL does for any + concurrent build that fails. Drop that index before you build it again. +- `ALTER TABLE ... SET TABLESPACE` is not supported on a columnar table. It is + refused, and the table stays where it was. ## Constraints on the import path diff --git a/src/columnar.h b/src/columnar.h index cb0f2c359..aea068ac8 100644 --- a/src/columnar.h +++ b/src/columnar.h @@ -1065,6 +1065,11 @@ extern bool PgColumnarRowWriteConflict(Relation rel, ItemPointer otid, TM_FailureData *tmfd, TM_Result *result); extern void PgColumnarSerializeFlushRows(uint64 storageId, const uint64 *rows, int nrows); +extern void PgColumnarSerializeRowNow(Relation rel, uint64 rowNumber); +extern bool PgColumnarHasUniqueIndex(Relation rel); +extern NativeRowGroupMetadata *PgColumnarLookupRowGroup(uint64 storageId, + Snapshot metaSnapshot, + uint64 rowNumber); /* ------------------------------------------------------------------------- * planner integration (pgcolumnar_customscan.c, spec 8.3, 9) diff --git a/src/columnar_reader.c b/src/columnar_reader.c index 6fa69e843..98b29594a 100644 --- a/src/columnar_reader.c +++ b/src/columnar_reader.c @@ -4472,6 +4472,18 @@ pgcolumnar_lookup_row_group(uint64 storageId, Snapshot metaSnapshot, return pgcolumnar_group_memo_walk(m, rowNumber); } +/* + * PgColumnarLookupRowGroup + * The row group covering rowNumber under metaSnapshot, or NULL, through the + * same memo the fetch path uses. For the update path's single-row conflict + * check (#1421), which needs one group, not the whole list. + */ +NativeRowGroupMetadata * +PgColumnarLookupRowGroup(uint64 storageId, Snapshot metaSnapshot, uint64 rowNumber) +{ + return pgcolumnar_lookup_row_group(storageId, metaSnapshot, rowNumber); +} + /* * Find the entry for this group in this command, or prepare an empty one. * diff --git a/src/columnar_row_lock.c b/src/columnar_row_lock.c index 6fcacf5c0..f43206b54 100644 --- a/src/columnar_row_lock.c +++ b/src/columnar_row_lock.c @@ -343,3 +343,98 @@ PgColumnarSerializeFlushRows(uint64 storageId, const uint64 *rows, int nrows) errmsg("columnar: could not serialize access due to concurrent update or delete of the same row"), errhint("Retry the transaction."))); } + +/* + * PgColumnarSerializeRowNow + * Serialize one row on its identity NOW, during the UPDATE, rather than at + * the flush (#1421). Take the row's bucket lock, waiting for a concurrent + * writer of the row to finish, then raise the same retryable + * serialization_failure if that writer committed the row as deleted. + * + * WHY NOT WAIT FOR THE FLUSH. An UPDATE of a table with a unique index takes + * the new version's unique-key lock before the flush. When a concurrent + * transaction updated the same row, its new version holds that key: the + * loser waited on it, and when the winner committed the executor's unique + * check raised unique_violation (23505), a duplicate the user never wrote, + * before the flush that raises 40001 was reached. docs/limitations.md + * promises 40001 and says the serialization "needs no unique index". + * + * Called only for a relation with a unique index, so a table without one + * keeps the batched, sorted acquisition at flush. The flush still runs and + * re-takes this lock, which the lock manager makes a no-op. Taking locks in + * row order during the statement can deadlock two multi-row UPDATEs that + * reach shared rows in opposite orders (40P01, also retryable), as heap can. + * + * ONE GROUP, NOT THE LIST. The batch reads every group's committed mask; per + * row that would be O(groups). The covering group comes through the fetch + * path's memo, and only its delete vector is read. + */ +void +PgColumnarSerializeRowNow(Relation rel, uint64 rowNumber) +{ + uint64 storageId; + uint32 numBuckets; + LOCKTAG tag; + Snapshot meta; + NativeRowGroupMetadata *rg; + bool deleted = false; + MemoryContext tmp; + MemoryContext old; + + if (!pgcolumnar_enable_row_update_lock) + return; + + storageId = PgColumnarStorageId(rel); + numBuckets = (uint32) Max(1, pgcolumnar_row_lock_buckets); + SET_LOCKTAG_ADVISORY(tag, MyDatabaseId, (uint32) storageId, + (uint32) (row_identity_lock_key(storageId, rowNumber) % numBuckets), + PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY); + (void) LockAcquire(&tag, ExclusiveLock, false /* transaction lock */ , + false /* wait */ ); + + /* + * With the lock held, the latest committed state: a delete committed after + * this statement's snapshot is visible here. This transaction's own deletes + * are in its buffer, not in this committed view, so they are not conflicts. + * + * The reads go in a context of their own, deleted before returning. This runs + * once per updated row, and in the caller's context each call would keep a copy + * of the group's delete bitmap until the statement ends: 402 MB for 25,000 rows + * of a 50,000-row group (OffgridwithJD, review of #1422). + */ + tmp = AllocSetContextCreate(CurrentMemoryContext, + "pgcolumnar row serialization check", + ALLOCSET_SMALL_SIZES); + old = MemoryContextSwitchTo(tmp); + PushActiveSnapshot(GetLatestSnapshot()); + meta = PgColumnarCatalogSnapshot(GetActiveSnapshot()); + rg = PgColumnarLookupRowGroup(storageId, meta, rowNumber); + if (rg != NULL) + { + List *maskList = PgColumnarReadDeleteVectorList(storageId, + rg->groupNumber, meta); + uint64 bit = rowNumber - rg->firstRowNumber; + ListCell *mc; + + foreach(mc, maskList) + { + DeleteVectorMetadata *rm = (DeleteVectorMetadata *) lfirst(mc); + + if (rm->bitmap != NULL && (bit >> 3) < rm->bitmapLen && + (rm->bitmap[bit >> 3] & (1 << (bit & 7))) != 0) + { + deleted = true; + break; + } + } + } + PopActiveSnapshot(); + MemoryContextSwitchTo(old); + MemoryContextDelete(tmp); + + if (deleted) + ereport(ERROR, + (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE), + errmsg("columnar: could not serialize access due to concurrent update or delete of the same row"), + errhint("Retry the transaction."))); +} diff --git a/src/columnar_tableam.c b/src/columnar_tableam.c index 6c7d94207..9bbff6453 100644 --- a/src/columnar_tableam.c +++ b/src/columnar_tableam.c @@ -2236,6 +2236,19 @@ pgcolumnar_tuple_delete(COLUMNAR_TUPLE_DELETE_ARGS) if (PgColumnarRowWriteConflict(rel, tid, cid, wait, tmfd, &conflict)) return conflict; + /* + * A row moving to another partition is deleted here and inserted there, and + * the insert reaches the destination's unique check before this delete is + * flushed. Serialize it now, as the update path does, so a concurrent mover + * of the same row loses with 40001 rather than 23505 (#1421). + */ +#if PG_VERSION_NUM >= 190000 + if ((options & TABLE_DELETE_CHANGING_PARTITION) != 0) +#else + if (changingPart) +#endif + PgColumnarSerializeRowNow(rel, rowNumber); + PgColumnarMarkRowDeleted(rel, rowNumber); return TM_Ok; } @@ -2270,6 +2283,14 @@ pgcolumnar_tuple_update(COLUMNAR_TUPLE_UPDATE_ARGS) return conflict; } + /* + * #1421: with a unique index, the new version's unique-key lock and check + * come before the flush that serializes the old row, so a loser waited on + * the winner's key and failed with 23505. Serialize the old row first. + */ + if (PgColumnarHasUniqueIndex(rel)) + PgColumnarSerializeRowNow(rel, oldRowNumber); + PgColumnarMarkRowDeleted(rel, oldRowNumber); writeState = PgColumnarGetWriteState(rel); diff --git a/src/columnar_unique.c b/src/columnar_unique.c index 70b0d0d20..8a421d4c8 100644 --- a/src/columnar_unique.c +++ b/src/columnar_unique.c @@ -326,6 +326,21 @@ pgcolumnar_acquire_key_lock(Oid indexOid, uint32 bucket) false /* wait */ ); } +/* + * PgColumnarHasUniqueIndex + * Does rel have a unique index that a new row version is checked against? + * The same set PgColumnarLockUniqueKeys locks for (unique, immediate, valid, + * ready), read from the same cache. The update path asks it to decide + * whether a row must be serialized before its new version is inserted + * (#1421). Independent of enable_unique_insert_lock: without that lock the + * btree's own uniqueness check still waits on the winner's key. + */ +bool +PgColumnarHasUniqueIndex(Relation rel) +{ + return pgcolumnar_unique_lookup(rel)->nIndexes > 0; +} + /* * PgColumnarLockUniqueKeys * For a row about to be inserted into rel (through the table AM), take a diff --git a/test/advisory_lock_class.sh b/test/advisory_lock_class.sh index db4f5a058..03f3caa8e 100755 --- a/test/advisory_lock_class.sh +++ b/test/advisory_lock_class.sh @@ -245,4 +245,126 @@ check "row_lock_buckets is fixed at server start, so a session cannot change it "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At -v VERBOSITY=sqlstate -c "SET pgcolumnar.row_lock_buckets = 7;" 2>&1 | sed -n 's/^.*ERROR:[[:space:]]*\([0-9A-Z]\{5\}\).*$/\1/p')|$(q "SELECT context || '|' || setting FROM pg_settings WHERE name = 'pgcolumnar.row_lock_buckets'")" \ "55P02|postmaster|1024" +# The documented bound (docs/configuration.md): row_lock_buckets "bounds the row locks +# a transaction holds, so a bulk update cannot exhaust the lock table". One transaction +# updates 200000 rows and then counts its own class-104 locks (one -c string is one +# transaction): one per bucket, not one per row. A row lock per row would not fit the +# default lock table at all. +psql_run "CREATE TABLE rlb (id int, v int) USING pgcolumnar; + INSERT INTO rlb SELECT g, g FROM generate_series(1, 200000) g;" >/dev/null +check "a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them" \ + "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \ + -c "UPDATE rlb SET v = v + 1; + SELECT count(*) || '|' || current_setting('pgcolumnar.row_lock_buckets') FROM pg_locks + WHERE pid = pg_backend_pid() AND locktype = 'advisory' AND objsubid = 104;" 2>&1 | + grep -E '^[0-9]+[|]|ERROR' | tail -1)" "1024|1024" +check "and the UPDATE committed every row" "$(q "SELECT count(*) || '|' || sum(v - id) FROM rlb")" "200000|200000" + +# ---- the row-identity lock cannot be reached from SQL (#1341 item 56) ------------ +# The INSERT arms above ask this of the unique-key and storage-row locks; nothing +# asked it of class 104. An UPDATE is held open and every advisory lock it took is +# read. None may be in field 1 or 2, the only classes SQL can produce. Then a second +# session takes each of those exact tags with pg_advisory_xact_lock(int4, int4), and +# a columnar UPDATE of the same row must not wait on it. An oid above the int4 range +# is addressed by its negative int4, the same 32 bits lockfuncs.c puts in the tag. +rl_wait_for() { # rl_wait_for APPLICATION_NAME -> its pid, once it has named itself + local i pid="" + for i in $(seq 1 100); do + pid="$(q "SELECT pid FROM pg_stat_activity WHERE application_name = '$1'")" + [ -n "$pid" ] && break + sleep 0.1 + done + echo "$pid" +} +rl_end() { # rl_end PID BGPID: end a holder's backend and its psql + q "SELECT pg_terminate_backend($1)" >/dev/null 2>&1 + kill "$2" 2>/dev/null; wait "$2" 2>/dev/null + local i + for i in $(seq 1 60); do + [ "$(q "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND pid <> pg_backend_pid()")" = 0 ] && break + sleep 0.2 + done +} +env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -c "BEGIN; UPDATE rl SET v = v + 1 WHERE id = 7; SET application_name = 'rl_tagger'; SELECT pg_sleep(30); COMMIT;" \ + >/dev/null 2>&1 & +rl_bg=$! +rl_pid="$(rl_wait_for rl_tagger)" +RL_TAGS="$(q "SELECT CASE WHEN classid::bigint > 2147483647 THEN classid::bigint - 4294967296 ELSE classid::bigint END + || ' ' || CASE WHEN objid::bigint > 2147483647 THEN objid::bigint - 4294967296 ELSE objid::bigint END + || ' ' || objsubid + FROM pg_locks WHERE locktype = 'advisory' AND granted AND pid = ${rl_pid:-0}")" +rl_end "${rl_pid:-0}" "$rl_bg" +RL_N="$(printf '%s\n' "$RL_TAGS" | grep -c '[0-9]')" +rl_reach="" +while read -r _c _o _f; do + case "${_f:-}" in 1|2) rl_reach="$rl_reach ($_c,$_o,field4=$_f)" ;; esac +done <<<"$RL_TAGS" +check "the locks an UPDATE takes are not in a SQL-reachable class" \ + "$(if [ "$RL_N" -eq 0 ]; then echo "unknown (no advisory lock seen; holder pid ${rl_pid:-none})" + elif [ -n "$rl_reach" ]; then echo "reachable:$rl_reach" + else echo unreachable; fi)" "unreachable" +rl_grab="BEGIN;" +while read -r _c _o _f; do + [ -z "${_f:-}" ] && continue + rl_grab="$rl_grab SELECT pg_advisory_xact_lock($_c::int, $_o::int);" +done <<<"$RL_TAGS" +env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -c "$rl_grab SET application_name = 'rl_user'; SELECT pg_sleep(30); COMMIT;" >/dev/null 2>&1 & +rl_bg=$! +rl_pid="$(rl_wait_for rl_user)" +check "premise: the other session holds every one of those tags through the SQL functions" \ + "$(q "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND granted AND objsubid = 2 + AND pid = ${rl_pid:-0}")" "$RL_N" +rl_upd="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \ + -c "SET statement_timeout = '10s';" -c "UPDATE rl SET v = v + 1 WHERE id = 7;" 2>&1)" +case "$rl_upd" in + *timeout*|*canceling*) rl_verdict="BLOCKED by the user lock" ;; + *ERROR*) rl_verdict="ERROR: $(head -1 <<<"$rl_upd")" ;; + *) rl_verdict=ok ;; +esac +check "a user advisory lock on those tags does not block a columnar UPDATE" "$rl_verdict" "ok" +rl_end "${rl_pid:-0}" "$rl_bg" + +# ---- bulk_parallel_writer set by hand still serialises a first write (#1341 item 55) -- +# The setting lets parallel_copy's loaders skip the storage-row creation lock (class +# 102). The reference says setting it by hand is safe, because the skip fires only +# once the storage row is committed. Two sessions with it on each make the first +# write to a committed, empty table: the second must wait for the first and then +# land its row. A skip that did not wait for the row to be committed lets both +# create it, and the second fails on storage_pkey. +# +# The first writer commits only once the suite has looked for the second one waiting +# and then inserted a committed flag row, not after a fixed sleep: a slow second +# writer would otherwise start after the first had committed, and the premise below +# would fail on timing (OffgridwithJD, review of #1415). The pytest twin does the same. +psql_run "CREATE TABLE bw (id int) USING pgcolumnar; CREATE TABLE bw_go (go int) USING heap;" +env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -c "SET statement_timeout = '60s'; BEGIN; SET LOCAL pgcolumnar.bulk_parallel_writer = on; INSERT INTO bw VALUES (1); SET application_name = 'bw_first'; + DO \$\$ BEGIN WHILE NOT EXISTS (SELECT 1 FROM bw_go) LOOP PERFORM pg_sleep(0.1); END LOOP; END \$\$; COMMIT;" \ + >/dev/null 2>&1 & +bw_bg=$! +bw_pid="$(rl_wait_for bw_first)" +env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \ + -c "SET pgcolumnar.bulk_parallel_writer = on; SET statement_timeout = '30s';" -c "INSERT INTO bw VALUES (2);" \ + >"$PGC_WORKDIR/bw_second.out" 2>&1 & +bw_bg2=$! +# The second write must be waiting on the first before the first commits, or the arm +# measures two writes in sequence. +bw_waited=no +for _i in $(seq 1 100); do + [ "$(q "SELECT count(*) FROM pg_stat_activity WHERE wait_event_type = 'Lock' AND query LIKE 'INSERT INTO bw%'")" -ge 1 ] && { bw_waited=yes; break; } + sleep 0.1 +done +psql_run "INSERT INTO bw_go VALUES (1);" # release the first writer, whatever the poll saw +check "premise: the second first write waited on the first" "$bw_waited" "yes" +wait "$bw_bg2"; wait "$bw_bg" +bw_out="$(cat "$PGC_WORKDIR/bw_second.out")" +case "$bw_out" in + *ERROR*) bw_verdict="ERROR: $(grep -m1 ERROR <<<"$bw_out")" ;; + *) bw_verdict=ok ;; +esac +check "with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land" \ + "$([ -n "$bw_pid" ] && echo held || echo "no holder")|$bw_verdict|$(q "SELECT count(*) FROM bw")" "held|ok|2" + pgc_summary diff --git a/test/am_contracts.sh b/test/am_contracts.sh index 2a7c38bad..1f0d82da7 100755 --- a/test/am_contracts.sh +++ b/test/am_contracts.sh @@ -351,4 +351,23 @@ check "REINDEX after a committed delete of most rows reclaims their entries, as "$(ix_reclaim pgcolumnar)|$(ix_reclaim heap)|$(q "SET enable_seqscan = off; SELECT count(*) FROM ix_pgcolumnar WHERE a > 0;" | tail -1)" \ "reclaimed|reclaimed|10000" +# ---- documented refusals (#1341 items 44, 51, 77) ---------------------------------- +# +# Each is refused where it is chosen, as limitations.md says, and refused cleanly: +# nothing is left changed by the attempt. +rf_count() { q "SELECT count(*) || '|' || sum(a) FROM rf"; } +psql_run "CREATE TABLE rf (a int) USING pgcolumnar; INSERT INTO rf SELECT g FROM generate_series(1, 5000) g;" +check "CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table" \ + "$(state "CREATE UNLOGGED TABLE rf_u (a int) USING pgcolumnar")|$(q "SELECT count(*) FROM pg_class WHERE relname = 'rf_u'")" "0A000|0" +check "ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows" \ + "$(state "ALTER TABLE rf SET UNLOGGED")|$(q "SELECT relpersistence FROM pg_class WHERE relname = 'rf'")|$(rf_count)" "0A000|p|5000|12502500" +pgc_pg "mkdir -p -m 700 '$PGC_WORKDIR/rf_ts'" +psql_run "CREATE TABLESPACE rf_ts LOCATION '$PGC_WORKDIR/rf_ts';" +check "ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows" \ + "$(state "ALTER TABLE rf SET TABLESPACE rf_ts")|$(q "SELECT reltablespace FROM pg_class WHERE relname = 'rf'")|$(rf_count)" "0A000|0|5000|12502500" +# A failed concurrent build leaves its index behind, marked invalid, as core does for +# any CREATE INDEX CONCURRENTLY that fails after its first phase. +check "CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid" \ + "$(state "CREATE INDEX CONCURRENTLY rf_cic ON rf (a)")|$(q "SELECT indisvalid FROM pg_index WHERE indexrelid = 'rf_cic'::regclass")" "0A000|f" + pgc_summary diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index dda5e4fa5..0b0c702ee 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1,3 +1,20 @@ +advisory_lock_class advisory_lock_class a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them 15;16;17;18;19 2026-10-15 columnar_row_lock.c: the row-identity bucket taken as the key mod 4294967291, not mod row_lock_buckets (one lock per row) +advisory_lock_class advisory_lock_class a duplicate key is still rejected 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class a user advisory lock on that tag does not block a columnar insert 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h) +advisory_lock_class advisory_lock_class a user advisory lock on those tags does not block a columnar UPDATE 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h) +advisory_lock_class advisory_lock_class an UPDATE holds a row-identity lock (class 104) while enable_row_update_lock is on 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h) +advisory_lock_class advisory_lock_class and holds none with enable_row_update_lock off 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class and the UPDATE committed every row 15;16;17;18;19 2026-10-15 columnar_row_lock.c: the row-identity bucket taken as the key mod 4294967291, not mod row_lock_buckets (one lock per row) +advisory_lock_class advisory_lock_class premise: the discovering session is gone and holds nothing 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class premise: the insert took an advisory lock we can see 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class premise: the lock is enabled, or nothing below proves anything 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class premise: the other session holds every one of those tags through the SQL functions 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class premise: the other session really holds that exact tag in class 2 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class premise: the second first write waited on the first 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class row_lock_buckets is fixed at server start, so a session cannot change it (55P02) 15;16;17;18;19 never - +advisory_lock_class advisory_lock_class the lock an insert takes is not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h) +advisory_lock_class advisory_lock_class the locks an UPDATE takes are not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h) +advisory_lock_class advisory_lock_class with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land 15;16;17;18;19 2026-10-15 the storage-row creation lock skipped whenever bulk_parallel_writer is on, not only once the row is committed (src/columnar_metadata.c) alter_table_scans alter_table_scans ADD COLUMN with a volatile default rewrites the table, as on heap 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c) alter_table_scans alter_table_scans ADD PRIMARY KEY is refused over a duplicate (23505), as on heap 15;16;17;18;19 never - alter_table_scans alter_table_scans ADD PRIMARY KEY succeeds once the duplicate is deleted, as on heap 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c) @@ -13,6 +30,10 @@ alter_table_scans alter_table_scans and the rewrite kept the live rows and not t alter_table_scans alter_table_scans the deleted rows are gone from both tables, which hold the same rows 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c) alter_table_scans alter_table_scans the lookup of id 1 is an index scan on the primary key 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c) alter_table_scans alter_table_scans the primary key still finds id 1 once, through the index, after both rewrites 15;16;17;18;19 never - +am_contracts am_contracts ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows 15;16;17;18;19 2026-10-15 relation copy returning without copying (SET TABLESPACE) +am_contracts am_contracts ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows 15;16;17;18;19 2026-10-15 the storage hook accepting an unlogged columnar table +am_contracts am_contracts CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid 15;16;17;18;19 2026-10-15 the concurrent index validate scan doing nothing +am_contracts am_contracts CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table 15;16;17;18;19 2026-10-15 the storage hook accepting an unlogged columnar table am_contracts am_contracts ON CONFLICT DO NOTHING skips the duplicate and inserts the new row 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341 am_contracts am_contracts ON CONFLICT DO UPDATE is refused as unsupported (0A000), and changes nothing 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341 am_contracts am_contracts ON CONFLICT DO UPDATE with no conflicting row inserts it 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341 @@ -2026,6 +2047,43 @@ latest_reading latest_reading the GROUP BY form returns what DISTINCT ON returns latest_reading latest_reading the LATERAL form returns what DISTINCT ON returns 15;16;17;18;19 2026-10-12 the document's LATERAL recipe orders by ts ascending, so it returns each device's FIRST reading latest_reading latest_reading with the btree the LATERAL form reads under a tenth of the table 15;16;17;18;19 never - latest_reading latest_reading without the btree the LATERAL form reads the table once per device 15;16;17;18;19 never - +maintenance_crash maintenance_crash after a crash with compact committed, the table is as compact left it, with the same rows 15;16;17;18;19 never - +maintenance_crash maintenance_crash after a crash with compact uncommitted, the table is as it was, by scan and by index 15;16;17;18;19 never - +maintenance_crash maintenance_crash after a crash with compact_rewrite committed, the table is as compact_rewrite left it, with the same rows 15;16;17;18;19 2026-10-15 PgColumnarWriteLogicalData writes data pages without WAL, so a committed rewrite's pages are lost in a crash +maintenance_crash maintenance_crash after a crash with compact_rewrite uncommitted, the table is as it was, by scan and by index 15;16;17;18;19 never - +maintenance_crash maintenance_crash after a crash with recluster committed, the table is as recluster left it, with the same rows 15;16;17;18;19 2026-10-15 PgColumnarWriteLogicalData writes data pages without WAL, so a committed rewrite's pages are lost in a crash +maintenance_crash maintenance_crash after a crash with recluster uncommitted, the table is as it was, by scan and by index 15;16;17;18;19 never - +maintenance_crash maintenance_crash after a crash with vacuum committed, the table is as vacuum left it, with the same rows 15;16;17;18;19 2026-10-15 PgColumnarWriteLogicalData writes data pages without WAL, so a committed rewrite's pages are lost in a crash +maintenance_crash maintenance_crash after a crash with vacuum uncommitted, the table is as it was, by scan and by index 15;16;17;18;19 never - +maintenance_crash maintenance_crash compact changed the row groups before the crash 15;16;17;18;19 never - +maintenance_crash maintenance_crash compact changed the row groups in its open transaction 15;16;17;18;19 never - +maintenance_crash maintenance_crash compact committed, and the cluster came back after a crash before any checkpoint 15;16;17;18;19 never - +maintenance_crash maintenance_crash compact ran in its open transaction, and the cluster came back after a crash 15;16;17;18;19 never - +maintenance_crash maintenance_crash compact_rewrite changed the row groups before the crash 15;16;17;18;19 2026-10-15 compact_rewrite rewrites nothing (rewritten = 0) +maintenance_crash maintenance_crash compact_rewrite changed the row groups in its open transaction 15;16;17;18;19 2026-10-15 compact_rewrite rewrites nothing (rewritten = 0) +maintenance_crash maintenance_crash compact_rewrite committed, and the cluster came back after a crash before any checkpoint 15;16;17;18;19 never - +maintenance_crash maintenance_crash compact_rewrite ran in its open transaction, and the cluster came back after a crash 15;16;17;18;19 never - +maintenance_crash maintenance_crash recluster changed the row groups before the crash 15;16;17;18;19 never - +maintenance_crash maintenance_crash recluster changed the row groups in its open transaction 15;16;17;18;19 never - +maintenance_crash maintenance_crash recluster committed, and the cluster came back after a crash before any checkpoint 15;16;17;18;19 never - +maintenance_crash maintenance_crash recluster ran in its open transaction, and the cluster came back after a crash 15;16;17;18;19 never - +maintenance_crash maintenance_crash vacuum changed the row groups before the crash 15;16;17;18;19 never - +maintenance_crash maintenance_crash vacuum changed the row groups in its open transaction 15;16;17;18;19 never - +maintenance_crash maintenance_crash vacuum committed, and the cluster came back after a crash before any checkpoint 15;16;17;18;19 never - +maintenance_crash maintenance_crash vacuum ran in its open transaction, and the cluster came back after a crash 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback after a savepoint rolls back two rewrites, the rows written later in the transaction commit beside the old ones 15;16;17;18;19 2026-10-15 PgColumnarAllocateFreeSpace reuses space whatever its freeing xid, so a rolled-back maintenance function's freed bytes are overwritten +maintenance_rollback maintenance_rollback after compact rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback after compact_rewrite rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 2026-10-15 PgColumnarAllocateFreeSpace reuses space whatever its freeing xid, so a rolled-back maintenance function's freed bytes are overwritten +maintenance_rollback maintenance_rollback after recluster rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback after vacuum rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback after vacuum_sorted rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback and every function then runs for real and keeps the same rows 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback compact changed the row groups inside its transaction 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback compact_rewrite changed the row groups inside its transaction 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback premise: the table holds the rows it was given 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback recluster changed the row groups inside its transaction 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback vacuum changed the row groups inside its transaction 15;16;17;18;19 never - +maintenance_rollback maintenance_rollback vacuum_sorted changed the row groups inside its transaction 15;16;17;18;19 never - native_agg native_agg an aggregate that needs zone maps reads them 15;16;17;18;19 never - native_agg native_agg avg(int4,nulls) 15;16;17;18;19 never - native_agg native_agg control: decoding ts reads the table's blocks 15;16;17;18;19 never - @@ -2386,6 +2444,21 @@ partition_lifecycle partition_lifecycle running the plan leaves only the partiti partition_lifecycle partition_lifecycle the parent now has the partitions it reported 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it partition_lifecycle partition_lifecycle the plan retires the partitions wholly before the cutoff, in range order 15;16;17;18;19 2026-10-13 a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY;the plan in catalog order, not range order partition_lifecycle partition_lifecycle they are contiguous, each one width wide 15;16;17;18;19 2026-10-12 partition_create_ahead leaving a minute between partitions +partition_rows partition_rows a duplicate key inserted through the parent is refused (23505), as on heap 15;16;17;18;19 never - +partition_rows partition_rows a duplicate of a moved row's new key is refused in its new partition (23505), as on heap 15;16;17;18;19 never - +partition_rows partition_rows a key a moved row gave up can be used again, as on heap 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete) +partition_rows partition_rows a second insert of the same new key through the parent waits, then is refused (23505), and one row is left, as on heap 15;16;17;18;19 never - +partition_rows partition_rows a second session moving a row the first is moving waits, then fails with 40001, and the row exists once, as on heap 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete);the tree before #1421: a row moved between partitions is serialized only at flush, after its new version's unique check +partition_rows partition_rows an UPDATE in place of a row another session is moving waits, then fails with 40001, as on heap 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete) +partition_rows partition_rows an UPDATE moving 25,000 keyed rows to the other partition keeps under 32 MB likewise 15;16;17;18;19 2026-10-15 PgColumnarSerializeRowNow reads the delete vector into the caller's context, keeping a bitmap copy per row (the tree before the #1422 review fix) +partition_rows partition_rows an UPDATE moving 49 rows to the other partition succeeds, as on heap 15;16;17;18;19 never - +partition_rows partition_rows an UPDATE of 25,000 keyed rows keeps under 32 MB of memory between its first and last row 15;16;17;18;19 2026-10-15 PgColumnarSerializeRowNow reads the delete vector into the caller's context, keeping a bitmap copy per row (the tree before the #1422 review fix) +partition_rows partition_rows and after the races the parents hold the same rows 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete) +partition_rows partition_rows and each UPDATE reached every row once 15;16;17;18;19 never - +partition_rows partition_rows and each partition holds what heap's does 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete) +partition_rows partition_rows and the parent holds the same rows as heap's 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete) +partition_rows partition_rows each keyed table's group carries delete marks before its UPDATE 15;16;17;18;19 never - +partition_rows partition_rows the primary key finds a moved row at its new key, and not at its old one, as on heap 15;16;17;18;19 2026-10-15 a row moving to another partition is not marked deleted in the partition it leaves (src/columnar_tableam.c tuple_delete) partman_columnar partman_columnar and no row-group row behind 15;16;17;18;19 2026-10-12 the drop hook stops deleting row_group rows, so they outlive their deleted storage row partman_columnar partman_columnar and read back as heap does 15;16;17;18;19 2026-10-12 a sequential scan drops the last row group from its list, so every one-group partition reads empty partman_columnar partman_columnar and the rows land in columnar partitions 15;16;17;18;19 2026-10-12 a sequential scan drops the last row group from its list, so every one-group partition reads empty @@ -2398,6 +2471,18 @@ partman_columnar partman_columnar premise: with default_table_access_method ever partman_columnar partman_columnar premise: without default_table_access_method pg_partman makes heap partitions 15;16;17;18;19 never - partman_columnar partman_columnar retention leaves no pgcolumnar storage row behind 15;16;17;18;19 2026-10-12 the drop hook stops deleting the storage row, so a partition pg_partman drops leaves it behind partman_columnar partman_columnar the rows went in 15;16;17;18;19 2026-10-12 a sequential scan drops the last row group from its list, so every one-group partition reads empty +prepared_across_sessions prepared_across_sessions ROLLBACK PREPARED of a DELETE leaves every row as it was 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions ROLLBACK PREPARED of an UPDATE leaves every row as it was, none duplicated 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions a prepared insert is invisible to another session 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions a prepared insert survives a crash: the cluster came back and still holds the prepared transaction 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions a second insert of a key the prepared transaction holds waits, then fails with 23505 after COMMIT PREPARED 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions an UPDATE of a row a prepared transaction updated waits, then fails with 40001 after COMMIT PREPARED 15;16;17;18;19 2026-10-15 the tree before #1421: a same-row UPDATE on a keyed table fails with 23505 rather than 40001 +prepared_across_sessions prepared_across_sessions and after COMMIT PREPARED its rows are heap's 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions and after COMMIT PREPARED the other session sees all the prepared rows 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions premise: prepared transactions are enabled 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions premise: the DELETE is prepared 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions premise: the UPDATE is prepared 15;16;17;18;19 never - +prepared_across_sessions prepared_across_sessions while a DELETE is prepared, another session still sees every row 15;16;17;18;19 never - prepared_writes prepared_writes a delete prepared and committed in one session leaves the session answering 15;16;17;18;19 2026-10-13 PREPARE letting go of neither (main before #1323);PREPARE without letting go of the delete-vector buffers prepared_writes prepared_writes an index fetch before PREPARE, and the same row fetched first after COMMIT PREPARED, in one session 15;16;17;18;19 never - prepared_writes prepared_writes an insert prepared and committed in one session leaves the session answering 15;16;17;18;19 2026-10-13 PREPARE letting go of neither (main before #1323);PREPARE without letting go of the pending write states @@ -2520,14 +2605,14 @@ range_pruning range_pruning premise: the range type collates differently from it range_pruning range_pruning so does a containment probe 15;16;17;18;19 never - range_pruning range_pruning the overlap qual is pushed down, where it used to be dropped 15;16;17;18;19 never - range_pruning range_pruning the same for a containment probe far to its right 15;16;17;18;19 never - -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered; this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered recovery_minimal_wal recovery_minimal_wal premise: the cluster runs at wal_level=minimal 15;16;17;18;19 never - rewrite_storage_oid rewrite_storage_oid a matview created WITH DATA leaves its storage row pointing at itself 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite repair gate no longer covers CreateTableAsStmt, so a matview created WITH DATA keeps the transient's OID rewrite_storage_oid rewrite_storage_oid a type rewrite leaves the storage row pointing at the live table 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite hook does not retarget storage.relation_oid @@ -2652,6 +2737,15 @@ rollup rollup the rollup to be dumped folds its source 15;16;17;18;19 never - rollup rollup the second refresh of the NULL-key rollup folded the append 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's rollup rollup the zone rollups hold a row per key of the recompute 15;16;17;18;19 2026-10-13 refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's rollup rollup two refreshes started together fold the new rows once between them 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup without the self-conflicting lock;row_tid mapping 290 rows a block instead of the AM's +serializable serializable a serializable read of a columnar table holds SIRead locks on columnar's row_group and delete_vector catalog 15;16;17;18;19 2026-10-15 PgColumnarCatalogSnapshot returns a SNAPSHOT_SELF copy, so a serializable read takes no SIRead lock on columnar's catalog +serializable serializable at REPEATABLE READ both doctors' transactions commit and no one stays on call, in one row group and in two, as on heap 15;16;17;18;19 never - +serializable serializable at REPEATABLE READ both ward transactions commit and two are on call, by a scan and by an index, as on heap 15;16;17;18;19 never - +serializable serializable premise: the second doctors table holds its two rows in two row groups, the first in one 15;16;17;18;19 never - +serializable serializable premise: the ward lookup is an index scan on both tables 15;16;17;18;19 never - +serializable serializable the same when both read the ward through an index 15;16;17;18;19 never - +serializable serializable the same with the two doctors in two row groups 15;16;17;18;19 2026-10-15 PgColumnarCatalogSnapshot returns a SNAPSHOT_SELF copy, so a serializable read takes no SIRead lock on columnar's catalog +serializable serializable two doctors in one row group each taken off call: SSI refuses one of the two with 40001, and one stays on call, as on heap 15;16;17;18;19 2026-10-15 PgColumnarCatalogSnapshot returns a SNAPSHOT_SELF copy, so a serializable read takes no SIRead lock on columnar's catalog +serializable serializable two sessions that each see an empty ward by a scan of the whole table and each put someone on call: SSI refuses one of the two with 40001, as on heap 15;16;17;18;19 never - server_file_privilege server_file_privilege coverage check found the file functions (not an empty scan) 15;16;17;18;19 never - server_file_privilege server_file_privilege no role is refused by the pg_read_server_files gate: file_split_offsets 15;16;17;18;19 never - server_file_privilege server_file_privilege no role is refused by the pg_read_server_files gate: iceberg_current_snapshot 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index da01b2ca4..95f2bdc18 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 231 +suites_not_covered 230 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2414,4 +2414,112 @@ suites_not_covered 231 # never 2282 + 0 = 2282 # dated 726 + 9 = 735 # 2282 + 735 == 3017 -checks_never_observed_red 2282 +# +# #1341 items 44, 51 and 77, the documented refusals (jdatcmd): 2282 -> 2282. +# am_contracts gains four checks from five-major runs (31/31 on each of 15-19), all +# dated: CREATE UNLOGGED and SET UNLOGGED by the storage hook accepting an unlogged +# table, CREATE INDEX CONCURRENTLY by its validate scan doing nothing, and SET +# TABLESPACE by the relation copy returning without copying. +# +# ledger rows 3017 + 4 = 3021 +# never 2282 + 0 = 2282 +# dated 735 + 4 = 739 +# 2282 + 739 == 3021 +# +# #1341 items 55, 56 and 16, the row-identity lock out of SQL's reach, +# bulk_parallel_writer set by hand, and the row_lock_buckets bound (jdatcmd): 2282 -> +# 2291, and suites_not_covered 231 -> 230. advisory_lock_class had no rows; it enters +# with all 17 checks from five-major runs (17/17 on each of 15-19). Eight are dated on +# PG17: +# - the row-identity class put at 2: the class-104 arm, the UPDATE's reachable-class +# arm and its contention arm; +# - the unique-key class put back at 2 (#430): the INSERT's reachable-class arm and +# its contention arm; +# - the storage-row lock skipped whenever bulk_parallel_writer is on: the second +# first write fails on storage_pkey; +# - the bucket taken as the raw key, one row lock per row: the bulk UPDATE fails +# with "out of shared memory", which reddens its lock-count arm and its row arm. +# The other nine arrive never: seeding a suite moves the census up by its undated rows. +# +# ledger rows 3021 + 17 = 3038 +# never 2282 + 9 = 2291 +# dated 739 + 8 = 747 +# 2291 + 747 == 3038 +# +# #1341, a maintenance function that rolls back leaves the table as it was +# (jdatcmd): 2291 -> 2302. maintenance_rollback enters with all 13 checks from +# five-major runs (13/13 on each of 15-19). Each function now runs on a fresh table +# (OffgridwithJD, review of #1419), so each date was observed on its own arm. Two +# are dated on PG17 by free space reused whatever its freeing xid: compact_rewrite's +# rollback, whose freed bytes the rewrite overwrote, and the savepoint arm. The other +# eleven arrive never, including the other four functions' arms, which that mutant +# does not reach. +# +# ledger rows 3038 + 13 = 3051 +# never 2291 + 11 = 2302 +# dated 747 + 2 = 749 +# 2302 + 749 == 3051 +# +# #1421, a concurrent UPDATE of the same row gets 40001 with a unique index too, and +# #1341's partition shapes (jdatcmd): 2302 -> 2308. partition_rows enters with all 11 +# checks from five-major runs (11/11 on each of 15-19). Seven are dated on PG17: +# - the tree before #1421: the move-versus-move race (23505 where heap gives 40001); +# - a moving row not marked deleted in the partition it leaves: the split, the +# content, the index lookup, the freed key, both 40001 races and the final +# content. +# The other four arrive never: the three duplicate-key refusals and the move itself. +# +# +# Four more from the #1422 review, a keyed UPDATE's memory over groups with +# deletes, from five-major runs (15/15 on each of 15-19). The two bounds are dated +# by the tree before the fix, where each UPDATE grew by 402 MB; the delete-marks +# premise and the every-row-once check arrive never. +# +# ledger rows 3051 + 15 = 3066 +# never 2302 + 6 = 2308 +# dated 749 + 9 = 758 +# 2308 + 758 == 3066 +# +# #1341, a prepared transaction seen from other sessions and across a crash +# (jdatcmd, stacked on #1422): 2308 -> 2319. prepared_across_sessions enters with all +# 12 checks from five-major runs (12/12 on each of 15-19). One is dated on PG17 by +# the tree before #1421, where an UPDATE of a row a prepared transaction updated +# failed with 23505. The other nine arrive never: a discard at PRE_PREPARE did not +# redden them, because each statement's writes are flushed at its executor end. +# Two more premises from the #1428 review, that the DELETE and the UPDATE are each +# prepared before their ROLLBACK PREPARED arms, arrive never. +# +# ledger rows 3066 + 12 = 3078 +# never 2308 + 11 = 2319 +# dated 758 + 1 = 759 +# 2319 + 759 == 3078 +# +# #1341, a crash part-way through maintenance loses nothing (jdatcmd): 2319 -> +# 2338. maintenance_crash enters with all 24 checks from five-major runs (24/24 on +# each of 15-19). Five are dated on PG17 by two mutants: +# - data pages written without WAL: the committed-crash arms of compact_rewrite, +# recluster and vacuum; +# - compact_rewrite rewriting nothing: its two did-work arms, uncommitted and +# committed, which the review of #1423 asked for, since a function that changes +# nothing is trivially crash-safe (OffgridwithJD). +# The other nineteen arrive never. +# +# ledger rows 3078 + 24 = 3102 +# never 2319 + 19 = 2338 +# dated 759 + 5 = 764 +# 2338 + 764 == 3102 +# +# #1341 item 2, SERIALIZABLE refuses a write skew on a columnar table (jdatcmd): +# 2338 -> 2344. serializable enters with all 9 checks from five-major runs (9/9 on +# each of 15-19). Three are dated on PG17 by one mutant: PgColumnarCatalogSnapshot +# returning a SNAPSHOT_SELF copy. Heap SSI then takes no SIRead locks on the +# catalog (0/0), and both doctors go off call. The other six arrive never: two +# premises, the two empty-ward skews, which that mutant did not redden (they are +# protected through the storage row; OffgridwithJD, review of #1424), and the two +# REPEATABLE READ controls from that review. +# +# ledger rows 3102 + 9 = 3111 +# never 2338 + 6 = 2344 +# dated 764 + 3 = 767 +# 2344 + 767 == 3111 +checks_never_observed_red 2344 diff --git a/test/maintenance_crash.sh b/test/maintenance_crash.sh new file mode 100755 index 000000000..72d613be0 --- /dev/null +++ b/test/maintenance_crash.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# +# A crash part-way through maintenance loses nothing (#1341). compact_rewrite, +# recluster, compact and vacuum retire row groups and write new ones. Two crash +# points, each followed by crash recovery: +# +# - the function has run but its transaction has not committed, and another +# session has checkpointed, so its new groups and catalog rows may be on disk. +# Recovery must leave the table as it was before the function ran; +# - the function has committed, and nothing has checkpointed since, so its work +# exists only in WAL. Recovery must leave the table as the function left it. +# +# The table has row groups of 1000 rows, two rows in five deleted, one group deleted +# whole, and an index. After each recovery the rows must match by content, the row +# groups by count and rows, and an index scan must agree with them. +# +# Usage: test/maintenance_crash.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +# crash_cluster: SIGKILL the postmaster's process group, as recovery.sh does, and +# wait for it to be gone. +crash_cluster() { + local pm i + pm="$(head -1 "$PGC_PGDATA/postmaster.pid" 2>/dev/null || true)" + if [ -n "$pm" ]; then + kill -9 -"$pm" 2>/dev/null || true + kill -9 "$pm" 2>/dev/null || true + fi + for i in $(seq 1 100); do + { [ -n "$pm" ] && kill -0 "$pm" 2>/dev/null; } || break + sleep 0.1 + done + sleep 1 +} +restart_cluster() { # -> ok, once the cluster is back after crash recovery + local attempt + for attempt in 1 2 3 4 5 6; do + if pgc_pg "pg_ctl -D '$PGC_PGDATA' -l '$PGC_LOGFILE' start -w" >/dev/null 2>&1; then + echo ok; return 0 + fi + sleep 2 + done + echo "restart failed" +} +redo_count() { # how many times the server has logged a crash recovery so far + local n + n="$(pgc_pg "grep -c 'database system was not properly shut down' '$PGC_LOGFILE'" 2>/dev/null | tail -1)" + echo "${n:-0}" # grep -c prints 0 and exits 1 when nothing matches: no second 0 +} + +rows() { + q "SELECT count(*) || '/' || md5(string_agg(id || ':' || v || ':' || w, ',' ORDER BY id)) FROM mc" | grep -E '^[0-9]+/' | tail -1 +} +RG_SQL="SELECT count(*) || '/' || coalesce(sum(row_count), 0) FROM pgcolumnar.row_group + WHERE storage_id = pgcolumnar.get_storage_id('mc')" +by_index() { + q "SET enable_seqscan = off; SET enable_bitmapscan = off; + SELECT count(*) || '/' || md5(string_agg(id || ':' || v || ':' || w, ',' ORDER BY id)) + FROM mc WHERE id BETWEEN 1 AND 10000" | grep -E '^[0-9]+/' | tail -1 +} +state() { # the table, as "||" + echo "$(rows)|$(q "$RG_SQL" | grep -E '^[0-9]+/' | tail -1)|$(by_index)" +} +fresh_table() { + psql_run "DROP TABLE IF EXISTS mc; + SET pgcolumnar.stripe_row_limit = 1000; + CREATE TABLE mc (id int, v int, w text) USING pgcolumnar; + INSERT INTO mc SELECT g, (g * 7919) % 10007, md5(g::text) FROM generate_series(1, 10000) g; + CREATE INDEX mc_id ON mc (id); + DELETE FROM mc WHERE id % 5 IN (1, 2) OR id BETWEEN 3001 AND 4000; + CHECKPOINT;" >/dev/null +} + +FUNCS=("pgcolumnar.compact_rewrite('mc', 0.0)" "pgcolumnar.recluster('mc', 'v')" "pgcolumnar.compact('mc')" "pgcolumnar.vacuum('mc')") + +# ---- uncommitted at the crash --------------------------------------------------------- +for f in "${FUNCS[@]}"; do + name="${f#pgcolumnar.}"; name="${name%%(*}" + fresh_table + before="$(state)" + redo0="$(redo_count)" + # The open session names itself with its row groups as it sees them, so the suite + # can tell the function did some work without reading output a SIGKILL may lose. + # A function that changed nothing would be trivially crash-safe (OffgridwithJD, + # review of #1423). + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -c "BEGIN; SELECT $f; SELECT set_config('application_name', 'mc_open ' || ($RG_SQL), false); SELECT pg_sleep(600);" >/dev/null 2>&1 & + bg=$! + opened=no; inside="" + for _i in $(seq 1 300); do + inside="$(q "SELECT substr(application_name, 9) FROM pg_stat_activity WHERE application_name LIKE 'mc_open %'")" + [ -n "$inside" ] && { opened=yes; break; } + sleep 0.1 + done + check "$name changed the row groups in its open transaction" \ + "$([ -n "$inside" ] && [ "$inside" != "$(cut -d'|' -f2 <<<"$before")" ] && echo changed || echo "unchanged (${inside:-not seen})")" "changed" + psql_run "CHECKPOINT;" >/dev/null + crash_cluster + wait "$bg" 2>/dev/null + check "$name ran in its open transaction, and the cluster came back after a crash" \ + "$opened|$(restart_cluster)|$(r1="$(redo_count)"; [ "$r1" -gt "$redo0" ] && echo recovered || echo "no recovery logged ($redo0 before, $r1 after)")" \ + "yes|ok|recovered" + check "after a crash with $name uncommitted, the table is as it was, by scan and by index" \ + "$(state)" "$before" +done + +# ---- committed, not checkpointed ---------------------------------------------------- +for f in "${FUNCS[@]}"; do + name="${f#pgcolumnar.}"; name="${name%%(*}" + fresh_table + before_rows="$(rows)" + before_rg="$(q "$RG_SQL" | grep -E '^[0-9]+/' | tail -1)" + psql_run "SELECT $f;" >/dev/null + after="$(state)" + check "$name changed the row groups before the crash" \ + "$(a="$(cut -d'|' -f2 <<<"$after")"; [ -n "$a" ] && [ "$a" != "$before_rg" ] && echo changed || echo "unchanged ($a)")" "changed" + redo0="$(redo_count)" + crash_cluster + check "$name committed, and the cluster came back after a crash before any checkpoint" \ + "$(restart_cluster)|$(r1="$(redo_count)"; [ "$r1" -gt "$redo0" ] && echo recovered || echo "no recovery logged ($redo0 before, $r1 after)")" "ok|recovered" + check "after a crash with $name committed, the table is as $name left it, with the same rows" \ + "$(state)|$(rows)" "$after|$before_rows" +done + +pgc_summary diff --git a/test/maintenance_rollback.sh b/test/maintenance_rollback.sh new file mode 100755 index 000000000..ccc030146 --- /dev/null +++ b/test/maintenance_rollback.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# +# A maintenance function that rolls back leaves the table as it was (#1341). +# compact_rewrite, recluster, compact, vacuum and vacuum_sorted each retire row +# groups and most write new ones. Retirement is a catalog delete, so a rollback +# must bring every retired group back, with the bytes it points at intact: the +# space a retired group frees may be reused only once no snapshot can need it, +# and never by the transaction that freed it, which can still roll back. +# +# The table has row groups of 1000 rows, two in five rows deleted, and one group +# deleted whole, so every function has work. Each function runs inside a +# transaction, its effect on the row groups is read there, and the transaction +# rolls back. The table must then hold the same rows, by content, and the same +# row groups, and an index scan must agree. The last arm does the same under a +# savepoint, then commits further writes, and runs every function for real. +# +# EACH FUNCTION GETS A FRESH TABLE. On one shared table, a rollback that corrupts it +# fails every function after it for that reason, not its own (OffgridwithJD, review +# of #1419). +# +# Usage: test/maintenance_rollback.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +build_mr() { # the table, made again: 10 row groups of 1000, 5400 rows left + psql_run "SET pgcolumnar.stripe_row_limit = 1000; + DROP TABLE IF EXISTS mr; + CREATE TABLE mr (id int, v int, w text) USING pgcolumnar; + INSERT INTO mr SELECT g, (g * 7919) % 10007, md5(g::text) FROM generate_series(1, 10000) g; + CREATE INDEX mr_id ON mr (id); + DELETE FROM mr WHERE id % 5 IN (1, 2) OR id BETWEEN 3001 AND 4000;" +} +build_mr + +rows() { # the rows, by content + q "SELECT count(*) || '/' || md5(string_agg(id || ':' || v || ':' || w, ',' ORDER BY id)) FROM mr" | tail -1 +} +RG_SQL="SELECT count(*) || '/' || coalesce(sum(row_count), 0) FROM pgcolumnar.row_group + WHERE storage_id = pgcolumnar.get_storage_id('mr')" +by_index() { # the rows an index scan returns, by content + q "SET enable_seqscan = off; SET enable_bitmapscan = off; + SELECT count(*) || '/' || md5(string_agg(id || ':' || v || ':' || w, ',' ORDER BY id)) + FROM mr WHERE id BETWEEN 1 AND 10000" | grep -E '^[0-9]+/' | tail -1 +} +ROWS0="$(rows)" +RG0="$(q "$RG_SQL")" +check "premise: the table holds the rows it was given" "${ROWS0%%/*}|$RG0" "5400|10/10000" + +# rolled_back FUNCTION -> "|||" +rolled_back() { + local inside + inside="$(q "BEGIN; SELECT $1; $RG_SQL; ROLLBACK;" | grep -E '^[0-9]+/[0-9]+$' | tail -1)" + echo "$inside|$(rows)|$(q "$RG_SQL")|$(by_index)" +} +for f in "pgcolumnar.compact_rewrite('mr', 0.0)" "pgcolumnar.recluster('mr', 'v')" "pgcolumnar.compact('mr')" \ + "pgcolumnar.vacuum('mr')" "pgcolumnar.vacuum_sorted('mr', 'v')"; do + build_mr + got="$(rolled_back "$f")" + name="${f#pgcolumnar.}"; name="${name%%(*}" + check "$name changed the row groups inside its transaction" \ + "$([ "${got%%|*}" != "$RG0" ] && [ -n "${got%%|*}" ] && echo changed || echo "unchanged (${got%%|*})")" "changed" + check "after $name rolls back, the table holds the same rows and row groups, by scan and by index" \ + "${got#*|}" "$ROWS0|$RG0|$ROWS0" +done + +# ---- under a savepoint, then committed work, then every function for real ---------- +build_mr +psql_run "BEGIN; + SAVEPOINT s; SELECT pgcolumnar.compact_rewrite('mr', 0.0); SELECT pgcolumnar.recluster('mr', 'v'); + ROLLBACK TO s; + SET LOCAL pgcolumnar.stripe_row_limit = 1000; + INSERT INTO mr SELECT g, g, 'new' FROM generate_series(20001, 21000) g; + COMMIT;" >/dev/null +ROWS1="$(rows)" +check "after a savepoint rolls back two rewrites, the rows written later in the transaction commit beside the old ones" \ + "$(q "SELECT count(*) FILTER (WHERE w = 'new') || '/' || count(*) FROM mr" | tail -1)|$(by_index | cut -d/ -f1)" "1000/6400|5400" +for f in "pgcolumnar.compact_rewrite('mr', 0.0)" "pgcolumnar.recluster('mr', 'v')" "pgcolumnar.compact('mr')" \ + "pgcolumnar.vacuum('mr')"; do + psql_run "SELECT $f;" >/dev/null +done +check "and every function then runs for real and keeps the same rows" "$(rows)" "$ROWS1" + +pgc_summary diff --git a/test/partition_rows.sh b/test/partition_rows.sh new file mode 100755 index 000000000..9e6be2d15 --- /dev/null +++ b/test/partition_rows.sh @@ -0,0 +1,170 @@ +#!/usr/bin/env bash +# +# Rows that cross columnar partitions, and keys enforced through a partitioned +# parent (#1341). No test moved a row between columnar partitions with an UPDATE, +# or put a PRIMARY KEY on a parent whose partitions are columnar. +# +# A row-moving UPDATE is a delete in one partition and an insert in another, and +# PostgreSQL asks the access method to report a concurrent writer of the moved row +# as a serialization failure rather than letting it find nothing. A key on the +# parent is one unique index per partition, each reached through the partition's +# own access method and its unique-insert lock. +# +# Each arm runs on a parent with two columnar partitions and on one with two heap +# partitions, and heap is the answer. +# +# Usage: test/partition_rows.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +for am in pgcolumnar heap; do + psql_run "CREATE TABLE pr_$am (id int PRIMARY KEY, k int, v text) PARTITION BY RANGE (id); + CREATE TABLE pr_${am}_1 PARTITION OF pr_$am FOR VALUES FROM (0) TO (1000) USING $am; + CREATE TABLE pr_${am}_2 PARTITION OF pr_$am FOR VALUES FROM (1000) TO (3000) USING $am; + INSERT INTO pr_$am SELECT g, g % 7, 'r' || g FROM generate_series(1, 999) g;" +done + +outcome() { # outcome SQL -> ok, or the SQLSTATE the statement failed with + local out st + out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" \ + -X -qtA -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$1" 2>&1)" + st="$(sed -n 's/^ERROR: *\([0-9A-Z]\{5\}\).*/\1/p' <<<"$out")" + echo "${st:-ok}" +} +both() { # both SQL -> "|", %p standing for the parent + echo "$(outcome "${1//%p/pr_pgcolumnar}")|$(outcome "${1//%p/pr_heap}")" +} +answers() { # answers SQL -> "|", %p standing for the parent + echo "$(q "${1//%p/pr_pgcolumnar}" | tail -1)|$(q "${1//%p/pr_heap}" | tail -1)" +} +same() { # same SQL -> "same" when both parents answer alike, else both answers + local a; a="$(answers "$1")" + [ "${a%%|*}" = "${a#*|}" ] && echo same || echo "$a" +} +CONTENT="SELECT count(*) || '/' || md5(string_agg(id || ':' || k || ':' || v, ',' ORDER BY id)) FROM %p" +SPLIT="SELECT (SELECT count(*) FROM %p_1) || '/' || (SELECT count(*) FROM %p_2)" + +# ---- a row-moving UPDATE ---------------------------------------------------------- +check "an UPDATE moving 49 rows to the other partition succeeds, as on heap" \ + "$(both "UPDATE %p SET id = id + 1000 WHERE id < 50")" "ok|ok" +check "and each partition holds what heap's does" "$(answers "$SPLIT")" "950/49|950/49" +check "and the parent holds the same rows as heap's" "$(same "$CONTENT")" "same" +IDX="SET enable_seqscan = off; SET enable_bitmapscan = off;" +check "the primary key finds a moved row at its new key, and not at its old one, as on heap" \ + "$(answers "$IDX SELECT count(*) FILTER (WHERE id = 1001) || '/' || count(*) FILTER (WHERE id = 1) FROM %p WHERE id IN (1, 1001)")" \ + "1/0|1/0" + +# ---- the key on the parent ---------------------------------------------------------- +check "a duplicate key inserted through the parent is refused (23505), as on heap" \ + "$(both "INSERT INTO %p VALUES (500, 0, 'dup')")" "23505|23505" +check "a key a moved row gave up can be used again, as on heap" \ + "$(both "INSERT INTO %p VALUES (1, 0, 'again')")" "ok|ok" +check "a duplicate of a moved row's new key is refused in its new partition (23505), as on heap" \ + "$(both "INSERT INTO %p VALUES (1001, 0, 'dup')")" "23505|23505" + +# race PARENT HOLDER CONTENDER KEY -> "||". HOLDER runs in its own transaction, then queues for an advisory +# lock a gate session holds. The contender must be seen waiting before the gate ends, +# and the holder then commits. Events, not sleeps. +pr_wait_for() { # pr_wait_for APPLICATION_NAME -> its pid + local i pid="" + for i in $(seq 1 300); do + pid="$(q "SELECT pid FROM pg_stat_activity WHERE application_name = '$1'")" + [ -n "$pid" ] && break; sleep 0.1 + done + echo "$pid" +} +race() { + local p="$1" holder="$2" contender="$3" key="$4" out gate i waited=no + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -c "SELECT pg_advisory_lock(1341); SET application_name = 'pr_gate'; SELECT pg_sleep(600);" >/dev/null 2>&1 & + local gb=$! + gate="$(pr_wait_for pr_gate)" + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq -v ON_ERROR_STOP=1 \ + -c "BEGIN; $holder; SET application_name = 'pr_holder'; SELECT pg_advisory_lock_shared(1341); COMMIT;" \ + >"$PGC_WORKDIR/pr_holder.out" 2>&1 & + local hb=$! + pr_wait_for pr_holder >/dev/null + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \ + -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate \ + -c "SET application_name = 'pr_contender'; SET statement_timeout = '60s'; $contender" \ + >"$PGC_WORKDIR/pr_contender.out" 2>&1 & + local cb=$! + for i in $(seq 1 300); do + [ "$(q "SELECT count(*) FROM pg_stat_activity WHERE application_name = 'pr_contender' AND wait_event_type = 'Lock'")" = 1 ] && { waited=yes; break; } + sleep 0.1 + done + q "SELECT pg_terminate_backend(${gate:-0})" >/dev/null + wait "$gb" 2>/dev/null; wait "$hb"; wait "$cb" + out="$(sed -n 's/^ERROR: *\([0-9A-Z]\{5\}\).*/\1/p' "$PGC_WORKDIR/pr_contender.out")" + echo "$waited|${out:-ok}|$(q "SELECT count(*) FROM $p WHERE $key" | tail -1)" +} +both_race() { # both_race HOLDER CONTENDER KEY -> " ", %p standing for the parent + echo "$(race pr_pgcolumnar "${1//%p/pr_pgcolumnar}" "${2//%p/pr_pgcolumnar}" "$3") $(race pr_heap "${1//%p/pr_heap}" "${2//%p/pr_heap}" "$3")" +} + +# ---- concurrent writers ---------------------------------------------------------- +check "a second insert of the same new key through the parent waits, then is refused (23505), and one row is left, as on heap" \ + "$(both_race "INSERT INTO %p VALUES (1500, 0, 'a')" "INSERT INTO %p VALUES (1500, 0, 'b')" "id = 1500")" \ + "yes|23505|1 yes|23505|1" +check "a second session moving a row the first is moving waits, then fails with 40001, and the row exists once, as on heap" \ + "$(both_race "UPDATE %p SET id = id + 1000 WHERE id = 60" "UPDATE %p SET id = id + 1000 WHERE id = 60" "id IN (60, 1060)")" \ + "yes|40001|1 yes|40001|1" +check "an UPDATE in place of a row another session is moving waits, then fails with 40001, as on heap" \ + "$(both_race "UPDATE %p SET id = id + 1000 WHERE id = 70" "UPDATE %p SET v = 'changed' WHERE id = 70" "id IN (70, 1070) AND v <> 'changed'")" \ + "yes|40001|1 yes|40001|1" +check "and after the races the parents hold the same rows" "$(same "$CONTENT")" "same" + +# ---- memory through a keyed UPDATE --------------------------------------------- +# On a keyed table each updated or moved row is checked at once against the +# committed delete marks of its row group (#1421). Those reads must not stay in the +# statement's memory: kept, they cost one copy of the group's delete bitmap per row, +# 1.2 GB for 50,000 rows in OffgridwithJD's measurement (review of #1422). One +# group of 50,000 rows with every even row deleted, so its bitmap is in the +# catalog; pm_mem records the backend's context bytes at the first and the last row +# an UPDATE reaches, and the growth between them is what the statement kept. +# pm_build NAME: a keyed parent NAME, its first partition holding rows 1..50000 in one +# row group with every even row deleted +pm_build() { + psql_run "CREATE TABLE $1 (id int PRIMARY KEY, v int) PARTITION BY RANGE (id); + CREATE TABLE $1_1 PARTITION OF $1 FOR VALUES FROM (0) TO (100000) USING pgcolumnar; + CREATE TABLE $1_2 PARTITION OF $1 FOR VALUES FROM (100000) TO (300000) USING pgcolumnar; + INSERT INTO $1 SELECT g, 0 FROM generate_series(1, 50000) g; + DELETE FROM $1 WHERE id % 2 = 0;" +} +psql_run "CREATE TABLE pm_mem (at text, id int, bytes bigint); + CREATE FUNCTION pm_mem(at text, id int) RETURNS int LANGUAGE plpgsql AS \$\$ + BEGIN + IF id IN (1, 49999) THEN + INSERT INTO pm_mem SELECT at, id, sum(total_bytes) FROM pg_backend_memory_contexts; + END IF; + RETURN 0; + END \$\$;" +pm_build pmu; pm_build pmm +growth() { # growth AT -> "readings|growth in MB" for the UPDATE labelled AT + q "SELECT count(*) || '|' || coalesce(((max(bytes) FILTER (WHERE id = 49999) - + max(bytes) FILTER (WHERE id = 1)) / 1048576)::text, '') FROM pm_mem WHERE at = '$1'" +} +bounded() { # bounded AT -> yes when both readings exist and the growth is under 32 MB + local g; g="$(growth "$1")" + [ "${g%%|*}" = 2 ] && [ -n "${g#*|}" ] && [ "${g#*|}" -lt 32 ] && echo yes || echo "no (readings|MB: $g)" +} +marks() { # marks PARENT -> "live rows|delete_vector rows of its first partition" + q "SELECT count(*) || '|' || (SELECT count(*) FROM pgcolumnar.delete_vector + WHERE storage_id = pgcolumnar.get_storage_id('$1_1')) FROM $1" +} +check "each keyed table's group carries delete marks before its UPDATE" \ + "$(marks pmu) $(marks pmm)" "25000|1 25000|1" +psql_run "UPDATE pmu SET v = v + 1 + pm_mem('update', id);" >/dev/null +check "an UPDATE of 25,000 keyed rows keeps under 32 MB of memory between its first and last row" \ + "$(bounded update)" "yes" +psql_run "UPDATE pmm SET id = id + 100000 + pm_mem('move', id);" >/dev/null +check "an UPDATE moving 25,000 keyed rows to the other partition keeps under 32 MB likewise" \ + "$(bounded move)" "yes" +check "and each UPDATE reached every row once" \ + "$(q "SELECT (SELECT sum(v) FROM pmu) || '/' || (SELECT count(*) FROM pmm_1) || '/' || (SELECT count(*) FROM pmm_2)")" "25000/0/25000" + +pgc_summary diff --git a/test/prepared_across_sessions.sh b/test/prepared_across_sessions.sh new file mode 100755 index 000000000..40c32c718 --- /dev/null +++ b/test/prepared_across_sessions.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# +# A prepared transaction, seen from other sessions and across a crash (#1341). +# prepared_writes.sh covers PREPARE and COMMIT PREPARED in one session. Untested +# until now: +# +# - what another session sees while a transaction is prepared; +# - the unique-key and row locks a prepared transaction still holds: a second +# writer of the same key or row waits, then fails once COMMIT PREPARED runs; +# - ROLLBACK PREPARED of a DELETE and of an UPDATE; +# - a prepared insert surviving a crash and committing after recovery. +# +# The columnar table has a primary key; a heap table loaded with the same rows is the +# answer where one is needed. Waits are observed, not slept on. +# +# Usage: test/prepared_across_sessions.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail +export PGC_EXTRA_CONF='max_prepared_transactions=4' +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +check "premise: prepared transactions are enabled" "$(q "SHOW max_prepared_transactions")" "4" +psql_run "CREATE TABLE px (k int PRIMARY KEY, v text) USING pgcolumnar; + INSERT INTO px SELECT g, 'r' || g FROM generate_series(1, 1000) g;" +rows() { q "SELECT count(*) || '/' || md5(string_agg(k || ':' || v, ',' ORDER BY k)) FROM px" | grep -E '^[0-9]+/' | tail -1; } +P() { env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq -v VERBOSITY=sqlstate "$@"; } +sqlstate_of() { sed -n 's/^ERROR: *\([0-9A-Z]\{5\}\).*/\1/p' "$1" | tail -1; } +waits_on_lock() { # waits_on_lock APPLICATION_NAME -> yes once that session waits on a lock + local i + for i in $(seq 1 300); do + [ "$(q "SELECT count(*) FROM pg_stat_activity WHERE application_name = '$1' AND wait_event_type = 'Lock'")" = 1 ] && { echo yes; return; } + sleep 0.1 + done + echo no +} + +# ---- a prepared insert, from another session ------------------------------------------ +psql_run "BEGIN; INSERT INTO px SELECT g, 'p' || g FROM generate_series(1001, 1100) g; PREPARE TRANSACTION 'px_ins';" >/dev/null +check "a prepared insert is invisible to another session" "$(q "SELECT count(*) FROM px WHERE k > 1000")" "0" +P -c "SET application_name = 'px_dup'; SET statement_timeout = '60s';" -c "INSERT INTO px VALUES (1050, 'dup');" \ + >"$PGC_WORKDIR/px_dup.out" 2>&1 & +dup=$! +waited="$(waits_on_lock px_dup)" +psql_run "COMMIT PREPARED 'px_ins';" >/dev/null +wait "$dup" +check "a second insert of a key the prepared transaction holds waits, then fails with 23505 after COMMIT PREPARED" \ + "$waited|$(sqlstate_of "$PGC_WORKDIR/px_dup.out")|$(q "SELECT count(*) || '/' || min(v) FROM px WHERE k = 1050")" "yes|23505|1/p1050" +check "and after COMMIT PREPARED the other session sees all the prepared rows" "$(q "SELECT count(*) FROM px WHERE k > 1000")" "100" + +# ---- ROLLBACK PREPARED of a DELETE and of an UPDATE ----------------------------------- +before="$(rows)" +psql_run "BEGIN; DELETE FROM px WHERE k % 20 = 0; PREPARE TRANSACTION 'px_del';" >/dev/null +# Without these premises a PREPARE that failed would leave the table unchanged, and +# the checks after it would compare it with itself (OffgridwithJD, review of #1428). +check "premise: the DELETE is prepared" "$(q "SELECT count(*) FROM pg_prepared_xacts WHERE gid = 'px_del'")" "1" +check "while a DELETE is prepared, another session still sees every row" "$(rows)" "$before" +psql_run "ROLLBACK PREPARED 'px_del';" >/dev/null +check "ROLLBACK PREPARED of a DELETE leaves every row as it was" "$(rows)" "$before" +psql_run "BEGIN; UPDATE px SET v = v || '!' WHERE k % 20 = 1; PREPARE TRANSACTION 'px_upd';" >/dev/null +check "premise: the UPDATE is prepared" "$(q "SELECT count(*) FROM pg_prepared_xacts WHERE gid = 'px_upd'")" "1" +psql_run "ROLLBACK PREPARED 'px_upd';" >/dev/null +check "ROLLBACK PREPARED of an UPDATE leaves every row as it was, none duplicated" "$(rows)" "$before" + +# ---- a row lock carried across PREPARE -------------------------------------------------- +psql_run "BEGIN; UPDATE px SET v = 'prepared' WHERE k = 7; PREPARE TRANSACTION 'px_row';" >/dev/null +P -c "SET application_name = 'px_rowc'; SET statement_timeout = '60s';" -c "UPDATE px SET v = 'other' WHERE k = 7;" \ + >"$PGC_WORKDIR/px_rowc.out" 2>&1 & +rc=$! +waited="$(waits_on_lock px_rowc)" +psql_run "COMMIT PREPARED 'px_row';" >/dev/null +wait "$rc" +check "an UPDATE of a row a prepared transaction updated waits, then fails with 40001 after COMMIT PREPARED" \ + "$waited|$(sqlstate_of "$PGC_WORKDIR/px_rowc.out")|$(q "SELECT count(*) || '/' || min(v) FROM px WHERE k = 7")" "yes|40001|1/prepared" + +# ---- a prepared insert across a crash ------------------------------------------------- +crash_cluster() { + local pm i + pm="$(head -1 "$PGC_PGDATA/postmaster.pid" 2>/dev/null || true)" + [ -n "$pm" ] && { kill -9 -"$pm" 2>/dev/null || true; kill -9 "$pm" 2>/dev/null || true; } + for i in $(seq 1 100); do { [ -n "$pm" ] && kill -0 "$pm" 2>/dev/null; } || break; sleep 0.1; done + sleep 1 +} +restart_cluster() { + local attempt + for attempt in 1 2 3 4 5 6; do + pgc_pg "pg_ctl -D '$PGC_PGDATA' -l '$PGC_LOGFILE' start -w" >/dev/null 2>&1 && { echo ok; return; } + sleep 2 + done + echo "restart failed" +} +psql_run "CREATE TABLE px_heap (k int PRIMARY KEY, v text); + INSERT INTO px_heap SELECT g, md5(g::text) FROM generate_series(5001, 10000) g; + BEGIN; INSERT INTO px SELECT g, md5(g::text) FROM generate_series(5001, 10000) g; PREPARE TRANSACTION 'px_crash';" >/dev/null +crash_cluster +back="$(restart_cluster)" +check "a prepared insert survives a crash: the cluster came back and still holds the prepared transaction" \ + "$back|$(q "SELECT count(*) FROM pg_prepared_xacts WHERE gid = 'px_crash'")" "ok|1" +psql_run "COMMIT PREPARED 'px_crash';" >/dev/null +check "and after COMMIT PREPARED its rows are heap's" \ + "$(q "SELECT count(*) || '/' || md5(string_agg(k || ':' || v, ',' ORDER BY k)) FROM px WHERE k > 5000")" \ + "$(q "SELECT count(*) || '/' || md5(string_agg(k || ':' || v, ',' ORDER BY k)) FROM px_heap")" + +pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 97d277b4a..7c7ae453a 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -165,6 +165,11 @@ behaviour, the source of that number is named. - [117. test_large_values.py: megabyte-scale values are stored exactly](#117-test_large_valuespy-megabyte-scale-values-are-stored-exactly) - [118. test_recovery_minimal_wal.py: tables that skip WAL survive a crash](#118-test_recovery_minimal_walpy-tables-that-skip-wal-survive-a-crash) - [119. test_statement_self_visibility.py: a statement does not see its own writes](#119-test_statement_self_visibilitypy-a-statement-does-not-see-its-own-writes) +- [120. test_maintenance_rollback.py: a maintenance function that rolls back leaves the table as it was](#120-test_maintenance_rollbackpy-a-maintenance-function-that-rolls-back-leaves-the-table-as-it-was) +- [121. test_partition_rows.py: rows cross columnar partitions, and keys hold through the parent](#121-test_partition_rowspy-rows-cross-columnar-partitions-and-keys-hold-through-the-parent) +- [122. test_prepared_across_sessions.py: a prepared transaction, seen from other sessions and across a crash](#122-test_prepared_across_sessionspy-a-prepared-transaction-seen-from-other-sessions-and-across-a-crash) +- [123. test_maintenance_crash.py: a crash part-way through maintenance loses nothing](#123-test_maintenance_crashpy-a-crash-part-way-through-maintenance-loses-nothing) +- [124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table](#124-test_serializablepy-serializable-refuses-a-write-skew-on-a-columnar-table) ## 1. How to read a test in here @@ -5397,7 +5402,9 @@ regardless. | `test_the_lock_an_insert_takes_is_not_in_a_sql_reachable_class` | the lock is enabled, the insert's locks are visible, none is SQL-reachable, and the discovering session leaves nothing held | | `test_a_user_advisory_lock_on_that_tag_does_not_block_an_insert` | a user session holding every addressable tag in class 2 does not block a columnar insert | | `test_a_duplicate_key_is_still_rejected` | the lock still does its job, so removing the collision by removing the lock does not pass | -| `test_the_row_identity_lock_and_its_settings` | an UPDATE holds a class-104 row-identity lock with `enable_row_update_lock` on and none with it off; `row_lock_buckets` refuses `SET` (55P02) because it is fixed at server start (#1341) | +| `test_the_row_identity_lock_and_its_settings` | an UPDATE holds a class-104 row-identity lock with `enable_row_update_lock` on and none with it off; `row_lock_buckets` refuses `SET` (55P02) because it is fixed at server start; a 200000-row UPDATE holds `row_lock_buckets` class-104 locks, one per bucket, not one per row (#1341 item 16) | +| `test_the_row_identity_lock_cannot_be_reached_from_sql` | every advisory lock an open UPDATE holds is outside field4 1 and 2, and a second session holding each of those tags through `pg_advisory_xact_lock(int4, int4)` does not block an UPDATE of the same row (#1341 item 56) | +| `test_bulk_parallel_writer_set_by_hand_still_serialises_a_first_write` | two sessions with `bulk_parallel_writer` on each make the first write to a committed, empty table; the second waits for the first, and both rows land (#1341 item 55) | ## 69. test_index_am_support.py: the index methods the page claims must work `docs/features.md` said "`CREATE INDEX` builds btree and hash indexes over a @@ -6925,6 +6932,7 @@ THE WITHDRAWAL IS REACHED ONLY BY A RACE. When one session holds a key uncommitt | `test_an_insert_racing_an_uncommitted_delete_of_its_key` | an insert of a key whose delete (or key-changing update) is uncommitted waits, then fails with 23505 if the delete rolls back and succeeds if it commits: never two rows with the key. Before #1341's fix the rollback left two | | `test_an_index_built_after_a_delete_serves_an_older_snapshot` | a REPEATABLE READ snapshot from before a committed DELETE still finds the row through an index built after it (an Index Scan, asserted), and a new snapshot does not; a UNIQUE index still builds over a key whose other holder is deleted. Before #1341's fix the old snapshot's lookup found 0 | | `test_reindex_reclaims_the_entries_of_rows_no_snapshot_sees` | after a committed delete of 9 rows in 10, with no older snapshot, `REINDEX` takes the index under half its size, as it does on heap, and the index still finds the 10,000 live rows (#1407) | +| `test_the_documented_refusals_change_nothing` | `CREATE UNLOGGED TABLE` and `SET UNLOGGED` are refused (`0A000`) and leave nothing changed; `SET TABLESPACE` is refused and the table stays where it was; `CREATE INDEX CONCURRENTLY` is refused and leaves its index marked invalid, as core does (#1341 items 44, 51, 77) | ## 111. test_objstore_credentials.py: object-store credentials live on a user mapping, and the environment is a privilege @@ -7018,3 +7026,54 @@ The pytest twin of `test/statement_self_visibility.sh` (#1341), every check by t | test | what it holds | | --- | --- | | `test_a_statement_does_not_see_its_own_writes` | the four shapes run to completion and answer as heap does, and the mid-statement flush | + +## 120. test_maintenance_rollback.py: a maintenance function that rolls back leaves the table as it was + +The pytest twin of `test/maintenance_rollback.sh` (#1341), every check by the shell's name. `compact_rewrite`, `recluster`, `compact`, `vacuum` and `vacuum_sorted` each retire row groups, and most write new ones. Each runs inside a transaction on a table with row groups of 1000 rows, two rows in five deleted, and one group deleted whole. Its effect on the row groups is read inside the transaction, and the transaction rolls back. The table must then hold the same rows by content and the same row groups, and an index scan must agree. The same then runs under a savepoint, later writes commit, and every function runs for real. The rollback is issued as SQL statements, because `pgc_conn` is a watching wrapper and an autocommit flag set on it does not reach the connection. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_a_maintenance_function_that_rolls_back_leaves_the_table_as_it_was` | each function's effect inside its transaction, the table after each rollback by scan and by index, the savepoint case, and the real runs | + +## 121. test_partition_rows.py: rows cross columnar partitions, and keys hold through the parent + +The pytest twin of `test/partition_rows.sh` (#1341, #1421), every check by the shell's name. A parent with a PRIMARY KEY and two columnar partitions, and the same with heap partitions; heap is the answer. An `UPDATE` moves 49 rows to the other partition: each partition and the parent then hold what heap's do, and the key finds a moved row at its new key only. A duplicate key through the parent is refused (23505), a key a moved row gave up is free again, and a duplicate of a moved row's new key is refused in its new partition. Three races, ordered by a gate connection rather than by sleeps: a second insert of the same key waits and gets 23505; a second session moving the row the first is moving waits and gets 40001, which columnar used to answer with 23505 (#1421); and an in-place `UPDATE` of a row being moved waits and gets 40001. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_rows_cross_columnar_partitions_and_keys_hold_through_the_parent` | the row-moving update against heap, the three key arms, and the three races with each contender seen waiting | +| `test_a_keyed_update_keeps_its_memory_bounded` | an UPDATE of 25,000 keyed rows, in place and moving partition, over a row group with deletes, grows the backend's memory by under 32 MB (#1422 review) | + +## 122. test_prepared_across_sessions.py: a prepared transaction, seen from other sessions and across a crash + +The pytest twin of `test/prepared_across_sessions.sh` (#1341), every check by the shell's name, on a private cluster because the last arm crashes the postmaster. While a transaction is prepared, another session does not see its insert and still sees the rows its DELETE removed. A second insert of a key it holds waits and gets 23505 after COMMIT PREPARED; an UPDATE of a row it updated waits and gets 40001, which needs #1421's fix (it got 23505 before, the table having a primary key). ROLLBACK PREPARED of a DELETE and of an UPDATE leaves every row as it was. A prepared insert survives an immediate stop, is still listed in `pg_prepared_xacts`, and after COMMIT PREPARED its rows equal a heap table's. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_a_prepared_transaction_across_sessions_and_a_crash` | visibility while prepared, the two waits and their errors, both ROLLBACK PREPAREDs, and the crash | + +## 123. test_maintenance_crash.py: a crash part-way through maintenance loses nothing + +The pytest twin of `test/maintenance_crash.sh` (#1341), every check by the shell's name. A private cluster, because each arm crashes the postmaster: the shell kills its process group, and this side stops it with `-m immediate`, which leaves the same redo for the next start. `compact_rewrite`, `recluster`, `compact` and `vacuum` each run twice on a fresh table with row groups of 1000 rows, two rows in five deleted, one group deleted whole, and an index. First the function runs in an open transaction, another session checkpoints, and the server crashes: after recovery the table is as it was. Then the function commits and the server crashes before any checkpoint: after recovery the table is as the function left it. Each arm also checks that the server logged a crash recovery. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_a_crash_part_way_through_maintenance_loses_nothing` | for each function, the uncommitted crash and the committed crash, each with its recovery logged | + +## 124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table + +The pytest twin of `test/serializable.sh` (#1341), every check by the shell's name. Two serializable transactions each read what the other is about to change, then each write; SSI must refuse the second with 40001 ("read/write dependencies") and the invariant must hold, on a columnar table as on heap. Each error is tagged with its step: columnar raises at the second write, which waits for the first commit, and heap at the second COMMIT, so the checks hold the refusal, not the step. At REPEATABLE READ the same interleavings must commit both transactions, on both tables, so a 40001 from anything but SSI cannot pass. Four shapes: two doctors taken off call in one row group and in two, and someone put on call in an empty ward after a scan of the whole table and after an index scan. The protection comes from heap SSI on pgColumnar's catalog tables, and an arm pins the SIRead locks a serializable read takes on `row_group` and `delete_vector`. The second write runs in a thread, because on columnar a write to a row group another session is changing waits for that session to commit. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_serializable_refuses_a_write_skew` | the catalog SIRead locks, the four write skews refused by SSI with 40001 against heap, and the REPEATABLE READ controls | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 316bd78ae..eb87b1ffe 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -689,4 +689,15 @@ guard_tests 435 # test_rollup.py, by collection over the cluster files on this tree. # cluster_tests 577 -> 578 for #1341's test_statement_self_visibility.py, one test, by collection # over the cluster files on this tree. -cluster_tests 578 +# cluster_tests 578 -> 579 for #1341's test_the_documented_refusals_change_nothing in +# cluster_tests 579 -> 581 for #1341 items 56 and 55: test_the_row_identity_lock_cannot_be_reached_from_sql +# and test_bulk_parallel_writer_set_by_hand_still_serialises_a_first_write in +# cluster_tests 581 -> 582 for #1341's test_maintenance_rollback.py, one test, by collection +# cluster_tests 582 -> 585 for #1421's test_partition_rows.py, one test, by collection +# cluster_tests 568 -> 569 for #1422 review's test_a_keyed_update_keeps_its_memory_bounded +# in test_partition_rows.py, by collection over the cluster files on this tree. +# cluster_tests 569 -> 570 for #1341's test_prepared_across_sessions.py, one test, by +# collection over the cluster files on this tree (stacked on #1422). +# cluster_tests 585 -> 586 for #1341's test_maintenance_crash.py, one test, by collection +# cluster_tests 586 -> 587 for #1341's test_serializable.py, one test, by collection +cluster_tests 587 diff --git a/test/pytest/test_advisory_lock_class.py b/test/pytest/test_advisory_lock_class.py index ddbe11dd0..bf3dd3ff9 100644 --- a/test/pytest/test_advisory_lock_class.py +++ b/test/pytest/test_advisory_lock_class.py @@ -244,7 +244,8 @@ def test_a_duplicate_key_is_still_rejected(pgc_conn, expect): def test_the_row_identity_lock_and_its_settings(pgc_cluster, pgc_conn, expect): """An UPDATE serialises on its row through an advisory lock in class 104; the session setting enable_row_update_lock turns that off, and row_lock_buckets is - part of the lock tag, so it is fixed at server start (#1341).""" + part of the lock tag, so it is fixed at server start. A bulk UPDATE holds one lock + per bucket, not one per row (#1341 item 16).""" with pgc_conn.cursor() as cur: cur.execute("CREATE TABLE rl (id int, v int) USING pgcolumnar") cur.execute("INSERT INTO rl SELECT g, g FROM generate_series(1, 100) g") @@ -280,3 +281,154 @@ def row_locks_while_updating(setting): ctx = cur.fetchone()[0] expect.text(f"{refused}|{ctx}", "55P02|postmaster|1024", "row_lock_buckets is fixed at server start, so a session cannot change it (55P02)") + + # The documented bound: row_lock_buckets bounds the row locks a transaction holds. + # One transaction updates 200000 rows, then counts its own class-104 locks. + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE rlb (id int, v int) USING pgcolumnar") + cur.execute("INSERT INTO rlb SELECT g, g FROM generate_series(1, 200000) g") + bulk = psycopg.connect(pgc_cluster.dsn(), autocommit=False) + try: + with bulk.cursor() as bc: + bc.execute(f'SET search_path TO "{schema}"') + try: + bc.execute("UPDATE rlb SET v = v + 1") + bc.execute("SELECT count(*) || '|' || current_setting('pgcolumnar.row_lock_buckets') FROM pg_locks" + " WHERE pid = pg_backend_pid() AND locktype = 'advisory' AND objsubid = 104") + held = bc.fetchone()[0] + bulk.commit() + except psycopg.Error as e: + held = f"ERROR: {e.sqlstate}" + bulk.rollback() + finally: + bulk.close() + expect.text(held, "1024|1024", + "a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them") + with pgc_conn.cursor() as cur: + cur.execute("SELECT count(*) || '|' || sum(v - id) FROM rlb") + expect.text(cur.fetchone()[0], "200000|200000", "and the UPDATE committed every row") + + +def _int4(x): + """An oid as the int4 with the same 32 bits, which is how lockfuncs.c puts a + two-int4 key into the tag.""" + return x - 4294967296 if x > 2147483647 else x + + +def test_the_row_identity_lock_cannot_be_reached_from_sql(pgc_cluster, pgc_conn, expect): + """#1341 item 56: the INSERT arms ask this of the unique-key and storage-row locks, + and nothing asked it of class 104. Every advisory lock an open UPDATE holds is read; + none may be in field 1 or 2. A second session then takes each of those exact tags + through pg_advisory_xact_lock(int4, int4), and an UPDATE of the same row must not + wait on it.""" + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE rl (id int, v int) USING pgcolumnar") + cur.execute("INSERT INTO rl SELECT g, g FROM generate_series(1, 100) g") + cur.execute("SELECT current_schema()") + schema = cur.fetchone()[0] + + updater = _holder(pgc_cluster, schema) + try: + with updater.cursor() as uc: + uc.execute("UPDATE rl SET v = v + 1 WHERE id = 7") + uc.execute("SELECT pg_backend_pid()") + upid = uc.fetchone()[0] + with pgc_conn.cursor() as cur: + cur.execute("SELECT classid, objid, objsubid FROM pg_locks " + "WHERE locktype = 'advisory' AND granted AND pid = %s", (upid,)) + tags = cur.fetchall() + finally: + updater.rollback() + updater.close() + + reachable = [t for t in tags if t[2] in SQL_REACHABLE] + expect.text("unknown (no advisory lock seen)" if not tags + else (f"reachable: {reachable}" if reachable else "unreachable"), + "unreachable", "the locks an UPDATE takes are not in a SQL-reachable class") + + user = _holder(pgc_cluster, schema) + try: + with user.cursor() as hc: + for classid, objid, _f in tags: + hc.execute("SELECT pg_advisory_xact_lock(%s::int, %s::int)", + (_int4(classid), _int4(objid))) + hc.execute("SELECT pg_backend_pid()") + hpid = hc.fetchone()[0] + expect.num(_one(pgc_conn, "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' " + "AND granted AND objsubid = 2 AND pid = %s", (hpid,)), + len(tags), + "premise: the other session holds every one of those tags through the SQL functions") + verdict = "ok" + try: + with pgc_conn.cursor() as cur: + cur.execute("SET statement_timeout = '10s'") + cur.execute("UPDATE rl SET v = v + 1 WHERE id = 7") + except psycopg.errors.QueryCanceled: + verdict = "BLOCKED by the user lock" + except psycopg.Error as exc: + verdict = f"ERROR: {str(exc).splitlines()[0]}" + finally: + with pgc_conn.cursor() as cur: + cur.execute("RESET statement_timeout") + expect.text(verdict, "ok", "a user advisory lock on those tags does not block a columnar UPDATE") + finally: + user.rollback() + user.close() + + +def test_bulk_parallel_writer_set_by_hand_still_serialises_a_first_write( + pgc_cluster, pgc_conn, expect): + """#1341 item 55: the reference says setting bulk_parallel_writer by hand is safe, + because the loaders' skip of the storage-row creation lock fires only once the + storage row is committed. Two sessions with it on each make the first write to a + committed, empty table; the second must wait for the first and then land its row. + A skip that did not wait for the row to be committed lets both create it, and the + second fails on storage_pkey.""" + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE bw (id int) USING pgcolumnar") + cur.execute("SELECT current_schema()") + schema = cur.fetchone()[0] + + first = _holder(pgc_cluster, schema) + second = _holder(pgc_cluster, schema) + second.commit() + second.autocommit = True + try: + with first.cursor() as fc: + fc.execute("SET LOCAL pgcolumnar.bulk_parallel_writer = on") + fc.execute("INSERT INTO bw VALUES (1)") + result = {} + + def write_second(): + try: + with second.cursor() as sc: + sc.execute("SET pgcolumnar.bulk_parallel_writer = on") + sc.execute("SET statement_timeout = '30s'") + sc.execute("INSERT INTO bw VALUES (2)") + result["verdict"] = "ok" + except psycopg.Error as exc: + result["verdict"] = f"ERROR: {str(exc).splitlines()[0]}" + + import threading + import time + t = threading.Thread(target=write_second) + t.start() + # The second write must be waiting on the first, not finished, before the + # first commits: otherwise the arm measures two writes in sequence. + waited = "no" + for _ in range(100): + if _one(pgc_conn, "SELECT count(*) FROM pg_stat_activity WHERE wait_event_type = 'Lock' " + "AND query LIKE 'INSERT INTO bw%%'"): + waited = "yes" + break + time.sleep(0.1) + first.commit() + t.join(60) + expect.text(waited, "yes", "premise: the second first write waited on the first") + expect.text(f"{result.get('verdict', 'still running')}|" + f"{_one(pgc_conn, 'SELECT count(*) FROM bw')}", "ok|2", + "with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land") + finally: + first.rollback() + first.close() + second.close() diff --git a/test/pytest/test_am_contracts.py b/test/pytest/test_am_contracts.py index 5780897d5..7e373c8d2 100644 --- a/test/pytest/test_am_contracts.py +++ b/test/pytest/test_am_contracts.py @@ -448,3 +448,53 @@ def reclaim(am): cur.execute("RESET enable_seqscan") expect.text(f"{col}|{heap}|{found}", "reclaimed|reclaimed|10000", "REINDEX after a committed delete of most rows reclaims their entries, as heap's does") + + +def test_the_documented_refusals_change_nothing(pgc_conn, expect): + """Each is refused where it is chosen, as limitations.md says, and nothing is left + changed by the attempt (#1341 items 44, 51, 77). A failed concurrent build leaves + its index behind, marked invalid, as core does for any such failure.""" + import os + import shutil + import tempfile + + def state(sql): + try: + with pgc_conn.cursor() as cur: + cur.execute(sql) + return "ok" + except psycopg.Error as e: + return e.sqlstate or "?" + + count = "SELECT count(*) || '|' || sum(a) FROM rf" + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE rf (a int) USING pgcolumnar") + cur.execute("INSERT INTO rf SELECT g FROM generate_series(1, 5000) g") + expect.text(f"{state('CREATE UNLOGGED TABLE rf_u (a int) USING pgcolumnar')}|" + f"{_one(pgc_conn, 'SELECT count(*) FROM pg_class WHERE relname = ' + repr('rf_u'))}", + "0A000|0", "CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table") + expect.text(f"{state('ALTER TABLE rf SET UNLOGGED')}|" + f"{_one(pgc_conn, 'SELECT relpersistence FROM pg_class WHERE relname = ' + repr('rf'))}|" + f"{_one(pgc_conn, count)}", + "0A000|p|5000|12502500", "ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows") + + # A tablespace directory the server's own user owns, outside any root-only parent. + ts = tempfile.mkdtemp(prefix="pgc-rf-ts-", dir="/tmp") + os.chmod(ts, 0o700) + if os.geteuid() == 0: + shutil.chown(ts, user="postgres") + try: + with pgc_conn.cursor() as cur: + cur.execute(f"CREATE TABLESPACE rf_ts_py LOCATION '{ts}'") + expect.text(f"{state('ALTER TABLE rf SET TABLESPACE rf_ts_py')}|" + f"{_one(pgc_conn, 'SELECT reltablespace FROM pg_class WHERE relname = ' + repr('rf'))}|" + f"{_one(pgc_conn, count)}", + "0A000|0|5000|12502500", "ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows") + finally: + with pgc_conn.cursor() as cur: + cur.execute("DROP TABLESPACE IF EXISTS rf_ts_py") + shutil.rmtree(ts, ignore_errors=True) + + expect.text(f"{state('CREATE INDEX CONCURRENTLY rf_cic ON rf (a)')}|" + f"{_one(pgc_conn, 'SELECT indisvalid FROM pg_index WHERE indexrelid = ' + repr('rf_cic') + '::regclass')}", + "0A000|False", "CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid") diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index baa7f5f1b..36270ee64 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -114,6 +114,8 @@ "index_fetch_penalty_crossover", "index_fetch_penalty_width", "latest_reading", + "maintenance_crash", + "maintenance_rollback", "native_chunk_length_bound", "native_delete_vector_index", "native_delete_visibility_paths", @@ -130,7 +132,9 @@ "parallel_am_scan", "parallel_scan_cost", "partition_lifecycle", + "partition_rows", "partman_columnar", + "prepared_across_sessions", "prepared_writes", "projection_drop_column", "projection_parallel", @@ -143,6 +147,7 @@ "rewrite_storage_oid", "rollup", "scan_decode_cost", + "serializable", "settings_reference", "sorted_pathkeys", "statement_self_visibility", diff --git a/test/pytest/test_maintenance_crash.py b/test/pytest/test_maintenance_crash.py new file mode 100644 index 000000000..97cdb0426 --- /dev/null +++ b/test/pytest/test_maintenance_crash.py @@ -0,0 +1,132 @@ +"""A crash part-way through maintenance loses nothing (#1341). The pytest twin of +`test/maintenance_crash.sh`, every check by the shell's name. + +compact_rewrite, recluster, compact and vacuum retire row groups and write new ones. +Two crash points, each followed by crash recovery: + + - the function has run but its transaction has not committed, and another session + has checkpointed. Recovery must leave the table as it was before; + - the function has committed, and nothing has checkpointed since, so its work exists + only in WAL. Recovery must leave the table as the function left it. + +A PRIVATE CLUSTER, because each crash takes the postmaster down. The shell kills the +postmaster's process group; this side stops it with `-m immediate`, which leaves the +same work for the next start: "database system was not properly shut down", and redo. +""" +import pathlib +import shutil +import tempfile + +import psycopg + +ROWS = ("SELECT count(*) || '/' || md5(string_agg(id || ':' || v || ':' || w, ',' ORDER BY id)) FROM mc") +GROUPS = ("SELECT count(*) || '/' || coalesce(sum(row_count), 0) FROM pgcolumnar.row_group " + "WHERE storage_id = pgcolumnar.get_storage_id('mc')") + + +def _one(dsn, sql, setup=()): + try: + with psycopg.connect(dsn, autocommit=True) as c: + for s in setup: + c.execute(s) + return c.execute(sql).fetchone()[0] + except psycopg.Error as exc: + return f"ERROR: {str(exc).splitlines()[0]}" + + +def _state(dsn): + by_index = _one(dsn, ROWS + " WHERE id BETWEEN 1 AND 10000", + ("SET enable_seqscan = off", "SET enable_bitmapscan = off")) + return f"{_one(dsn, ROWS)}|{_one(dsn, GROUPS)}|{by_index}" + + +def _fresh_table(dsn): + with psycopg.connect(dsn, autocommit=True) as c: + for s in ("DROP TABLE IF EXISTS mc", "SET pgcolumnar.stripe_row_limit = 1000", + "CREATE TABLE mc (id int, v int, w text) USING pgcolumnar", + "INSERT INTO mc SELECT g, (g * 7919) % 10007, md5(g::text) FROM generate_series(1, 10000) g", + "CREATE INDEX mc_id ON mc (id)", + "DELETE FROM mc WHERE id % 5 IN (1, 2) OR id BETWEEN 3001 AND 4000", + "CHECKPOINT"): + c.execute(s) + + +def _redo_count(log): + try: + return log.read_text(errors="replace").count("database system was not properly shut down") + except OSError: + return 0 + + +def test_a_crash_part_way_through_maintenance_loses_nothing(pgc_cluster, expect): + from pgc_cluster import Cluster, pick_port + + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-mcrash-")) + work.chmod(0o777) + datadir = work / "data" + datadir.mkdir() + cluster = Cluster(pgc_cluster.pg_config, "mcrash", datadir, pick_port(11)) + log = datadir / "server.log" + try: + cluster.initdb() + cluster.start() + dsn = cluster.dsn() + with psycopg.connect(dsn, autocommit=True) as c: + c.execute("CREATE EXTENSION IF NOT EXISTS pgcolumnar") + + for f in ("compact_rewrite('mc', 0.0)", "recluster('mc', 'v')", "compact('mc')", "vacuum('mc')"): + name = f.split("(")[0] + _fresh_table(dsn) + before = _state(dsn) + redo0 = _redo_count(log) + holder = psycopg.connect(dsn, autocommit=True) + holder.execute("BEGIN") + holder.execute(f"SELECT pgcolumnar.{f}") + opened = "yes" + # A function that changed nothing would be trivially crash-safe. + inside = holder.execute(GROUPS).fetchone()[0] + expect.text("changed" if inside != before.split("|")[1] else f"unchanged ({inside})", "changed", + f"{name} changed the row groups in its open transaction") + _one(dsn, "CHECKPOINT") + cluster.stop() + try: + holder.close() + except psycopg.Error: + pass + try: + cluster.start() + came_back = "ok" + except RuntimeError as exc: + came_back = f"restart failed: {exc}" + r1 = _redo_count(log) + recovered = "recovered" if r1 > redo0 else f"no recovery logged ({redo0} before, {r1} after)" + expect.text(f"{opened}|{came_back}|{recovered}", "yes|ok|recovered", + f"{name} ran in its open transaction, and the cluster came back after a crash") + expect.text(_state(dsn), before, + f"after a crash with {name} uncommitted, the table is as it was, by scan and by index") + + for f in ("compact_rewrite('mc', 0.0)", "recluster('mc', 'v')", "compact('mc')", "vacuum('mc')"): + name = f.split("(")[0] + _fresh_table(dsn) + before_rows = _one(dsn, ROWS) + before_groups = _one(dsn, GROUPS) + _one(dsn, f"SELECT pgcolumnar.{f}") + after = _state(dsn) + expect.text("changed" if after.split("|")[1] != before_groups else f"unchanged ({after.split('|')[1]})", + "changed", f"{name} changed the row groups before the crash") + redo0 = _redo_count(log) + cluster.stop() + try: + cluster.start() + came_back = "ok" + except RuntimeError as exc: + came_back = f"restart failed: {exc}" + r1 = _redo_count(log) + recovered = "recovered" if r1 > redo0 else f"no recovery logged ({redo0} before, {r1} after)" + expect.text(f"{came_back}|{recovered}", "ok|recovered", + f"{name} committed, and the cluster came back after a crash before any checkpoint") + expect.text(f"{_state(dsn)}|{_one(dsn, ROWS)}", f"{after}|{before_rows}", + f"after a crash with {name} committed, the table is as {name} left it, with the same rows") + finally: + cluster.stop() + shutil.rmtree(work, ignore_errors=True) diff --git a/test/pytest/test_maintenance_rollback.py b/test/pytest/test_maintenance_rollback.py new file mode 100644 index 000000000..9d4925209 --- /dev/null +++ b/test/pytest/test_maintenance_rollback.py @@ -0,0 +1,107 @@ +"""A maintenance function that rolls back leaves the table as it was (#1341). The +pytest twin of `test/maintenance_rollback.sh`, every check by the shell's name. + +compact_rewrite, recluster, compact, vacuum and vacuum_sorted each retire row groups, +and most write new ones. Retirement is a catalog delete, so a rollback must bring +every retired group back with the bytes it points at intact. The space a retired +group frees may be reused only once no snapshot can need it, and never by the +transaction that freed it, which can still roll back. + +The table has row groups of 1000 rows, two in five rows deleted, and one group +deleted whole, so every function has work. Each runs inside a transaction, its effect +on the row groups is read there, and the transaction rolls back. The table must then +hold the same rows by content and the same row groups, and an index scan must agree. +The last arms do the same under a savepoint, commit further writes, and run every +function for real. +""" +import psycopg + +ROWS = ("SELECT count(*) || '/' || md5(string_agg(id || ':' || v || ':' || w, ',' ORDER BY id)) " + "FROM mr") +GROUPS = ("SELECT count(*) || '/' || coalesce(sum(row_count), 0) FROM pgcolumnar.row_group " + "WHERE storage_id = pgcolumnar.get_storage_id('mr')") + + +def _one(conn, sql): + """-> the first column of the only row, or the error's first line.""" + try: + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + except psycopg.Error as exc: + return f"ERROR: {str(exc).splitlines()[0]}" + + +def _by_index(conn): + with conn.cursor() as cur: + cur.execute("SET enable_seqscan = off") + cur.execute("SET enable_bitmapscan = off") + got = _one(conn, ROWS + " WHERE id BETWEEN 1 AND 10000") + with conn.cursor() as cur: + cur.execute("RESET enable_seqscan") + cur.execute("RESET enable_bitmapscan") + return got + + +def _build_mr(conn): + """The table, made again: 10 row groups of 1000, 5400 rows left. Each function + gets a fresh one, so a rollback that corrupts it fails only its own arm.""" + with conn.cursor() as cur: + cur.execute("SET pgcolumnar.stripe_row_limit = 1000") + cur.execute("DROP TABLE IF EXISTS mr") + cur.execute("CREATE TABLE mr (id int, v int, w text) USING pgcolumnar") + cur.execute("INSERT INTO mr SELECT g, (g * 7919) % 10007, md5(g::text) FROM generate_series(1, 10000) g") + cur.execute("CREATE INDEX mr_id ON mr (id)") + cur.execute("DELETE FROM mr WHERE id % 5 IN (1, 2) OR id BETWEEN 3001 AND 4000") + cur.execute("RESET pgcolumnar.stripe_row_limit") + + +def test_a_maintenance_function_that_rolls_back_leaves_the_table_as_it_was(pgc_conn, expect): + funcs = ("compact_rewrite('mr', 0.0)", "recluster('mr', 'v')", "compact('mr')", + "vacuum('mr')", "vacuum_sorted('mr', 'v')") + _build_mr(pgc_conn) + rows0, groups0 = _one(pgc_conn, ROWS), _one(pgc_conn, GROUPS) + expect.text(f"{rows0.split('/')[0]}|{groups0}", "5400|10/10000", + "premise: the table holds the rows it was given") + + for f in funcs: + name = f.split("(")[0] + _build_mr(pgc_conn) + # BEGIN and ROLLBACK as statements: pgc_conn is a watching wrapper, and an + # autocommit flag set on it would not reach the connection, so the + # function would commit and the rollback would do nothing. + with pgc_conn.cursor() as cur: + cur.execute("BEGIN") + try: + cur.execute(f"SELECT pgcolumnar.{f}") + inside = _one(pgc_conn, GROUPS) + finally: + cur.execute("ROLLBACK") + expect.text("changed" if inside != groups0 and "ERROR" not in inside else f"unchanged ({inside})", + "changed", f"{name} changed the row groups inside its transaction") + expect.text(f"{_one(pgc_conn, ROWS)}|{_one(pgc_conn, GROUPS)}|{_by_index(pgc_conn)}", + f"{rows0}|{groups0}|{rows0}", + f"after {name} rolls back, the table holds the same rows and row groups, by scan and by index") + + _build_mr(pgc_conn) + with pgc_conn.cursor() as cur: + cur.execute("BEGIN") + try: + cur.execute("SAVEPOINT s") + cur.execute("SELECT pgcolumnar.compact_rewrite('mr', 0.0)") + cur.execute("SELECT pgcolumnar.recluster('mr', 'v')") + cur.execute("ROLLBACK TO s") + cur.execute("SET LOCAL pgcolumnar.stripe_row_limit = 1000") + cur.execute("INSERT INTO mr SELECT g, g, 'new' FROM generate_series(20001, 21000) g") + cur.execute("COMMIT") + except psycopg.Error: + cur.execute("ROLLBACK") + raise + rows1 = _one(pgc_conn, ROWS) + new = _one(pgc_conn, "SELECT count(*) FILTER (WHERE w = 'new') || '/' || count(*) FROM mr") + expect.text(f"{new}|{str(_by_index(pgc_conn)).split('/')[0]}", "1000/6400|5400", + "after a savepoint rolls back two rewrites, the rows written later in the transaction commit beside the old ones") + for f in ("compact_rewrite('mr', 0.0)", "recluster('mr', 'v')", "compact('mr')", "vacuum('mr')"): + with pgc_conn.cursor() as cur: + cur.execute(f"SELECT pgcolumnar.{f}") + expect.text(_one(pgc_conn, ROWS), rows1, "and every function then runs for real and keeps the same rows") diff --git a/test/pytest/test_partition_rows.py b/test/pytest/test_partition_rows.py new file mode 100644 index 000000000..533d95257 --- /dev/null +++ b/test/pytest/test_partition_rows.py @@ -0,0 +1,200 @@ +"""Rows that cross columnar partitions, and keys enforced through a partitioned +parent (#1341, #1421). The pytest twin of `test/partition_rows.sh`, every check by +the shell's name. + +A row-moving UPDATE is a delete in one partition and an insert in another. A key on +the parent is one unique index per partition. Each arm runs on a parent with two +columnar partitions and on one with two heap partitions, and heap is the answer. + +THE RACES are ordered by events, not sleeps. A gate connection holds an advisory +lock. The holder writes, then queues for that lock. The contender starts and must be +seen waiting on a lock; only then does the gate let go and the holder commit. A +second session moving a row the first is moving, with a primary key on the parent, +used to fail with 23505 on columnar, where heap fails with 40001 (#1421). +""" +import threading +import time + +import psycopg + +AMS = ("pgcolumnar", "heap") +CONTENT = ("SELECT count(*) || '/' || md5(string_agg(id || ':' || k || ':' || v, ',' ORDER BY id)) " + "FROM %p") +SPLIT = "SELECT (SELECT count(*) FROM %p_1) || '/' || (SELECT count(*) FROM %p_2)" +IDX = ("SET enable_seqscan = off", "SET enable_bitmapscan = off") + + +def _connect(cluster, schema, name): + conn = psycopg.connect(cluster.dsn(), autocommit=True) + conn.execute(f'SET search_path TO "{schema}", public') + conn.execute(f"SET application_name = '{name}'") + return conn + + +def _outcome(conn, sql): + try: + conn.execute(sql) + return "ok" + except psycopg.Error as exc: + return exc.sqlstate or f"no sqlstate: {exc}" + + +def _value(conn, sql, setup=()): + with conn.cursor() as cur: + for s in setup: + cur.execute(s) + cur.execute(sql) + got = cur.fetchone()[0] + for s in setup: + cur.execute(s.replace("SET ", "RESET ").split(" =")[0]) + return got + + +def _both(conn, sql): + return "|".join(_outcome(conn, sql.replace("%p", f"pr_{am}")) for am in AMS) + + +def _answers(conn, sql, setup=()): + return "|".join(str(_value(conn, sql.replace("%p", f"pr_{am}"), setup)) for am in AMS) + + +def _race(cluster, schema, conn, holder_sql, contender_sql, count_sql): + """-> '||'.""" + gate = _connect(cluster, schema, "pr_gate") + holder = _connect(cluster, schema, "pr_holder") + contender = _connect(cluster, schema, "pr_contender") + result = {} + try: + gate.execute("SELECT pg_advisory_lock(1341)") + holder.execute("BEGIN") + holder.execute(holder_sql) + + def hold(): + holder.execute("SELECT pg_advisory_lock_shared(1341)") + holder.execute("COMMIT") + + def contend(): + contender.execute("SET statement_timeout = '60s'") + result["out"] = _outcome(contender, contender_sql) + + th = threading.Thread(target=hold) + th.start() + tc = threading.Thread(target=contend) + tc.start() + waited = "no" + for _ in range(300): + if _value(conn, "SELECT count(*) FROM pg_stat_activity WHERE application_name = 'pr_contender' " + "AND wait_event_type = 'Lock'") == 1: + waited = "yes" + break + time.sleep(0.1) + gate.execute("SELECT pg_advisory_unlock(1341)") + th.join(60) + tc.join(60) + return f"{waited}|{result.get('out', 'still running')}|{_value(conn, count_sql)}" + finally: + for c in (contender, holder, gate): + c.close() + + +def _both_race(cluster, schema, conn, holder_sql, contender_sql, key): + return " ".join(_race(cluster, schema, conn, holder_sql.replace("%p", f"pr_{am}"), + contender_sql.replace("%p", f"pr_{am}"), + f"SELECT count(*) FROM pr_{am} WHERE {key}") for am in AMS) + + +def test_rows_cross_columnar_partitions_and_keys_hold_through_the_parent(pgc_cluster, pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("SELECT current_schema()") + schema = cur.fetchone()[0] + for am in AMS: + cur.execute(f"CREATE TABLE pr_{am} (id int PRIMARY KEY, k int, v text) PARTITION BY RANGE (id)") + cur.execute(f"CREATE TABLE pr_{am}_1 PARTITION OF pr_{am} FOR VALUES FROM (0) TO (1000) USING {am}") + cur.execute(f"CREATE TABLE pr_{am}_2 PARTITION OF pr_{am} FOR VALUES FROM (1000) TO (3000) USING {am}") + cur.execute(f"INSERT INTO pr_{am} SELECT g, g % 7, 'r' || g FROM generate_series(1, 999) g") + + expect.text(_both(pgc_conn, "UPDATE %p SET id = id + 1000 WHERE id < 50"), "ok|ok", + "an UPDATE moving 49 rows to the other partition succeeds, as on heap") + expect.text(_answers(pgc_conn, SPLIT), "950/49|950/49", "and each partition holds what heap's does") + c, h = _answers(pgc_conn, CONTENT).split("|") + expect.text("same" if c == h else f"{c}|{h}", "same", "and the parent holds the same rows as heap's") + expect.text(_answers(pgc_conn, "SELECT count(*) FILTER (WHERE id = 1001) || '/' || " + "count(*) FILTER (WHERE id = 1) FROM %p WHERE id IN (1, 1001)", IDX), + "1/0|1/0", "the primary key finds a moved row at its new key, and not at its old one, as on heap") + + expect.text(_both(pgc_conn, "INSERT INTO %p VALUES (500, 0, 'dup')"), "23505|23505", + "a duplicate key inserted through the parent is refused (23505), as on heap") + expect.text(_both(pgc_conn, "INSERT INTO %p VALUES (1, 0, 'again')"), "ok|ok", + "a key a moved row gave up can be used again, as on heap") + expect.text(_both(pgc_conn, "INSERT INTO %p VALUES (1001, 0, 'dup')"), "23505|23505", + "a duplicate of a moved row's new key is refused in its new partition (23505), as on heap") + + expect.text(_both_race(pgc_cluster, schema, pgc_conn, "INSERT INTO %p VALUES (1500, 0, 'a')", + "INSERT INTO %p VALUES (1500, 0, 'b')", "id = 1500"), + "yes|23505|1 yes|23505|1", + "a second insert of the same new key through the parent waits, then is refused (23505), and one row is left, as on heap") + expect.text(_both_race(pgc_cluster, schema, pgc_conn, "UPDATE %p SET id = id + 1000 WHERE id = 60", + "UPDATE %p SET id = id + 1000 WHERE id = 60", "id IN (60, 1060)"), + "yes|40001|1 yes|40001|1", + "a second session moving a row the first is moving waits, then fails with 40001, and the row exists once, as on heap") + expect.text(_both_race(pgc_cluster, schema, pgc_conn, "UPDATE %p SET id = id + 1000 WHERE id = 70", + "UPDATE %p SET v = 'changed' WHERE id = 70", "id IN (70, 1070) AND v <> 'changed'"), + "yes|40001|1 yes|40001|1", + "an UPDATE in place of a row another session is moving waits, then fails with 40001, as on heap") + c, h = _answers(pgc_conn, CONTENT).split("|") + expect.text("same" if c == h else f"{c}|{h}", "same", "and after the races the parents hold the same rows") + + +PM_MEM = """CREATE FUNCTION pm_mem(at text, id int) RETURNS int LANGUAGE plpgsql AS $$ +BEGIN + IF id IN (1, 49999) THEN + INSERT INTO pm_mem SELECT at, id, sum(total_bytes) FROM pg_backend_memory_contexts; + END IF; + RETURN 0; +END $$""" + + +def _pm_build(conn, name): + """A keyed parent whose first partition holds 1..50000 in one row group, evens deleted.""" + conn.execute(f"CREATE TABLE {name} (id int PRIMARY KEY, v int) PARTITION BY RANGE (id)") + conn.execute(f"CREATE TABLE {name}_1 PARTITION OF {name} FOR VALUES FROM (0) TO (100000) USING pgcolumnar") + conn.execute(f"CREATE TABLE {name}_2 PARTITION OF {name} FOR VALUES FROM (100000) TO (300000) USING pgcolumnar") + conn.execute(f"INSERT INTO {name} SELECT g, 0 FROM generate_series(1, 50000) g") + conn.execute(f"DELETE FROM {name} WHERE id % 2 = 0") + + +def _marks(conn, name): + return _value(conn, f"SELECT count(*) || '|' || (SELECT count(*) FROM pgcolumnar.delete_vector " + f"WHERE storage_id = pgcolumnar.get_storage_id('{name}_1')) FROM {name}") + + +def _bounded(conn, at): + readings, mb = conn.execute( + "SELECT count(*), (max(bytes) FILTER (WHERE id = 49999) - max(bytes) FILTER (WHERE id = 1)) / 1048576 " + "FROM pm_mem WHERE at = %s", (at,)).fetchone() + print(f" {at}: {readings} readings, growth {mb} MB") + return "yes" if readings == 2 and mb is not None and mb < 32 else f"no (readings|MB: {readings}|{mb})" + + +def test_a_keyed_update_keeps_its_memory_bounded(pgc_conn, expect): + """On a keyed table each updated or moved row is checked at once against the + committed delete marks of its row group (#1421). Kept in the statement's memory, + those reads cost one copy of the group's delete bitmap per row: 402 MB for + 25,000 rows of a 50,000-row group (OffgridwithJD, review of #1422). pm_mem records + the backend's context bytes at the first and the last row an UPDATE reaches. + """ + pgc_conn.execute("CREATE TABLE pm_mem (at text, id int, bytes bigint)") + pgc_conn.execute(PM_MEM) + _pm_build(pgc_conn, "pmu") + _pm_build(pgc_conn, "pmm") + expect.text(f"{_marks(pgc_conn, 'pmu')} {_marks(pgc_conn, 'pmm')}", "25000|1 25000|1", + "each keyed table's group carries delete marks before its UPDATE") + pgc_conn.execute("UPDATE pmu SET v = v + 1 + pm_mem('update', id)") + expect.text(_bounded(pgc_conn, "update"), "yes", + "an UPDATE of 25,000 keyed rows keeps under 32 MB of memory between its first and last row") + pgc_conn.execute("UPDATE pmm SET id = id + 100000 + pm_mem('move', id)") + expect.text(_bounded(pgc_conn, "move"), "yes", + "an UPDATE moving 25,000 keyed rows to the other partition keeps under 32 MB likewise") + expect.text(_value(pgc_conn, "SELECT (SELECT sum(v) FROM pmu) || '/' || (SELECT count(*) FROM pmm_1) " + "|| '/' || (SELECT count(*) FROM pmm_2)"), "25000/0/25000", + "and each UPDATE reached every row once") diff --git a/test/pytest/test_prepared_across_sessions.py b/test/pytest/test_prepared_across_sessions.py new file mode 100644 index 000000000..8f40f70dc --- /dev/null +++ b/test/pytest/test_prepared_across_sessions.py @@ -0,0 +1,152 @@ +"""A prepared transaction, seen from other sessions and across a crash (#1341). The +pytest twin of `test/prepared_across_sessions.sh`, every check by the shell's name. + +prepared_writes covers PREPARE and COMMIT PREPARED in one session. This covers what +another session sees while a transaction is prepared; the unique-key and row locks a +prepared transaction still holds (a second writer waits, then fails once COMMIT +PREPARED runs); ROLLBACK PREPARED of a DELETE and an UPDATE; and a prepared insert +surviving a crash and committing after recovery. + +A PRIVATE CLUSTER, because the last arm crashes the postmaster (`-m immediate`). +""" +import pathlib +import shutil +import tempfile +import threading +import time + +import psycopg + +ROWS = "SELECT count(*) || '/' || md5(string_agg(k || ':' || v, ',' ORDER BY k)) FROM px" + + +def _one(dsn, sql): + try: + with psycopg.connect(dsn, autocommit=True) as c: + row = c.execute(sql).fetchone() + return row[0] if row else None + except psycopg.Error as exc: + return f"ERROR: {exc.sqlstate}" + + +def _run(dsn, *stmts): + with psycopg.connect(dsn, autocommit=True) as c: + for s in stmts: + c.execute(s) + + +def _try(dsn, *stmts): + """_run, but a failure is left for the premise after it to report, as the shell does.""" + try: + _run(dsn, *stmts) + except psycopg.Error: + pass + + +def _contend(dsn, name, sql, release): + """Run `sql` as session `name`; once it waits on a lock, run `release`. + -> '|'.""" + result = {} + + def go(): + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(f"SET application_name = '{name}'") + c.execute("SET statement_timeout = '60s'") + c.execute(sql) + result["out"] = "ok" + except psycopg.Error as exc: + result["out"] = exc.sqlstate or "?" + + t = threading.Thread(target=go) + t.start() + waited = "no" + for _ in range(300): + if _one(dsn, f"SELECT count(*) FROM pg_stat_activity WHERE application_name = '{name}' " + "AND wait_event_type = 'Lock'") == 1: + waited = "yes" + break + time.sleep(0.1) + _run(dsn, release) + t.join(60) + return f"{waited}|{result.get('out', 'still running')}" + + +def test_a_prepared_transaction_across_sessions_and_a_crash(pgc_cluster, expect): + from pgc_cluster import Cluster, pick_port + + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-prepx-")) + work.chmod(0o777) + datadir = work / "data" + datadir.mkdir() + cluster = Cluster(pgc_cluster.pg_config, "prepx", datadir, pick_port(14)) + try: + cluster.initdb() + with open(datadir / "postgresql.conf", "a") as fh: + fh.write("\nmax_prepared_transactions=4\n") + cluster.start() + dsn = cluster.dsn() + _run(dsn, "CREATE EXTENSION IF NOT EXISTS pgcolumnar") + expect.text(_one(dsn, "SHOW max_prepared_transactions"), "4", "premise: prepared transactions are enabled") + _run(dsn, "CREATE TABLE px (k int PRIMARY KEY, v text) USING pgcolumnar", + "INSERT INTO px SELECT g, 'r' || g FROM generate_series(1, 1000) g") + + _run(dsn, "BEGIN", "INSERT INTO px SELECT g, 'p' || g FROM generate_series(1001, 1100) g", + "PREPARE TRANSACTION 'px_ins'") + expect.num(_one(dsn, "SELECT count(*) FROM px WHERE k > 1000"), 0, + "a prepared insert is invisible to another session") + got = _contend(dsn, "px_dup", "INSERT INTO px VALUES (1050, 'dup')", "COMMIT PREPARED 'px_ins'") + row = _one(dsn, "SELECT count(*) || '/' || min(v) FROM px WHERE k = 1050") + expect.text(f"{got}|{row}", + "yes|23505|1/p1050", + "a second insert of a key the prepared transaction holds waits, then fails with 23505 after COMMIT PREPARED") + expect.num(_one(dsn, "SELECT count(*) FROM px WHERE k > 1000"), 100, + "and after COMMIT PREPARED the other session sees all the prepared rows") + + before = _one(dsn, ROWS) + # A PREPARE that failed would leave the table unchanged and the checks after + # it comparing it with itself, so its failure is reported by the premise. + _try(dsn, "BEGIN", "DELETE FROM px WHERE k % 20 = 0", "PREPARE TRANSACTION 'px_del'") + expect.num(_one(dsn, "SELECT count(*) FROM pg_prepared_xacts WHERE gid = 'px_del'"), 1, + "premise: the DELETE is prepared") + expect.text(_one(dsn, ROWS), before, "while a DELETE is prepared, another session still sees every row") + _run(dsn, "ROLLBACK PREPARED 'px_del'") + expect.text(_one(dsn, ROWS), before, "ROLLBACK PREPARED of a DELETE leaves every row as it was") + _try(dsn, "BEGIN", "UPDATE px SET v = v || '!' WHERE k % 20 = 1", "PREPARE TRANSACTION 'px_upd'") + expect.num(_one(dsn, "SELECT count(*) FROM pg_prepared_xacts WHERE gid = 'px_upd'"), 1, + "premise: the UPDATE is prepared") + _run(dsn, "ROLLBACK PREPARED 'px_upd'") + expect.text(_one(dsn, ROWS), before, "ROLLBACK PREPARED of an UPDATE leaves every row as it was, none duplicated") + + _run(dsn, "BEGIN", "UPDATE px SET v = 'prepared' WHERE k = 7", "PREPARE TRANSACTION 'px_row'") + got = _contend(dsn, "px_rowc", "UPDATE px SET v = 'other' WHERE k = 7", "COMMIT PREPARED 'px_row'") + row = _one(dsn, "SELECT count(*) || '/' || min(v) FROM px WHERE k = 7") + expect.text(f"{got}|{row}", + "yes|40001|1/prepared", + "an UPDATE of a row a prepared transaction updated waits, then fails with 40001 after COMMIT PREPARED") + + _run(dsn, "CREATE TABLE px_heap (k int PRIMARY KEY, v text)", + "INSERT INTO px_heap SELECT g, md5(g::text) FROM generate_series(5001, 10000) g", + "BEGIN", "INSERT INTO px SELECT g, md5(g::text) FROM generate_series(5001, 10000) g", + "PREPARE TRANSACTION 'px_crash'") + cluster.stop() + try: + cluster.start() + back = "ok" + except RuntimeError as exc: + back = f"restart failed: {exc}" + held = _one(dsn, "SELECT count(*) FROM pg_prepared_xacts WHERE gid = 'px_crash'") + expect.text(f"{back}|{held}", + "ok|1", + "a prepared insert survives a crash: the cluster came back and still holds the prepared transaction") + _run(dsn, "COMMIT PREPARED 'px_crash'") + expect.text(_one(dsn, "SELECT count(*) || '/' || md5(string_agg(k || ':' || v, ',' ORDER BY k)) FROM px WHERE k > 5000"), + _one(dsn, "SELECT count(*) || '/' || md5(string_agg(k || ':' || v, ',' ORDER BY k)) FROM px_heap"), + "and after COMMIT PREPARED its rows are heap's") + finally: + try: + for gid in ("px_ins", "px_del", "px_upd", "px_row", "px_crash"): + _one(cluster.dsn(), f"ROLLBACK PREPARED '{gid}'") + finally: + cluster.stop() + shutil.rmtree(work, ignore_errors=True) diff --git a/test/pytest/test_serializable.py b/test/pytest/test_serializable.py new file mode 100644 index 000000000..90c441e46 --- /dev/null +++ b/test/pytest/test_serializable.py @@ -0,0 +1,175 @@ +"""SERIALIZABLE refuses a write skew on a columnar table, as on heap (#1341). The +pytest twin of `test/serializable.sh`, every check by the shell's name. + +Two transactions each read what the other is about to change, then each writes +something the other's read depended on. At SERIALIZABLE one must fail with 40001. +Four shapes, each on a columnar table and on heap: two doctors taken off call (in one +row group, then in two), and two sessions putting someone on call in an empty ward +(read by a scan of the whole table, then through an index). The first transaction +commits; SSI refuses the second with 40001 ("read/write dependencies"); the +invariant holds. Which statement raises differs: heap's second COMMIT, columnar's +second write, which waits for the first commit. The checks hold the guarantee, not +the step; each error is tagged with its step, and a failure prints them. At +REPEATABLE READ the same interleavings must commit both and break the invariant, +the control that makes a 40001 from anything but SSI fail the checks. + +WHERE THE PROTECTION COMES FROM: heap SSI on columnar's catalog tables, which a +serializable read of a columnar table reads and a write later changes. An arm pins +the SIRead locks there, so a change that stopped reading the catalog that way would +turn it red rather than drop the protection silently. + +The two sessions are two connections. The second write runs in a thread, because +on columnar it can wait for the first session to commit (see `_skew`). +""" +import psycopg + +AMS = ("pgcolumnar", "heap") + + +def _error(conn, sql, tag): + """'' when sql runs, else '::'.""" + try: + conn.execute(sql) + return "" + except psycopg.Error as exc: + msg = str(exc).splitlines()[0] + return f"{tag}:{exc.sqlstate or '?'}:{'ssi' if 'read/write dependencies' in msg else msg}" + + +def _skew(dsn, schema, read, write1, write2, level="SERIALIZABLE"): + """-> '|' for the interleaving in the module docstring, at `level`. + + The second write runs in a thread. On heap it returns at once; on columnar a + write to a row group the first session is changing waits for that session to + commit, as limitations.md says, so the first COMMIT is sent once the second write + has returned or is seen waiting on a lock. Driving the two in strict turn would + wait on itself. + """ + import threading + import time + + s1 = psycopg.connect(dsn, autocommit=True) + s2 = psycopg.connect(dsn, autocommit=True) + try: + errs = {1: [], 2: []} + for c in (s1, s2): + c.execute(f'SET search_path TO "{schema}", public') + pid2 = s2.execute("SELECT pg_backend_pid()").fetchone()[0] + for n, c, sql, tag in ((1, s1, f"BEGIN ISOLATION LEVEL {level}", "r1"), (2, s2, f"BEGIN ISOLATION LEVEL {level}", "r2"), + (1, s1, read, "r1"), (2, s2, read, "r2"), (1, s1, write1, "w1")): + st = _error(c, sql, tag) + if st: + errs[n].append(st) + t = threading.Thread(target=lambda: (lambda st: st and errs[2].append(st))(_error(s2, write2, "w2"))) + t.start() + with psycopg.connect(dsn, autocommit=True) as watch: + for _ in range(300): + if not t.is_alive() or watch.execute( + "SELECT wait_event_type = 'Lock' FROM pg_stat_activity WHERE pid = %s", (pid2,)).fetchone()[0]: + break + time.sleep(0.1) + st = _error(s1, "COMMIT", "c1") + if st: + errs[1].append(st) + t.join(60) + st = _error(s2, "COMMIT", "c2") + if st: + errs[2].append(st) + e2 = " ".join(errs[2]) + # s2 refused once by SSI, at its write or its COMMIT, reads as 40001/ssi + if len(errs[2]) == 1 and errs[2][0] in ("w2:40001:ssi", "c2:40001:ssi"): + e2 = "40001/ssi" + return f"{' '.join(errs[1])}|{e2}" + finally: + s1.close() + s2.close() + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +def test_serializable_refuses_a_write_skew(pgc_cluster, pgc_conn, expect): + schema = _one(pgc_conn, "SELECT current_schema()") + with pgc_conn.cursor() as cur: + for am in AMS: + cur.execute(f"CREATE TABLE doc1_{am} (id int, on_call bool) USING {am}") + cur.execute(f"INSERT INTO doc1_{am} VALUES (1, true), (2, true)") + cur.execute(f"CREATE TABLE doc2_{am} (id int, on_call bool) USING {am}") + cur.execute(f"INSERT INTO doc2_{am} VALUES (1, true)") + cur.execute(f"INSERT INTO doc2_{am} VALUES (2, true)") + cur.execute(f"CREATE TABLE ward_{am} (id int, ward int, on_call bool) USING {am}") + cur.execute(f"INSERT INTO ward_{am} SELECT g, g % 50, false FROM generate_series(1, 5000) g") + cur.execute(f"CREATE TABLE wardi_{am} (id int, ward int, on_call bool) USING {am}") + cur.execute(f"INSERT INTO wardi_{am} SELECT g, g % 50, false FROM generate_series(1, 5000) g") + cur.execute(f"CREATE INDEX wardi_{am}_w ON wardi_{am} (ward)") + cur.execute(f"ANALYZE wardi_{am}") + groups = "/".join(str(_one(pgc_conn, f"SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = " + f"pgcolumnar.get_storage_id('{t}_pgcolumnar')")) for t in ("doc1", "doc2")) + expect.text(groups, "1/2", "premise: the second doctors table holds its two rows in two row groups, the first in one") + + reader = psycopg.connect(pgc_cluster.dsn(), autocommit=True) + try: + reader.execute(f'SET search_path TO "{schema}", public') + reader.execute("BEGIN ISOLATION LEVEL SERIALIZABLE") + reader.execute("SELECT count(*) FROM doc1_pgcolumnar WHERE on_call") + rg, dv = reader.execute( + "SELECT count(*) FILTER (WHERE c.relname IN ('row_group', 'row_group_pkey')), " + "count(*) FILTER (WHERE c.relname IN ('delete_vector', 'delete_vector_pkey')) " + "FROM pg_locks l JOIN pg_class c ON c.oid = l.relation " + "WHERE l.mode = 'SIReadLock' AND l.pid = pg_backend_pid()").fetchone() + reader.execute("COMMIT") + finally: + reader.close() + expect.text("held" if rg >= 1 and dv >= 1 else f"{rg}/{dv}", "held", + "a serializable read of a columnar table holds SIRead locks on columnar's row_group and delete_vector catalog") + + def both(stem, read, w1, w2, invariant, level="SERIALIZABLE"): + out = [] + for am in AMS: + t = f"{stem}_{am}" + got = _skew(pgc_cluster.dsn(), schema, read.replace("%t", t), w1.replace("%t", t), w2.replace("%t", t), + level) + out.append(f"{got}|{_one(pgc_conn, invariant.replace('%t', t))}") + return " ".join(out) + + off = ("SELECT count(*) FROM %t WHERE on_call", "UPDATE %t SET on_call = false WHERE id = 1", + "UPDATE %t SET on_call = false WHERE id = 2", "SELECT count(*) FROM %t WHERE on_call") + expect.text(both("doc1", *off), "|40001/ssi|1 |40001/ssi|1", + "two doctors in one row group each taken off call: SSI refuses one of the two with 40001, and one stays on call, as on heap") + expect.text(both("doc2", *off), "|40001/ssi|1 |40001/ssi|1", "the same with the two doctors in two row groups") + on = ("INSERT INTO %t VALUES (101, 7, true)", "INSERT INTO %t VALUES (102, 7, true)", + "SELECT count(*) FROM %t WHERE ward = 7 AND on_call") + expect.text(both("ward", "SELECT count(*) FROM %t WHERE ward = 7 AND on_call", *on), "|40001/ssi|1 |40001/ssi|1", + "two sessions that each see an empty ward by a scan of the whole table and each put someone on call: " + "SSI refuses one of the two with 40001, as on heap") + plans = [] + for am in AMS: + with pgc_conn.cursor() as cur: + for s in ("SET enable_seqscan = off", "SET enable_bitmapscan = off", "SET pgcolumnar.enable_custom_scan = off"): + cur.execute(s) + cur.execute(f"EXPLAIN (COSTS OFF) SELECT count(*) FROM wardi_{am} WHERE ward = 7 AND on_call") + plan = "\n".join(r[0] for r in cur.fetchall()) + for s in ("enable_seqscan", "enable_bitmapscan", "pgcolumnar.enable_custom_scan"): + cur.execute(f"RESET {s}") + plans.append(str(int(f"Scan using wardi_{am}_w" in plan and "Index" in plan))) + expect.text("|".join(plans), "1|1", "premise: the ward lookup is an index scan on both tables") + idx = ("SET LOCAL enable_seqscan = off; SET LOCAL enable_bitmapscan = off; " + "SET LOCAL pgcolumnar.enable_custom_scan = off; SELECT count(*) FROM %t WHERE ward = 7 AND on_call") + expect.text(both("wardi", idx, *on), "|40001/ssi|1 |40001/ssi|1", "the same when both read the ward through an index") + + # ---- the control: REPEATABLE READ lets both commit ---- + for am in AMS: + for sql in (f"UPDATE doc1_{am} SET on_call = true", f"UPDATE doc2_{am} SET on_call = true", + f"DELETE FROM ward_{am} WHERE on_call", f"DELETE FROM wardi_{am} WHERE on_call"): + pgc_conn.execute(sql) + rr = "REPEATABLE READ" + expect.text(f"{both('doc1', *off, rr)} / {both('doc2', *off, rr)}", "||0 ||0 / ||0 ||0", + "at REPEATABLE READ both doctors' transactions commit and no one stays on call, in one row group and " + "in two, as on heap") + expect.text(f"{both('ward', 'SELECT count(*) FROM %t WHERE ward = 7 AND on_call', *on, rr)} / " + f"{both('wardi', idx, *on, rr)}", "||2 ||2 / ||2 ||2", + "at REPEATABLE READ both ward transactions commit and two are on call, by a scan and by an index, " + "as on heap") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 7e53924d1..bd7adbd97 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -140,7 +140,9 @@ SUITES=( logical_decoding_cdc_recipe logical_decoding_source logical_subscriber + maintenance_crash maintenance_due + maintenance_rollback native_agg native_agg_addcolumn native_agg_deletes @@ -257,6 +259,7 @@ SUITES=( parquet_nested parquet_nested_import partition_lifecycle + partition_rows partman_columnar pg19_vacuum_options pg_dump_roundtrip @@ -267,6 +270,7 @@ SUITES=( phase6 planner_choice_quality preimage_rewrite + prepared_across_sessions prepared_writes projection_drop_column projection_parallel @@ -294,6 +298,7 @@ SUITES=( row_triggers scan_decode_cost scan_direction + serializable server_file_privilege settings_reference smoke diff --git a/test/serializable.sh b/test/serializable.sh new file mode 100755 index 000000000..a7b93273a --- /dev/null +++ b/test/serializable.sh @@ -0,0 +1,172 @@ +#!/usr/bin/env bash +# +# SERIALIZABLE refuses a write skew on a columnar table, as on heap (#1341). Two +# transactions each read what the other is about to change, then each writes +# something the other's read depended on. Run one after the other, the second would +# have read differently, so at SERIALIZABLE one of them must fail with +# serialization_failure (40001). +# +# Four shapes, each on a columnar table and on heap, and heap is the answer: +# - the on-call doctors: both read who is on call, each takes a different doctor +# off call. Once with the two rows in one row group, once in two; +# - the empty ward: both see no one on call in a ward, each inserts someone. +# Once read by a scan of the whole table (columnar's custom scan), once through +# an index. +# In each, the first transaction commits, the second is refused by SSI with 40001 +# ("could not serialize access due to read/write dependencies"), and the invariant +# holds afterwards. WHICH STATEMENT raises differs: on heap the second COMMIT; on +# columnar the second write, which waits for the first session to commit and then +# raises, after which its COMMIT returns cleanly (measured on PG17 and PG18 by +# OffgridwithJD, review of #1424). The checks hold the guarantee, not the step: each +# error is tagged with its step (r = BEGIN and read, w = write, c = COMMIT), and a +# failure prints them. At REPEATABLE READ the same interleavings must commit both +# transactions and break the invariant, on both tables -- the control that makes a +# 40001 from anything but SSI fail the checks above. +# +# Usage: test/serializable.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +W="$PGC_WORKDIR" +session() { # a psql reading statements from a fifo, reporting SQLSTATEs + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq -v VERBOSITY=verbose +} +# skew READ WRITE1 WRITE2 [LEVEL] -> "|", running at LEVEL (SERIALIZABLE) +# s1: BEGIN LEVEL; READ s2: BEGIN LEVEL; READ +# s1: WRITE1 s2: WRITE2 +# s1: COMMIT s2: COMMIT +# Each statement is sent once the session has answered the one before, which a +# marker line echoed after it proves. +skew() { + local i n + rm -f "$W/sk1" "$W/sk2" + mkfifo "$W/sk1" "$W/sk2" + session <"$W/sk1" >"$W/so1" 2>&1 & local p1=$! + session <"$W/sk2" >"$W/so2" 2>&1 & local p2=$! + exec 7>"$W/sk1" 8>"$W/sk2" + step() { # step FD OUT MARK SQL [NAME]: send SQL, wait until OUT shows MARK, or + # until the session named NAME is seen waiting on a lock. On columnar a write + # to a row group another session is changing waits for that session to + # commit, as limitations.md says, so the next step must not wait for it. + echo "$4" >&"$1"; echo "\\echo $3" >&"$1" + for i in $(seq 1 300); do + grep -qx "$3" "$2" 2>/dev/null && return 0 + [ -n "${5:-}" ] && [ "$(q "SELECT count(*) FROM pg_stat_activity WHERE application_name = '$5' AND wait_event_type = 'Lock'")" = 1 ] && return 0 + sleep 0.1 + done + echo "timeout at $3" >>"$2" + } + local lvl="${4:-SERIALIZABLE}" + step 7 "$W/so1" m1 "SET application_name = 'sk_s1'; BEGIN ISOLATION LEVEL $lvl; $1" + step 8 "$W/so2" m2 "SET application_name = 'sk_s2'; BEGIN ISOLATION LEVEL $lvl; $1" + step 7 "$W/so1" m3 "$2" + step 8 "$W/so2" m4 "$3" sk_s2 + step 7 "$W/so1" m5 "COMMIT;" + step 8 "$W/so2" m6 "COMMIT;" + exec 7>&- 8>&- + wait "$p1" "$p2" + # tagged FILE N WMARK CMARK -> "::" per error + tagged() { + awk -v n="$2" -v wm="$3" -v cm="$4" 'BEGIN { st = "r" } + $0 == wm { st = "w" } $0 == cm { st = "c" } + /^timeout at / { printf "%s ", $0 } + /^ERROR: +[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]:/ { + code = $2; sub(/:$/, "", code); msg = $0; sub(/^ERROR: +[0-9A-Z]+: /, "", msg) + printf "%s%s:%s:%s ", st, n, code, (msg ~ /read\/write dependencies/ ? "ssi" : msg) }' "$1" | sed 's/ $//' + } + local e1 e2; e1="$(tagged "$W/so1" 1 m1 m3)"; e2="$(tagged "$W/so2" 2 m2 m4)" + # s2 refused once by SSI, at its write or its COMMIT, reads as 40001/ssi + [[ "$e2" =~ ^[wc]2:40001:ssi$ ]] && e2="40001/ssi" + echo "$e1|$e2" +} + +for am in pgcolumnar heap; do + psql_run "CREATE TABLE doc1_$am (id int, on_call bool) USING $am; + INSERT INTO doc1_$am VALUES (1, true), (2, true); + CREATE TABLE doc2_$am (id int, on_call bool) USING $am; + INSERT INTO doc2_$am VALUES (1, true); + INSERT INTO doc2_$am VALUES (2, true); + CREATE TABLE ward_$am (id int, ward int, on_call bool) USING $am; + INSERT INTO ward_$am SELECT g, g % 50, false FROM generate_series(1, 5000) g; + CREATE TABLE wardi_$am (id int, ward int, on_call bool) USING $am; + INSERT INTO wardi_$am SELECT g, g % 50, false FROM generate_series(1, 5000) g; + CREATE INDEX wardi_${am}_w ON wardi_$am (ward); + ANALYZE wardi_$am;" +done +check "premise: the second doctors table holds its two rows in two row groups, the first in one" \ + "$(q "SELECT (SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('doc1_pgcolumnar')) + || '/' || (SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('doc2_pgcolumnar'))")" "1/2" + +# WHERE THE PROTECTION COMES FROM. The access method takes no predicate locks of its +# own. A serializable read of a columnar table reads columnar's catalog -- row_group, +# delete_vector, column_chunk, which are heap tables -- and heap SSI takes SIRead +# locks there; a write later updates those catalog rows, and heap finds the conflict. +# So a change that stopped reading the catalog per statement (a cache) or read it +# with a non-MVCC snapshot would drop the protection silently. This arm pins it. +check "a serializable read of a columnar table holds SIRead locks on columnar's row_group and delete_vector catalog" \ + "$(q "BEGIN ISOLATION LEVEL SERIALIZABLE; SELECT count(*) FROM doc1_pgcolumnar WHERE on_call; + SELECT count(*) FILTER (WHERE c.relname IN ('row_group', 'row_group_pkey')) || '/' || + count(*) FILTER (WHERE c.relname IN ('delete_vector', 'delete_vector_pkey')) + FROM pg_locks l JOIN pg_class c ON c.oid = l.relation + WHERE l.mode = 'SIReadLock' AND l.pid = pg_backend_pid(); COMMIT;" | grep -E '^[0-9]+/[0-9]+$' | sed 's#^\([1-9][0-9]*\)/\([1-9][0-9]*\)$#held#')" "held" + +both() { # both TABLESTEM READ WRITE1 WRITE2 INVARIANT [LEVEL] -> " " + local am out="" + for am in pgcolumnar heap; do + t="$1_$am" + out="$out$(skew "${2//%t/$t}" "${3//%t/$t}" "${4//%t/$t}" "${6:-}")|$(q "${5//%t/$t}" | tail -1) " + done + echo "${out% }" +} +reset_tables() { # the doctors back on call, the wards empty again + local am + for am in pgcolumnar heap; do + psql_run "UPDATE doc1_$am SET on_call = true; UPDATE doc2_$am SET on_call = true; + DELETE FROM ward_$am WHERE on_call; DELETE FROM wardi_$am WHERE on_call;" + done +} +check "two doctors in one row group each taken off call: SSI refuses one of the two with 40001, and one stays on call, as on heap" \ + "$(both doc1 "SELECT count(*) FROM %t WHERE on_call;" "UPDATE %t SET on_call = false WHERE id = 1;" \ + "UPDATE %t SET on_call = false WHERE id = 2;" "SELECT count(*) FROM %t WHERE on_call")" \ + "|40001/ssi|1 |40001/ssi|1" +check "the same with the two doctors in two row groups" \ + "$(both doc2 "SELECT count(*) FROM %t WHERE on_call;" "UPDATE %t SET on_call = false WHERE id = 1;" \ + "UPDATE %t SET on_call = false WHERE id = 2;" "SELECT count(*) FROM %t WHERE on_call")" \ + "|40001/ssi|1 |40001/ssi|1" +check "two sessions that each see an empty ward by a scan of the whole table and each put someone on call: SSI refuses one of the two with 40001, as on heap" \ + "$(both ward "SET LOCAL enable_indexscan = off; SELECT count(*) FROM %t WHERE ward = 7 AND on_call;" \ + "INSERT INTO %t VALUES (101, 7, true);" "INSERT INTO %t VALUES (102, 7, true);" \ + "SELECT count(*) FROM %t WHERE ward = 7 AND on_call")" \ + "|40001/ssi|1 |40001/ssi|1" +# The columnar custom scan wins over an index scan even with seqscan off, so it is +# turned off for the index arm. The sequential arm above reads columnar through it. +IDX="SET LOCAL enable_seqscan = off; SET LOCAL enable_bitmapscan = off; SET LOCAL pgcolumnar.enable_custom_scan = off;" +check "premise: the ward lookup is an index scan on both tables" \ + "$(for t in wardi_pgcolumnar wardi_heap; do q "SET enable_seqscan = off; SET enable_bitmapscan = off; SET pgcolumnar.enable_custom_scan = off; EXPLAIN (COSTS OFF) SELECT count(*) FROM $t WHERE ward = 7 AND on_call" | grep -c "Index.* Scan using ${t}_w"; done | paste -sd'|')" "1|1" +check "the same when both read the ward through an index" \ + "$(both wardi "$IDX SELECT count(*) FROM %t WHERE ward = 7 AND on_call;" \ + "INSERT INTO %t VALUES (101, 7, true);" "INSERT INTO %t VALUES (102, 7, true);" \ + "SELECT count(*) FROM %t WHERE ward = 7 AND on_call")" \ + "|40001/ssi|1 |40001/ssi|1" + +# ---- the control: REPEATABLE READ lets both commit -------------------------------- +# Snapshot isolation without SSI allows the write skew, on heap and on columnar, so +# both transactions commit and the invariant breaks. A 40001 above that came from +# anything but SSI's read/write tracking would show up here as a failure too. +reset_tables +DOFF=("SELECT count(*) FROM %t WHERE on_call;" "UPDATE %t SET on_call = false WHERE id = 1;" + "UPDATE %t SET on_call = false WHERE id = 2;" "SELECT count(*) FROM %t WHERE on_call") +WON=("INSERT INTO %t VALUES (101, 7, true);" "INSERT INTO %t VALUES (102, 7, true);" + "SELECT count(*) FROM %t WHERE ward = 7 AND on_call") +r1="$(both doc1 "${DOFF[@]}" "REPEATABLE READ")"; r2="$(both doc2 "${DOFF[@]}" "REPEATABLE READ")" +check "at REPEATABLE READ both doctors' transactions commit and no one stays on call, in one row group and in two, as on heap" \ + "$r1 / $r2" "||0 ||0 / ||0 ||0" +r1="$(both ward "SET LOCAL enable_indexscan = off; SELECT count(*) FROM %t WHERE ward = 7 AND on_call;" "${WON[@]}" "REPEATABLE READ")" +r2="$(both wardi "$IDX SELECT count(*) FROM %t WHERE ward = 7 AND on_call;" "${WON[@]}" "REPEATABLE READ")" +check "at REPEATABLE READ both ward transactions commit and two are on call, by a scan and by an index, as on heap" \ + "$r1 / $r2" "||2 ||2 / ||2 ||2" + +pgc_summary diff --git a/test/update_conc.sh b/test/update_conc.sh index db3651e35..7243146cd 100755 --- a/test/update_conc.sh +++ b/test/update_conc.sh @@ -466,14 +466,15 @@ check "CONTROL serial columnar keeps one row (GREEN today; proves the RED is the check "CONTROL serial columnar keeps the last writer's value" "$(vals t_col)" "20" # --------------------------------------------------------------------------- -# CHARACTERIZATION: a covering UNIQUE(id) index makes the shipped unique-key -# advisory lock (columnar_unique.c) serialize the two NEW rows, so the loser gets -# a unique_violation (23505) instead of duplicating -- but a heap COMPOSES with no -# error. This documents why the primary RED fixture above has NO unique index (an -# index would mask the lost update) and that the unique path is a partial -# mitigation, not heap-transparent serialization. Stable before and after the fix. +# A covering UNIQUE(id) index. The loser must get the same retryable +# serialization_failure (40001) as without one, as docs/limitations.md promises +# ("needs no unique index"). It used to get unique_violation (23505): the new +# version's unique-key lock waited on the winner's identical key, and the unique +# check fired before the flush that raises 40001 (#1421). A heap composes with no +# error. The primary fixture above still has no unique index, so this arm and that +# one exercise the two paths separately. # --------------------------------------------------------------------------- -echo; echo "### CHARACTERIZATION: covering UNIQUE(id) is only a partial mitigation" +echo; echo "### a covering UNIQUE(id): the loser still gets 40001 (#1421)" ctl_q "CREATE TABLE t_colu (id int, v int) USING pgcolumnar;" >/dev/null ctl_q "CREATE UNIQUE INDEX t_colu_uidx ON t_colu (id);" >/dev/null ctl_q "CREATE TABLE t_heapu (id int, v int);" >/dev/null @@ -482,8 +483,8 @@ uu t_heapu "READ COMMITTED" uxh_; H_CNT=$RES_CNT; H_SS=$RES_SS uu t_colu "READ COMMITTED" uxc_; C_CNT=$RES_CNT; C_SS=$RES_SS echo "-- heap+uniq: count=$H_CNT sqlstate=$H_SS ; col+uniq: count=$C_CNT sqlstate=$C_SS" check "unique/RC heap composes with no error (sqlstate 00000)" "$H_SS" "00000" -check "unique/RC columnar loser errors where heap composes (not transparent)" \ - "$([ "$C_SS" != "00000" ] && echo errored)" "errored" +check "unique/RC columnar loser gets serialization_failure (40001), as without a unique index" \ + "$C_SS|$C_CNT" "40001|1" # --------------------------------------------------------------------------- # savepoint: a subtransaction that updates a row then rolls back must discard both