test: replication gets its pytest twin, and its vacuum and compact arms replay a write (#1341)

CHANGELOG.md

@@ -400,6 +400,18 @@ true until the next version shipped.
   Reported by @OffgridwithJD.
 
 ### Fixed
+- **In the upgrade window, a refusal for a missing catalog table names `ALTER
+  EXTENSION UPDATE`.** After the new library is installed and the server restarted,
+  but before `ALTER EXTENSION pgcolumnar UPDATE`, `DROP TABLE`, `TRUNCATE` and
+  `pgcolumnar.vacuum()` on a columnar table fail, because the 1.0-alpha5 library
+  needs `pgcolumnar.retired_group` and a 1.0-alpha4 catalog lacks it. They change
+  nothing. The error used to name only the missing table, and it now carries a hint:
+  `The pgcolumnar catalog in this database may be older than the installed library.
+  Run ALTER EXTENSION pgcolumnar UPDATE in this database.` Reads and writes work
+  throughout, and after the update every one of these succeeds on the same rows
+  (#1341 item 66). `test/native_upgrade_converge.sh` and
+  `test_native_upgrade_converge.py`.
+
 
 - **A join whose runtime-filter build is empty no longer reads the fact table.** When
   a large dimension filtered to no row was joined to a columnar table, the hash join

docs/installation.md

@@ -152,6 +152,18 @@ The fix is step 3. Run `ALTER EXTENSION pgcolumnar UPDATE;` in that database and
 the error goes away. **Your data is not affected.** The tables are intact and no
 conversion happens. Only the catalog entry is stale.
 
+Between releases that add a catalog table, the new library can also need a table
+the old catalog does not have. Upgrading from `1.0-alpha4` to `1.0-alpha5`,
+`DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()` on a columnar table fail until
+step 3, and change nothing; reads and writes keep working:
+
+```
+ERROR:  columnar metadata table "pgcolumnar.retired_group" does not exist
+HINT:  The pgcolumnar catalog in this database may be older than the installed library. Run ALTER EXTENSION pgcolumnar UPDATE in this database.
+```
+
+The fix is the same: step 3.
+
 Do not use `DROP EXTENSION` instead. A plain `DROP EXTENSION pgcolumnar` fails
 while columnar tables exist, because they depend on the access method. The
 form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every

src/columnar_metadata.c

@@ -239,11 +239,22 @@ open_columnar_table(const char *name, LOCKMODE lockmode)
 	nspOid = pgcolumnar_schema_oid();
 	relOid = get_relname_relid(name, nspOid);
 
+	/*
+	 * NAME THE REMEDY (#1341 item 66). The usual way here is the upgrade window:
+	 * the new library is installed and the server restarted (docs/installation.md
+	 * steps 1 and 2), but this database's catalog is still the previous release's
+	 * until ALTER EXTENSION UPDATE (step 3). A catalog table this library added,
+	 * such as retired_group in 1.0-alpha5, is then missing, and DROP TABLE,
+	 * TRUNCATE and pgcolumnar.vacuum() stop here. The error alone named only the
+	 * table, which says neither why nor what to do.
+	 */
 	if (!OidIsValid(relOid))
 		ereport(ERROR,
 				(errcode(ERRCODE_UNDEFINED_TABLE),
 				 errmsg("columnar metadata table \"%s.%s\" does not exist",
-						COLUMNAR_SCHEMA_NAME, name)));
+						COLUMNAR_SCHEMA_NAME, name),
+				 errhint("The pgcolumnar catalog in this database may be older than the installed library. "
+						 "Run ALTER EXTENSION pgcolumnar UPDATE in this database.")));
 
 	rel = table_open(relOid, lockmode);
 

test/check_ledger_budget.txt

@@ -15,7 +15,7 @@
 #
 #   Adding a check to a suite that is already covered does not move it, which is
 #   what makes it safe to bound.
-suites_not_covered 219
+suites_not_covered 211
 #
 # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
 #   one.
@@ -2943,4 +2943,127 @@ suites_not_covered 219
 #     never         2676 - 1 = 2675
 #     dated          943 + 1 = 944
 #     2675 + 944 == 3619
-checks_never_observed_red 2675
+#
+# #1341, fk_referencing seeded, with its twin (OffgridwithJD): 2675 -> 2686, and
+# suites_not_covered 219 -> 218. The suite had no rows. It enters with 28 rows
+# from five-major runs: 27 checks on PG17-19 and 26 on 15-16, where the
+# partitioned-control branch asserts core's refusal instead of a conversion. 17 are
+# dated by four mutations of the foreign-key guards: the referenced-side refusal
+# removed, the guard keyed on the referencing table, the conversion refusal
+# removed, and the conversion refused whatever the target access method (the new
+# referenced-table-to-row-store arm). Eleven arrive never: the over-rejection
+# controls that no single guard reaches (another constraint, a heap-to-heap key,
+# an unreferenced conversion, an unrelated ALTER, a conversion back to heap, the
+# referencing side, the partitioned control on either side of 17) and the primary
+# key's own arms.
+#
+#     ledger rows   3619 + 28 = 3647
+#     never         2675 + 11 = 2686
+#     dated          944 + 17 = 961
+#     2686 + 961 == 3647
+#
+# #1341, truncate_cleanup seeded, with its twin (OffgridwithJD): 2686 -> 2695, and
+# suites_not_covered 218 -> 217. The suite had no rows. It enters with all 25 checks
+# from five-major runs (25/25 on each of 15-19). 16 are dated by four mutations of
+# the truncate paths: a rewrite that no longer deletes the retired storage's rows
+# (7), nor drops the stale write state (1), an in-place truncate that skips the
+# projections' storages (8), and a storage-tree delete that skips them (1). Nine
+# arrive never: the premises that each fixture holds rows or metadata, the
+# surviving projection declaration, the empty table after TRUNCATE, and the
+# rewrite that keeps every row.
+#
+#     ledger rows   3647 + 25 = 3672
+#     never         2686 +  9 = 2695
+#     dated          961 + 16 = 977
+#     2695 + 977 == 3672
+#
+# #1341, native_dml seeded, with its twin (OffgridwithJD): 2695 -> 2702, and
+# suites_not_covered 217 -> 216. The suite had no rows. It enters with all 15 checks
+# from five-major runs (15/15 on each of 15-19). 8 are dated by three mutations: a
+# row group holding deletes still answering min and max from its zone maps (the
+# min/max arm, which could not see it before this change), count(*) from metadata
+# not subtracting deleted rows (3), and the index fetch ignoring the delete vector
+# (5). Seven arrive never: the two premises, the rows and sum after the delete, the
+# deleted rows being gone, the live row's index lookup, and the primary key arm.
+#
+#     ledger rows   3672 + 15 = 3687
+#     never         2695 +  7 = 2702
+#     dated          977 +  8 = 985
+#     2702 + 985 == 3687
+#
+# #1341, native_dict_underfill seeded, with its twin (OffgridwithJD): 2702 -> 2704,
+# and suites_not_covered 216 -> 215. The suite had no rows. It enters with all 4
+# checks from five-major runs (4/4 on each of 15-19). 2 are dated by two mutations
+# of the DICT fill check: removed, so the garbage tail is read back with no error,
+# and replaced by a backend crash, which the rejection and the log-counted survival
+# arm both catch. The two premises arrive never.
+#
+#     ledger rows   3687 +  4 = 3691
+#     never         2702 +  2 = 2704
+#     dated          985 +  2 = 987
+#     2704 + 987 == 3691
+#
+# #1341, native_decode_gating seeded, with its twin (OffgridwithJD): 2704 -> 2722, and
+# suites_not_covered 215 -> 214. The suite had no rows. It enters with all 29 checks
+# from five-major runs (29/29 on each of 15-19). 11 are dated by four mutations of
+# #452 phase 2: the per-vector evaluation never run (4), a deleted row counted as
+# filtered (1, the new arm), every vector marked skipped (7), and the producer
+# counting a gated group's rejections twice (2, one of them new). Eighteen arrive
+# never: the thirteen premises, the observable's presence, the full scan across
+# every group, the nomatch query's empty answer, and two exact counters that no
+# single mutation here moves.
+#
+#     ledger rows   3691 + 29 = 3720
+#     never         2704 + 18 = 2722
+#     dated          987 + 11 = 998
+#     2722 + 998 == 3720
+#
+# #1341, native_decode_gate_width seeded, with its twin (OffgridwithJD): 2722 -> 2754,
+# and suites_not_covered 214 -> 213. The suite had no rows. It enters with all 35
+# checks from five-major runs (35/35 on each of 15-19). 3 are dated by three
+# mutations of the #595 width gate: gating whatever the width (the two "not gated"
+# arms), > for >= (the boundary arm), and a payload count one too high (the
+# raised-threshold arm). Thirty-two arrive never: the twenty-one premises, and the arms
+# the width policy does not move -- the gated counts, the gate-off and controls, the
+# correctness arms -- which the sibling native_decode_gating dates.
+#
+#     ledger rows   3720 + 35 = 3755
+#     never         2722 + 32 = 2754
+#     dated          998 +  3 = 1001
+#     2754 + 1001 == 3755
+#
+# #1341, pg_dump_roundtrip seeded, with its twin (OffgridwithJD): 2754 -> 2778, and
+# suites_not_covered 213 -> 212. The suite had no rows. It enters with all 40 checks
+# from five-major runs (40/40 on each of 15-19). 16 are dated by five mutations: the
+# options and the projection-declaration catalogs no longer registered with
+# pg_extension_config_dump, rebuild_projections() skipping a missing projection,
+# drop_projection keeping its declaration, and vacuum_sorted sorting on column 1
+# (the strengthened sort arm). Twenty-four arrive never: the dump and restore exit
+# codes and the dump's columnar AM, the access method, the row count and checksum,
+# the index premise and its lookup, the pinned loss of projection storage, the
+# inversion premise, the second rebuild's no-op, and the source's own premises.
+#
+#     ledger rows   3755 + 40 = 3795
+#     never         2754 + 24 = 2778
+#     dated         1001 + 16 = 1017
+#     2778 + 1017 == 3795
+#
+# #1341, replication seeded, with its twin (OffgridwithJD): 2778 -> 2811, and
+# suites_not_covered 212 -> 211. The suite had no rows. It enters with all 44 checks
+# from five-major runs (44/44 on each of 15-19). 11 are dated by four mutations: the
+# SMGR truncate record not emitted (the standby's fork stays long), data and gap pages
+# written without WAL (seven replay comparisons and the fork size), vacuum returning
+# before it rewrites, and compact retiring nothing (its new arm and the truncate's
+# precondition). Thirty-three arrive never: the backup, restore and standby controls
+# (the data directories, connections, recovery state, slot and upstream), the
+# primary's pins and the r2 premise, the projection arms a bytewise copy carries
+# without any columnar WAL, the base backup's rows, the row-count comparisons that
+# count(*) answers from the catalog, the paused-replay control, the standby's
+# recovery state after the truncate, and the module-less standby's arms, which no
+# product mutation can redden while every record is a core type.
+#
+#     ledger rows   3795 + 44 = 3839
+#     never         2778 + 33 = 2811
+#     dated         1017 + 11 = 1028
+#     2811 + 1028 == 3839
+checks_never_observed_red 2811

test/fk_referencing.sh

@@ -220,6 +220,16 @@ check "converting back to heap is unaffected" \
 	"$(case "$back" in *ERROR*) echo "no ($back)" ;; *) echo yes ;; esac)" \
 	"yes"
 
+# AND ON THE REFERENCED TABLE ITSELF (#1341). The control above converts a table
+# with no foreign key, so it cannot see a rule that refuses every SET ACCESS METHOD
+# on a referenced table whatever the target: with the guard's columnar-target test
+# removed, this file stayed 26/26. Only a columnar target needs the row lock the
+# constraint cannot get.
+refheap="$(psql_run "ALTER TABLE fk_conv_parent SET ACCESS METHOD heap;" 2>&1 || true)"
+check "an FK-referenced table may still be set to a row store" \
+	"$(case "$refheap" in *ERROR*) echo "no ($refheap)" ;; *) echo yes ;; esac)" \
+	"yes"
+
 # --- 7. a partitioned parent is refused too (#201) -----------------------------
 
 # A partitioned table has no storage, so setting its access method breaks nothing

test/native_decode_gate_width.sh

@@ -106,10 +106,17 @@ check "premise: the width-gate GUC exists and defaults to the shipped $DEFAULT_G
 # unknown parameter errors; q swallows the error and returns empty, so an empty
 # result is the proof the parameter does not exist. The paired positive premise
 # above (min_payload_cols == 20) rules out an empty result from a dead connection.
+# By SQLSTATE rather than by an empty result (#1341): q returns empty for a dead
+# connection as well, and an empty expectation is the shape the pytest layer refuses.
+show_state() {	# show_state PARAM -> the SQLSTATE SHOW raised, or "shown"
+	env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" \
+		-qAtX -v VERBOSITY=verbose -c "SHOW $1;" >/dev/null 2>"$PGC_WORKDIR/show.err" && { echo shown; return; }
+	sed -nE 's/^ERROR:  ([0-9A-Z]{5}): .*/\1/p' "$PGC_WORKDIR/show.err" | head -1
+}
 check "premise: the rejected probe GUC qual_skipvec_probe_vecs was NOT shipped" \
-	"$(q "SHOW pgcolumnar.qual_skipvec_probe_vecs;")" ""
+	"$(show_state pgcolumnar.qual_skipvec_probe_vecs)" "42704"
 check "premise: the rejected probe GUC qual_skipvec_min_skip_pct was NOT shipped" \
-	"$(q "SHOW pgcolumnar.qual_skipvec_min_skip_pct;")" ""
+	"$(show_state pgcolumnar.qual_skipvec_min_skip_pct)" "42704"
 check "premise: one row group, so every count below is one group's" \
 	"$(q "SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('nw');")" \
 	"1"

test/native_decode_gating.sh

@@ -172,6 +172,20 @@ check "the zero-match query removes every row by filter, counted once" \
 check "and skips every vector" \
 	"$(counter_in "$PLAN_NOMATCH" 'Columnar Vectors Skipped')" "$VECTORS"
 
+# A SURVIVING VECTOR WITH REJECTS IN IT (#1341). Every row of the needle's
+# surviving vector matches, so the producer, re-testing that vector to find its
+# rows, finds nothing to reject: a producer that counted those rejections a second
+# time read exactly right above. Measured, with the producer's non-counting filter
+# disabled, this file stayed 26/26. Half of vector 0 passes this qual (p1 % 2 = 0 is
+# an expression, so it is no skip predicate either), so its 512 rejections are
+# counted twice by that mistake.
+HALFQ="SELECT * FROM n WHERE tag LIKE '%needle%' AND p1 % 2 = 0"
+PLAN_HALF="$(explain_of "$HALFQ")"
+check "premise: the half-needle survives in one vector, half of its rows" \
+	"$(q "SELECT count(*) || '|' || count(DISTINCT (p1 - 1) / 1024) FROM n WHERE tag LIKE '%needle%' AND p1 % 2 = 0;")" "512|1"
+check "rejections inside a surviving vector are counted once" \
+	"$(counter_in "$PLAN_HALF" 'Rows Removed by Filter')" "$(( ROWS - 512 ))"
+
 # ---- correctness -----------------------------------------------------------
 #
 # The mask must never drop a row. Without these, an implementation that marks
@@ -205,13 +219,18 @@ MGEN="SELECT 'x' || CASE WHEN g % 4096 = 7 THEN 'needle' ELSE 'miss' END || g::t
 psql_run "INSERT INTO hm $MGEN;"
 psql_run "INSERT INTO m  $MGEN;"
 psql_run "DELETE FROM hm WHERE p1 % 17 = 0;"
-psql_run "DELETE FROM m  WHERE p1 % 17 = 0;"
+# The fixture's DELETE must not depend on the gating under test: its WHERE is an
+# unprunable qual too, so it is run with the width gate out of reach (#1341).
+psql_run "SET pgcolumnar.qual_skipvec_min_payload_cols = 100000; DELETE FROM m  WHERE p1 % 17 = 0;"
 
 check "premise: the columnar table spans more than one row group" \
 	"$(q "SELECT count(*) > 1 FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('m');")" \
 	"t"
+# count(*) rather than a count WHERE p1 % 17 = 0: that WHERE is an unprunable qual
+# too, so it ran through the very gating under test, and a gating mutant broke this
+# premise rather than the arms below (#1341).
 check "premise: rows really were deleted (so the mask is non-NULL)" \
-	"$(q "SELECT count(*) FROM m WHERE p1 % 17 = 0;")" "0"
+	"$(q "SELECT count(*) FROM m;")" "$(( MROWS - MROWS / 17 ))"
 
 check "deletes + multiple groups + unprunable qual: columnar matches the heap" \
 	"$(pgc_set_hash "SELECT * FROM m  WHERE tag LIKE '%needle%'")" \
@@ -220,4 +239,13 @@ check "and the full scan across every group's mask matches too" \
 	"$(pgc_set_hash "SELECT * FROM m")" \
 	"$(pgc_set_hash "SELECT * FROM hm")"
 
+# AND THE COUNTER SKIPS DELETED ROWS (#1341). The evaluator must not count a deleted
+# row as filtered: it is invisible. Only the single-group table above had its
+# counters checked, and it has no deletes, so with the evaluator's deleted-row test
+# removed this file stayed 26/26.
+PLAN_M="$(explain_of "SELECT * FROM m WHERE tag LIKE '%needle%'")"
+check "rows removed by filter across deleted rows count only live rows" \
+	"$(counter_in "$PLAN_M" 'Rows Removed by Filter')" \
+	"$(q "SELECT count(*) FROM hm WHERE tag NOT LIKE '%needle%';")"
+
 pgc_summary

test/native_dict_underfill.sh

@@ -27,23 +27,31 @@ echo "-- v-column vector0 encoding type byte (DICT=6): $ENC"
 check "the text column is DICT-encoded (precondition)" "$ENC" "6"
 
 # read old rawLen (uint32 LE at descriptor offset 11), inflate by 64, write back
-q "UPDATE pgcolumnar.column_chunk c SET encoding_descriptor =
+crashes() {
+	# grep -c prints 0 itself when nothing matches, and then exits 1
+	pgc_pg "grep -c 'terminated by signal' '$PGC_LOGFILE'" 2>/dev/null || true
+}
+crashes0="$(crashes)"
+q "WITH u AS (UPDATE pgcolumnar.column_chunk c SET encoding_descriptor =
      set_byte(set_byte(set_byte(set_byte(encoding_descriptor,
        11, ( (get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64 )       & 255),
        12, (((get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64) >> 8) & 255),
        13, (((get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64) >>16) & 255),
        14, (((get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64) >>24) & 255)
-     WHERE c.storage_id=$SID AND c.column_index=1" >/dev/null
+     WHERE c.storage_id=$SID AND c.column_index=1 RETURNING 1) SELECT count(*) FROM u" > "$PGC_WORKDIR/uf.n"
+check "premise: the inflation rewrote the DICT chunk's descriptor" "$(cat "$PGC_WORKDIR/uf.n")" "1"
 
 env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" \
-    -c "SELECT v FROM dt" >/dev/null 2>"$PGC_WORKDIR/uf.err"
+    -v VERBOSITY=verbose -c "SELECT v FROM dt" >/dev/null 2>"$PGC_WORKDIR/uf.err"
 echo "-- select stderr:"; sed 's/^/     /' "$PGC_WORKDIR/uf.err" | head -3
-rejected=no
-grep -qiE 'does not match|corrupt|invalid|out of' "$PGC_WORKDIR/uf.err" && rejected=yes
-crash=no
-grep -qiE 'terminated by signal|segmentation|server closed' "$PGC_WORKDIR/uf.err" && crash=yes
-grep -qiE 'terminated by signal|segmentation' "$PGC_LOGFILE" 2>/dev/null && crash=yes
-echo "-- rejected=$rejected crash=$crash"
-check "under-filled DICT chunk is rejected with a clean error" "$rejected" "yes"
-check "backend survives the under-filled DICT chunk" "$(q 'SELECT 1')" "1"
+# A CLEAN ERROR IS XX001, AND A CRASH IS READ FROM THE SERVER LOG (#1341). The
+# rejection used to match 'does not match|corrupt|invalid|out of' in any message,
+# which "out of memory" satisfies too; and survival was a fresh SELECT 1, which a
+# crashed backend passes once crash recovery has finished. The crash flag below was
+# computed and never checked.
+rejected="$(sed -nE 's/^ERROR:  ([0-9A-Z]{5}): .*/\1/p' "$PGC_WORKDIR/uf.err" | head -1 | grep . || echo "no SQLSTATE ($(head -1 "$PGC_WORKDIR/uf.err"))")"
+echo "-- rejected=$rejected crashes=$(( $(crashes) - crashes0 ))"
+check "under-filled DICT chunk is rejected with a clean error" "$rejected" "XX001"
+check "backend survives the under-filled DICT chunk" \
+	"$(q 'SELECT 1')|$(( $(crashes) - crashes0 )) backend crashes" "1|0 backend crashes"
 pgc_summary