test: native_parquet_schema gets its pytest twin, and bad magic must be refused by the magic check (#1341)

CHANGELOG.md

@@ -400,6 +400,18 @@ true until the next version shipped.
   Reported by @OffgridwithJD.
 
 ### Fixed
+- **In the upgrade window, a refusal for a missing catalog table names `ALTER
+  EXTENSION UPDATE`.** After the new library is installed and the server restarted,
+  but before `ALTER EXTENSION pgcolumnar UPDATE`, `DROP TABLE`, `TRUNCATE` and
+  `pgcolumnar.vacuum()` on a columnar table fail, because the 1.0-alpha5 library
+  needs `pgcolumnar.retired_group` and a 1.0-alpha4 catalog lacks it. They change
+  nothing. The error used to name only the missing table, and it now carries a hint:
+  `The pgcolumnar catalog in this database may be older than the installed library.
+  Run ALTER EXTENSION pgcolumnar UPDATE in this database.` Reads and writes work
+  throughout, and after the update every one of these succeeds on the same rows
+  (#1341 item 66). `test/native_upgrade_converge.sh` and
+  `test_native_upgrade_converge.py`.
+
 
 - **A join whose runtime-filter build is empty no longer reads the fact table.** When
   a large dimension filtered to no row was joined to a columnar table, the hash join

docs/installation.md

@@ -152,6 +152,18 @@ The fix is step 3. Run `ALTER EXTENSION pgcolumnar UPDATE;` in that database and
 the error goes away. **Your data is not affected.** The tables are intact and no
 conversion happens. Only the catalog entry is stale.
 
+Between releases that add a catalog table, the new library can also need a table
+the old catalog does not have. Upgrading from `1.0-alpha4` to `1.0-alpha5`,
+`DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()` on a columnar table fail until
+step 3, and change nothing; reads and writes keep working:
+
+```
+ERROR:  columnar metadata table "pgcolumnar.retired_group" does not exist
+HINT:  The pgcolumnar catalog in this database may be older than the installed library. Run ALTER EXTENSION pgcolumnar UPDATE in this database.
+```
+
+The fix is the same: step 3.
+
 Do not use `DROP EXTENSION` instead. A plain `DROP EXTENSION pgcolumnar` fails
 while columnar tables exist, because they depend on the access method. The
 form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every

src/columnar_metadata.c

@@ -239,11 +239,22 @@ open_columnar_table(const char *name, LOCKMODE lockmode)
 	nspOid = pgcolumnar_schema_oid();
 	relOid = get_relname_relid(name, nspOid);
 
+	/*
+	 * NAME THE REMEDY (#1341 item 66). The usual way here is the upgrade window:
+	 * the new library is installed and the server restarted (docs/installation.md
+	 * steps 1 and 2), but this database's catalog is still the previous release's
+	 * until ALTER EXTENSION UPDATE (step 3). A catalog table this library added,
+	 * such as retired_group in 1.0-alpha5, is then missing, and DROP TABLE,
+	 * TRUNCATE and pgcolumnar.vacuum() stop here. The error alone named only the
+	 * table, which says neither why nor what to do.
+	 */
 	if (!OidIsValid(relOid))
 		ereport(ERROR,
 				(errcode(ERRCODE_UNDEFINED_TABLE),
 				 errmsg("columnar metadata table \"%s.%s\" does not exist",
-						COLUMNAR_SCHEMA_NAME, name)));
+						COLUMNAR_SCHEMA_NAME, name),
+				 errhint("The pgcolumnar catalog in this database may be older than the installed library. "
+						 "Run ALTER EXTENSION pgcolumnar UPDATE in this database.")));
 
 	rel = table_open(relOid, lockmode);
 

test/alter_am_cleanup.sh

@@ -55,7 +55,13 @@ psql_run "DROP TABLE aac_t;"
 check "DROP after SET ACCESS METHOD heap leaves no options or declarations" \
 	"$(snapshot)" "$base"
 
+# Each cleanup arm below compares with the snapshot taken just before ITS table
+# was made, not with the suite's starting one (#1341). Against $base, one leak
+# above reddened every arm after it: measured, with the cleanup gated on the
+# columnar AM, the never-columnar heap arm went red on aac_t's leftover rows.
+#
 # Round-trip: convert away and back, then DROP a still-columnar table.
+pre_rt="$(snapshot)"
 psql_run "CREATE TABLE aac_rt (id int) USING pgcolumnar;"
 psql_run "SELECT pgcolumnar.set_options('aac_rt', stripe_row_limit => 8000);"
 psql_run "ALTER TABLE aac_rt SET ACCESS METHOD heap;"
@@ -64,15 +70,16 @@ check "options survive a heap round-trip" \
 	"$(q "SELECT stripe_row_limit FROM pgcolumnar.options
 	      WHERE regclass = 'aac_rt'::regclass")" "8000"
 psql_run "DROP TABLE aac_rt;"
-check "a round-tripped table still cleans up on DROP" "$(snapshot)" "$base"
+check "a round-tripped table still cleans up on DROP" "$(snapshot)" "$pre_rt"
 
 # Control: a never-columnar heap must not be charged for this either.
+pre_h="$(snapshot)"
 psql_run "CREATE TABLE aac_h (id int);"
 psql_run "DROP TABLE aac_h;"
 # kept alive for the VACUUM FULL control further down
 psql_run "CREATE TABLE aac_h_ctl (id int);"
 check "dropping a heap table that was never columnar leaves the catalogs alone" \
-	"$(snapshot)" "$base"
+	"$(snapshot)" "$pre_h"
 
 # ---- the hook is armed in EVERY database, not only where the extension is ----
 #
@@ -136,8 +143,12 @@ check "ALTER COLUMN TYPE succeeds without the extension" \
 	"$(aac_dbrun aac_nocx 'ALTER TABLE rw ALTER COLUMN t TYPE varchar(64);')" "rc=0"
 check "CREATE MATERIALIZED VIEW succeeds without the extension" \
 	"$(aac_dbrun aac_nocx 'CREATE MATERIALIZED VIEW rwm AS SELECT * FROM rw;')" "rc=0"
+# REFRESH gets a matview of its own, made WITH NO DATA, so it does not inherit the
+# CREATE arm's failure: measured, it went red as 'relation "rwm" does not exist'.
+check "premise: a matview to refresh exists, made WITH NO DATA" \
+	"$(aac_dbrun aac_nocx 'CREATE MATERIALIZED VIEW rwr AS SELECT * FROM rw WITH NO DATA;')" "rc=0"
 check "and REFRESH MATERIALIZED VIEW does too, without the extension" \
-	"$(aac_dbrun aac_nocx 'REFRESH MATERIALIZED VIEW rwm;')" "rc=0"
+	"$(aac_dbrun aac_nocx 'REFRESH MATERIALIZED VIEW rwr;')" "rc=0"
 check "TRUNCATE succeeds without the extension" \
 	"$(aac_dbrun aac_nocx 'TRUNCATE rw;')" "rc=0"
 
@@ -153,14 +164,15 @@ check "control: VACUUM FULL still succeeds where the extension IS installed" \
 # different set: DROP EXTENSION drops those as ordinary relations, and opening
 # options while options is itself being dropped would fail.
 
+pre_in="$(snapshot)"
 psql_run "CREATE TABLE pgcolumnar.aac_inschema (id int) USING pgcolumnar;"
 psql_run "SELECT pgcolumnar.set_options('pgcolumnar.aac_inschema', stripe_row_limit => 7000);"
 check "premise: a table in the extension's schema recorded its options" \
 	"$(q "SELECT count(*) FROM pgcolumnar.options
 	      WHERE regclass = 'pgcolumnar.aac_inschema'::regclass")" "1"
 psql_run "DROP TABLE pgcolumnar.aac_inschema;"
 check "dropping a table that lives in the extension's schema still cleans up" \
-	"$(snapshot)" "$base"
+	"$(snapshot)" "$pre_in"
 
 # ---- a relation that predates CREATE EXTENSION is not an extension member ----
 #
@@ -226,7 +238,9 @@ check "DROP EXTENSION succeeds with those catalogs populated" \
 
 check "premise: DROP EXTENSION left the schema behind" \
 	"$(aac_dbq aac_dropx "SELECT count(*) FROM pg_namespace WHERE nspname='pgcolumnar'")" "1"
-check "premise: and took pgcolumnar.options with it" \
+# An arm, not a premise (#1236): a hook that opened options while DROP EXTENSION
+# was dropping it aborted the DROP, and this read the options table still there.
+check "and took pgcolumnar.options with it" \
 	"$(aac_dbq aac_dropx "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
 	                      WHERE n.nspname = 'pgcolumnar' AND c.relname = 'options'")" "0"
 
@@ -248,8 +262,10 @@ check "ALTER COLUMN TYPE succeeds after DROP EXTENSION" \
 	"$(aac_dbrun aac_dropx 'ALTER TABLE dxr ALTER COLUMN t TYPE varchar(64);')" "rc=0"
 check "CREATE MATERIALIZED VIEW succeeds after DROP EXTENSION" \
 	"$(aac_dbrun aac_dropx 'CREATE MATERIALIZED VIEW dxm AS SELECT * FROM dxr;')" "rc=0"
+check "premise: a matview to refresh exists after DROP EXTENSION, made WITH NO DATA" \
+	"$(aac_dbrun aac_dropx 'CREATE MATERIALIZED VIEW dxr_m AS SELECT * FROM dxr WITH NO DATA;')" "rc=0"
 check "and REFRESH MATERIALIZED VIEW does too, after DROP EXTENSION" \
-	"$(aac_dbrun aac_dropx 'REFRESH MATERIALIZED VIEW dxm;')" "rc=0"
+	"$(aac_dbrun aac_dropx 'REFRESH MATERIALIZED VIEW dxr_m;')" "rc=0"
 check "TRUNCATE succeeds after DROP EXTENSION" \
 	"$(aac_dbrun aac_dropx 'TRUNCATE dxr;')" "rc=0"