test: native_dml gets its pytest twin, and its min/max arm can see a zone-map answer that ignores deletes (#1341)

CHANGELOG.md

@@ -400,6 +400,18 @@ true until the next version shipped.
   Reported by @OffgridwithJD.
 
 ### Fixed
+- **In the upgrade window, a refusal for a missing catalog table names `ALTER
+  EXTENSION UPDATE`.** After the new library is installed and the server restarted,
+  but before `ALTER EXTENSION pgcolumnar UPDATE`, `DROP TABLE`, `TRUNCATE` and
+  `pgcolumnar.vacuum()` on a columnar table fail, because the 1.0-alpha5 library
+  needs `pgcolumnar.retired_group` and a 1.0-alpha4 catalog lacks it. They change
+  nothing. The error used to name only the missing table, and it now carries a hint:
+  `The pgcolumnar catalog in this database may be older than the installed library.
+  Run ALTER EXTENSION pgcolumnar UPDATE in this database.` Reads and writes work
+  throughout, and after the update every one of these succeeds on the same rows
+  (#1341 item 66). `test/native_upgrade_converge.sh` and
+  `test_native_upgrade_converge.py`.
+
 
 - **A join whose runtime-filter build is empty no longer reads the fact table.** When
   a large dimension filtered to no row was joined to a columnar table, the hash join

docs/installation.md

@@ -152,6 +152,18 @@ The fix is step 3. Run `ALTER EXTENSION pgcolumnar UPDATE;` in that database and
 the error goes away. **Your data is not affected.** The tables are intact and no
 conversion happens. Only the catalog entry is stale.
 
+Between releases that add a catalog table, the new library can also need a table
+the old catalog does not have. Upgrading from `1.0-alpha4` to `1.0-alpha5`,
+`DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()` on a columnar table fail until
+step 3, and change nothing; reads and writes keep working:
+
+```
+ERROR:  columnar metadata table "pgcolumnar.retired_group" does not exist
+HINT:  The pgcolumnar catalog in this database may be older than the installed library. Run ALTER EXTENSION pgcolumnar UPDATE in this database.
+```
+
+The fix is the same: step 3.
+
 Do not use `DROP EXTENSION` instead. A plain `DROP EXTENSION pgcolumnar` fails
 while columnar tables exist, because they depend on the access method. The
 form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every

src/columnar_metadata.c

@@ -239,11 +239,22 @@ open_columnar_table(const char *name, LOCKMODE lockmode)
 	nspOid = pgcolumnar_schema_oid();
 	relOid = get_relname_relid(name, nspOid);
 
+	/*
+	 * NAME THE REMEDY (#1341 item 66). The usual way here is the upgrade window:
+	 * the new library is installed and the server restarted (docs/installation.md
+	 * steps 1 and 2), but this database's catalog is still the previous release's
+	 * until ALTER EXTENSION UPDATE (step 3). A catalog table this library added,
+	 * such as retired_group in 1.0-alpha5, is then missing, and DROP TABLE,
+	 * TRUNCATE and pgcolumnar.vacuum() stop here. The error alone named only the
+	 * table, which says neither why nor what to do.
+	 */
 	if (!OidIsValid(relOid))
 		ereport(ERROR,
 				(errcode(ERRCODE_UNDEFINED_TABLE),
 				 errmsg("columnar metadata table \"%s.%s\" does not exist",
-						COLUMNAR_SCHEMA_NAME, name)));
+						COLUMNAR_SCHEMA_NAME, name),
+				 errhint("The pgcolumnar catalog in this database may be older than the installed library. "
+						 "Run ALTER EXTENSION pgcolumnar UPDATE in this database.")));
 
 	rel = table_open(relOid, lockmode);
 

test/check_ledger.tsv

@@ -804,6 +804,34 @@ fdw_plan_privilege	fdw_plan_privilege	and a metadata_path that does not exist pl
 fdw_plan_privilege	fdw_plan_privilege	premise: the Parquet file was written	15;16;17;18;19	never	-
 fdw_plan_privilege	fdw_plan_privilege	premise: the reader is not a member of pg_read_server_files	15;16;17;18;19	never	-
 fdw_plan_privilege	fdw_plan_privilege	the unprivileged Iceberg scan is still refused (42501)	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	ALTER TABLE ADD CONSTRAINT is refused by the same rule	15;16;17;18;19	2026-10-16	columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	a columnar partition of an FK-referenced parent is refused	15;16;17;18;19	2026-10-16	columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	a columnar table may reference a heap table	15;16;17;18;19	2026-10-16	columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	a foreign key between two heap tables is unaffected	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	a foreign key referencing a columnar table is refused	15;16;17;18;19	2026-10-16	columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	a partitioned table with no foreign key still converts	17;18;19	never	-
+fk_referencing	fk_referencing	a row that violates it is still rejected	15;16;17;18;19	2026-10-16	columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	a table with no foreign key still converts	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	an FK-referenced table may still be set to a row store	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table refused whatever the target access method
+fk_referencing	fk_referencing	an unrelated ALTER TABLE on a referenced table is unaffected	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	and a partition can still be created the ordinary way	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	and its partitions inherit the columnar access method	17;18;19	never	-
+fk_referencing	fk_referencing	and rows that satisfy it are accepted	15;16;17;18;19	2026-10-16	columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	and setting the parent's access method is refused as well	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	and the error names the constraint	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	and the error says why	15;16;17;18;19	2026-10-16	columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
+fk_referencing	fk_referencing	and they still work	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	before 17, core refuses this for every partitioned table	15;16	never	-
+fk_referencing	fk_referencing	converting an FK-referenced table to columnar is refused	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	converting back to heap is unaffected	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	converting the referencing side is still allowed	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	primary key, unique, check and not-null on a columnar table are unaffected	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	so the parent did not become columnar	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	the child is still writable afterwards	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	the conversion helper is refused identically	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	the parent is still a row store	15;16;17;18;19	2026-10-16	columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
+fk_referencing	fk_referencing	the primary key is still enforced	15;16;17;18;19	never	-
+fk_referencing	fk_referencing	the refused table was not left behind	15;16;17;18;19	2026-10-16	columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
 harness_selftest	030-assertions	nothing leaked into the squatter	15;16;17;18;19	never	-
 harness_selftest	030-assertions	pgc_port_free says the squatter's port is busy	15;16;17;18;19	never	-
 harness_selftest	030-assertions	squatter survived untouched	15;16;17;18;19	never	-
@@ -2288,6 +2316,21 @@ native_delete_visibility_paths	native_delete_visibility_paths	aggregate over the
 native_delete_visibility_paths	native_delete_visibility_paths	index / bitmap scan sees the live set	15;16;17;18;19	never	-
 native_delete_visibility_paths	native_delete_visibility_paths	index-only scan sees the live set	15;16;17;18;19	never	-
 native_delete_visibility_paths	native_delete_visibility_paths	sequential scan sees the live set (row-emitting path)	15;16;17;18;19	never	-
+native_dml	native_dml	PK still enforced after DML	15;16;17;18;19	never	-
+native_dml	native_dml	count after delete	15;16;17;18;19	2026-10-16	columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows
+native_dml	native_dml	count after merge	15;16;17;18;19	2026-10-16	columnar_tableam.c: the index fetch no longer checks the delete vector;columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows
+native_dml	native_dml	count after update	15;16;17;18;19	2026-10-16	columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows
+native_dml	native_dml	deleted rows gone	15;16;17;18;19	never	-
+native_dml	native_dml	index finds live row	15;16;17;18;19	never	-
+native_dml	native_dml	index skips deleted row	15;16;17;18;19	2026-10-16	columnar_tableam.c: the index fetch no longer checks the delete vector
+native_dml	native_dml	min/max after delete	15;16;17;18;19	2026-10-16	columnar_vector.c: a row group holding deletes still answers min and max from its zone maps
+native_dml	native_dml	premise: the deletes moved the smallest and largest id and k	15;16;17;18;19	never	-
+native_dml	native_dml	premise: the lookups take the primary-key index	15;16;17;18;19	never	-
+native_dml	native_dml	rows after delete match heap	15;16;17;18;19	never	-
+native_dml	native_dml	rows after merge match heap	15;16;17;18;19	2026-10-16	columnar_tableam.c: the index fetch no longer checks the delete vector
+native_dml	native_dml	rows after update match heap	15;16;17;18;19	2026-10-16	columnar_tableam.c: the index fetch no longer checks the delete vector
+native_dml	native_dml	sum(k) after delete	15;16;17;18;19	never	-
+native_dml	native_dml	sum(k) after update	15;16;17;18;19	2026-10-16	columnar_tableam.c: the index fetch no longer checks the delete vector
 native_fetch_coalesce	native_fetch_coalesce	a wide index fetch does not pin once per column	15;16;17;18;19	2026-10-13	the coalesced fetch merging no adjacent ranges, so each column is read on its own
 native_fetch_coalesce	native_fetch_coalesce	premise: a point lookup uses the index	15;16;17;18;19	never	-
 native_fetch_coalesce	native_fetch_coalesce	premise: fetching every projected column touched a measurable number of buffers	15;16;17;18;19	never	-
@@ -3341,6 +3384,31 @@ timeseries_recipes	timeseries_recipes	the how-to gives the LOCF recipe	15;16;17;
 timeseries_recipes	timeseries_recipes	the how-to gives the gap-fill recipe	15;16;17;18;19	2026-10-13	the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found
 timeseries_recipes	timeseries_recipes	the how-to gives the interpolation recipe	15;16;17;18;19	2026-10-13	the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found
 timeseries_recipes	timeseries_recipes	the interpolation recipe fills a gap linearly, and leaves both ends NULL	15;16;17;18;19	2026-10-13	the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found;the how-to's interpolation recipe uses prev_v in place of next_v, so every gap is filled flat;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN
+truncate_cleanup	truncate_cleanup	DROP of a loaded projected table leaves nothing	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows;columnar_tableam.c: deleting a storage tree no longer reaches its projections' storages
+truncate_cleanup	truncate_cleanup	DROP of a loaded table after TRUNCATE leaves nothing	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows
+truncate_cleanup	truncate_cleanup	INSERT, TRUNCATE, INSERT in one transaction commits and keeps the new rows	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer drops the cached write state of the retired storage
+truncate_cleanup	truncate_cleanup	PREMISE the in-place transaction committed	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	PREMISE the projection has its own storage, distinct from the base	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	PREMISE the table exists after it	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	TRUNCATE leaves no projection row under a retired storage id	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows
+truncate_cleanup	truncate_cleanup	TRUNCATE leaves no rows	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	TRUNCATE returns the catalog to the empty baseline	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows
+truncate_cleanup	truncate_cleanup	a full rewrite keeps every row, and they are still readable	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	and dropping it afterwards still leaves nothing	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows
+truncate_cleanup	truncate_cleanup	and that group describes only the rows written after it	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	and the declaration survives the TRUNCATE, so the projection is rebuildable	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	and the projection reads them back	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	insert did add metadata	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	premise: five thousand rows before the rewrite	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	premise: the projected table carries metadata again before the DROP	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	premise: the table carries metadata again, so the DROP has work to do	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	premise: the table holds its first ten rows	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	ten insert-and-truncate cycles on one table leave nothing	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows
+truncate_cleanup	truncate_cleanup	the projection agrees with the base table	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	the projection did add metadata	15;16;17;18;19	never	-
+truncate_cleanup	truncate_cleanup	the projection storage holds exactly one row group after the truncate	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	the rows written after the in-place truncate are all there	15;16;17;18;19	2026-10-16	columnar_tableam.c: the in-place truncate no longer clears a projection's own storage
+truncate_cleanup	truncate_cleanup	three truncate-and-reload cycles still sit at the baseline	15;16;17;18;19	2026-10-16	columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows
 ttl_expire	ttl_expire	NO row still inside the retention was dropped (#403 item 5a)	15;16;17;18;19	never	-
 ttl_expire	ttl_expire	NO row still inside the retention was dropped, on a date column	15;16;17;18;19	never	-
 ttl_expire	ttl_expire	a compaction's retired groups are not recorded as retention	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum

test/check_ledger_budget.txt

@@ -15,7 +15,7 @@
 #
 #   Adding a check to a suite that is already covered does not move it, which is
 #   what makes it safe to bound.
-suites_not_covered 219
+suites_not_covered 216
 #
 # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
 #   one.
@@ -2943,4 +2943,51 @@ suites_not_covered 219
 #     never         2676 - 1 = 2675
 #     dated          943 + 1 = 944
 #     2675 + 944 == 3619
-checks_never_observed_red 2675
+#
+# #1341, fk_referencing seeded, with its twin (OffgridwithJD): 2675 -> 2686, and
+# suites_not_covered 219 -> 218. The suite had no rows. It enters with 28 rows
+# from five-major runs: 27 checks on PG17-19 and 26 on 15-16, where the
+# partitioned-control branch asserts core's refusal instead of a conversion. 17 are
+# dated by four mutations of the foreign-key guards: the referenced-side refusal
+# removed, the guard keyed on the referencing table, the conversion refusal
+# removed, and the conversion refused whatever the target access method (the new
+# referenced-table-to-row-store arm). Eleven arrive never: the over-rejection
+# controls that no single guard reaches (another constraint, a heap-to-heap key,
+# an unreferenced conversion, an unrelated ALTER, a conversion back to heap, the
+# referencing side, the partitioned control on either side of 17) and the primary
+# key's own arms.
+#
+#     ledger rows   3619 + 28 = 3647
+#     never         2675 + 11 = 2686
+#     dated          944 + 17 = 961
+#     2686 + 961 == 3647
+#
+# #1341, truncate_cleanup seeded, with its twin (OffgridwithJD): 2686 -> 2695, and
+# suites_not_covered 218 -> 217. The suite had no rows. It enters with all 25 checks
+# from five-major runs (25/25 on each of 15-19). 16 are dated by four mutations of
+# the truncate paths: a rewrite that no longer deletes the retired storage's rows
+# (7), nor drops the stale write state (1), an in-place truncate that skips the
+# projections' storages (8), and a storage-tree delete that skips them (1). Nine
+# arrive never: the premises that each fixture holds rows or metadata, the
+# surviving projection declaration, the empty table after TRUNCATE, and the
+# rewrite that keeps every row.
+#
+#     ledger rows   3647 + 25 = 3672
+#     never         2686 +  9 = 2695
+#     dated          961 + 16 = 977
+#     2695 + 977 == 3672
+#
+# #1341, native_dml seeded, with its twin (OffgridwithJD): 2695 -> 2702, and
+# suites_not_covered 217 -> 216. The suite had no rows. It enters with all 15 checks
+# from five-major runs (15/15 on each of 15-19). 8 are dated by three mutations: a
+# row group holding deletes still answering min and max from its zone maps (the
+# min/max arm, which could not see it before this change), count(*) from metadata
+# not subtracting deleted rows (3), and the index fetch ignoring the delete vector
+# (5). Seven arrive never: the two premises, the rows and sum after the delete, the
+# deleted rows being gone, the live row's index lookup, and the primary key arm.
+#
+#     ledger rows   3672 + 15 = 3687
+#     never         2695 +  7 = 2702
+#     dated          977 +  8 = 985
+#     2702 + 985 == 3687
+checks_never_observed_red 2702