Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -400,6 +400,18 @@ true until the next version shipped.
Reported by @OffgridwithJD.

### Fixed
- **In the upgrade window, a refusal for a missing catalog table names `ALTER
EXTENSION UPDATE`.** After the new library is installed and the server restarted,
but before `ALTER EXTENSION pgcolumnar UPDATE`, `DROP TABLE`, `TRUNCATE` and
`pgcolumnar.vacuum()` on a columnar table fail, because the 1.0-alpha5 library
needs `pgcolumnar.retired_group` and a 1.0-alpha4 catalog lacks it. They change
nothing. The error used to name only the missing table, and it now carries a hint:
`The pgcolumnar catalog in this database may be older than the installed library.
Run ALTER EXTENSION pgcolumnar UPDATE in this database.` Reads and writes work
throughout, and after the update every one of these succeeds on the same rows
(#1341 item 66). `test/native_upgrade_converge.sh` and
`test_native_upgrade_converge.py`.


- **A join whose runtime-filter build is empty no longer reads the fact table.** When
a large dimension filtered to no row was joined to a columnar table, the hash join
Expand Down
12 changes: 12 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,18 @@ The fix is step 3. Run `ALTER EXTENSION pgcolumnar UPDATE;` in that database and
the error goes away. **Your data is not affected.** The tables are intact and no
conversion happens. Only the catalog entry is stale.

Between releases that add a catalog table, the new library can also need a table
the old catalog does not have. Upgrading from `1.0-alpha4` to `1.0-alpha5`,
`DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()` on a columnar table fail until
step 3, and change nothing; reads and writes keep working:

```
ERROR: columnar metadata table "pgcolumnar.retired_group" does not exist
HINT: The pgcolumnar catalog in this database may be older than the installed library. Run ALTER EXTENSION pgcolumnar UPDATE in this database.
```

The fix is the same: step 3.

Do not use `DROP EXTENSION` instead. A plain `DROP EXTENSION pgcolumnar` fails
while columnar tables exist, because they depend on the access method. The
form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every
Expand Down
13 changes: 12 additions & 1 deletion src/columnar_metadata.c
Original file line number Diff line number Diff line change
Expand Up @@ -239,11 +239,22 @@ open_columnar_table(const char *name, LOCKMODE lockmode)
nspOid = pgcolumnar_schema_oid();
relOid = get_relname_relid(name, nspOid);

/*
* NAME THE REMEDY (#1341 item 66). The usual way here is the upgrade window:
* the new library is installed and the server restarted (docs/installation.md
* steps 1 and 2), but this database's catalog is still the previous release's
* until ALTER EXTENSION UPDATE (step 3). A catalog table this library added,
* such as retired_group in 1.0-alpha5, is then missing, and DROP TABLE,
* TRUNCATE and pgcolumnar.vacuum() stop here. The error alone named only the
* table, which says neither why nor what to do.
*/
if (!OidIsValid(relOid))
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_TABLE),
errmsg("columnar metadata table \"%s.%s\" does not exist",
COLUMNAR_SCHEMA_NAME, name)));
COLUMNAR_SCHEMA_NAME, name),
errhint("The pgcolumnar catalog in this database may be older than the installed library. "
"Run ALTER EXTENSION pgcolumnar UPDATE in this database.")));

rel = table_open(relOid, lockmode);

Expand Down
28 changes: 28 additions & 0 deletions test/check_ledger.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -804,6 +804,34 @@ fdw_plan_privilege fdw_plan_privilege and a metadata_path that does not exist pl
fdw_plan_privilege fdw_plan_privilege premise: the Parquet file was written 15;16;17;18;19 never -
fdw_plan_privilege fdw_plan_privilege premise: the reader is not a member of pg_read_server_files 15;16;17;18;19 never -
fdw_plan_privilege fdw_plan_privilege the unprivileged Iceberg scan is still refused (42501) 15;16;17;18;19 never -
fk_referencing fk_referencing ALTER TABLE ADD CONSTRAINT is refused by the same rule 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing a columnar partition of an FK-referenced parent is refused 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing a columnar table may reference a heap table 15;16;17;18;19 2026-10-16 columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing a foreign key between two heap tables is unaffected 15;16;17;18;19 never -
fk_referencing fk_referencing a foreign key referencing a columnar table is refused 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing a partitioned table with no foreign key still converts 17;18;19 never -
fk_referencing fk_referencing a row that violates it is still rejected 15;16;17;18;19 2026-10-16 columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing a table with no foreign key still converts 15;16;17;18;19 never -
fk_referencing fk_referencing an FK-referenced table may still be set to a row store 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table refused whatever the target access method
fk_referencing fk_referencing an unrelated ALTER TABLE on a referenced table is unaffected 15;16;17;18;19 never -
fk_referencing fk_referencing and a partition can still be created the ordinary way 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing and its partitions inherit the columnar access method 17;18;19 never -
fk_referencing fk_referencing and rows that satisfy it are accepted 15;16;17;18;19 2026-10-16 columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing and setting the parent's access method is refused as well 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing and the error names the constraint 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing and the error says why 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
fk_referencing fk_referencing and they still work 15;16;17;18;19 never -
fk_referencing fk_referencing before 17, core refuses this for every partitioned table 15;16 never -
fk_referencing fk_referencing converting an FK-referenced table to columnar is refused 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing converting back to heap is unaffected 15;16;17;18;19 never -
fk_referencing fk_referencing converting the referencing side is still allowed 15;16;17;18;19 never -
fk_referencing fk_referencing primary key, unique, check and not-null on a columnar table are unaffected 15;16;17;18;19 never -
fk_referencing fk_referencing so the parent did not become columnar 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing the child is still writable afterwards 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing the conversion helper is refused identically 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing the parent is still a row store 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused
fk_referencing fk_referencing the primary key is still enforced 15;16;17;18;19 never -
fk_referencing fk_referencing the refused table was not left behind 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one
harness_selftest 030-assertions nothing leaked into the squatter 15;16;17;18;19 never -
harness_selftest 030-assertions pgc_port_free says the squatter's port is busy 15;16;17;18;19 never -
harness_selftest 030-assertions squatter survived untouched 15;16;17;18;19 never -
Expand Down
22 changes: 20 additions & 2 deletions test/check_ledger_budget.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
#
# Adding a check to a suite that is already covered does not move it, which is
# what makes it safe to bound.
suites_not_covered 219
suites_not_covered 218
#
# checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
# one.
Expand Down Expand Up @@ -2943,4 +2943,22 @@ suites_not_covered 219
# never 2676 - 1 = 2675
# dated 943 + 1 = 944
# 2675 + 944 == 3619
checks_never_observed_red 2675
#
# #1341, fk_referencing seeded, with its twin (OffgridwithJD): 2675 -> 2686, and
# suites_not_covered 219 -> 218. The suite had no rows. It enters with 28 rows
# from five-major runs: 27 checks on PG17-19 and 26 on 15-16, where the
# partitioned-control branch asserts core's refusal instead of a conversion. 17 are
# dated by four mutations of the foreign-key guards: the referenced-side refusal
# removed, the guard keyed on the referencing table, the conversion refusal
# removed, and the conversion refused whatever the target access method (the new
# referenced-table-to-row-store arm). Eleven arrive never: the over-rejection
# controls that no single guard reaches (another constraint, a heap-to-heap key,
# an unreferenced conversion, an unrelated ALTER, a conversion back to heap, the
# referencing side, the partitioned control on either side of 17) and the primary
# key's own arms.
#
# ledger rows 3619 + 28 = 3647
# never 2675 + 11 = 2686
# dated 944 + 17 = 961
# 2686 + 961 == 3647
checks_never_observed_red 2686
10 changes: 10 additions & 0 deletions test/fk_referencing.sh
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,16 @@ check "converting back to heap is unaffected" \
"$(case "$back" in *ERROR*) echo "no ($back)" ;; *) echo yes ;; esac)" \
"yes"

# AND ON THE REFERENCED TABLE ITSELF (#1341). The control above converts a table
# with no foreign key, so it cannot see a rule that refuses every SET ACCESS METHOD
# on a referenced table whatever the target: with the guard's columnar-target test
# removed, this file stayed 26/26. Only a columnar target needs the row lock the
# constraint cannot get.
refheap="$(psql_run "ALTER TABLE fk_conv_parent SET ACCESS METHOD heap;" 2>&1 || true)"
check "an FK-referenced table may still be set to a row store" \
"$(case "$refheap" in *ERROR*) echo "no ($refheap)" ;; *) echo yes ;; esac)" \
"yes"

# --- 7. a partitioned parent is refused too (#201) -----------------------------

# A partitioned table has no storage, so setting its access method breaks nothing
Expand Down
49 changes: 49 additions & 0 deletions test/native_upgrade_converge.sh
Original file line number Diff line number Diff line change
Expand Up @@ -310,4 +310,53 @@ done
rm -f "$REF"
P -d postgres -c "DROP DATABASE IF EXISTS conv_fresh;" >/dev/null

# ---- the window: this library over the previous release's catalog (#1341 item 66) ----
#
# docs/installation.md steps 1 and 2 put the new library in place; step 3,
# ALTER EXTENSION UPDATE, brings the catalog up to it. Between them a database
# runs this library over its previous release's catalog. The library deletes
# from retired_group unconditionally (columnar_metadata.c, PgColumnarDeleteMetadata),
# which that catalog does not have, so DROP TABLE, TRUNCATE and pgcolumnar.vacuum()
# fail there: measured, with 42P01 'columnar metadata table
# "pgcolumnar.retired_group" does not exist', which names neither the cause nor step 3.
#
# The arms hold what the window must keep: reads and writes work, a refused
# operation changes nothing and names the remedy, and step 3 makes every one of
# them work on the same rows. PREV is the newest fixture, so these follow the
# release they describe rather than naming it.
PREV="${_NUC_FIXTURES[${#_NUC_FIXTURES[@]}-1]}"
WIN=conv_window
P -d postgres -c "DROP DATABASE IF EXISTS $WIN;" >/dev/null
P -d postgres -c "CREATE DATABASE $WIN;" >/dev/null
P -d "$WIN" -c "CREATE EXTENSION pgcolumnar VERSION '$PREV';" >/dev/null
check "premise: the window database runs this library over the previous release's catalog" \
"$(P -d "$WIN" -c "SELECT extversion FROM pg_extension WHERE extname='pgcolumnar';")" "$PREV"
check "premise: that catalog has no retired_group, which this library deletes from" \
"$(P -d "$WIN" -c "SELECT count(*) FROM pg_class WHERE relname = 'retired_group' AND relnamespace = 'pgcolumnar'::regnamespace;")" "0"
P -d "$WIN" -c "CREATE TABLE w (a int, b text) USING pgcolumnar;" >/dev/null
P -d "$WIN" -c "INSERT INTO w SELECT g, 'x' || g FROM generate_series(1, 20000) g;" >/dev/null
P -d "$WIN" -c "DELETE FROM w WHERE a % 3 = 0;" >/dev/null
win_rows() { P -d "$WIN" -c "SELECT count(*) || '|' || sum(a) FROM w;"; }
check "in the window, a columnar table is written, deleted from and read" "$(win_rows)" "13334|133346667"
win_refusal() { # win_refusal SQL -> SQLSTATE|whether the HINT names step 3, or what happened instead
local out
out="$(P -d "$WIN" -v VERBOSITY=verbose -c "$1" 2>&1)" && { echo accepted; return; }
echo "$(sed -nE 's/^ERROR: ([0-9A-Z]{5}): .*/\1/p' <<<"$out" | head -1)|$(grep -q '^HINT: .*ALTER EXTENSION pgcolumnar UPDATE' <<<"$out" && echo names-the-update || echo no-hint)"
}
check "in the window, DROP TABLE is refused and names ALTER EXTENSION UPDATE" \
"$(win_refusal "DROP TABLE w;")" "42P01|names-the-update"
check "in the window, TRUNCATE is refused and names ALTER EXTENSION UPDATE" \
"$(win_refusal "TRUNCATE w;")" "42P01|names-the-update"
check "in the window, pgcolumnar.vacuum() is refused and names ALTER EXTENSION UPDATE" \
"$(win_refusal "SELECT pgcolumnar.vacuum('w');")" "42P01|names-the-update"
check "in the window, the refused operations changed nothing" "$(win_rows)" "13334|133346667"

P -d "$WIN" -c "ALTER EXTENSION pgcolumnar UPDATE;" >/dev/null
check "after ALTER EXTENSION UPDATE, the same rows are there" "$(win_rows)" "13334|133346667"
check "after ALTER EXTENSION UPDATE, pgcolumnar.vacuum() succeeds and keeps them" \
"$(P -d "$WIN" -c "SELECT pgcolumnar.vacuum('w');" >/dev/null 2>&1 && win_rows || echo refused)" "13334|133346667"
check "after ALTER EXTENSION UPDATE, TRUNCATE and DROP TABLE succeed" \
"$(P -d "$WIN" -c "TRUNCATE w;" >/dev/null 2>&1 && P -d "$WIN" -c "DROP TABLE w;" >/dev/null 2>&1 && echo succeeded || echo refused)" "succeeded"
P -d postgres -c "DROP DATABASE IF EXISTS $WIN;" >/dev/null

pgc_summary
26 changes: 26 additions & 0 deletions test/pytest/TESTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ behaviour, the source of that number is named.
- [140. test_parquet_import.py: a partitioned columnar parent is refused by import_parquet](#140-test_parquet_importpy-a-partitioned-columnar-parent-is-refused-by-import_parquet)
- [141. test_phase5.py: set_options and reset_options agree](#141-test_phase5py-set_options-and-reset_options-agree)
- [142. test_drop_cleanup.py: a columnar matview's options row goes with its DROP](#142-test_drop_cleanuppy-a-columnar-matviews-options-row-goes-with-its-drop)
- [143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE](#143-test_native_upgrade_convergepy-the-upgrade-window-before-alter-extension-update)
- [144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen](#144-test_fk_referencingpy-a-foreign-key-referencing-a-columnar-table-is-refused-where-it-is-chosen)

## 1. How to read a test in here

Expand Down Expand Up @@ -7377,3 +7379,27 @@ Part of the pytest twin of `test/drop_cleanup.sh` (#1265, ported for #1341): its
| `test_a_columnar_matview_takes_its_options_row_with_it` | a planted options row is gone after DROP MATERIALIZED VIEW |
| `test_refresh_keeps_a_matviews_options_row` | a REFRESH that rewrote the matview leaves its set_options row in place |
| `test_an_unrelated_options_row_survives` | another table's row survives a matview's DROP, and an ordinary columnar table still takes its own |

## 143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE

Part of the pytest twin of `test/native_upgrade_converge.sh` (#1341 item 66): its upgrade-window arms only, by the shell's names, and the pair is declared INCOMPLETE. Between installing a new library and running `ALTER EXTENSION pgcolumnar UPDATE`, a database runs the new library over its previous release's catalog. Reads and writes must work there. `DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()`, which need a catalog table the old one lacks, must be refused with 42P01, change nothing, and carry a hint naming the update; after the update all three succeed on the same rows. The previous release is the version the shipped upgrade script to the current `default_version` starts from; its base script is staged from `test/fixtures` for the test and removed after.

### Every test

| test | what it holds |
| --- | --- |
| `test_the_upgrade_window` | at the previous catalog: rows written, deleted and read; the three operations refused with the hint and nothing changed; after the update the rows are intact and all three succeed |

## 144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen

The pytest twin of `test/fk_referencing.sh` (ported for #1341), every check by the shell's name. Enforcing a foreign key locks the referenced row, and columnar tables cannot lock rows, so a constraint referencing one would make every insert into the referencing table fail. It is refused at DDL by either route into that state: creating the constraint against a columnar table, or converting a referenced table (or partitioned parent) to columnar. Only the referenced side is refused, and the controls hold everything else steady, including converting the referenced table itself to a row store.

### Every test

| test | what it holds |
| --- | --- |
| `test_the_referenced_side_is_refused_at_ddl` | CREATE TABLE and ALTER TABLE ADD CONSTRAINT referencing a columnar table are refused, saying why, and nothing is left behind |
| `test_the_referencing_side_still_works` | a columnar table referencing a heap table is created, accepts satisfying rows and rejects a violating one |
| `test_constraints_that_need_no_row_lock_are_unaffected` | primary key, unique, check and not-null on a columnar table, and a heap-to-heap foreign key, are unaffected |
| `test_converting_a_referenced_table_is_refused_and_nothing_else_is` | SET ACCESS METHOD pgcolumnar on a referenced table and the helper are refused, the table stays heap and its child writable; an unreferenced table, the referencing side, an unrelated ALTER, a conversion back to heap and the referenced table to a row store all still work |
| `test_a_partitioned_referenced_parent` | a columnar partition and setting the parent's access method are refused, the parent does not become columnar, an ordinary partition still works; an unreferenced partitioned table converts from PG17, and before 17 core refuses it |
6 changes: 5 additions & 1 deletion test/pytest/expected_tests.txt
Original file line number Diff line number Diff line change
Expand Up @@ -747,4 +747,8 @@ guard_tests 438
# seven tests, by collection over the cluster files on this tree.
# cluster_tests 653 -> 660 for #1341 item 29's partial twins: test_inheritance.py (2),
# test_parquet_import.py (1), test_phase5.py (1) and test_drop_cleanup.py (3), by collection.
cluster_tests 660
# cluster_tests 660 -> 661 for #1341 item 66's test_native_upgrade_converge.py, one test,
# by collection over the cluster files on this tree.
# cluster_tests 661 -> 666 for #1341's test_fk_referencing.py, the twin of fk_referencing.sh:
# five tests, by collection over the cluster files on this tree.
cluster_tests 666
Loading
Loading