From 7ac79e58c87aea1a4bd3e6b06207e1a6daf27bab Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:05 +0000 Subject: [PATCH 01/36] fix: in the upgrade window, a refusal for a missing catalog table names ALTER EXTENSION UPDATE (#1341 item 66) Between installing a new library and running ALTER EXTENSION pgcolumnar UPDATE, a database runs the 1.0-alpha5 library over its 1.0-alpha4 catalog. The library deletes from pgcolumnar.retired_group, which that catalog lacks, so DROP TABLE, TRUNCATE and pgcolumnar.vacuum() on a columnar table fail there with 42P01 'columnar metadata table "pgcolumnar.retired_group" does not exist', which names neither the cause nor the remedy. Measured: each refusal changes nothing, reads and writes work throughout, and after the update all three succeed on the same rows. open_columnar_table's error now carries a hint naming the update. docs/installation.md's "If you skipped step 3" shows the error and the hint. Red first, both twins, PG18: native_upgrade_converge gains the window's arms in a database at the previous release's catalog (the newest fixture), and the three refusal arms fail on main with `got [42P01|no-hint]`. test_native_upgrade_converge.py is their partial twin, declared INCOMPLETE; it stages the previous release's base script from test/fixtures as the shell does. Green on 15-19 in both twins. cluster_tests 660 -> 661 by collection; TESTS.md gets section 143. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 12 ++ docs/installation.md | 12 ++ src/columnar_metadata.c | 13 +- test/native_upgrade_converge.sh | 49 ++++++++ test/pytest/TESTS.md | 11 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 4 + test/pytest/test_native_upgrade_converge.py | 126 ++++++++++++++++++++ 8 files changed, 229 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_native_upgrade_converge.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d2b779a6d..c0dffb95f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -400,6 +400,18 @@ true until the next version shipped. Reported by @OffgridwithJD. ### Fixed +- **In the upgrade window, a refusal for a missing catalog table names `ALTER + EXTENSION UPDATE`.** After the new library is installed and the server restarted, + but before `ALTER EXTENSION pgcolumnar UPDATE`, `DROP TABLE`, `TRUNCATE` and + `pgcolumnar.vacuum()` on a columnar table fail, because the 1.0-alpha5 library + needs `pgcolumnar.retired_group` and a 1.0-alpha4 catalog lacks it. They change + nothing. The error used to name only the missing table, and it now carries a hint: + `The pgcolumnar catalog in this database may be older than the installed library. + Run ALTER EXTENSION pgcolumnar UPDATE in this database.` Reads and writes work + throughout, and after the update every one of these succeeds on the same rows + (#1341 item 66). `test/native_upgrade_converge.sh` and + `test_native_upgrade_converge.py`. + - **A join whose runtime-filter build is empty no longer reads the fact table.** When a large dimension filtered to no row was joined to a columnar table, the hash join diff --git a/docs/installation.md b/docs/installation.md index 2618c47bf..6cf867108 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -152,6 +152,18 @@ The fix is step 3. Run `ALTER EXTENSION pgcolumnar UPDATE;` in that database and the error goes away. **Your data is not affected.** The tables are intact and no conversion happens. Only the catalog entry is stale. +Between releases that add a catalog table, the new library can also need a table +the old catalog does not have. Upgrading from `1.0-alpha4` to `1.0-alpha5`, +`DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()` on a columnar table fail until +step 3, and change nothing; reads and writes keep working: + +``` +ERROR: columnar metadata table "pgcolumnar.retired_group" does not exist +HINT: The pgcolumnar catalog in this database may be older than the installed library. Run ALTER EXTENSION pgcolumnar UPDATE in this database. +``` + +The fix is the same: step 3. + Do not use `DROP EXTENSION` instead. A plain `DROP EXTENSION pgcolumnar` fails while columnar tables exist, because they depend on the access method. The form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every diff --git a/src/columnar_metadata.c b/src/columnar_metadata.c index bd0120efb..eb5bdc53f 100644 --- a/src/columnar_metadata.c +++ b/src/columnar_metadata.c @@ -239,11 +239,22 @@ open_columnar_table(const char *name, LOCKMODE lockmode) nspOid = pgcolumnar_schema_oid(); relOid = get_relname_relid(name, nspOid); + /* + * NAME THE REMEDY (#1341 item 66). The usual way here is the upgrade window: + * the new library is installed and the server restarted (docs/installation.md + * steps 1 and 2), but this database's catalog is still the previous release's + * until ALTER EXTENSION UPDATE (step 3). A catalog table this library added, + * such as retired_group in 1.0-alpha5, is then missing, and DROP TABLE, + * TRUNCATE and pgcolumnar.vacuum() stop here. The error alone named only the + * table, which says neither why nor what to do. + */ if (!OidIsValid(relOid)) ereport(ERROR, (errcode(ERRCODE_UNDEFINED_TABLE), errmsg("columnar metadata table \"%s.%s\" does not exist", - COLUMNAR_SCHEMA_NAME, name))); + COLUMNAR_SCHEMA_NAME, name), + errhint("The pgcolumnar catalog in this database may be older than the installed library. " + "Run ALTER EXTENSION pgcolumnar UPDATE in this database."))); rel = table_open(relOid, lockmode); diff --git a/test/native_upgrade_converge.sh b/test/native_upgrade_converge.sh index 0249d5655..454900137 100755 --- a/test/native_upgrade_converge.sh +++ b/test/native_upgrade_converge.sh @@ -310,4 +310,53 @@ done rm -f "$REF" P -d postgres -c "DROP DATABASE IF EXISTS conv_fresh;" >/dev/null +# ---- the window: this library over the previous release's catalog (#1341 item 66) ---- +# +# docs/installation.md steps 1 and 2 put the new library in place; step 3, +# ALTER EXTENSION UPDATE, brings the catalog up to it. Between them a database +# runs this library over its previous release's catalog. The library deletes +# from retired_group unconditionally (columnar_metadata.c, PgColumnarDeleteMetadata), +# which that catalog does not have, so DROP TABLE, TRUNCATE and pgcolumnar.vacuum() +# fail there: measured, with 42P01 'columnar metadata table +# "pgcolumnar.retired_group" does not exist', which names neither the cause nor step 3. +# +# The arms hold what the window must keep: reads and writes work, a refused +# operation changes nothing and names the remedy, and step 3 makes every one of +# them work on the same rows. PREV is the newest fixture, so these follow the +# release they describe rather than naming it. +PREV="${_NUC_FIXTURES[${#_NUC_FIXTURES[@]}-1]}" +WIN=conv_window +P -d postgres -c "DROP DATABASE IF EXISTS $WIN;" >/dev/null +P -d postgres -c "CREATE DATABASE $WIN;" >/dev/null +P -d "$WIN" -c "CREATE EXTENSION pgcolumnar VERSION '$PREV';" >/dev/null +check "premise: the window database runs this library over the previous release's catalog" \ + "$(P -d "$WIN" -c "SELECT extversion FROM pg_extension WHERE extname='pgcolumnar';")" "$PREV" +check "premise: that catalog has no retired_group, which this library deletes from" \ + "$(P -d "$WIN" -c "SELECT count(*) FROM pg_class WHERE relname = 'retired_group' AND relnamespace = 'pgcolumnar'::regnamespace;")" "0" +P -d "$WIN" -c "CREATE TABLE w (a int, b text) USING pgcolumnar;" >/dev/null +P -d "$WIN" -c "INSERT INTO w SELECT g, 'x' || g FROM generate_series(1, 20000) g;" >/dev/null +P -d "$WIN" -c "DELETE FROM w WHERE a % 3 = 0;" >/dev/null +win_rows() { P -d "$WIN" -c "SELECT count(*) || '|' || sum(a) FROM w;"; } +check "in the window, a columnar table is written, deleted from and read" "$(win_rows)" "13334|133346667" +win_refusal() { # win_refusal SQL -> SQLSTATE|whether the HINT names step 3, or what happened instead + local out + out="$(P -d "$WIN" -v VERBOSITY=verbose -c "$1" 2>&1)" && { echo accepted; return; } + echo "$(sed -nE 's/^ERROR: ([0-9A-Z]{5}): .*/\1/p' <<<"$out" | head -1)|$(grep -q '^HINT: .*ALTER EXTENSION pgcolumnar UPDATE' <<<"$out" && echo names-the-update || echo no-hint)" +} +check "in the window, DROP TABLE is refused and names ALTER EXTENSION UPDATE" \ + "$(win_refusal "DROP TABLE w;")" "42P01|names-the-update" +check "in the window, TRUNCATE is refused and names ALTER EXTENSION UPDATE" \ + "$(win_refusal "TRUNCATE w;")" "42P01|names-the-update" +check "in the window, pgcolumnar.vacuum() is refused and names ALTER EXTENSION UPDATE" \ + "$(win_refusal "SELECT pgcolumnar.vacuum('w');")" "42P01|names-the-update" +check "in the window, the refused operations changed nothing" "$(win_rows)" "13334|133346667" + +P -d "$WIN" -c "ALTER EXTENSION pgcolumnar UPDATE;" >/dev/null +check "after ALTER EXTENSION UPDATE, the same rows are there" "$(win_rows)" "13334|133346667" +check "after ALTER EXTENSION UPDATE, pgcolumnar.vacuum() succeeds and keeps them" \ + "$(P -d "$WIN" -c "SELECT pgcolumnar.vacuum('w');" >/dev/null 2>&1 && win_rows || echo refused)" "13334|133346667" +check "after ALTER EXTENSION UPDATE, TRUNCATE and DROP TABLE succeed" \ + "$(P -d "$WIN" -c "TRUNCATE w;" >/dev/null 2>&1 && P -d "$WIN" -c "DROP TABLE w;" >/dev/null 2>&1 && echo succeeded || echo refused)" "succeeded" +P -d postgres -c "DROP DATABASE IF EXISTS $WIN;" >/dev/null + pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 9212ee1a5..46d0a068c 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -188,6 +188,7 @@ behaviour, the source of that number is named. - [140. test_parquet_import.py: a partitioned columnar parent is refused by import_parquet](#140-test_parquet_importpy-a-partitioned-columnar-parent-is-refused-by-import_parquet) - [141. test_phase5.py: set_options and reset_options agree](#141-test_phase5py-set_options-and-reset_options-agree) - [142. test_drop_cleanup.py: a columnar matview's options row goes with its DROP](#142-test_drop_cleanuppy-a-columnar-matviews-options-row-goes-with-its-drop) +- [143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE](#143-test_native_upgrade_convergepy-the-upgrade-window-before-alter-extension-update) ## 1. How to read a test in here @@ -7377,3 +7378,13 @@ Part of the pytest twin of `test/drop_cleanup.sh` (#1265, ported for #1341): its | `test_a_columnar_matview_takes_its_options_row_with_it` | a planted options row is gone after DROP MATERIALIZED VIEW | | `test_refresh_keeps_a_matviews_options_row` | a REFRESH that rewrote the matview leaves its set_options row in place | | `test_an_unrelated_options_row_survives` | another table's row survives a matview's DROP, and an ordinary columnar table still takes its own | + +## 143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE + +Part of the pytest twin of `test/native_upgrade_converge.sh` (#1341 item 66): its upgrade-window arms only, by the shell's names, and the pair is declared INCOMPLETE. Between installing a new library and running `ALTER EXTENSION pgcolumnar UPDATE`, a database runs the new library over its previous release's catalog. Reads and writes must work there. `DROP TABLE`, `TRUNCATE` and `pgcolumnar.vacuum()`, which need a catalog table the old one lacks, must be refused with 42P01, change nothing, and carry a hint naming the update; after the update all three succeed on the same rows. The previous release is the version the shipped upgrade script to the current `default_version` starts from; its base script is staged from `test/fixtures` for the test and removed after. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_upgrade_window` | at the previous catalog: rows written, deleted and read; the three operations refused with the hint and nothing changed; after the update the rows are intact and all three succeed | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index a6a77ccba..f758b74f1 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -747,4 +747,6 @@ guard_tests 438 # seven tests, by collection over the cluster files on this tree. # cluster_tests 653 -> 660 for #1341 item 29's partial twins: test_inheritance.py (2), # test_parquet_import.py (1), test_phase5.py (1) and test_drop_cleanup.py (3), by collection. -cluster_tests 660 +# cluster_tests 660 -> 661 for #1341 item 66's test_native_upgrade_converge.py, one test, +# by collection over the cluster files on this tree. +cluster_tests 661 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 4140fbf32..4e0dbf689 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -179,6 +179,10 @@ # about the PORT rather than a licence: the standing arm does not grade it, so the # reason is the only thing standing between a declared gap and a forgotten one. INCOMPLETE = { + # #1341 item 66 ported the arms it added: the upgrade window, this library over + # the previous release's catalog before ALTER EXTENSION UPDATE. + "native_upgrade_converge": "the upgrade-window arms are ported; the fixture-blob, " + "staging and catalog-convergence arms are not", # #1341 item 29 ported the arms four fixes added with shell arms only. Each # pair below holds that fix's arms and nothing else of its shell suite. "inheritance": "the #1259 arms (a partitioned columnar parent is refused with " diff --git a/test/pytest/test_native_upgrade_converge.py b/test/pytest/test_native_upgrade_converge.py new file mode 100644 index 000000000..5a2c1d54b --- /dev/null +++ b/test/pytest/test_native_upgrade_converge.py @@ -0,0 +1,126 @@ +"""The upgrade window: this library over the previous release's catalog, before ALTER +EXTENSION UPDATE (#1341 item 66). Part of the pytest twin of +`test/native_upgrade_converge.sh`: these arms only, by the shell's names, and the pair +is declared INCOMPLETE. + +docs/installation.md steps 1 and 2 put a new library in place; step 3, ALTER EXTENSION +UPDATE, brings each database's catalog up to it. Between them a database runs the new +library over the old catalog, and the library deletes from catalog tables the old one +does not have (retired_group, new in 1.0-alpha5). Reads and writes must work there; an +operation that needs a missing table must be refused cleanly, change nothing and name +step 3; and after step 3 every one of them must work on the same rows. + +The previous release is the version the shipped upgrade script to the current +default_version starts from, and its base script is staged from test/fixtures into the +extension directory for the test, as the shell does, then removed. +""" +import pathlib +import re +import shutil +import subprocess + +import psycopg +import pytest + +REPO = pathlib.Path(__file__).resolve().parents[2] + + +def _target_and_previous(): + target = re.search(r"^default_version\s*=\s*'([^']+)'", + (REPO / "pgcolumnar.control").read_text(), re.M).group(1) + prev = [p.name[len("pgcolumnar--"):-len(f"--{target}.sql")] + for p in REPO.glob(f"pgcolumnar--*--{target}.sql")] + return target, prev + + +@pytest.fixture +def window_db(pgc_cluster): + """-> (dsn, previous version) of a database at the previous release's catalog.""" + target, prev = _target_and_previous() + assert len(prev) == 1, f"one upgrade script should lead to {target}, found {prev}" + prev = prev[0] + sharedir = subprocess.run([pgc_cluster.pg_config, "--sharedir"], capture_output=True, + text=True, check=True).stdout.strip() + dst = pathlib.Path(sharedir) / "extension" / f"pgcolumnar--{prev}.sql" + backup = dst.with_name(dst.name + ".pytest-bak") + if dst.exists(): + shutil.copy2(dst, backup) + shutil.copy2(REPO / "test" / "fixtures" / f"pgcolumnar--{prev}.sql", dst) + + def admin(sql): + with psycopg.connect(pgc_cluster.dsn(), autocommit=True) as c: + c.execute(sql) + + admin("DROP DATABASE IF EXISTS pgc_upgrade_window") + admin("CREATE DATABASE pgc_upgrade_window") + dsn = pgc_cluster.dsn(dbname="pgc_upgrade_window") + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(f"CREATE EXTENSION pgcolumnar VERSION '{prev}'") + yield dsn, prev + finally: + admin("DROP DATABASE IF EXISTS pgc_upgrade_window WITH (FORCE)") + if backup.exists(): + shutil.move(backup, dst) + else: + dst.unlink(missing_ok=True) + + +def _one(dsn, sql): + with psycopg.connect(dsn, autocommit=True) as c: + return c.execute(sql).fetchone()[0] + + +def _refusal(dsn, sql): + """-> 'SQLSTATE|names-the-update' or '|no-hint', or 'accepted'.""" + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(sql) + return "accepted" + except psycopg.Error as exc: + hint = exc.diag.message_hint or "" + return f"{exc.sqlstate}|{'names-the-update' if 'ALTER EXTENSION pgcolumnar UPDATE' in hint else 'no-hint'}" + + +def _succeeds(dsn, *sqls): + try: + with psycopg.connect(dsn, autocommit=True) as c: + for sql in sqls: + c.execute(sql) + return "succeeded" + except psycopg.Error as exc: + return f"refused ({exc.sqlstate}: {str(exc).splitlines()[0]})" + + +def test_the_upgrade_window(window_db, expect): + dsn, prev = window_db + expect.text(_one(dsn, "SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar'"), prev, + "premise: the window database runs this library over the previous release's catalog") + expect.num(_one(dsn, "SELECT count(*) FROM pg_class WHERE relname = 'retired_group' " + "AND relnamespace = 'pgcolumnar'::regnamespace"), 0, + "premise: that catalog has no retired_group, which this library deletes from") + with psycopg.connect(dsn, autocommit=True) as c: + c.execute("CREATE TABLE w (a int, b text) USING pgcolumnar") + c.execute("INSERT INTO w SELECT g, 'x' || g FROM generate_series(1, 20000) g") + c.execute("DELETE FROM w WHERE a % 3 = 0") + + def rows(): + return _one(dsn, "SELECT count(*) || '|' || sum(a) FROM w") + + expect.text(rows(), "13334|133346667", "in the window, a columnar table is written, deleted from and read") + expect.text(_refusal(dsn, "DROP TABLE w"), "42P01|names-the-update", + "in the window, DROP TABLE is refused and names ALTER EXTENSION UPDATE") + expect.text(_refusal(dsn, "TRUNCATE w"), "42P01|names-the-update", + "in the window, TRUNCATE is refused and names ALTER EXTENSION UPDATE") + expect.text(_refusal(dsn, "SELECT pgcolumnar.vacuum('w')"), "42P01|names-the-update", + "in the window, pgcolumnar.vacuum() is refused and names ALTER EXTENSION UPDATE") + expect.text(rows(), "13334|133346667", "in the window, the refused operations changed nothing") + + with psycopg.connect(dsn, autocommit=True) as c: + c.execute("ALTER EXTENSION pgcolumnar UPDATE") + expect.text(rows(), "13334|133346667", "after ALTER EXTENSION UPDATE, the same rows are there") + vacuumed = _succeeds(dsn, "SELECT pgcolumnar.vacuum('w')") + expect.text(rows() if vacuumed == "succeeded" else vacuumed, "13334|133346667", + "after ALTER EXTENSION UPDATE, pgcolumnar.vacuum() succeeds and keeps them") + expect.text(_succeeds(dsn, "TRUNCATE w", "DROP TABLE w"), "succeeded", + "after ALTER EXTENSION UPDATE, TRUNCATE and DROP TABLE succeed") From 110aa0bf584f065f14ef3e068548d41827e58fdd Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:05 +0000 Subject: [PATCH 02/36] test: fk_referencing gets its pytest twin, and a referenced table may still be set to a row store (#1341) test_fk_referencing.py asserts every shell check by name; compare_to_bash grades it as covering all 28 (27 checks on PG17+, and the pre-17 branch's own), and it joins COMPLETE. The conversion guard's over-rejection control converted a table with no foreign key back to heap, so it could not see a guard that refused every SET ACCESS METHOD on a referenced table whatever the target: with the guard's columnar-target test removed, the suite stayed 26/26. A new arm sets the referenced table itself to heap; under that mutant both twins now fail it. cluster_tests 661 -> 666 by collection; TESTS.md gets section 144. Co-Authored-By: Claude Opus 5.5 --- test/fk_referencing.sh | 10 ++ test/pytest/TESTS.md | 15 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_fk_referencing.py | 158 ++++++++++++++++++++++++++++ 5 files changed, 187 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_fk_referencing.py diff --git a/test/fk_referencing.sh b/test/fk_referencing.sh index 73b59c41d..cd49dcecd 100755 --- a/test/fk_referencing.sh +++ b/test/fk_referencing.sh @@ -220,6 +220,16 @@ check "converting back to heap is unaffected" \ "$(case "$back" in *ERROR*) echo "no ($back)" ;; *) echo yes ;; esac)" \ "yes" +# AND ON THE REFERENCED TABLE ITSELF (#1341). The control above converts a table +# with no foreign key, so it cannot see a rule that refuses every SET ACCESS METHOD +# on a referenced table whatever the target: with the guard's columnar-target test +# removed, this file stayed 26/26. Only a columnar target needs the row lock the +# constraint cannot get. +refheap="$(psql_run "ALTER TABLE fk_conv_parent SET ACCESS METHOD heap;" 2>&1 || true)" +check "an FK-referenced table may still be set to a row store" \ + "$(case "$refheap" in *ERROR*) echo "no ($refheap)" ;; *) echo yes ;; esac)" \ + "yes" + # --- 7. a partitioned parent is refused too (#201) ----------------------------- # A partitioned table has no storage, so setting its access method breaks nothing diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 46d0a068c..579d1b082 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -189,6 +189,7 @@ behaviour, the source of that number is named. - [141. test_phase5.py: set_options and reset_options agree](#141-test_phase5py-set_options-and-reset_options-agree) - [142. test_drop_cleanup.py: a columnar matview's options row goes with its DROP](#142-test_drop_cleanuppy-a-columnar-matviews-options-row-goes-with-its-drop) - [143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE](#143-test_native_upgrade_convergepy-the-upgrade-window-before-alter-extension-update) +- [144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen](#144-test_fk_referencingpy-a-foreign-key-referencing-a-columnar-table-is-refused-where-it-is-chosen) ## 1. How to read a test in here @@ -7388,3 +7389,17 @@ Part of the pytest twin of `test/native_upgrade_converge.sh` (#1341 item 66): it | test | what it holds | | --- | --- | | `test_the_upgrade_window` | at the previous catalog: rows written, deleted and read; the three operations refused with the hint and nothing changed; after the update the rows are intact and all three succeed | + +## 144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen + +The pytest twin of `test/fk_referencing.sh` (ported for #1341), every check by the shell's name. Enforcing a foreign key locks the referenced row, and columnar tables cannot lock rows, so a constraint referencing one would make every insert into the referencing table fail. It is refused at DDL by either route into that state: creating the constraint against a columnar table, or converting a referenced table (or partitioned parent) to columnar. Only the referenced side is refused, and the controls hold everything else steady, including converting the referenced table itself to a row store. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_referenced_side_is_refused_at_ddl` | CREATE TABLE and ALTER TABLE ADD CONSTRAINT referencing a columnar table are refused, saying why, and nothing is left behind | +| `test_the_referencing_side_still_works` | a columnar table referencing a heap table is created, accepts satisfying rows and rejects a violating one | +| `test_constraints_that_need_no_row_lock_are_unaffected` | primary key, unique, check and not-null on a columnar table, and a heap-to-heap foreign key, are unaffected | +| `test_converting_a_referenced_table_is_refused_and_nothing_else_is` | SET ACCESS METHOD pgcolumnar on a referenced table and the helper are refused, the table stays heap and its child writable; an unreferenced table, the referencing side, an unrelated ALTER, a conversion back to heap and the referenced table to a row store all still work | +| `test_a_partitioned_referenced_parent` | a columnar partition and setting the parent's access method are refused, the parent does not become columnar, an ordinary partition still works; an unreferenced partitioned table converts from PG17, and before 17 core refuses it | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index f758b74f1..db4d7c30f 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -749,4 +749,6 @@ guard_tests 438 # test_parquet_import.py (1), test_phase5.py (1) and test_drop_cleanup.py (3), by collection. # cluster_tests 660 -> 661 for #1341 item 66's test_native_upgrade_converge.py, one test, # by collection over the cluster files on this tree. -cluster_tests 661 +# cluster_tests 661 -> 666 for #1341's test_fk_referencing.py, the twin of fk_referencing.sh: +# five tests, by collection over the cluster files on this tree. +cluster_tests 666 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 4e0dbf689..a0fd43008 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -106,6 +106,7 @@ "differential", "encode_post_codec", "entry_point_privilege", + "fk_referencing", "format_downgrade", "hilbert_cluster", "hilbert_locality", diff --git a/test/pytest/test_fk_referencing.py b/test/pytest/test_fk_referencing.py new file mode 100644 index 000000000..3e7a60274 --- /dev/null +++ b/test/pytest/test_fk_referencing.py @@ -0,0 +1,158 @@ +"""A foreign key that references a columnar table is refused where it is chosen +(#1341). The pytest twin of `test/fk_referencing.sh`, every check by the shell's name. + +Referential integrity reads the referenced row with FOR KEY SHARE, and row locking is +not implemented for columnar tables, so a constraint referencing one would make every +insert into the referencing table fail. It is refused at DDL instead, by either route +into that state: creating the constraint against a columnar table, or converting a +referenced table to columnar. Only the REFERENCED side is refused: a columnar table may +reference a heap table, and the controls hold everything else steady -- other +constraints, heap-to-heap keys, conversions of an unreferenced table, of the +referencing side, back to heap, and of the referenced table to a row store. +""" +import psycopg + + +def _err(conn, sql): + """-> '' when sql succeeds, else the error's message, detail and hint, joined.""" + try: + with conn.cursor() as cur: + cur.execute(sql) + return "" + except psycopg.Error as exc: + d = exc.diag + return " ".join(x for x in (d.message_primary, d.message_detail, d.message_hint) if x) or str(exc) + + +def _has(text, needle): + return "yes" if needle in text else f"no ({text})" + + +def _ok(text): + return "yes" if not text else f"no ({text})" + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +def _run(conn, *sqls): + with conn.cursor() as cur: + for sql in sqls: + cur.execute(sql) + + +def test_the_referenced_side_is_refused_at_ddl(pgc_conn, expect): + conn = pgc_conn + _run(conn, "CREATE TABLE fk_parent (id int PRIMARY KEY) USING pgcolumnar", + "INSERT INTO fk_parent VALUES (1), (2)") + err = _err(conn, "CREATE TABLE fk_child (id int REFERENCES fk_parent(id))") + expect.text(_has(err, "cannot create a foreign key referencing columnar table"), "yes", + "a foreign key referencing a columnar table is refused") + expect.text(_has(err, "row locking is not supported"), "yes", "and the error says why") + expect.num(_one(conn, "SELECT count(*) FROM pg_class WHERE relname = 'fk_child' " + "AND relnamespace = current_schema()::regnamespace"), 0, + "the refused table was not left behind") + + _run(conn, "CREATE TABLE fk_child2 (id int)") + err2 = _err(conn, "ALTER TABLE fk_child2 ADD CONSTRAINT fk_c2 FOREIGN KEY (id) REFERENCES fk_parent(id)") + expect.text(_has(err2, "cannot create a foreign key referencing columnar table"), "yes", + "ALTER TABLE ADD CONSTRAINT is refused by the same rule") + + +def test_the_referencing_side_still_works(pgc_conn, expect): + conn = pgc_conn + _run(conn, "CREATE TABLE fk_h_parent (id int PRIMARY KEY)", "INSERT INTO fk_h_parent VALUES (1), (2), (3)") + expect.text(_ok(_err(conn, "CREATE TABLE fk_c_child (id int REFERENCES fk_h_parent(id), v text) " + "USING pgcolumnar")), "yes", + "a columnar table may reference a heap table") + _err(conn, "INSERT INTO fk_c_child VALUES (1, 'a'), (2, 'b')") + expect.num(_one(conn, "SELECT count(*) FROM fk_c_child"), 2, "and rows that satisfy it are accepted") + expect.text(_has(_err(conn, "INSERT INTO fk_c_child VALUES (99, 'nope')"), "violates foreign key constraint"), + "yes", "a row that violates it is still rejected") + + +def test_constraints_that_need_no_row_lock_are_unaffected(pgc_conn, expect): + conn = pgc_conn + expect.text(_ok(_err(conn, "CREATE TABLE fk_other (id int PRIMARY KEY, k int UNIQUE, v int CHECK (v > 0), " + "w int NOT NULL) USING pgcolumnar")), "yes", + "primary key, unique, check and not-null on a columnar table are unaffected") + _err(conn, "INSERT INTO fk_other VALUES (1, 10, 5, 7)") + expect.num(_one(conn, "SELECT count(*) FROM fk_other"), 1, "and they still work") + expect.text(_has(_err(conn, "INSERT INTO fk_other VALUES (1, 11, 5, 7)"), "duplicate key"), "yes", + "the primary key is still enforced") + _run(conn, "CREATE TABLE fk_hh_parent (id int PRIMARY KEY)") + expect.text(_ok(_err(conn, "CREATE TABLE fk_hh_child (id int REFERENCES fk_hh_parent(id))")), "yes", + "a foreign key between two heap tables is unaffected") + + +def test_converting_a_referenced_table_is_refused_and_nothing_else_is(pgc_conn, expect): + conn = pgc_conn + _run(conn, "CREATE TABLE fk_conv_parent (id int PRIMARY KEY)", + "CREATE TABLE fk_conv_child (id int REFERENCES fk_conv_parent(id))", + "INSERT INTO fk_conv_parent VALUES (1)", "INSERT INTO fk_conv_child VALUES (1)") + conv = _err(conn, "ALTER TABLE fk_conv_parent SET ACCESS METHOD pgcolumnar") + expect.text(_has(conv, "referenced by a foreign key"), "yes", + "converting an FK-referenced table to columnar is refused") + expect.text(_has(conv, "row locking is not supported"), "yes", "and the error names the constraint") + expect.text(_one(conn, "SELECT am.amname::text FROM pg_class c JOIN pg_am am ON am.oid = c.relam " + "WHERE c.oid = 'fk_conv_parent'::regclass"), "heap", "the parent is still a row store") + _err(conn, "INSERT INTO fk_conv_parent VALUES (2)") + expect.text(_ok(_err(conn, "INSERT INTO fk_conv_child VALUES (2)")), "yes", + "the child is still writable afterwards") + expect.text(_has(_err(conn, "SELECT pgcolumnar.alter_table_set_access_method('fk_conv_parent', 'pgcolumnar')"), + "referenced by a foreign key"), "yes", "the conversion helper is refused identically") + + # The over-rejection controls: a rule keyed on the wrong side, or firing for any + # ALTER TABLE or any target, passes every arm above and fails one of these. + _run(conn, "CREATE TABLE fk_conv_plain (id int PRIMARY KEY)", "INSERT INTO fk_conv_plain VALUES (1)") + expect.text(_ok(_err(conn, "ALTER TABLE fk_conv_plain SET ACCESS METHOD pgcolumnar")), "yes", + "a table with no foreign key still converts") + expect.text(_ok(_err(conn, "ALTER TABLE fk_conv_child SET ACCESS METHOD pgcolumnar")), "yes", + "converting the referencing side is still allowed") + expect.text(_ok(_err(conn, "ALTER TABLE fk_conv_parent ADD COLUMN note text")), "yes", + "an unrelated ALTER TABLE on a referenced table is unaffected") + expect.text(_ok(_err(conn, "ALTER TABLE fk_conv_plain SET ACCESS METHOD heap")), "yes", + "converting back to heap is unaffected") + # The control above has no foreign key; this one is the referenced table itself. + expect.text(_ok(_err(conn, "ALTER TABLE fk_conv_parent SET ACCESS METHOD heap")), "yes", + "an FK-referenced table may still be set to a row store") + + +def test_a_partitioned_referenced_parent(pgc_conn, expect): + """Setting a partitioned parent's access method chooses what every later partition + inherits, and core clones the foreign key to each, so it is refused too (#201).""" + conn = pgc_conn + _run(conn, "CREATE TABLE fk_pp (id int PRIMARY KEY) PARTITION BY RANGE (id)", + "CREATE TABLE fk_pp1 PARTITION OF fk_pp FOR VALUES FROM (1) TO (100)", + "CREATE TABLE fk_pc (id int REFERENCES fk_pp(id))") + expect.text(_has(_err(conn, "CREATE TABLE fk_pp2 PARTITION OF fk_pp FOR VALUES FROM (100) TO (200) " + "USING pgcolumnar"), "cannot create a foreign key referencing columnar table"), + "yes", "a columnar partition of an FK-referenced parent is refused") + expect.text(_has(_err(conn, "ALTER TABLE fk_pp SET ACCESS METHOD pgcolumnar"), "cannot convert table"), "yes", + "and setting the parent's access method is refused as well") + # "did it become columnar", not "is it heap": a partitioned table never given an + # access method has relam = 0, which joins to no pg_am row. + expect.num(_one(conn, "SELECT count(*) FROM pg_class c JOIN pg_am a ON a.oid = c.relam " + "WHERE c.oid = 'fk_pp'::regclass AND a.amname = 'pgcolumnar'"), 0, + "so the parent did not become columnar") + expect.num(0 if not _err(conn, "CREATE TABLE fk_pp3 PARTITION OF fk_pp FOR VALUES FROM (200) TO (300)") else 1, + 0, "and a partition can still be created the ordinary way") + + _run(conn, "CREATE TABLE fk_free (id int) PARTITION BY RANGE (id)") + major = int(_one(conn, "SHOW server_version_num")) // 10000 + if major >= 17: + expect.num(0 if not _err(conn, "ALTER TABLE fk_free SET ACCESS METHOD pgcolumnar") else 1, 0, + "a partitioned table with no foreign key still converts") + _err(conn, "CREATE TABLE fk_free1 PARTITION OF fk_free FOR VALUES FROM (1) TO (100)") + expect.text(_one(conn, "SELECT amname::text FROM pg_class c JOIN pg_am a ON a.oid = c.relam " + "WHERE c.oid = 'fk_free1'::regclass"), "pgcolumnar", + "and its partitions inherit the columnar access method") + else: + # Before 17 core refuses SET ACCESS METHOD on every partitioned table, so the + # refusal must be core's, by core's message. + expect.text(_has(_err(conn, "ALTER TABLE fk_free SET ACCESS METHOD pgcolumnar"), + "cannot change access method of a partitioned table"), "yes", + "before 17, core refuses this for every partitioned table") From 4ee6e1fdc3377e33e1d3bf6017a716c63446f3ce Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:05 +0000 Subject: [PATCH 03/36] test: fk_referencing seeds the ledger, 17 of 28 arms dated by four mutations of the foreign-key guards (#1341) 28 rows from five-major runs (27/27 on 17-19, 26/26 on 15-16). Dated on PG18 by: the referenced-side refusal removed (5), the guard keyed on the referencing table (8), the conversion refusal removed (8), and the conversion refused whatever the target (1). suites_not_covered 219 -> 218; the census is 3647 rows, 2686 never, 961 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 28 ++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 22 ++++++++++++++++++++-- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 74fcd205f..2c094ebec 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -804,6 +804,34 @@ fdw_plan_privilege fdw_plan_privilege and a metadata_path that does not exist pl fdw_plan_privilege fdw_plan_privilege premise: the Parquet file was written 15;16;17;18;19 never - fdw_plan_privilege fdw_plan_privilege premise: the reader is not a member of pg_read_server_files 15;16;17;18;19 never - fdw_plan_privilege fdw_plan_privilege the unprivileged Iceberg scan is still refused (42501) 15;16;17;18;19 never - +fk_referencing fk_referencing ALTER TABLE ADD CONSTRAINT is refused by the same rule 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing a columnar partition of an FK-referenced parent is refused 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing a columnar table may reference a heap table 15;16;17;18;19 2026-10-16 columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing a foreign key between two heap tables is unaffected 15;16;17;18;19 never - +fk_referencing fk_referencing a foreign key referencing a columnar table is refused 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing a partitioned table with no foreign key still converts 17;18;19 never - +fk_referencing fk_referencing a row that violates it is still rejected 15;16;17;18;19 2026-10-16 columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing a table with no foreign key still converts 15;16;17;18;19 never - +fk_referencing fk_referencing an FK-referenced table may still be set to a row store 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table refused whatever the target access method +fk_referencing fk_referencing an unrelated ALTER TABLE on a referenced table is unaffected 15;16;17;18;19 never - +fk_referencing fk_referencing and a partition can still be created the ordinary way 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing and its partitions inherit the columnar access method 17;18;19 never - +fk_referencing fk_referencing and rows that satisfy it are accepted 15;16;17;18;19 2026-10-16 columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing and setting the parent's access method is refused as well 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing and the error names the constraint 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing and the error says why 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one +fk_referencing fk_referencing and they still work 15;16;17;18;19 never - +fk_referencing fk_referencing before 17, core refuses this for every partitioned table 15;16 never - +fk_referencing fk_referencing converting an FK-referenced table to columnar is refused 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing converting back to heap is unaffected 15;16;17;18;19 never - +fk_referencing fk_referencing converting the referencing side is still allowed 15;16;17;18;19 never - +fk_referencing fk_referencing primary key, unique, check and not-null on a columnar table are unaffected 15;16;17;18;19 never - +fk_referencing fk_referencing so the parent did not become columnar 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing the child is still writable afterwards 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing the conversion helper is refused identically 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing the parent is still a row store 15;16;17;18;19 2026-10-16 columnar_tableam.c: converting an FK-referenced table to columnar no longer refused +fk_referencing fk_referencing the primary key is still enforced 15;16;17;18;19 never - +fk_referencing fk_referencing the refused table was not left behind 15;16;17;18;19 2026-10-16 columnar_tableam.c: a foreign key referencing a columnar table no longer refused;columnar_tableam.c: the foreign-key guard keyed on the referencing table, not the referenced one harness_selftest 030-assertions nothing leaked into the squatter 15;16;17;18;19 never - harness_selftest 030-assertions pgc_port_free says the squatter's port is busy 15;16;17;18;19 never - harness_selftest 030-assertions squatter survived untouched 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 3f8e29cde..42a7cd12d 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 219 +suites_not_covered 218 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2943,4 +2943,22 @@ suites_not_covered 219 # never 2676 - 1 = 2675 # dated 943 + 1 = 944 # 2675 + 944 == 3619 -checks_never_observed_red 2675 +# +# #1341, fk_referencing seeded, with its twin (OffgridwithJD): 2675 -> 2686, and +# suites_not_covered 219 -> 218. The suite had no rows. It enters with 28 rows +# from five-major runs: 27 checks on PG17-19 and 26 on 15-16, where the +# partitioned-control branch asserts core's refusal instead of a conversion. 17 are +# dated by four mutations of the foreign-key guards: the referenced-side refusal +# removed, the guard keyed on the referencing table, the conversion refusal +# removed, and the conversion refused whatever the target access method (the new +# referenced-table-to-row-store arm). Eleven arrive never: the over-rejection +# controls that no single guard reaches (another constraint, a heap-to-heap key, +# an unreferenced conversion, an unrelated ALTER, a conversion back to heap, the +# referencing side, the partitioned control on either side of 17) and the primary +# key's own arms. +# +# ledger rows 3619 + 28 = 3647 +# never 2675 + 11 = 2686 +# dated 944 + 17 = 961 +# 2686 + 961 == 3647 +checks_never_observed_red 2686 From 110ccd434c808c6da37fcf179825d71d3b3302e1 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:05 +0000 Subject: [PATCH 04/36] test: truncate_cleanup gets its pytest twin (#1341) test_truncate_cleanup.py asserts every shell check by name; compare_to_bash grades it as covering all 25, and it joins COMPLETE. It runs in a private database, since the snapshot counts every pgcolumnar catalog row and the shared one holds every other test's. The shell suite needed no change: each of its arms already goes red under a mutation of the path it names. cluster_tests 666 -> 672 by collection; TESTS.md gets section 145. Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 16 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_truncate_cleanup.py | 181 +++++++++++++++++++++++++++ 4 files changed, 201 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_truncate_cleanup.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 579d1b082..95dee4a68 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -190,6 +190,7 @@ behaviour, the source of that number is named. - [142. test_drop_cleanup.py: a columnar matview's options row goes with its DROP](#142-test_drop_cleanuppy-a-columnar-matviews-options-row-goes-with-its-drop) - [143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE](#143-test_native_upgrade_convergepy-the-upgrade-window-before-alter-extension-update) - [144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen](#144-test_fk_referencingpy-a-foreign-key-referencing-a-columnar-table-is-refused-where-it-is-chosen) +- [145. test_truncate_cleanup.py: TRUNCATE leaves no catalog rows behind, and keeps what it must](#145-test_truncate_cleanuppy-truncate-leaves-no-catalog-rows-behind-and-keeps-what-it-must) ## 1. How to read a test in here @@ -7403,3 +7404,18 @@ The pytest twin of `test/fk_referencing.sh` (ported for #1341), every check by t | `test_constraints_that_need_no_row_lock_are_unaffected` | primary key, unique, check and not-null on a columnar table, and a heap-to-heap foreign key, are unaffected | | `test_converting_a_referenced_table_is_refused_and_nothing_else_is` | SET ACCESS METHOD pgcolumnar on a referenced table and the helper are refused, the table stays heap and its child writable; an unreferenced table, the referencing side, an unrelated ALTER, a conversion back to heap and the referenced table to a row store all still work | | `test_a_partitioned_referenced_parent` | a columnar partition and setting the parent's access method are refused, the parent does not become columnar, an ordinary partition still works; an unreferenced partitioned table converts from PG17, and before 17 core refuses it | + +## 145. test_truncate_cleanup.py: TRUNCATE leaves no catalog rows behind, and keeps what it must + +The pytest twin of `test/truncate_cleanup.sh` (ported for #1341), every check by the shell's name. TRUNCATE installs a new storage id, and the old storage's catalog rows used to stay behind, so each truncate-and-reload cycle leaked one storage's worth and a later DROP removed only the empty current one. Retiring them made a cached write state stale, which is dropped with them. A relation created in the current transaction is truncated in place, on a different path, which must clear each projection's storage too. A private database, since the snapshot counts every pgcolumnar catalog row. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_truncate_and_drop_leave_the_catalog_as_they_found_it` | one TRUNCATE and three truncate-and-reload cycles return the catalog to the baseline, and a DROP of a reloaded table leaves nothing | +| `test_a_projection_is_not_leaked_by_truncate` | no projection row names a retired storage id, the declaration survives, and a DROP of the reloaded table leaves nothing | +| `test_repeated_truncates_accumulate_nothing` | ten transactional truncates of one table, and its DROP, leave the baseline | +| `test_insert_truncate_insert_in_one_transaction` | the transaction commits and holds exactly the rows written after the TRUNCATE | +| `test_a_rewrite_that_is_not_a_truncate_loses_nothing` | a type-changing rewrite keeps all 5000 rows readable | +| `test_an_in_place_truncate_clears_the_projection_too` | the in-place truncate commits, the later rows are all there, the projection reads them back and its storage holds one group of exactly them | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index db4d7c30f..fd24e418c 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -751,4 +751,6 @@ guard_tests 438 # by collection over the cluster files on this tree. # cluster_tests 661 -> 666 for #1341's test_fk_referencing.py, the twin of fk_referencing.sh: # five tests, by collection over the cluster files on this tree. -cluster_tests 666 +# cluster_tests 666 -> 672 for #1341's test_truncate_cleanup.py, the twin of truncate_cleanup.sh: +# six tests, by collection over the cluster files on this tree. +cluster_tests 672 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index a0fd43008..185ee0529 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -167,6 +167,7 @@ "tid_range_scan", "tier_partition", "timeseries_recipes", + "truncate_cleanup", "ttl_expire", "type_matrix", "unique_conc", diff --git a/test/pytest/test_truncate_cleanup.py b/test/pytest/test_truncate_cleanup.py new file mode 100644 index 000000000..c3c34ab9e --- /dev/null +++ b/test/pytest/test_truncate_cleanup.py @@ -0,0 +1,181 @@ +"""TRUNCATE leaves no catalog rows behind, and keeps what it must (#1341). The pytest +twin of `test/truncate_cleanup.sh`, every check by the shell's name. + +TRUNCATE installs a new relfilenode and a new storage id. It used to leave the old +storage's row_group, column_chunk, zone_map and bloom rows behind, keyed by the retired +id, so each truncate-and-reload cycle leaked one storage's worth, and a DROP after it +removed only the current, empty one. Retiring them made a cached write state stale, so +a transaction that wrote, truncated and wrote again committed into storage nothing +reads; the state is now dropped with the rows. A relation created in the current +transaction is truncated in place instead, on a different path, which must clear each +projection's storage as well as the base's. + +A PRIVATE DATABASE, because the snapshot counts every pgcolumnar catalog row, and the +shared one holds every other test's. +""" +import psycopg + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +def _run(conn, *sqls): + with conn.cursor() as cur: + for sql in sqls: + cur.execute(sql) + + +def _snapshot(conn): + return _one(conn, "SELECT (SELECT count(*) FROM pgcolumnar.storage) || '/' || " + "(SELECT count(*) FROM pgcolumnar.projection) || '/' || " + "(SELECT count(*) FROM pgcolumnar.row_group) || '/' || " + "(SELECT count(*) FROM pgcolumnar.column_chunk) || '/' || " + "(SELECT count(*) FROM pgcolumnar.zone_map) || '/' || " + "(SELECT count(*) FROM pgcolumnar.bloom) || '/' || " + "(SELECT count(*) FROM pgcolumnar.delete_vector)") + + +def _grew(base, now): + return "no" if base == now else "yes" + + +LOAD = "INSERT INTO {t} SELECT g, 'x' || g FROM generate_series(1, {n}) g" + + +def test_truncate_and_drop_leave_the_catalog_as_they_found_it(pgc_own_db, expect): + conn = pgc_own_db + base = _snapshot(conn) + _run(conn, "CREATE TABLE tcl_plain (id int, v text) USING pgcolumnar", LOAD.format(t="tcl_plain", n=5000)) + expect.text(_grew(base, _snapshot(conn)), "yes", "insert did add metadata") + _run(conn, "TRUNCATE tcl_plain") + expect.text(_snapshot(conn), base, "TRUNCATE returns the catalog to the empty baseline") + expect.num(_one(conn, "SELECT count(*) FROM tcl_plain"), 0, "TRUNCATE leaves no rows") + for _ in range(2): + _run(conn, LOAD.format(t="tcl_plain", n=5000), "TRUNCATE tcl_plain") + expect.text(_snapshot(conn), base, "three truncate-and-reload cycles still sit at the baseline") + # loaded first, so the DROP has work to do: a DROP of an empty table could not fail + _run(conn, LOAD.format(t="tcl_plain", n=5000)) + expect.text(_grew(base, _snapshot(conn)), "yes", + "premise: the table carries metadata again, so the DROP has work to do") + _run(conn, "DROP TABLE tcl_plain") + expect.text(_snapshot(conn), base, "DROP of a loaded table after TRUNCATE leaves nothing") + + +def test_a_projection_is_not_leaked_by_truncate(pgc_own_db, expect): + """A projection has its own storage id. What must hold is that no projection row + names a storage id the table no longer has, not that projection is empty.""" + conn = pgc_own_db + base = _snapshot(conn) + _run(conn, "CREATE TABLE tcl_proj (id int, a int, b text) USING pgcolumnar", + "SELECT pgcolumnar.add_projection('tcl_proj', 'tcl_p', ARRAY['a', 'b'], ARRAY['a'])", + "INSERT INTO tcl_proj SELECT g, g % 50, 'b' || g FROM generate_series(1, 5000) g") + expect.text(_grew(base, _snapshot(conn)), "yes", "the projection did add metadata") + _run(conn, "TRUNCATE tcl_proj") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.projection p " + "WHERE p.storage_id <> pgcolumnar.get_storage_id('tcl_proj'::regclass)"), 0, + "TRUNCATE leaves no projection row under a retired storage id") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.projection_declaration " + "WHERE rel = 'tcl_proj'::regclass"), 1, + "and the declaration survives the TRUNCATE, so the projection is rebuildable") + _run(conn, "INSERT INTO tcl_proj SELECT g, g % 50, 'b' || g FROM generate_series(1, 5000) g") + expect.text(_grew(base, _snapshot(conn)), "yes", "premise: the projected table carries metadata again before the DROP") + _run(conn, "DROP TABLE tcl_proj") + expect.text(_snapshot(conn), base, "DROP of a loaded projected table leaves nothing") + + +def test_repeated_truncates_accumulate_nothing(pgc_own_db, expect): + """Created once outside the loop and one statement at a time, so every TRUNCATE + takes the transactional path and any leak accumulates.""" + conn = pgc_own_db + base = _snapshot(conn) + _run(conn, "CREATE TABLE tcl_rep (id int, v text) USING pgcolumnar") + for _ in range(10): + _run(conn, LOAD.format(t="tcl_rep", n=2000), "TRUNCATE tcl_rep") + expect.text(_snapshot(conn), base, "ten insert-and-truncate cycles on one table leave nothing") + _run(conn, "DROP TABLE tcl_rep") + expect.text(_snapshot(conn), base, "and dropping it afterwards still leaves nothing") + + +def test_insert_truncate_insert_in_one_transaction(pgc_own_db, expect): + """The cached write state named the storage the TRUNCATE retired, so the second + INSERT flushed into storage nothing reads, and the transaction committed an empty + table. The status and the surviving rows are one arm: split, a commit that keeps + nothing passes the status half, and 1..10 passes a bare count of 10.""" + conn = pgc_own_db + _run(conn, "CREATE TABLE tc_txn (id int) USING pgcolumnar", + "INSERT INTO tc_txn SELECT g FROM generate_series(1, 10) g") + expect.num(_one(conn, "SELECT count(*) FROM tc_txn"), 10, "premise: the table holds its first ten rows") + rc = 0 + try: + _run(conn, "BEGIN", "INSERT INTO tc_txn SELECT g FROM generate_series(11, 20) g", "TRUNCATE tc_txn", + "INSERT INTO tc_txn SELECT g FROM generate_series(21, 30) g", "COMMIT") + except psycopg.Error: + rc = 1 + _run(conn, "ROLLBACK") + expect.text(f"rc={rc} " + _one(conn, "SELECT coalesce(min(id), -1) || '..' || coalesce(max(id), -1) || " + "' n=' || count(*) FROM tc_txn"), + "rc=0 21..30 n=10", "INSERT, TRUNCATE, INSERT in one transaction commits and keeps the new rows") + + +def test_a_rewrite_that_is_not_a_truncate_loses_nothing(pgc_own_db, expect): + """The retire runs from relation_set_new_filelocator, which every rewrite calls.""" + conn = pgc_own_db + _run(conn, "CREATE TABLE tc_rw (id int, v text) USING pgcolumnar", + "INSERT INTO tc_rw SELECT g, 'v' || g FROM generate_series(1, 5000) g") + expect.num(_one(conn, "SELECT count(*) FROM tc_rw"), 5000, "premise: five thousand rows before the rewrite") + _run(conn, "ALTER TABLE tc_rw ALTER COLUMN v TYPE varchar(64)") + expect.text(_one(conn, "SELECT count(*) || '/' || count(v) || '/' || max(id) FROM tc_rw"), "5000/5000/5000", + "a full rewrite keeps every row, and they are still readable") + + +def test_an_in_place_truncate_clears_the_projection_too(pgc_own_db, expect): + """A relation created in the current transaction is truncated in place, through + nontransactional_truncate, which cleared the base storage and left the + projection's. The write after the truncate is what makes the leak observable: it + collided on row_group_pkey (#896).""" + conn = pgc_own_db + _run(conn, "CREATE TABLE tcl_inplace_pre (id int)") + committed = "committed" + try: + _run(conn, "BEGIN", + "CREATE TABLE tcl_inplace (id int, a int, b text) USING pgcolumnar", + "INSERT INTO tcl_inplace SELECT g, g % 50, 'b' || g FROM generate_series(1, 2000) g", + "SELECT pgcolumnar.add_projection('tcl_inplace', 'tcl_ip', ARRAY['a', 'b'], ARRAY['a'])", + "TRUNCATE tcl_inplace", + "INSERT INTO tcl_inplace SELECT g, g % 50, 'c' || g FROM generate_series(1, 500) g", + "COMMIT") + except psycopg.Error as exc: + committed = f"rolled back ({str(exc).splitlines()[0]})" + _run(conn, "ROLLBACK") + # vacuous without these: a rolled-back transaction leaves no table to count + expect.num(0 if committed == "committed" else 1, 0, "PREMISE the in-place transaction committed") + expect.num(_one(conn, "SELECT count(*) FROM pg_class WHERE relname = 'tcl_inplace'"), 1, + "PREMISE the table exists after it") + if committed != "committed": + return + expect.num(_one(conn, "SELECT count(*) FROM tcl_inplace"), 500, + "the rows written after the in-place truncate are all there") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('tcl_inplace', 'tcl_ip')"), 500, + "and the projection reads them back") + with conn.cursor() as cur: + cur.execute("SELECT pgcolumnar.read_projection('tcl_inplace', 'tcl_ip')") + proj = cur.fetchall() + cur.execute("SELECT a::text || '|' || b FROM tcl_inplace") + base = cur.fetchall() + expect.row_set(proj, base, "the projection agrees with the base table") + proj_storage = _one(conn, "SELECT proj_storage_id FROM pgcolumnar.projection " + "WHERE storage_id = pgcolumnar.get_storage_id('tcl_inplace'::regclass) " + "AND projection_id > 0") + base_storage = _one(conn, "SELECT pgcolumnar.get_storage_id('tcl_inplace'::regclass)") + expect.text("yes" if proj_storage is not None and proj_storage != base_storage else "no", "yes", + "PREMISE the projection has its own storage, distinct from the base") + # one group of 500 rows, not a group NUMBER: the base's counter is reset and the + # projection's has no metapage to reset, so the numbers differ by design + expect.num(_one(conn, f"SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = {proj_storage}"), 1, + "the projection storage holds exactly one row group after the truncate") + expect.num(_one(conn, f"SELECT coalesce(sum(row_count), 0) FROM pgcolumnar.row_group " + f"WHERE storage_id = {proj_storage}"), 500, + "and that group describes only the rows written after it") From 50a8c31f557526b412b3fad4012b36579d75cee1 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:05 +0000 Subject: [PATCH 05/36] test: truncate_cleanup seeds the ledger, 16 of 25 arms dated by four mutations of the truncate paths (#1341) 25 rows from five-major runs (25/25 on each of 15-19). Dated on PG18 by a rewrite that keeps the retired storage's rows (7) or its stale write state (1), an in-place truncate that skips the projections' storages (8), and a storage-tree delete that skips them (1). suites_not_covered 218 -> 217; the census is 3672 rows, 2695 never, 977 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 25 +++++++++++++++++++++++++ test/check_ledger_budget.txt | 19 +++++++++++++++++-- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 2c094ebec..714a816d1 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -3369,6 +3369,31 @@ timeseries_recipes timeseries_recipes the how-to gives the LOCF recipe 15;16;17; timeseries_recipes timeseries_recipes the how-to gives the gap-fill recipe 15;16;17;18;19 2026-10-13 the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found timeseries_recipes timeseries_recipes the how-to gives the interpolation recipe 15;16;17;18;19 2026-10-13 the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found timeseries_recipes timeseries_recipes the interpolation recipe fills a gap linearly, and leaves both ends NULL 15;16;17;18;19 2026-10-13 the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found;the how-to's interpolation recipe uses prev_v in place of next_v, so every gap is filled flat;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN +truncate_cleanup truncate_cleanup DROP of a loaded projected table leaves nothing 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows;columnar_tableam.c: deleting a storage tree no longer reaches its projections' storages +truncate_cleanup truncate_cleanup DROP of a loaded table after TRUNCATE leaves nothing 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows +truncate_cleanup truncate_cleanup INSERT, TRUNCATE, INSERT in one transaction commits and keeps the new rows 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer drops the cached write state of the retired storage +truncate_cleanup truncate_cleanup PREMISE the in-place transaction committed 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup PREMISE the projection has its own storage, distinct from the base 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup PREMISE the table exists after it 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup TRUNCATE leaves no projection row under a retired storage id 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows +truncate_cleanup truncate_cleanup TRUNCATE leaves no rows 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup TRUNCATE returns the catalog to the empty baseline 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows +truncate_cleanup truncate_cleanup a full rewrite keeps every row, and they are still readable 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup and dropping it afterwards still leaves nothing 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows +truncate_cleanup truncate_cleanup and that group describes only the rows written after it 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup and the declaration survives the TRUNCATE, so the projection is rebuildable 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup and the projection reads them back 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup insert did add metadata 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup premise: five thousand rows before the rewrite 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup premise: the projected table carries metadata again before the DROP 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup premise: the table carries metadata again, so the DROP has work to do 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup premise: the table holds its first ten rows 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup ten insert-and-truncate cycles on one table leave nothing 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows +truncate_cleanup truncate_cleanup the projection agrees with the base table 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup the projection did add metadata 15;16;17;18;19 never - +truncate_cleanup truncate_cleanup the projection storage holds exactly one row group after the truncate 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup the rows written after the in-place truncate are all there 15;16;17;18;19 2026-10-16 columnar_tableam.c: the in-place truncate no longer clears a projection's own storage +truncate_cleanup truncate_cleanup three truncate-and-reload cycles still sit at the baseline 15;16;17;18;19 2026-10-16 columnar_tableam.c: a rewrite no longer deletes the retired storage's catalog rows ttl_expire ttl_expire NO row still inside the retention was dropped (#403 item 5a) 15;16;17;18;19 never - ttl_expire ttl_expire NO row still inside the retention was dropped, on a date column 15;16;17;18;19 never - ttl_expire ttl_expire a compaction's retired groups are not recorded as retention 15;16;17;18;19 2026-10-13 the early exit compares the extent's maximum to the cutoff instead of its minimum diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 42a7cd12d..a03e1dfcc 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 218 +suites_not_covered 217 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2961,4 +2961,19 @@ suites_not_covered 218 # never 2675 + 11 = 2686 # dated 944 + 17 = 961 # 2686 + 961 == 3647 -checks_never_observed_red 2686 +# +# #1341, truncate_cleanup seeded, with its twin (OffgridwithJD): 2686 -> 2695, and +# suites_not_covered 218 -> 217. The suite had no rows. It enters with all 25 checks +# from five-major runs (25/25 on each of 15-19). 16 are dated by four mutations of +# the truncate paths: a rewrite that no longer deletes the retired storage's rows +# (7), nor drops the stale write state (1), an in-place truncate that skips the +# projections' storages (8), and a storage-tree delete that skips them (1). Nine +# arrive never: the premises that each fixture holds rows or metadata, the +# surviving projection declaration, the empty table after TRUNCATE, and the +# rewrite that keeps every row. +# +# ledger rows 3647 + 25 = 3672 +# never 2686 + 9 = 2695 +# dated 961 + 16 = 977 +# 2695 + 977 == 3672 +checks_never_observed_red 2695 From c0f426372c34ebf1f88929d825d535316926337c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:06 +0000 Subject: [PATCH 06/36] test: native_dml gets its pytest twin, and its min/max arm can see a zone-map answer that ignores deletes (#1341) test_native_dml.py asserts every shell check by name; compare_to_bash grades it as covering all 15, and it joins COMPLETE. The min/max arm could not fail as its comment claimed. The deletes (id % 7 = 0) removed neither the smallest nor the largest id, so min and max were the same whether or not the deleted rows counted; and measured with groups that hold deletes left on the zone-map path, min/max(id) still came out right while min/max(k) came out wrong (10|50000 for 20|49990). The suite stayed 13/13. The deletes now include ids 1 and 5000, a premise asserts on heap that min and max moved, and the arm takes min/max over k as well as id; under that mutant both twins fail it. The index arms gain a premise that the lookups take the primary-key index, so they test the index fetch's delete check rather than the scan. cluster_tests 672 -> 673 by collection; TESTS.md gets section 146. Co-Authored-By: Claude Opus 5.5 --- test/native_dml.sh | 22 +++++-- test/pytest/TESTS.md | 11 ++++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_dml.py | 92 +++++++++++++++++++++++++++++ 5 files changed, 124 insertions(+), 6 deletions(-) create mode 100644 test/pytest/test_native_dml.py diff --git a/test/native_dml.sh b/test/native_dml.sh index 9e67578b6..ae68bc317 100755 --- a/test/native_dml.sh +++ b/test/native_dml.sh @@ -21,9 +21,15 @@ GEN="SELECT g, (g * 10)::bigint, 'row-' || g FROM generate_series(1, 5000) g" psql_run "INSERT INTO h $GEN;" psql_run "INSERT INTO n $GEN;" -# DELETE across several row groups. -psql_run "DELETE FROM h WHERE id % 7 = 0;" -psql_run "DELETE FROM n WHERE id % 7 = 0;" +# DELETE across several row groups, and the table's smallest and largest ids with +# them. Without those two, min and max are the same whether or not the deleted rows +# count, so the min/max arm below could not see a zone-map answer that ignored the +# deletes (#1341): measured, with groups that hold deletes left on the zone-map path, +# this file stayed 13/13. +psql_run "DELETE FROM h WHERE id % 7 = 0 OR id IN (1, 5000);" +psql_run "DELETE FROM n WHERE id % 7 = 0 OR id IN (1, 5000);" +check "premise: the deletes moved the smallest and largest id and k" \ + "$(q 'SELECT min(id) || '"'"'|'"'"' || max(id) || '"'"'|'"'"' || min(k) || '"'"'|'"'"' || max(k) FROM h;')" "2|4999|20|49990" check "count after delete" "$(q 'SELECT count(*) FROM n;')" "$(q 'SELECT count(*) FROM h;')" check "rows after delete match heap" \ @@ -32,14 +38,20 @@ check "rows after delete match heap" \ # Aggregates after delete: the zone-map path is bypassed (deletes present), so this # exercises the delete-applying scan fold. check "sum(k) after delete" "$(q 'SELECT sum(k) FROM n;')" "$(q 'SELECT sum(k) FROM h;')" +# k as well as id: measured, a zone-map answer that ignored the deletes still gave +# min/max(id) right and min/max(k) wrong (10|50000 for 20|49990). check "min/max after delete" \ - "$(q 'SELECT min(id), max(id) FROM n;')" \ - "$(q 'SELECT min(id), max(id) FROM h;')" + "$(q 'SELECT min(id), max(id), min(k), max(k) FROM n;')" \ + "$(q 'SELECT min(id), max(id), min(k), max(k) FROM h;')" check "deleted rows gone" "$(q 'SELECT count(*) FROM n WHERE id % 7 = 0;')" "0" # Index lookup of a deleted row returns nothing; a live row still resolves. idx() { env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -q -At -c "SET enable_seqscan=off" -c "$1" 2>/dev/null; } # id 4444 is live (4444 % 7 = 6); id 4445 was deleted (4445 % 7 = 0). +# The lookups must take the primary-key index, or these arms test the scan instead +# of the index fetch's delete check. +check "premise: the lookups take the primary-key index" \ + "$(idx 'EXPLAIN (COSTS off) SELECT count(*) FROM n WHERE id = 4445;' | grep -c 'Index.* using n_pkey')" "1" check "index finds live row" "$(idx 'SELECT v FROM n WHERE id = 4444;')" "$(q 'SELECT v FROM h WHERE id = 4444;')" check "index skips deleted row" "$(idx 'SELECT count(*) FROM n WHERE id = 4445;')" "0" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 95dee4a68..a855303d9 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -191,6 +191,7 @@ behaviour, the source of that number is named. - [143. test_native_upgrade_converge.py: the upgrade window, before ALTER EXTENSION UPDATE](#143-test_native_upgrade_convergepy-the-upgrade-window-before-alter-extension-update) - [144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen](#144-test_fk_referencingpy-a-foreign-key-referencing-a-columnar-table-is-refused-where-it-is-chosen) - [145. test_truncate_cleanup.py: TRUNCATE leaves no catalog rows behind, and keeps what it must](#145-test_truncate_cleanuppy-truncate-leaves-no-catalog-rows-behind-and-keeps-what-it-must) +- [146. test_native_dml.py: DELETE, UPDATE and MERGE on a native table agree with heap](#146-test_native_dmlpy-delete-update-and-merge-on-a-native-table-agree-with-heap) ## 1. How to read a test in here @@ -7419,3 +7420,13 @@ The pytest twin of `test/truncate_cleanup.sh` (ported for #1341), every check by | `test_insert_truncate_insert_in_one_transaction` | the transaction commits and holds exactly the rows written after the TRUNCATE | | `test_a_rewrite_that_is_not_a_truncate_loses_nothing` | a type-changing rewrite keeps all 5000 rows readable | | `test_an_in_place_truncate_clears_the_projection_too` | the in-place truncate commits, the later rows are all there, the projection reads them back and its storage holds one group of exactly them | + +## 146. test_native_dml.py: DELETE, UPDATE and MERGE on a native table agree with heap + +The pytest twin of `test/native_dml.sh` (ported for #1341), every check by the shell's name. Each statement runs on a heap table and on a columnar one with 1000-row stripes, and every result is compared. After deletes an aggregate must not come from zone maps that still describe the deleted rows: the deletes include the smallest and largest ids, so min and max move, and they are taken over `k`, a column the primary key does not order. A deleted row's index lookup finds nothing, and the lookups are asserted to take the index. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_delete_update_and_merge_match_heap` | after DELETE, UPDATE and MERGE, counts, rows, sum and min/max match heap; the index finds a live row and skips a deleted one; the primary key still rejects a duplicate | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index fd24e418c..383f58447 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -753,4 +753,6 @@ guard_tests 438 # five tests, by collection over the cluster files on this tree. # cluster_tests 666 -> 672 for #1341's test_truncate_cleanup.py, the twin of truncate_cleanup.sh: # six tests, by collection over the cluster files on this tree. -cluster_tests 672 +# cluster_tests 672 -> 673 for #1341's test_native_dml.py, the twin of native_dml.sh: one test, +# by collection over the cluster files on this tree. +cluster_tests 673 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 185ee0529..42b4f5f1f 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -122,6 +122,7 @@ "native_chunk_length_bound", "native_delete_vector_index", "native_delete_visibility_paths", + "native_dml", "native_fetch_coalesce", "native_groupagg", "native_index_fetch_stripe_cost", diff --git a/test/pytest/test_native_dml.py b/test/pytest/test_native_dml.py new file mode 100644 index 000000000..3a97bde8a --- /dev/null +++ b/test/pytest/test_native_dml.py @@ -0,0 +1,92 @@ +"""DELETE, UPDATE and MERGE on a native-format columnar table agree with a heap mirror +(#1341). The pytest twin of `test/native_dml.sh`, every check by the shell's name. + +Each statement runs on a heap table and on a columnar one with stripes of 1000 rows, +and every result is compared: the surviving rows, counts, aggregates and index lookups. +After deletes, an aggregate must not be answered from zone maps that still describe the +deleted rows. The deletes include the table's smallest and largest ids, so min and max +move and an answer that ignored the deletes differs from heap's; min and max are taken +over k too, a column the primary key does not order. An index lookup of a deleted row +must find nothing, and the lookups are asserted to take the index. +""" + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +def _row(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone() + + +def _rows(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchall() + + +def _run(conn, *sqls): + with conn.cursor() as cur: + for sql in sqls: + cur.execute(sql) + + +GEN = "SELECT g, (g * 10)::bigint, 'row-' || g FROM generate_series(1, 5000) g" + + +def test_delete_update_and_merge_match_heap(pgc_conn, expect): + conn = pgc_conn + _run(conn, "CREATE TABLE h (id int PRIMARY KEY, k bigint, v text)", + "CREATE TABLE n (id int PRIMARY KEY, k bigint, v text) USING pgcolumnar", + "SELECT pgcolumnar.set_options('n', stripe_row_limit => 1000)", + f"INSERT INTO h {GEN}", f"INSERT INTO n {GEN}") + + for t in ("h", "n"): + _run(conn, f"DELETE FROM {t} WHERE id % 7 = 0 OR id IN (1, 5000)") + expect.text(_one(conn, "SELECT min(id) || '|' || max(id) || '|' || min(k) || '|' || max(k) FROM h"), + "2|4999|20|49990", "premise: the deletes moved the smallest and largest id and k") + expect.num(_one(conn, "SELECT count(*) FROM n"), _one(conn, "SELECT count(*) FROM h"), "count after delete") + expect.row_set(_rows(conn, "SELECT id, k, v FROM n"), _rows(conn, "SELECT id, k, v FROM h"), + "rows after delete match heap") + expect.num(_one(conn, "SELECT sum(k) FROM n"), _one(conn, "SELECT sum(k) FROM h"), "sum(k) after delete") + expect.rows([_row(conn, "SELECT min(id), max(id), min(k), max(k) FROM n")], + [_row(conn, "SELECT min(id), max(id), min(k), max(k) FROM h")], "min/max after delete") + expect.num(_one(conn, "SELECT count(*) FROM n WHERE id % 7 = 0"), 0, "deleted rows gone") + + # id 4444 is live (4444 % 7 = 6); id 4445 was deleted (4445 % 7 = 0) + _run(conn, "SET enable_seqscan = off") + plan = "\n".join(r[0] for r in _rows(conn, "EXPLAIN (COSTS off) SELECT count(*) FROM n WHERE id = 4445")) + expect.num(sum(1 for line in plan.splitlines() if "Index" in line and "using n_pkey" in line), 1, + "premise: the lookups take the primary-key index") + expect.text(_one(conn, "SELECT v FROM n WHERE id = 4444"), _one(conn, "SELECT v FROM h WHERE id = 4444"), + "index finds live row") + expect.num(_one(conn, "SELECT count(*) FROM n WHERE id = 4445"), 0, "index skips deleted row") + _run(conn, "RESET enable_seqscan") + + for t in ("h", "n"): + _run(conn, f"UPDATE {t} SET k = k + 1 WHERE id % 5 = 0") + expect.num(_one(conn, "SELECT count(*) FROM n"), _one(conn, "SELECT count(*) FROM h"), "count after update") + expect.row_set(_rows(conn, "SELECT id, k, v FROM n"), _rows(conn, "SELECT id, k, v FROM h"), + "rows after update match heap") + expect.num(_one(conn, "SELECT sum(k) FROM n"), _one(conn, "SELECT sum(k) FROM h"), "sum(k) after update") + + import psycopg + try: + _run(conn, "INSERT INTO n VALUES (4444, 0, 'x')") + dup = "accepted" + except psycopg.Error as exc: + dup = exc.sqlstate + expect.text(dup, "23505", "PK still enforced after DML") + + _run(conn, "CREATE TABLE src (id int, k bigint)", "INSERT INTO src VALUES (1, 999), (999001, 5), (999002, 6)") + for t in ("h", "n"): + _run(conn, f"MERGE INTO {t} t USING src s ON t.id = s.id " + "WHEN MATCHED THEN UPDATE SET k = s.k " + "WHEN NOT MATCHED THEN INSERT (id, k, v) VALUES (s.id, s.k, 'm')") + expect.num(_one(conn, "SELECT count(*) FROM n"), _one(conn, "SELECT count(*) FROM h"), "count after merge") + expect.row_set(_rows(conn, "SELECT id, k, v FROM n"), _rows(conn, "SELECT id, k, v FROM h"), + "rows after merge match heap") From 091f98267ab3805ec971ab85d3baec0b1be4bf5c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:06 +0000 Subject: [PATCH 07/36] test: native_dml seeds the ledger, 8 of 15 arms dated by three mutations (#1341) 15 rows from five-major runs (15/15 on each of 15-19). Dated on PG18 by: a row group with deletes still answering min/max from zone maps (1), count(*) from metadata not subtracting deleted rows (3), and the index fetch ignoring the delete vector (5). suites_not_covered 217 -> 216; the census is 3687 rows, 2702 never, 985 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 15 +++++++++++++++ test/check_ledger_budget.txt | 18 ++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 714a816d1..7696edfc8 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2316,6 +2316,21 @@ native_delete_visibility_paths native_delete_visibility_paths aggregate over the native_delete_visibility_paths native_delete_visibility_paths index / bitmap scan sees the live set 15;16;17;18;19 never - native_delete_visibility_paths native_delete_visibility_paths index-only scan sees the live set 15;16;17;18;19 never - native_delete_visibility_paths native_delete_visibility_paths sequential scan sees the live set (row-emitting path) 15;16;17;18;19 never - +native_dml native_dml PK still enforced after DML 15;16;17;18;19 never - +native_dml native_dml count after delete 15;16;17;18;19 2026-10-16 columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows +native_dml native_dml count after merge 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector;columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows +native_dml native_dml count after update 15;16;17;18;19 2026-10-16 columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows +native_dml native_dml deleted rows gone 15;16;17;18;19 never - +native_dml native_dml index finds live row 15;16;17;18;19 never - +native_dml native_dml index skips deleted row 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector +native_dml native_dml min/max after delete 15;16;17;18;19 2026-10-16 columnar_vector.c: a row group holding deletes still answers min and max from its zone maps +native_dml native_dml premise: the deletes moved the smallest and largest id and k 15;16;17;18;19 never - +native_dml native_dml premise: the lookups take the primary-key index 15;16;17;18;19 never - +native_dml native_dml rows after delete match heap 15;16;17;18;19 never - +native_dml native_dml rows after merge match heap 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector +native_dml native_dml rows after update match heap 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector +native_dml native_dml sum(k) after delete 15;16;17;18;19 never - +native_dml native_dml sum(k) after update 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector native_fetch_coalesce native_fetch_coalesce a wide index fetch does not pin once per column 15;16;17;18;19 2026-10-13 the coalesced fetch merging no adjacent ranges, so each column is read on its own native_fetch_coalesce native_fetch_coalesce premise: a point lookup uses the index 15;16;17;18;19 never - native_fetch_coalesce native_fetch_coalesce premise: fetching every projected column touched a measurable number of buffers 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index a03e1dfcc..95506368c 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 217 +suites_not_covered 216 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2976,4 +2976,18 @@ suites_not_covered 217 # never 2686 + 9 = 2695 # dated 961 + 16 = 977 # 2695 + 977 == 3672 -checks_never_observed_red 2695 +# +# #1341, native_dml seeded, with its twin (OffgridwithJD): 2695 -> 2702, and +# suites_not_covered 217 -> 216. The suite had no rows. It enters with all 15 checks +# from five-major runs (15/15 on each of 15-19). 8 are dated by three mutations: a +# row group holding deletes still answering min and max from its zone maps (the +# min/max arm, which could not see it before this change), count(*) from metadata +# not subtracting deleted rows (3), and the index fetch ignoring the delete vector +# (5). Seven arrive never: the two premises, the rows and sum after the delete, the +# deleted rows being gone, the live row's index lookup, and the primary key arm. +# +# ledger rows 3672 + 15 = 3687 +# never 2695 + 7 = 2702 +# dated 977 + 8 = 985 +# 2702 + 985 == 3687 +checks_never_observed_red 2702 From 25ce0efeccb93f35ccf3ab4c09634cd947c3eb88 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:06 +0000 Subject: [PATCH 08/36] test: native_dict_underfill gets its pytest twin, and a crash can no longer pass its survival arm (#1341) test_native_dict_underfill.py asserts every shell check by name; compare_to_bash grades it as covering all 4, and it joins COMPLETE. It turns the post-codec encoding choice off for its own session, where the shell sets it for its database. The survival arm was a fresh SELECT 1, which a crashed backend passes once crash recovery has finished, and the suite computed a crash flag it never checked. It now counts "terminated by signal" in the server log. The rejection arm matched 'does not match|corrupt|invalid|out of' in any message, which "out of memory" satisfies too; it now requires XX001. A premise asserts the inflation rewrote the chunk's descriptor. cluster_tests 673 -> 674 by collection; TESTS.md gets section 147. Co-Authored-By: Claude Opus 5.5 --- test/native_dict_underfill.sh | 30 ++++++---- test/pytest/TESTS.md | 11 ++++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_dict_underfill.py | 70 +++++++++++++++++++++++ 5 files changed, 104 insertions(+), 12 deletions(-) create mode 100644 test/pytest/test_native_dict_underfill.py diff --git a/test/native_dict_underfill.sh b/test/native_dict_underfill.sh index bf32d082c..b5cf5bf8e 100755 --- a/test/native_dict_underfill.sh +++ b/test/native_dict_underfill.sh @@ -27,23 +27,31 @@ echo "-- v-column vector0 encoding type byte (DICT=6): $ENC" check "the text column is DICT-encoded (precondition)" "$ENC" "6" # read old rawLen (uint32 LE at descriptor offset 11), inflate by 64, write back -q "UPDATE pgcolumnar.column_chunk c SET encoding_descriptor = +crashes() { + # grep -c prints 0 itself when nothing matches, and then exits 1 + pgc_pg "grep -c 'terminated by signal' '$PGC_LOGFILE'" 2>/dev/null || true +} +crashes0="$(crashes)" +q "WITH u AS (UPDATE pgcolumnar.column_chunk c SET encoding_descriptor = set_byte(set_byte(set_byte(set_byte(encoding_descriptor, 11, ( (get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64 ) & 255), 12, (((get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64) >> 8) & 255), 13, (((get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64) >>16) & 255), 14, (((get_byte(encoding_descriptor,11) + (get_byte(encoding_descriptor,12)<<8) + (get_byte(encoding_descriptor,13)<<16) + (get_byte(encoding_descriptor,14)<<24)) + 64) >>24) & 255) - WHERE c.storage_id=$SID AND c.column_index=1" >/dev/null + WHERE c.storage_id=$SID AND c.column_index=1 RETURNING 1) SELECT count(*) FROM u" > "$PGC_WORKDIR/uf.n" +check "premise: the inflation rewrote the DICT chunk's descriptor" "$(cat "$PGC_WORKDIR/uf.n")" "1" env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" \ - -c "SELECT v FROM dt" >/dev/null 2>"$PGC_WORKDIR/uf.err" + -v VERBOSITY=verbose -c "SELECT v FROM dt" >/dev/null 2>"$PGC_WORKDIR/uf.err" echo "-- select stderr:"; sed 's/^/ /' "$PGC_WORKDIR/uf.err" | head -3 -rejected=no -grep -qiE 'does not match|corrupt|invalid|out of' "$PGC_WORKDIR/uf.err" && rejected=yes -crash=no -grep -qiE 'terminated by signal|segmentation|server closed' "$PGC_WORKDIR/uf.err" && crash=yes -grep -qiE 'terminated by signal|segmentation' "$PGC_LOGFILE" 2>/dev/null && crash=yes -echo "-- rejected=$rejected crash=$crash" -check "under-filled DICT chunk is rejected with a clean error" "$rejected" "yes" -check "backend survives the under-filled DICT chunk" "$(q 'SELECT 1')" "1" +# A CLEAN ERROR IS XX001, AND A CRASH IS READ FROM THE SERVER LOG (#1341). The +# rejection used to match 'does not match|corrupt|invalid|out of' in any message, +# which "out of memory" satisfies too; and survival was a fresh SELECT 1, which a +# crashed backend passes once crash recovery has finished. The crash flag below was +# computed and never checked. +rejected="$(sed -nE 's/^ERROR: ([0-9A-Z]{5}): .*/\1/p' "$PGC_WORKDIR/uf.err" | head -1 | grep . || echo "no SQLSTATE ($(head -1 "$PGC_WORKDIR/uf.err"))")" +echo "-- rejected=$rejected crashes=$(( $(crashes) - crashes0 ))" +check "under-filled DICT chunk is rejected with a clean error" "$rejected" "XX001" +check "backend survives the under-filled DICT chunk" \ + "$(q 'SELECT 1')|$(( $(crashes) - crashes0 )) backend crashes" "1|0 backend crashes" pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index a855303d9..35536ff8b 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -192,6 +192,7 @@ behaviour, the source of that number is named. - [144. test_fk_referencing.py: a foreign key referencing a columnar table is refused where it is chosen](#144-test_fk_referencingpy-a-foreign-key-referencing-a-columnar-table-is-refused-where-it-is-chosen) - [145. test_truncate_cleanup.py: TRUNCATE leaves no catalog rows behind, and keeps what it must](#145-test_truncate_cleanuppy-truncate-leaves-no-catalog-rows-behind-and-keeps-what-it-must) - [146. test_native_dml.py: DELETE, UPDATE and MERGE on a native table agree with heap](#146-test_native_dmlpy-delete-update-and-merge-on-a-native-table-agree-with-heap) +- [147. test_native_dict_underfill.py: an under-filled DICT chunk is refused and no backend dies](#147-test_native_dict_underfillpy-an-under-filled-dict-chunk-is-refused-and-no-backend-dies) ## 1. How to read a test in here @@ -7430,3 +7431,13 @@ The pytest twin of `test/native_dml.sh` (ported for #1341), every check by the s | test | what it holds | | --- | --- | | `test_delete_update_and_merge_match_heap` | after DELETE, UPDATE and MERGE, counts, rows, sum and min/max match heap; the index finds a live row and skips a deleted one; the primary key still rejects a duplicate | + +## 147. test_native_dict_underfill.py: an under-filled DICT chunk is refused and no backend dies + +The pytest twin of `test/native_dict_underfill.sh` (ported for #1341), every check by the shell's name. A DICT chunk's `value_raw_length` is inflated in its descriptor, so the codes decode to fewer bytes than declared; without the fill check the raw buffer's tail is uninitialized and a varlena length is read out of it. The read must raise XX001, and survival is read from the server log's crash count, not from a fresh connection, which succeeds once crash recovery has finished. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_an_under_filled_dict_chunk_is_refused` | the column is DICT-encoded, the inflation rewrites its descriptor, the read is refused with XX001, and no backend died | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 383f58447..641b37863 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -755,4 +755,6 @@ guard_tests 438 # six tests, by collection over the cluster files on this tree. # cluster_tests 672 -> 673 for #1341's test_native_dml.py, the twin of native_dml.sh: one test, # by collection over the cluster files on this tree. -cluster_tests 673 +# cluster_tests 673 -> 674 for #1341's test_native_dict_underfill.py, the twin of +# native_dict_underfill.sh: one test, by collection over the cluster files on this tree. +cluster_tests 674 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 42b4f5f1f..fe9b9ae8f 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -122,6 +122,7 @@ "native_chunk_length_bound", "native_delete_vector_index", "native_delete_visibility_paths", + "native_dict_underfill", "native_dml", "native_fetch_coalesce", "native_groupagg", diff --git a/test/pytest/test_native_dict_underfill.py b/test/pytest/test_native_dict_underfill.py new file mode 100644 index 000000000..98c3f2c4f --- /dev/null +++ b/test/pytest/test_native_dict_underfill.py @@ -0,0 +1,70 @@ +"""An under-filled DICT chunk is refused cleanly, and no backend dies (#1341). The pytest +twin of `test/native_dict_underfill.sh`, every check by the shell's name. + +The test inflates a DICT chunk's value_raw_length in its encoding descriptor, so the +codes decode to fewer bytes than declared. Without the fill check the raw buffer's +tail is left uninitialized and a varlena length is read out of it. The read must raise +XX001, and the crash is read from the server log, since a fresh connection succeeds +again once crash recovery has finished. + +The post-codec encoding choice (#1132) is turned off for this session, as the shell +does for its database, so the fixture is DICT-encoded whether or not DICT beats zstd. +""" +import psycopg + + +def _crashes(pgc_cluster): + try: + return (pgc_cluster.datadir / "server.log").read_text(errors="replace").count("terminated by signal") + except OSError: + return 0 + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +RAW = ("(get_byte(encoding_descriptor, 11) + (get_byte(encoding_descriptor, 12) << 8) + " + "(get_byte(encoding_descriptor, 13) << 16) + (get_byte(encoding_descriptor, 14) << 24) + 64)") + + +def test_an_under_filled_dict_chunk_is_refused(pgc_conn, pgc_cluster, expect): + conn = pgc_conn + with conn.cursor() as cur: + cur.execute("SET pgcolumnar.enable_post_codec_encoding_choice = off") + cur.execute("CREATE TABLE dt (id int, v text) USING pgcolumnar") + # low-cardinality short text: a DICT-encoded varlena column, one small chunk + cur.execute("INSERT INTO dt SELECT g, (ARRAY['aa', 'bb', 'cc', 'dd'])[1 + (g % 4)] " + "FROM generate_series(1, 300) g") + sid = _one(conn, "SELECT pgcolumnar.get_storage_id('dt')") + # column_index is 0-based: v is attribute 2, index 1; byte 6 is vector 0's encoding type + expect.num(_one(conn, f"SELECT get_byte(encoding_descriptor, 6) FROM pgcolumnar.column_chunk " + f"WHERE storage_id = {sid} AND column_index = 1 LIMIT 1"), 6, + "the text column is DICT-encoded (precondition)") + + crashes = _crashes(pgc_cluster) + # rawLen is the uint32 LE at descriptor offset 11: raise it by 64 + rewrote = _one(conn, f"""WITH u AS (UPDATE pgcolumnar.column_chunk c SET encoding_descriptor = + set_byte(set_byte(set_byte(set_byte(encoding_descriptor, + 11, {RAW} & 255), 12, ({RAW} >> 8) & 255), 13, ({RAW} >> 16) & 255), 14, ({RAW} >> 24) & 255) + WHERE c.storage_id = {sid} AND c.column_index = 1 RETURNING 1) SELECT count(*) FROM u""") + expect.num(rewrote, 1, "premise: the inflation rewrote the DICT chunk's descriptor") + + try: + with psycopg.connect(pgc_cluster.dsn(), autocommit=True) as c: + c.execute(f"SET search_path TO {conn.execute('SELECT current_schema()').fetchone()[0]}") + c.execute("SELECT v FROM dt").fetchall() + rejected = "no error" + except psycopg.Error as exc: + rejected = exc.sqlstate or f"no SQLSTATE ({str(exc).splitlines()[0]})" + expect.text(rejected, "XX001", "under-filled DICT chunk is rejected with a clean error") + + try: + with psycopg.connect(pgc_cluster.dsn(), autocommit=True) as c: + alive = c.execute("SELECT 1").fetchone()[0] == 1 + except psycopg.Error: + alive = False + expect.text(f"{'1' if alive else ''}|{_crashes(pgc_cluster) - crashes} backend crashes", "1|0 backend crashes", + "backend survives the under-filled DICT chunk") From 2548d989f6bdeb6ddc93c66dc6ca2832b00ea731 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:06 +0000 Subject: [PATCH 09/36] test: native_dict_underfill seeds the ledger, 2 of 4 arms dated (#1341) 4 rows from five-major runs (4/4 on each of 15-19). Dated on PG18 by the DICT fill check removed (1) and replaced by a crash (2, one shared). suites_not_covered 216 -> 215; the census is 3691 rows, 2704 never, 987 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 4 ++++ test/check_ledger_budget.txt | 16 ++++++++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 7696edfc8..e9f4d490f 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2316,6 +2316,10 @@ native_delete_visibility_paths native_delete_visibility_paths aggregate over the native_delete_visibility_paths native_delete_visibility_paths index / bitmap scan sees the live set 15;16;17;18;19 never - native_delete_visibility_paths native_delete_visibility_paths index-only scan sees the live set 15;16;17;18;19 never - native_delete_visibility_paths native_delete_visibility_paths sequential scan sees the live set (row-emitting path) 15;16;17;18;19 never - +native_dict_underfill native_dict_underfill backend survives the under-filled DICT chunk 15;16;17;18;19 2026-10-16 columnar_encoding.c: the DICT fill refusal replaced by a backend crash +native_dict_underfill native_dict_underfill premise: the inflation rewrote the DICT chunk's descriptor 15;16;17;18;19 never - +native_dict_underfill native_dict_underfill the text column is DICT-encoded (precondition) 15;16;17;18;19 never - +native_dict_underfill native_dict_underfill under-filled DICT chunk is rejected with a clean error 15;16;17;18;19 2026-10-16 columnar_encoding.c: a DICT chunk whose codes do not fill the raw length is no longer refused;columnar_encoding.c: the DICT fill refusal replaced by a backend crash native_dml native_dml PK still enforced after DML 15;16;17;18;19 never - native_dml native_dml count after delete 15;16;17;18;19 2026-10-16 columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows native_dml native_dml count after merge 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector;columnar_vector.c: count(*) from metadata no longer subtracts a group's deleted rows diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 95506368c..d3cc593b9 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 216 +suites_not_covered 215 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2990,4 +2990,16 @@ suites_not_covered 216 # never 2695 + 7 = 2702 # dated 977 + 8 = 985 # 2702 + 985 == 3687 -checks_never_observed_red 2702 +# +# #1341, native_dict_underfill seeded, with its twin (OffgridwithJD): 2702 -> 2704, +# and suites_not_covered 216 -> 215. The suite had no rows. It enters with all 4 +# checks from five-major runs (4/4 on each of 15-19). 2 are dated by two mutations +# of the DICT fill check: removed, so the garbage tail is read back with no error, +# and replaced by a backend crash, which the rejection and the log-counted survival +# arm both catch. The two premises arrive never. +# +# ledger rows 3687 + 4 = 3691 +# never 2702 + 2 = 2704 +# dated 985 + 2 = 987 +# 2704 + 987 == 3691 +checks_never_observed_red 2704 From e16ec6963757fc24a90f599c9d64903b2eb7381d Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:07 +0000 Subject: [PATCH 10/36] test: native_decode_gating gets its pytest twin, and its filter counters can see a double count and a deleted row (#1341) test_native_decode_gating.py asserts every shell check by name; compare_to_bash grades it as covering all 29, and it joins COMPLETE. It sets the width gate off for its session, where the shell pins it off for its cluster. Two of the filter counters' guarantees could not fail: - every row of the needle's surviving vector matched, so the producer's re-test of that vector rejected nothing, and a producer counting its rejections a second time read exactly right: with the non-counting filter disabled the suite stayed 26/26. A qual that half of vector 0 passes now asserts its rejections are counted once. - the counters were checked only on the table without deletes, so with the evaluator's deleted-row test removed the suite stayed 26/26. Rows Removed by Filter is now checked on the multi-group table with deletes, against heap's live count. The multi-group fixture's DELETE ran through the gating it was built to test (its WHERE is an unprunable qual too), so a gating mutant broke the fixture rather than the arms. It now runs with the width gate out of reach, and its premise counts the surviving rows with count(*) instead of a gated WHERE. cluster_tests 674 -> 676 by collection; TESTS.md gets section 148. Co-Authored-By: Claude Opus 5.5 --- test/native_decode_gating.sh | 32 ++++- test/pytest/TESTS.md | 12 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_decode_gating.py | 158 +++++++++++++++++++++++ 5 files changed, 204 insertions(+), 3 deletions(-) create mode 100644 test/pytest/test_native_decode_gating.py diff --git a/test/native_decode_gating.sh b/test/native_decode_gating.sh index fb18ac03c..2b8f9fc7b 100755 --- a/test/native_decode_gating.sh +++ b/test/native_decode_gating.sh @@ -172,6 +172,20 @@ check "the zero-match query removes every row by filter, counted once" \ check "and skips every vector" \ "$(counter_in "$PLAN_NOMATCH" 'Columnar Vectors Skipped')" "$VECTORS" +# A SURVIVING VECTOR WITH REJECTS IN IT (#1341). Every row of the needle's +# surviving vector matches, so the producer, re-testing that vector to find its +# rows, finds nothing to reject: a producer that counted those rejections a second +# time read exactly right above. Measured, with the producer's non-counting filter +# disabled, this file stayed 26/26. Half of vector 0 passes this qual (p1 % 2 = 0 is +# an expression, so it is no skip predicate either), so its 512 rejections are +# counted twice by that mistake. +HALFQ="SELECT * FROM n WHERE tag LIKE '%needle%' AND p1 % 2 = 0" +PLAN_HALF="$(explain_of "$HALFQ")" +check "premise: the half-needle survives in one vector, half of its rows" \ + "$(q "SELECT count(*) || '|' || count(DISTINCT (p1 - 1) / 1024) FROM n WHERE tag LIKE '%needle%' AND p1 % 2 = 0;")" "512|1" +check "rejections inside a surviving vector are counted once" \ + "$(counter_in "$PLAN_HALF" 'Rows Removed by Filter')" "$(( ROWS - 512 ))" + # ---- correctness ----------------------------------------------------------- # # The mask must never drop a row. Without these, an implementation that marks @@ -205,13 +219,18 @@ MGEN="SELECT 'x' || CASE WHEN g % 4096 = 7 THEN 'needle' ELSE 'miss' END || g::t psql_run "INSERT INTO hm $MGEN;" psql_run "INSERT INTO m $MGEN;" psql_run "DELETE FROM hm WHERE p1 % 17 = 0;" -psql_run "DELETE FROM m WHERE p1 % 17 = 0;" +# The fixture's DELETE must not depend on the gating under test: its WHERE is an +# unprunable qual too, so it is run with the width gate out of reach (#1341). +psql_run "SET pgcolumnar.qual_skipvec_min_payload_cols = 100000; DELETE FROM m WHERE p1 % 17 = 0;" check "premise: the columnar table spans more than one row group" \ "$(q "SELECT count(*) > 1 FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('m');")" \ "t" +# count(*) rather than a count WHERE p1 % 17 = 0: that WHERE is an unprunable qual +# too, so it ran through the very gating under test, and a gating mutant broke this +# premise rather than the arms below (#1341). check "premise: rows really were deleted (so the mask is non-NULL)" \ - "$(q "SELECT count(*) FROM m WHERE p1 % 17 = 0;")" "0" + "$(q "SELECT count(*) FROM m;")" "$(( MROWS - MROWS / 17 ))" check "deletes + multiple groups + unprunable qual: columnar matches the heap" \ "$(pgc_set_hash "SELECT * FROM m WHERE tag LIKE '%needle%'")" \ @@ -220,4 +239,13 @@ check "and the full scan across every group's mask matches too" \ "$(pgc_set_hash "SELECT * FROM m")" \ "$(pgc_set_hash "SELECT * FROM hm")" +# AND THE COUNTER SKIPS DELETED ROWS (#1341). The evaluator must not count a deleted +# row as filtered: it is invisible. Only the single-group table above had its +# counters checked, and it has no deletes, so with the evaluator's deleted-row test +# removed this file stayed 26/26. +PLAN_M="$(explain_of "SELECT * FROM m WHERE tag LIKE '%needle%'")" +check "rows removed by filter across deleted rows count only live rows" \ + "$(counter_in "$PLAN_M" 'Rows Removed by Filter')" \ + "$(q "SELECT count(*) FROM hm WHERE tag NOT LIKE '%needle%';")" + pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 35536ff8b..3ae974458 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -193,6 +193,7 @@ behaviour, the source of that number is named. - [145. test_truncate_cleanup.py: TRUNCATE leaves no catalog rows behind, and keeps what it must](#145-test_truncate_cleanuppy-truncate-leaves-no-catalog-rows-behind-and-keeps-what-it-must) - [146. test_native_dml.py: DELETE, UPDATE and MERGE on a native table agree with heap](#146-test_native_dmlpy-delete-update-and-merge-on-a-native-table-agree-with-heap) - [147. test_native_dict_underfill.py: an under-filled DICT chunk is refused and no backend dies](#147-test_native_dict_underfillpy-an-under-filled-dict-chunk-is-refused-and-no-backend-dies) +- [148. test_native_decode_gating.py: an unprunable qual gates payload decode per vector](#148-test_native_decode_gatingpy-an-unprunable-qual-gates-payload-decode-per-vector) ## 1. How to read a test in here @@ -7441,3 +7442,14 @@ The pytest twin of `test/native_dict_underfill.sh` (ported for #1341), every che | test | what it holds | | --- | --- | | `test_an_under_filled_dict_chunk_is_refused` | the column is DICT-encoded, the inflation rewrites its descriptor, the read is refused with XX001, and no backend died | + +## 148. test_native_decode_gating.py: an unprunable qual gates payload decode per vector + +The pytest twin of `test/native_decode_gating.sh` (#452 phase 2, ported for #1341), every check by the shell's name. A leading-wildcard LIKE is no skip predicate, so phase 2 evaluates the qual per 1024-row vector between the decode passes, and a vector no row passes is never decoded for its payload columns; "Columnar Vector Decodes" makes the saving exact arithmetic. The filter counters stay exact: a rejection is counted once, inside a surviving vector too, and a deleted row is not counted. The width gate is set off for the session. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_an_unprunable_qual_gates_payload_decode` | the premises (no skip predicate, the late-materialization path); the control decodes all 192 (vector, column) pairs, a zero-match LIKE 32 and the needle 37; every filter counter is exact, including rejections inside the surviving vector; the answers match heap | +| `test_deleted_rows_across_several_groups` | with deletes over several groups the answers match heap, and Rows Removed by Filter counts only live rows | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 641b37863..8741e7acf 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -757,4 +757,6 @@ guard_tests 438 # by collection over the cluster files on this tree. # cluster_tests 673 -> 674 for #1341's test_native_dict_underfill.py, the twin of # native_dict_underfill.sh: one test, by collection over the cluster files on this tree. -cluster_tests 674 +# cluster_tests 674 -> 676 for #1341's test_native_decode_gating.py, the twin of +# native_decode_gating.sh: two tests, by collection over the cluster files on this tree. +cluster_tests 676 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index fe9b9ae8f..7bd9cd799 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -120,6 +120,7 @@ "maintenance_rollback", "native_backend_crash", "native_chunk_length_bound", + "native_decode_gating", "native_delete_vector_index", "native_delete_visibility_paths", "native_dict_underfill", diff --git a/test/pytest/test_native_decode_gating.py b/test/pytest/test_native_decode_gating.py new file mode 100644 index 000000000..e2a6e5148 --- /dev/null +++ b/test/pytest/test_native_decode_gating.py @@ -0,0 +1,158 @@ +"""An unprunable qual gates payload-column decode per vector (#452 phase 2, #1341). +The pytest twin of `test/native_decode_gating.sh`, every check by the shell's name. + +A leading-wildcard LIKE is never a skip predicate, so the reader cannot rule a vector +out by value; phase 2 evaluates the executor qual per 1024-row vector between the two +decode passes, and a vector no row passes is never decoded for its payload columns. +"Columnar Vector Decodes" counts (vector, column) decodes, so the saving is exact +arithmetic. The filter counters must stay exact too: a rejection is counted once, +inside a surviving vector as well as a skipped one, and a deleted row is not counted at +all. The width gate (#595) is set off for this session, as the shell pins it off for +its cluster, so the five-column projection still gates. +""" +import re + +ROWS = 32768 +VECTORS = 32 # ROWS / 1024 +NCOLS = 6 # tag plus five payload columns +SURVIVING_VECTORS = 1 # the needle lives only in the first vector +MATCHES = 1024 +MROWS = 49152 + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +def _rows(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchall() + + +def _run(conn, *sqls): + with conn.cursor() as cur: + for sql in sqls: + cur.execute(sql) + + +def _plan(conn, sql): + return "\n".join(r[0] for r in _rows(conn, f"EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) {sql}")) + + +def _counter(plan, label): + """-> the first integer on the plan's line holding label, or None.""" + for line in plan.splitlines(): + if label in line: + m = re.search(r"\d+", line[line.index(label) + len(label):]) or re.search(r"\d+", line) + return int(m.group(0)) if m else None + return None + + +def _has(plan, label): + return "yes" if label in plan else "no" + + +NEEDLEQ = "SELECT * FROM n WHERE tag LIKE '%needle%'" +NOMATCHQ = "SELECT * FROM n WHERE tag LIKE '%zzzzz%'" +CONTROLQ = "SELECT * FROM n WHERE tag LIKE '%row%'" +HALFQ = "SELECT * FROM n WHERE tag LIKE '%needle%' AND p1 % 2 = 0" + + +def test_an_unprunable_qual_gates_payload_decode(pgc_conn, expect): + conn = pgc_conn + gen = (f"SELECT 'row' || CASE WHEN g <= 1024 THEN 'needle' ELSE 'miss' END, " + f"g, g + 1, g + 2, g + 3, g + 4 FROM generate_series(1, {ROWS}) g") + _run(conn, "SET pgcolumnar.qual_skipvec_min_payload_cols = 0", + "CREATE TABLE h (tag text, p1 int, p2 int, p3 int, p4 int, p5 int)", + "CREATE TABLE n (tag text, p1 int, p2 int, p3 int, p4 int, p5 int) USING pgcolumnar", + "SELECT pgcolumnar.set_options('n', stripe_row_limit => 65536, chunk_group_row_limit => 1024)", + f"INSERT INTO h {gen}", f"INSERT INTO n {gen}") + needle, nomatch, control = _plan(conn, NEEDLEQ), _plan(conn, NOMATCHQ), _plan(conn, CONTROLQ) + + expect.text(_has(needle, "Columnar Projected Columns"), "yes", "premise: the needle plan is a columnar custom scan") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.row_group " + "WHERE storage_id = pgcolumnar.get_storage_id('n')"), 1, + "premise: one row group, so every count below is one group's") + expect.text(_one(conn, "SHOW pgcolumnar.qual_skipvec_min_payload_cols"), "0", + "premise: the width gate is pinned off, so this narrow projection still gates") + expect.num(_counter(needle, "Columnar Usable Skip Predicates"), 0, + "premise: the reader has no usable skip predicate for this LIKE") + expect.num(_counter(needle, "Columnar Chunk Groups Removed by Filter"), 0, "premise: no whole row group is pruned") + expect.num(_counter(needle, "Columnar Vectors Ruled Out by Value"), 0, + "premise: no vector is ruled out by VALUE either (that is 1b-ii's path)") + expect.text(_has(needle, "Columnar Rows Filtered Before Materialization"), "yes", + "premise: the needle query takes the late-materialization path") + expect.num(_one(conn, "SELECT count(*) FROM n WHERE tag LIKE '%needle%'"), MATCHES, + "premise: the needle really matches one vector's worth of rows") + expect.num(_one(conn, "SELECT count(*) FROM n WHERE tag LIKE '%zzzzz%'"), 0, + "premise: the nomatch predicate matches no row") + expect.num(_one(conn, "SELECT count(*) FROM n WHERE tag LIKE '%row%'"), ROWS, + "premise: the control matches every row") + + expect.text(_has(control, "Columnar Vector Decodes"), "yes", "EXPLAIN reports Columnar Vector Decodes") + expect.num(_counter(control, "Columnar Vector Decodes"), VECTORS * NCOLS, + "the control decodes every column of every vector") + expect.num(_counter(nomatch, "Columnar Vector Decodes"), VECTORS, + "a zero-matching LIKE decodes the qual column and no payload column") + expect.num(_counter(needle, "Columnar Vector Decodes"), VECTORS + (NCOLS - 1) * SURVIVING_VECTORS, + "a one-vector LIKE decodes the qual column plus payload for that vector only") + + expect.num(_counter(needle, "Rows Removed by Filter"), ROWS - MATCHES, + "the needle removes exactly the non-matching rows, counted once") + expect.num(_counter(needle, "Columnar Rows Filtered Before Materialization"), ROWS - MATCHES, + "and reports them all filtered before materialization") + expect.num(_counter(needle, "Columnar Vectors Skipped"), VECTORS - SURVIVING_VECTORS, + "the needle skips exactly the vectors holding no match") + expect.num(_counter(nomatch, "Rows Removed by Filter"), ROWS, + "the zero-match query removes every row by filter, counted once") + expect.num(_counter(nomatch, "Columnar Vectors Skipped"), VECTORS, "and skips every vector") + + # Every row of the needle's surviving vector matches, so a producer that counted + # its re-test's rejections twice reads right above; half of vector 0 passes here. + expect.text(_one(conn, "SELECT count(*) || '|' || count(DISTINCT (p1 - 1) / 1024) FROM n " + "WHERE tag LIKE '%needle%' AND p1 % 2 = 0"), "512|1", + "premise: the half-needle survives in one vector, half of its rows") + expect.num(_counter(_plan(conn, HALFQ), "Rows Removed by Filter"), ROWS - 512, + "rejections inside a surviving vector are counted once") + + expect.row_set(_rows(conn, NEEDLEQ), _rows(conn, "SELECT * FROM h WHERE tag LIKE '%needle%'"), + "the needle query returns exactly the matching rows") + expect.num(_one(conn, "SELECT count(*) FROM n WHERE tag LIKE '%zzzzz%'"), 0, "the nomatch query returns nothing") + expect.rows(_rows(conn, "SELECT count(*), sum(p1), sum(p5) FROM n WHERE tag LIKE '%row%'"), + _rows(conn, "SELECT count(*), sum(p1), sum(p5) FROM h WHERE tag LIKE '%row%'"), + "the control returns every row, values intact") + + +def test_deleted_rows_across_several_groups(pgc_conn, expect): + """The evaluator reads the CURRENT group's delete mask; an earlier version read the + previous group's, freed one load earlier. It needs more than one group, deletes, + and the unprunable qual at once. A deleted row is not counted as filtered.""" + conn = pgc_conn + mgen = (f"SELECT 'x' || CASE WHEN g % 4096 = 7 THEN 'needle' ELSE 'miss' END || g::text, g, g * 2 " + f"FROM generate_series(1, {MROWS}) g") + _run(conn, "SET pgcolumnar.qual_skipvec_min_payload_cols = 0", + "CREATE TABLE hm (tag text, p1 int, p2 int)", + "CREATE TABLE m (tag text, p1 int, p2 int) USING pgcolumnar", + "SELECT pgcolumnar.set_options('m', stripe_row_limit => 8192, chunk_group_row_limit => 1024)", + f"INSERT INTO hm {mgen}", f"INSERT INTO m {mgen}", + "DELETE FROM hm WHERE p1 % 17 = 0", + # the fixture's DELETE must not depend on the gating under test + "SET pgcolumnar.qual_skipvec_min_payload_cols = 100000", "DELETE FROM m WHERE p1 % 17 = 0", + "SET pgcolumnar.qual_skipvec_min_payload_cols = 0") + expect.text(str(_one(conn, "SELECT count(*) > 1 FROM pgcolumnar.row_group " + "WHERE storage_id = pgcolumnar.get_storage_id('m')")), "True", + "premise: the columnar table spans more than one row group") + # count(*), not a count WHERE p1 % 17 = 0: that WHERE would run through the gating + expect.num(_one(conn, "SELECT count(*) FROM m"), MROWS - MROWS // 17, + "premise: rows really were deleted (so the mask is non-NULL)") + expect.row_set(_rows(conn, "SELECT * FROM m WHERE tag LIKE '%needle%'"), + _rows(conn, "SELECT * FROM hm WHERE tag LIKE '%needle%'"), + "deletes + multiple groups + unprunable qual: columnar matches the heap") + expect.row_set(_rows(conn, "SELECT * FROM m"), _rows(conn, "SELECT * FROM hm"), + "and the full scan across every group's mask matches too") + expect.num(_counter(_plan(conn, "SELECT * FROM m WHERE tag LIKE '%needle%'"), "Rows Removed by Filter"), + _one(conn, "SELECT count(*) FROM hm WHERE tag NOT LIKE '%needle%'"), + "rows removed by filter across deleted rows count only live rows") From fb1f6e238991c77d6d57548212ce16eceb9c09a0 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:07 +0000 Subject: [PATCH 11/36] test: native_decode_gating seeds the ledger, 11 of 29 arms dated by four mutations of phase 2 (#1341) 29 rows from five-major runs (29/29 on each of 15-19). Dated on PG18 by: the per-vector evaluation never run (4), a deleted row counted as filtered (1), every vector skipped (7), and a double count in the producer (2). No premise is dated: the fixture no longer depends on the gating under test. suites_not_covered 215 -> 214; the census is 3720 rows, 2722 never, 998 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 29 +++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 19 +++++++++++++++++-- 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index e9f4d490f..f5f554373 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2308,6 +2308,35 @@ native_chunk_length_bound native_chunk_length_bound backend survived the sequent native_chunk_length_bound native_chunk_length_bound backend survived the truncated-length fetch 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound premise: a point lookup uses the index, not a sequential columnar scan 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound that fetch returns the row 15;16;17;18;19 2026-10-16 the index fetch treated a flushed row as not live, so the lookup returned no row (mutation, #1339) +native_decode_gating native_decode_gating EXPLAIN reports Columnar Vector Decodes 15;16;17;18;19 never - +native_decode_gating native_decode_gating a one-vector LIKE decodes the qual column plus payload for that vector only 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluation between the decode passes never runs;columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating a zero-matching LIKE decodes the qual column and no payload column 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluation between the decode passes never runs +native_decode_gating native_decode_gating and reports them all filtered before materialization 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating and skips every vector 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluation between the decode passes never runs +native_decode_gating native_decode_gating and the full scan across every group's mask matches too 15;16;17;18;19 never - +native_decode_gating native_decode_gating deletes + multiple groups + unprunable qual: columnar matches the heap 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating premise: no vector is ruled out by VALUE either (that is 1b-ii's path) 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: no whole row group is pruned 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: one row group, so every count below is one group's 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: rows really were deleted (so the mask is non-NULL) 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the columnar table spans more than one row group 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the control matches every row 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the half-needle survives in one vector, half of its rows 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the needle plan is a columnar custom scan 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the needle query takes the late-materialization path 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the needle really matches one vector's worth of rows 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the nomatch predicate matches no row 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the reader has no usable skip predicate for this LIKE 15;16;17;18;19 never - +native_decode_gating native_decode_gating premise: the width gate is pinned off, so this narrow projection still gates 15;16;17;18;19 never - +native_decode_gating native_decode_gating rejections inside a surviving vector are counted once 15;16;17;18;19 2026-10-16 columnar_reader.c: the producer counts a gated group's rejections a second time +native_decode_gating native_decode_gating rows removed by filter across deleted rows count only live rows 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluator counts a deleted row as filtered;columnar_reader.c: the producer counts a gated group's rejections a second time +native_decode_gating native_decode_gating the control decodes every column of every vector 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating the control returns every row, values intact 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating the needle query returns exactly the matching rows 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating the needle removes exactly the non-matching rows, counted once 15;16;17;18;19 never - +native_decode_gating native_decode_gating the needle skips exactly the vectors holding no match 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluation between the decode passes never runs;columnar_reader.c: the per-vector qual evaluator marks every vector skipped +native_decode_gating native_decode_gating the nomatch query returns nothing 15;16;17;18;19 never - +native_decode_gating native_decode_gating the zero-match query removes every row by filter, counted once 15;16;17;18;19 never - native_delete_visibility_paths native_delete_visibility_paths a DELETE whose qual deletes another row of the table succeeds and leaves heap's rows (#1394) 15;16;17;18;19 2026-10-14 PgColumnarUpsertDeleteVector without the command advance (main before #1394's fix) native_delete_visibility_paths native_delete_visibility_paths a deleted row is invisible via the index 15;16;17;18;19 never - native_delete_visibility_paths native_delete_visibility_paths a live row is visible via the index 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index d3cc593b9..ef3e3c5d1 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 215 +suites_not_covered 214 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3002,4 +3002,19 @@ suites_not_covered 215 # never 2702 + 2 = 2704 # dated 985 + 2 = 987 # 2704 + 987 == 3691 -checks_never_observed_red 2704 +# +# #1341, native_decode_gating seeded, with its twin (OffgridwithJD): 2704 -> 2722, and +# suites_not_covered 215 -> 214. The suite had no rows. It enters with all 29 checks +# from five-major runs (29/29 on each of 15-19). 11 are dated by four mutations of +# #452 phase 2: the per-vector evaluation never run (4), a deleted row counted as +# filtered (1, the new arm), every vector marked skipped (7), and the producer +# counting a gated group's rejections twice (2, one of them new). Eighteen arrive +# never: the thirteen premises, the observable's presence, the full scan across +# every group, the nomatch query's empty answer, and two exact counters that no +# single mutation here moves. +# +# ledger rows 3691 + 29 = 3720 +# never 2704 + 18 = 2722 +# dated 987 + 11 = 998 +# 2722 + 998 == 3720 +checks_never_observed_red 2722 From d095dc9441137646e944f36fd866a7d2935ea199 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:07 +0000 Subject: [PATCH 12/36] test: native_decode_gate_width gets its pytest twin, and the unshipped settings are proven absent by SQLSTATE (#1341) test_native_decode_gate_width.py asserts every shell check by name; compare_to_bash grades it as covering all 35, and it joins COMPLETE. It holds the threshold for its session, where the shell sets it per database because every helper opens a new connection. The two premises that the rejected probe settings were not shipped compared SHOW's output with the empty string: q returns empty for a dead connection too, and an empty expectation is the shape the pytest layer refuses. Both twins now require SHOW to raise 42704 (undefined_object). cluster_tests 676 -> 677 by collection; TESTS.md gets section 149. Co-Authored-By: Claude Opus 5.5 --- test/native_decode_gate_width.sh | 11 +- test/pytest/TESTS.md | 11 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_decode_gate_width.py | 185 +++++++++++++++++++ 5 files changed, 209 insertions(+), 3 deletions(-) create mode 100644 test/pytest/test_native_decode_gate_width.py diff --git a/test/native_decode_gate_width.sh b/test/native_decode_gate_width.sh index 94bcd8189..176830fd6 100755 --- a/test/native_decode_gate_width.sh +++ b/test/native_decode_gate_width.sh @@ -106,10 +106,17 @@ check "premise: the width-gate GUC exists and defaults to the shipped $DEFAULT_G # unknown parameter errors; q swallows the error and returns empty, so an empty # result is the proof the parameter does not exist. The paired positive premise # above (min_payload_cols == 20) rules out an empty result from a dead connection. +# By SQLSTATE rather than by an empty result (#1341): q returns empty for a dead +# connection as well, and an empty expectation is the shape the pytest layer refuses. +show_state() { # show_state PARAM -> the SQLSTATE SHOW raised, or "shown" + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" \ + -qAtX -v VERBOSITY=verbose -c "SHOW $1;" >/dev/null 2>"$PGC_WORKDIR/show.err" && { echo shown; return; } + sed -nE 's/^ERROR: ([0-9A-Z]{5}): .*/\1/p' "$PGC_WORKDIR/show.err" | head -1 +} check "premise: the rejected probe GUC qual_skipvec_probe_vecs was NOT shipped" \ - "$(q "SHOW pgcolumnar.qual_skipvec_probe_vecs;")" "" + "$(show_state pgcolumnar.qual_skipvec_probe_vecs)" "42704" check "premise: the rejected probe GUC qual_skipvec_min_skip_pct was NOT shipped" \ - "$(q "SHOW pgcolumnar.qual_skipvec_min_skip_pct;")" "" + "$(show_state pgcolumnar.qual_skipvec_min_skip_pct)" "42704" check "premise: one row group, so every count below is one group's" \ "$(q "SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('nw');")" \ "1" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 3ae974458..02172f0cd 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -194,6 +194,7 @@ behaviour, the source of that number is named. - [146. test_native_dml.py: DELETE, UPDATE and MERGE on a native table agree with heap](#146-test_native_dmlpy-delete-update-and-merge-on-a-native-table-agree-with-heap) - [147. test_native_dict_underfill.py: an under-filled DICT chunk is refused and no backend dies](#147-test_native_dict_underfillpy-an-under-filled-dict-chunk-is-refused-and-no-backend-dies) - [148. test_native_decode_gating.py: an unprunable qual gates payload decode per vector](#148-test_native_decode_gatingpy-an-unprunable-qual-gates-payload-decode-per-vector) +- [149. test_native_decode_gate_width.py: decode gating runs only on a projection wide enough to pay](#149-test_native_decode_gate_widthpy-decode-gating-runs-only-on-a-projection-wide-enough-to-pay) ## 1. How to read a test in here @@ -7453,3 +7454,13 @@ The pytest twin of `test/native_decode_gating.sh` (#452 phase 2, ported for #134 | --- | --- | | `test_an_unprunable_qual_gates_payload_decode` | the premises (no skip predicate, the late-materialization path); the control decodes all 192 (vector, column) pairs, a zero-match LIKE 32 and the needle 37; every filter counter is exact, including rejections inside the surviving vector; the answers match heap | | `test_deleted_rows_across_several_groups` | with deletes over several groups the answers match heap, and Rows Removed by Filter counts only live rows | + +## 149. test_native_decode_gate_width.py: decode gating runs only on a projection wide enough to pay + +The pytest twin of `test/native_decode_gate_width.sh` (#595, ported for #1341), every check by the shell's name. Phase-2 gating runs only when the projection has at least `pgcolumnar.qual_skipvec_min_payload_cols` non-qual columns (default 20). The policy is pinned by moving the threshold across a fixed projection: a 24-payload `SELECT *` gates at the default, stops above 24 and gates again at exactly 24, and a three-column projection gates only once the setting is 0. The gate is output-invariant: the answers match heap with it on or off. The setting is held per session, where the shell sets it per database. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_width_gate` | the premises, including that the two rejected probe settings are undefined (42704); the decode counts at the default, with the gate off, above the width and at it; and correctness against heap with the gate on and off | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 8741e7acf..cd252950e 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -759,4 +759,6 @@ guard_tests 438 # native_dict_underfill.sh: one test, by collection over the cluster files on this tree. # cluster_tests 674 -> 676 for #1341's test_native_decode_gating.py, the twin of # native_decode_gating.sh: two tests, by collection over the cluster files on this tree. -cluster_tests 676 +# cluster_tests 676 -> 677 for #1341's test_native_decode_gate_width.py, the twin of +# native_decode_gate_width.sh: one test, by collection over the cluster files on this tree. +cluster_tests 677 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 7bd9cd799..f5965db19 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -120,6 +120,7 @@ "maintenance_rollback", "native_backend_crash", "native_chunk_length_bound", + "native_decode_gate_width", "native_decode_gating", "native_delete_vector_index", "native_delete_visibility_paths", diff --git a/test/pytest/test_native_decode_gate_width.py b/test/pytest/test_native_decode_gate_width.py new file mode 100644 index 000000000..56e3620eb --- /dev/null +++ b/test/pytest/test_native_decode_gate_width.py @@ -0,0 +1,185 @@ +"""Phase-2 decode gating runs only on a projection wide enough to pay for it (#595, +#1341). The pytest twin of `test/native_decode_gate_width.sh`, every check by the +shell's name. + +#452 phase 2 evaluates an unprunable qual per vector so a vector no row passes skips +its payload decode; it pays only when there is payload to spare. So it runs only when +the projection has at least pgcolumnar.qual_skipvec_min_payload_cols non-qual columns +(default 20). The policy is pinned by moving the threshold across a fixed projection: +a wide SELECT * (24 payload columns) gates at the default and stops once the setting is +above 24, and gates again at exactly 24 (the >= boundary); a narrow three-column +projection does not gate at the default and does once the setting is 0. "Columnar +Vector Decodes" is the observable: VECTORS when gated, every column of every vector +when not. The shell sets the threshold per database because each helper opens a new +connection; this side holds one connection and sets it for the session. +""" +import re + +import psycopg + +ROWS = 32768 +VECTORS = 32 +PCOLS = 24 +NCOLS_WIDE = PCOLS + 1 +NARROW_NCOLS = 3 +SURVIVING_VECTORS = 1 +DEFAULT_GUC = 20 + +COLDEFS = ", ".join(f"p{i} int" for i in range(1, PCOLS + 1)) +COLVALS = ", ".join(f"g + {i - 1}" for i in range(1, PCOLS + 1)) +WIDE_NOMATCH = "SELECT * FROM nw WHERE tag LIKE '%zzzzz%'" +WIDE_NEEDLE = "SELECT * FROM nw WHERE tag LIKE '%needle%'" +WIDE_CONTROL = "SELECT * FROM nw WHERE tag LIKE '%row%'" +NARROW_NOMATCH = "SELECT tag, p1, p2 FROM nw WHERE tag LIKE '%zzzzz%'" +NARROW_CONTROL = "SELECT tag, p1, p2 FROM nw WHERE tag LIKE '%row%'" +NARROW_NEEDLE = "SELECT tag, p1, p2 FROM nw WHERE tag LIKE '%needle%'" + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchone()[0] + + +def _rows(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchall() + + +def _show_state(conn, name): + """-> the SQLSTATE SHOW raised, or 'shown'. 42704 is undefined_object.""" + try: + _one(conn, f"SHOW {name}") + return "shown" + except psycopg.Error as exc: + return exc.sqlstate or "no SQLSTATE" + + +def _plan(conn, sql): + return "\n".join(r[0] for r in _rows(conn, f"EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) {sql}")) + + +def _counter(plan, label): + for line in plan.splitlines(): + if label in line: + m = re.search(r"\d+", line[line.index(label) + len(label):]) + return int(m.group(0)) if m else None + return None + + +def _has(plan, label): + return "yes" if label in plan else "no" + + +def _set_gate(conn, v): + with conn.cursor() as cur: + cur.execute(f"SET pgcolumnar.qual_skipvec_min_payload_cols = {v}") + + +def _reset_gate(conn): + with conn.cursor() as cur: + cur.execute("RESET pgcolumnar.qual_skipvec_min_payload_cols") + + +def _gate_now(conn): + return int(_one(conn, "SHOW pgcolumnar.qual_skipvec_min_payload_cols")) + + +def test_the_width_gate(pgc_conn, expect): + conn = pgc_conn + gen = (f"SELECT 'row' || CASE WHEN g <= 1024 THEN 'needle' ELSE 'miss' END, {COLVALS} " + f"FROM generate_series(1, {ROWS}) g") + with conn.cursor() as cur: + cur.execute(f"CREATE TABLE hw (tag text, {COLDEFS})") + cur.execute(f"CREATE TABLE nw (tag text, {COLDEFS}) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('nw', stripe_row_limit => 65536, chunk_group_row_limit => 1024)") + cur.execute(f"INSERT INTO hw {gen}") + cur.execute(f"INSERT INTO nw {gen}") + + expect.num(_gate_now(conn), DEFAULT_GUC, + f"premise: the width-gate GUC exists and defaults to the shipped {DEFAULT_GUC}") + expect.text(_show_state(conn, "pgcolumnar.qual_skipvec_probe_vecs"), "42704", + "premise: the rejected probe GUC qual_skipvec_probe_vecs was NOT shipped") + expect.text(_show_state(conn, "pgcolumnar.qual_skipvec_min_skip_pct"), "42704", + "premise: the rejected probe GUC qual_skipvec_min_skip_pct was NOT shipped") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.row_group " + "WHERE storage_id = pgcolumnar.get_storage_id('nw')"), 1, + "premise: one row group, so every count below is one group's") + + wide_needle = _plan(conn, WIDE_NEEDLE) + narrow_needle = _plan(conn, NARROW_NEEDLE) + narrow_nomatch = _plan(conn, NARROW_NOMATCH) + expect.text(_has(wide_needle, "Columnar Projected Columns"), "yes", + "premise: the wide plan is a columnar custom scan with projection") + expect.text(_has(narrow_nomatch, "Columnar Projected Columns"), "yes", + "premise: the narrow plan is a columnar custom scan with projection") + expect.num(_counter(wide_needle, "Columnar Usable Skip Predicates"), 0, + "premise: the LIKE is unprunable (no usable skip predicate)") + expect.num(_counter(wide_needle, "Columnar Chunk Groups Removed by Filter"), 0, + "premise: no whole row group is pruned") + expect.num(_counter(wide_needle, "Columnar Vectors Ruled Out by Value"), 0, + "premise: no vector is ruled out by VALUE (that is 1b-ii's path, not phase 2)") + expect.text(_has(wide_needle, "Columnar Rows Filtered Before Materialization"), "yes", + "premise: the wide query takes the late-materialization path") + expect.text(_has(narrow_needle, "Columnar Rows Filtered Before Materialization"), "yes", + "premise: the narrow query takes the late-materialization path") + expect.text(_has(narrow_nomatch, "Columnar Vector Decodes"), "yes", + "premise: EXPLAIN reports Columnar Vector Decodes (the observable)") + expect.num(_one(conn, "SELECT count(*) FROM nw WHERE tag LIKE '%needle%'"), 1024, + "premise: the needle matches exactly one vector's worth of rows") + expect.num(_one(conn, "SELECT count(*) FROM nw WHERE tag LIKE '%zzzzz%'"), 0, + "premise: the nomatch predicate matches no row") + expect.num(_one(conn, "SELECT count(*) FROM nw WHERE tag LIKE '%row%'"), ROWS, + "premise: the control matches every row") + + # arm 1: at the shipped default, wide gates and narrow does not + expect.num(_gate_now(conn), DEFAULT_GUC, "premise: still at the default before arm 1") + wide_nomatch = _plan(conn, WIDE_NOMATCH) + expect.num(_counter(wide_nomatch, "Columnar Vector Decodes"), VECTORS, + "wide, default: a zero-match LIKE decodes the qual column and NO payload -> gated") + expect.num(_counter(wide_nomatch, "Columnar Vectors Skipped"), VECTORS, + "wide, default: and reports the payload vectors skipped") + wide_needle_d = _plan(conn, WIDE_NEEDLE) + expect.num(_counter(wide_needle_d, "Columnar Vector Decodes"), VECTORS + (NCOLS_WIDE - 1) * SURVIVING_VECTORS, + "wide, default: a one-vector LIKE decodes the qual column plus payload for that vector only") + expect.num(_counter(wide_needle_d, "Columnar Vectors Skipped"), VECTORS - SURVIVING_VECTORS, + "wide, default: skips exactly the vectors holding no match") + expect.num(_counter(_plan(conn, NARROW_NOMATCH), "Columnar Vector Decodes"), VECTORS * NARROW_NCOLS, + "narrow, default: a zero-match LIKE decodes EVERY projected column of every vector -> NOT gated") + + # arm 2: the gate off, and the same narrow projection gates + _set_gate(conn, 0) + expect.num(_gate_now(conn), 0, "premise: the width gate is now disabled (0)") + expect.num(_counter(_plan(conn, NARROW_NOMATCH), "Columnar Vector Decodes"), VECTORS, + "narrow, gate off (GUC=0): the SAME narrow projection now gates -> qual column alone") + expect.num(_counter(_plan(conn, WIDE_CONTROL), "Columnar Vector Decodes"), VECTORS * NCOLS_WIDE, + "control, gate forced on: a matches-everything wide qual skips no vector (full decode)") + expect.num(_counter(_plan(conn, NARROW_CONTROL), "Columnar Vector Decodes"), VECTORS * NARROW_NCOLS, + "control, gate forced on: a matches-everything narrow qual skips no vector (full decode)") + + # arm 3: above the wide projection's width it stops; at exactly its width it gates + _set_gate(conn, NCOLS_WIDE) + expect.num(_gate_now(conn), NCOLS_WIDE, "premise: the width threshold is now above the projection width") + expect.num(_counter(_plan(conn, WIDE_NOMATCH), "Columnar Vector Decodes"), VECTORS * NCOLS_WIDE, + "wide, gate raised above width: the SAME wide SELECT * stops gating -> full decode") + _set_gate(conn, PCOLS) + expect.num(_gate_now(conn), PCOLS, "premise: the width threshold now equals the projection width") + expect.num(_counter(_plan(conn, WIDE_NOMATCH), "Columnar Vector Decodes"), VECTORS, + "wide, gate AT the width (>= boundary): gates again -> qual column alone") + + # correctness: the gate never changes a result + _reset_gate(conn) + expect.num(_gate_now(conn), DEFAULT_GUC, "premise: back to the shipped default") + gated = _rows(conn, WIDE_NEEDLE) + expect.row_set(gated, _rows(conn, "SELECT * FROM hw WHERE tag LIKE '%needle%'"), + "correctness: wide needle, gate ON, matches the heap oracle") + _set_gate(conn, NCOLS_WIDE) + expect.num(_gate_now(conn), NCOLS_WIDE, "premise: gate suppressed for the invariance arm") + expect.row_set(_rows(conn, WIDE_NEEDLE), gated, + "correctness: wide needle, gate OFF, returns the identical rows (output-invariant)") + expect.row_set(_rows(conn, NARROW_NEEDLE), _rows(conn, "SELECT tag, p1, p2 FROM hw WHERE tag LIKE '%needle%'"), + "correctness: narrow needle matches the heap either way") + expect.row_set(_rows(conn, "SELECT * FROM nw"), _rows(conn, "SELECT * FROM hw"), + "correctness: the full scan matches the heap") + _reset_gate(conn) From 0cb4e37a35e0a48f2199c0873328f5be97384cd8 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:07 +0000 Subject: [PATCH 13/36] test: native_decode_gate_width seeds the ledger, 3 of 35 arms dated by three mutations of the width gate (#1341) 35 rows from five-major runs (35/35 on each of 15-19). Dated on PG18 by gating whatever the width (2), > for >= (1) and a payload count one too high (1, shared). suites_not_covered 214 -> 213; the census is 3755 rows, 2754 never, 1001 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 35 +++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 18 ++++++++++++++++-- 2 files changed, 51 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index f5f554373..7aea84451 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2308,6 +2308,41 @@ native_chunk_length_bound native_chunk_length_bound backend survived the sequent native_chunk_length_bound native_chunk_length_bound backend survived the truncated-length fetch 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound premise: a point lookup uses the index, not a sequential columnar scan 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound that fetch returns the row 15;16;17;18;19 2026-10-16 the index fetch treated a flushed row as not live, so the lookup returned no row (mutation, #1339) +native_decode_gate_width native_decode_gate_width control, gate forced on: a matches-everything narrow qual skips no vector (full decode) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width control, gate forced on: a matches-everything wide qual skips no vector (full decode) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width correctness: narrow needle matches the heap either way 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width correctness: the full scan matches the heap 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width correctness: wide needle, gate OFF, returns the identical rows (output-invariant) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width correctness: wide needle, gate ON, matches the heap oracle 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width narrow, default: a zero-match LIKE decodes EVERY projected column of every vector -> NOT gated 15;16;17;18;19 2026-10-16 columnar_reader.c: phase-2 gating runs whatever the projection's payload width +native_decode_gate_width native_decode_gate_width narrow, gate off (GUC=0): the SAME narrow projection now gates -> qual column alone 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: EXPLAIN reports Columnar Vector Decodes (the observable) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: back to the shipped default 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: gate suppressed for the invariance arm 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: no vector is ruled out by VALUE (that is 1b-ii's path, not phase 2) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: no whole row group is pruned 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: one row group, so every count below is one group's 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: still at the default before arm 1 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the LIKE is unprunable (no usable skip predicate) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the control matches every row 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the narrow plan is a columnar custom scan with projection 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the narrow query takes the late-materialization path 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the needle matches exactly one vector's worth of rows 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the nomatch predicate matches no row 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the rejected probe GUC qual_skipvec_min_skip_pct was NOT shipped 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the rejected probe GUC qual_skipvec_probe_vecs was NOT shipped 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the wide plan is a columnar custom scan with projection 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the wide query takes the late-materialization path 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the width gate is now disabled (0) 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the width threshold is now above the projection width 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the width threshold now equals the projection width 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width premise: the width-gate GUC exists and defaults to the shipped 20 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width wide, default: a one-vector LIKE decodes the qual column plus payload for that vector only 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width wide, default: a zero-match LIKE decodes the qual column and NO payload -> gated 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width wide, default: and reports the payload vectors skipped 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width wide, default: skips exactly the vectors holding no match 15;16;17;18;19 never - +native_decode_gate_width native_decode_gate_width wide, gate AT the width (>= boundary): gates again -> qual column alone 15;16;17;18;19 2026-10-16 columnar_reader.c: the width gate requires more payload columns than the threshold, not at least as many +native_decode_gate_width native_decode_gate_width wide, gate raised above width: the SAME wide SELECT * stops gating -> full decode 15;16;17;18;19 2026-10-16 columnar_reader.c: phase-2 gating runs whatever the projection's payload width;columnar_reader.c: the payload-width count includes one column too many native_decode_gating native_decode_gating EXPLAIN reports Columnar Vector Decodes 15;16;17;18;19 never - native_decode_gating native_decode_gating a one-vector LIKE decodes the qual column plus payload for that vector only 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluation between the decode passes never runs;columnar_reader.c: the per-vector qual evaluator marks every vector skipped native_decode_gating native_decode_gating a zero-matching LIKE decodes the qual column and no payload column 15;16;17;18;19 2026-10-16 columnar_reader.c: the per-vector qual evaluation between the decode passes never runs diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index ef3e3c5d1..05119c5ab 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 214 +suites_not_covered 213 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3017,4 +3017,18 @@ suites_not_covered 214 # never 2704 + 18 = 2722 # dated 987 + 11 = 998 # 2722 + 998 == 3720 -checks_never_observed_red 2722 +# +# #1341, native_decode_gate_width seeded, with its twin (OffgridwithJD): 2722 -> 2754, +# and suites_not_covered 214 -> 213. The suite had no rows. It enters with all 35 +# checks from five-major runs (35/35 on each of 15-19). 3 are dated by three +# mutations of the #595 width gate: gating whatever the width (the two "not gated" +# arms), > for >= (the boundary arm), and a payload count one too high (the +# raised-threshold arm). Thirty-two arrive never: the twenty-one premises, and the arms +# the width policy does not move -- the gated counts, the gate-off and controls, the +# correctness arms -- which the sibling native_decode_gating dates. +# +# ledger rows 3720 + 35 = 3755 +# never 2722 + 32 = 2754 +# dated 998 + 3 = 1001 +# 2754 + 1001 == 3755 +checks_never_observed_red 2754 From 1d346dea13f8338aef4acbcc93464e9c9b1b5ccf Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:07 +0000 Subject: [PATCH 14/36] test: pg_dump_roundtrip gets its pytest twin, and its index and sort_by arms test what they name (#1341) test_pg_dump_roundtrip.py asserts every shell check by name; compare_to_bash grades it as covering all 25 names (40 checks over the two formats), and it joins COMPLETE. It dumps a private source database and restores it into fresh ones. Two arms could not fail as claimed: - "restored index answers a point lookup" disabled seqscan and bitmapscan, but the columnar scan is neither, and with the index dropped from the restored database the lookup still answered 12345: the suite stayed 36/36. A premise now asserts the lookup's plan uses dt_id_idx. - "restored sort_by resolves and applies" checked the row count after vacuum_sorted, which is 20000 whatever it sorted on: with the sort keys replaced by column 1 the suite stayed 36/36. It now counts (host, ts) inversions in scan order, which must be 0, with a premise that the restored table has some before the call. cluster_tests 677 -> 680 by collection; TESTS.md gets section 150. Co-Authored-By: Claude Opus 5.5 --- test/pg_dump_roundtrip.sh | 14 ++- test/pytest/TESTS.md | 13 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_pg_dump_roundtrip.py | 165 ++++++++++++++++++++++++++ 5 files changed, 195 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_pg_dump_roundtrip.py diff --git a/test/pg_dump_roundtrip.sh b/test/pg_dump_roundtrip.sh index 50e4ec894..60cf10d8a 100755 --- a/test/pg_dump_roundtrip.sh +++ b/test/pg_dump_roundtrip.sh @@ -95,6 +95,11 @@ verify() { # label db check "$label: row data survives (checksum)" "$(on "$db" "$sum_sql;")" "$before_sum" check "$label: access method is pgcolumnar" "$(on "$db" "$am_sql;")" "pgcolumnar" # tail -1: the two SET statements each emit a command tag before the result. + # The plan premise is what makes this arm about the INDEX (#1341): the columnar + # scan is not disabled by enable_seqscan, and with the index dropped from the + # restored database the same lookup still answered 12345. + check "$label: premise: the point lookup takes the restored index" \ + "$(on "$db" "SET enable_seqscan=off; SET enable_bitmapscan=off; EXPLAIN (COSTS off) SELECT id FROM dt WHERE id = 12345;" | grep -c 'Index.* using dt_id_idx')" "1" check "$label: restored index answers a point lookup" "$(on "$db" "$idx_sql;" | tail -1)" "12345" # Per-table options survive the round trip (#248). They did not until the # extension registered pgcolumnar.options with pg_extension_config_dump: @@ -119,8 +124,15 @@ verify() { # label db # the wrong or a missing column here. check "$label: declared sort_by survives restore" \ "$(on "$db" "SELECT sort_by::text FROM pgcolumnar.options WHERE regclass='ds'::regclass;")" "{host,ts}" + # APPLIES means the rows end up in (host, ts) order, not merely that the call + # returns (#1341): the row count alone is 20000 whichever columns it sorted on. + # Inversions are counted in scan order, with a premise that there are some + # before the call, so a no-op cannot pass. + inversions="SELECT count(*) FROM (SELECT host, ts, lag(host) OVER () AS lh, lag(ts) OVER () AS lt FROM ds) x WHERE (lh, lt) > (host, ts)" + check "$label: premise: the restored ds is not yet in its sort_by order" \ + "$(on "$db" "SELECT ($inversions) > 0;")" "t" check "$label: restored sort_by resolves and applies" \ - "$(on "$db" "SELECT pgcolumnar.vacuum_sorted('ds'); SELECT count(*) FROM ds;" 2>/dev/null | tail -1)" "20000" + "$(on "$db" "SELECT pgcolumnar.vacuum_sorted('ds'); SELECT count(*) || '|' || ($inversions) FROM ds;" 2>/dev/null | tail -1)" "20000|0" # The DECLARATION is carried, which is the half #266 fixed. It is keyed by # regclass and stores column names, so config_dump can restore it, and the diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 02172f0cd..cb4ff342d 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -195,6 +195,7 @@ behaviour, the source of that number is named. - [147. test_native_dict_underfill.py: an under-filled DICT chunk is refused and no backend dies](#147-test_native_dict_underfillpy-an-under-filled-dict-chunk-is-refused-and-no-backend-dies) - [148. test_native_decode_gating.py: an unprunable qual gates payload decode per vector](#148-test_native_decode_gatingpy-an-unprunable-qual-gates-payload-decode-per-vector) - [149. test_native_decode_gate_width.py: decode gating runs only on a projection wide enough to pay](#149-test_native_decode_gate_widthpy-decode-gating-runs-only-on-a-projection-wide-enough-to-pay) +- [150. test_pg_dump_roundtrip.py: a columnar table survives pg_dump and restore](#150-test_pg_dump_roundtrippy-a-columnar-table-survives-pg_dump-and-restore) ## 1. How to read a test in here @@ -7464,3 +7465,15 @@ The pytest twin of `test/native_decode_gate_width.sh` (#595, ported for #1341), | test | what it holds | | --- | --- | | `test_the_width_gate` | the premises, including that the two rejected probe settings are undefined (42704); the decode counts at the default, with the gate off, above the width and at it; and correctness against heap with the gate on and off | + +## 150. test_pg_dump_roundtrip.py: a columnar table survives pg_dump and restore + +The pytest twin of `test/pg_dump_roundtrip.sh` (ported for #1341), every check by the shell's name. A source database holds a columnar table with data, an option, an index and a projection, and a second table whose declared `sort_by` names columns after a dropped one (#288). It is dumped as plain SQL and as a custom archive and restored into fresh databases, where the rows, the access method, the option, the `sort_by` and the projection declaration survive, and `rebuild_projections()` recovers the projection whose storage pg_dump cannot carry (#266). The point lookup must take the restored index, and the restored `sort_by` must leave the rows in its order. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_dropping_a_projection_forgets_its_declaration` | a dropped projection's declaration goes with it, so a rebuild builds nothing | +| `test_a_plain_dump_round_trips` | pg_dump and psql exit 0, and the restored database passes every survival arm | +| `test_a_custom_archive_round_trips` | the same through `pg_dump -Fc` and `pg_restore --exit-on-error` | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index cd252950e..1ffe4fbed 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -761,4 +761,6 @@ guard_tests 438 # native_decode_gating.sh: two tests, by collection over the cluster files on this tree. # cluster_tests 676 -> 677 for #1341's test_native_decode_gate_width.py, the twin of # native_decode_gate_width.sh: one test, by collection over the cluster files on this tree. -cluster_tests 677 +# cluster_tests 677 -> 680 for #1341's test_pg_dump_roundtrip.py, the twin of pg_dump_roundtrip.sh: +# three tests, by collection over the cluster files on this tree. +cluster_tests 680 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index f5965db19..3aa3c0385 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -144,6 +144,7 @@ "partition_lifecycle", "partition_rows", "partman_columnar", + "pg_dump_roundtrip", "prepared_across_sessions", "prepared_writes", "projection_drop_column", diff --git a/test/pytest/test_pg_dump_roundtrip.py b/test/pytest/test_pg_dump_roundtrip.py new file mode 100644 index 000000000..0d776de10 --- /dev/null +++ b/test/pytest/test_pg_dump_roundtrip.py @@ -0,0 +1,165 @@ +"""A columnar table survives pg_dump and restore, in both formats (#1341). The pytest +twin of `test/pg_dump_roundtrip.sh`, every check by the shell's name. + +A source database holds a columnar table with data, a per-table option, an index and a +projection, and a second table whose declared sort_by names columns after a dropped +one, so a restore renumbers their attnums (#288). It is dumped as plain SQL and as a +custom archive, and each is restored into a fresh database, where the rows, the access +method, the option, the declared sort_by and the projection declaration must all +survive. The projection's storage is not carried (#266) but rebuild_projections() +recovers it. + +Two arms are about more than their call returning: the point lookup must take the +restored INDEX, since the columnar scan answers it too with the index gone; and the +restored sort_by must leave the rows in (host, ts) order, counted as inversions in scan +order, which is why the readers run with parallel workers off. +""" +import pathlib +import subprocess +import tempfile + +import psycopg +import pytest + +SRC = "pgc_dumprt" +SUM_SQL = ("SELECT coalesce(sum(hashtextextended(id::text || '|' || kind::text || '|' || payload, 0)), 0) " + "FROM dt") +AM_SQL = ("SELECT a.amname::text FROM pg_class c JOIN pg_am a ON a.oid = c.relam " + "WHERE c.oid = 'dt'::regclass") +OPT_SQL = ("SELECT coalesce((SELECT encode_effort FROM pgcolumnar.options WHERE regclass = 'dt'::regclass), " + "'')") +PROJ_SQL = ("SELECT count(*) FROM pgcolumnar.projection WHERE storage_id = pgcolumnar.get_storage_id('dt') " + "AND projection_id > 0") +DECL_SQL = "SELECT count(*) FROM pgcolumnar.projection_declaration WHERE rel = 'dt'::regclass" +INVERSIONS = ("SELECT count(*) FROM (SELECT host, ts, lag(host) OVER () AS lh, lag(ts) OVER () AS lt FROM ds) x " + "WHERE (lh, lt) > (host, ts)") + + +def _conn(cluster, db): + return psycopg.connect(cluster.dsn(dbname=db), autocommit=True, + options="-c max_parallel_workers_per_gather=0") + + +def _one(cluster, db, sql): + with _conn(cluster, db) as c: + return c.execute(sql).fetchone()[0] + + +def _admin(cluster, sql): + with psycopg.connect(cluster.dsn(), autocommit=True) as c: + c.execute(sql) + + +def _tool(cluster, name, *args): + return subprocess.run([str(pathlib.Path(cluster.bindir) / name), "-h", "127.0.0.1", "-p", str(cluster.port), + "-U", "postgres", *args], capture_output=True, text=True).returncode + + +@pytest.fixture(scope="module") +def src(pgc_cluster): + """-> (cluster, the values the source holds before any dump).""" + _admin(pgc_cluster, f"DROP DATABASE IF EXISTS {SRC}") + _admin(pgc_cluster, f"CREATE DATABASE {SRC}") + try: + with _conn(pgc_cluster, SRC) as c: + for sql in ("CREATE EXTENSION pgcolumnar", + "CREATE TABLE dt (id bigint, kind int, payload text) USING pgcolumnar", + "SELECT pgcolumnar.set_options('dt', encode_effort => 'fast')", + "INSERT INTO dt SELECT g, g % 7, 'p' || g FROM generate_series(1, 50000) g", + "CREATE INDEX dt_id_idx ON dt (id)", + "SELECT pgcolumnar.add_projection('dt', 'dt_proj', ARRAY['id', 'payload'], ARRAY['id'])", + # #288: sort_by names columns that sit after a dropped one + "CREATE TABLE ds (id bigint, doomed int, host text, ts timestamptz) USING pgcolumnar", + "INSERT INTO ds (id, host, ts) SELECT g, 'h' || (g % 20), " + "'2024-01-01'::timestamptz + (g % 100) * interval '1 minute' FROM generate_series(1, 20000) g", + "ALTER TABLE ds DROP COLUMN doomed", + "SELECT pgcolumnar.set_options('ds', sort_by => ARRAY['host', 'ts'])"): + c.execute(sql) + before = {k: _one(pgc_cluster, SRC, sql) for k, sql in + (("count", "SELECT count(*) FROM dt"), ("sum", SUM_SQL), ("opt", OPT_SQL), + ("proj", PROJ_SQL), ("decl", DECL_SQL))} + yield pgc_cluster, before + finally: + _admin(pgc_cluster, f"DROP DATABASE IF EXISTS {SRC} WITH (FORCE)") + + +def test_dropping_a_projection_forgets_its_declaration(src, expect): + cluster, before = src + expect.num(before["proj"], 1, "the base table has a projection to preserve") + expect.num(before["decl"], 1, "and a declaration recorded for it") + # If DROP kept the declaration, a rebuild would resurrect what the operator removed. + with _conn(cluster, SRC) as c: + c.execute("SELECT pgcolumnar.add_projection('dt', 'dt_tmp', ARRAY['id'], ARRAY['id'])") + expect.num(_one(cluster, SRC, DECL_SQL), 2, "a second projection is declared") + with _conn(cluster, SRC) as c: + c.execute("SELECT pgcolumnar.drop_projection('dt', 'dt_tmp')") + expect.num(_one(cluster, SRC, DECL_SQL), 1, + "and dropping it forgets the declaration, so a rebuild cannot resurrect it") + expect.num(_one(cluster, SRC, "SELECT pgcolumnar.rebuild_projections()"), 0, "a rebuild now builds nothing") + + +def _verify(cluster, db, label, before, expect): + expect.num(_one(cluster, db, "SELECT count(*) FROM dt"), before["count"], f"{label}: row count survives") + expect.num(_one(cluster, db, SUM_SQL), before["sum"], f"{label}: row data survives (checksum)") + expect.text(_one(cluster, db, AM_SQL), "pgcolumnar", f"{label}: access method is pgcolumnar") + with _conn(cluster, db) as c: + c.execute("SET enable_seqscan = off") + c.execute("SET enable_bitmapscan = off") + plan = "\n".join(r[0] for r in c.execute("EXPLAIN (COSTS off) SELECT id FROM dt WHERE id = 12345")) + expect.num(sum(1 for line in plan.splitlines() if "Index" in line and "using dt_id_idx" in line), 1, + f"{label}: premise: the point lookup takes the restored index") + expect.num(c.execute("SELECT id FROM dt WHERE id = 12345").fetchone()[0], 12345, + f"{label}: restored index answers a point lookup") + expect.text(_one(cluster, db, OPT_SQL), before["opt"], f"{label}: per-table options survive") + # #266: the projection's storage is keyed by a storage id a restore reassigns + expect.num(_one(cluster, db, PROJ_SQL), 0, f"{label}: projection storage is NOT carried by pg_dump (see #266)") + expect.text(_one(cluster, db, "SELECT sort_by::text FROM pgcolumnar.options WHERE regclass = 'ds'::regclass"), + "{host,ts}", f"{label}: declared sort_by survives restore") + expect.text(str(_one(cluster, db, f"SELECT ({INVERSIONS}) > 0")), "True", + f"{label}: premise: the restored ds is not yet in its sort_by order") + with _conn(cluster, db) as c: + c.execute("SELECT pgcolumnar.vacuum_sorted('ds')") + applied = c.execute(f"SELECT count(*) || '|' || ({INVERSIONS}) FROM ds").fetchone()[0] + expect.text(applied, "20000|0", f"{label}: restored sort_by resolves and applies") + expect.num(_one(cluster, db, DECL_SQL), before["decl"], f"{label}: the projection declaration IS carried") + expect.num(_one(cluster, db, "SELECT pgcolumnar.rebuild_projections()"), 1, + f"{label}: rebuild_projections() reports it rebuilt one") + expect.num(_one(cluster, db, PROJ_SQL), before["proj"], f"{label}: and the projection exists again after the rebuild") + expect.num(_one(cluster, db, "SELECT count(*) FROM pgcolumnar.read_projection('dt', 'dt_proj')"), + _one(cluster, db, "SELECT count(*) FROM dt"), f"{label}: and it reads the same rows as the base table") + expect.num(_one(cluster, db, "SELECT pgcolumnar.rebuild_projections()"), 0, f"{label}: a second rebuild is a no-op") + + +def test_a_plain_dump_round_trips(src, expect): + cluster, before = src + db = f"{SRC}_p" + with tempfile.TemporaryDirectory() as d: + path = str(pathlib.Path(d) / "rt.sql") + expect.num(_tool(cluster, "pg_dump", "-d", SRC, "-f", path), 0, "plain: pg_dump exited 0") + text = pathlib.Path(path).read_text() + expect.text("yes" if ("USING pgcolumnar" in text or "default_table_access_method = pgcolumnar" in text) + else "no", "yes", "plain: dump recreates the table on the columnar AM") + _admin(cluster, f"DROP DATABASE IF EXISTS {db}") + _admin(cluster, f"CREATE DATABASE {db}") + try: + expect.num(_tool(cluster, "psql", "-d", db, "-v", "ON_ERROR_STOP=1", "-q", "-f", path), 0, + "plain: restore exited 0 (ON_ERROR_STOP)") + _verify(cluster, db, "plain", before, expect) + finally: + _admin(cluster, f"DROP DATABASE IF EXISTS {db} WITH (FORCE)") + + +def test_a_custom_archive_round_trips(src, expect): + cluster, before = src + db = f"{SRC}_c" + with tempfile.TemporaryDirectory() as d: + path = str(pathlib.Path(d) / "rt.dump") + expect.num(_tool(cluster, "pg_dump", "-Fc", "-d", SRC, "-f", path), 0, "custom: pg_dump -Fc exited 0") + _admin(cluster, f"DROP DATABASE IF EXISTS {db}") + _admin(cluster, f"CREATE DATABASE {db}") + try: + expect.num(_tool(cluster, "pg_restore", "--exit-on-error", "-d", db, path), 0, + "custom: pg_restore exited 0 (--exit-on-error)") + _verify(cluster, db, "custom", before, expect) + finally: + _admin(cluster, f"DROP DATABASE IF EXISTS {db} WITH (FORCE)") From 1da9bbfeb961b0b78b7389af896c17c23f5301d4 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:08 +0000 Subject: [PATCH 15/36] test: pg_dump_roundtrip seeds the ledger, 16 of 40 arms dated by five mutations (#1341) 40 rows from five-major runs (40/40 on each of 15-19). Dated on PG18 by: options (6) and projection declarations (8) no longer registered with pg_extension_config_dump, rebuild_projections() skipping missing projections (7), drop_projection keeping its declaration (8), and vacuum_sorted on the wrong column (2). suites_not_covered 213 -> 212; the census is 3795 rows, 2778 never, 1017 dated, restated on main db561684. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 40 ++++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 20 ++++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 7aea84451..c0267df50 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2777,6 +2777,46 @@ partman_columnar partman_columnar premise: with default_table_access_method ever partman_columnar partman_columnar premise: without default_table_access_method pg_partman makes heap partitions 15;16;17;18;19 never - partman_columnar partman_columnar retention leaves no pgcolumnar storage row behind 15;16;17;18;19 2026-10-12 the drop hook stops deleting the storage row, so a partition pg_partman drops leaves it behind partman_columnar partman_columnar the rows went in 15;16;17;18;19 2026-10-12 a sequential scan drops the last row group from its list, so every one-group partition reads empty +pg_dump_roundtrip pg_dump_roundtrip a rebuild now builds nothing 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip a second projection is declared 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip and a declaration recorded for it 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip and dropping it forgets the declaration, so a rebuild cannot resurrect it 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration +pg_dump_roundtrip pg_dump_roundtrip custom: a second rebuild is a no-op 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: access method is pgcolumnar 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: and it reads the same rows as the base table 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip custom: and the projection exists again after the rebuild 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip custom: declared sort_by survives restore 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: pgcolumnar.options no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip custom: per-table options survive 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: pgcolumnar.options no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip custom: pg_dump -Fc exited 0 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: pg_restore exited 0 (--exit-on-error) 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: premise: the point lookup takes the restored index 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: premise: the restored ds is not yet in its sort_by order 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: projection storage is NOT carried by pg_dump (see #266) 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: rebuild_projections() reports it rebuilt one 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip custom: restored index answers a point lookup 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: restored sort_by resolves and applies 15;16;17;18;19 2026-10-16 columnar_vacuum.c: vacuum_sorted sorts on column 1 instead of the resolved sort keys;pgcolumnar--1.0-alpha5.sql: pgcolumnar.options no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip custom: row count survives 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: row data survives (checksum) 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip custom: the projection declaration IS carried 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip plain: a second rebuild is a no-op 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: access method is pgcolumnar 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: and it reads the same rows as the base table 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip plain: and the projection exists again after the rebuild 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip plain: declared sort_by survives restore 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: pgcolumnar.options no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip plain: dump recreates the table on the columnar AM 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: per-table options survive 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: pgcolumnar.options no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip plain: pg_dump exited 0 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: premise: the point lookup takes the restored index 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: premise: the restored ds is not yet in its sort_by order 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: projection storage is NOT carried by pg_dump (see #266) 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: rebuild_projections() reports it rebuilt one 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump;pgcolumnar--1.0-alpha5.sql: rebuild_projections() skips the declarations whose projection is missing +pg_dump_roundtrip pg_dump_roundtrip plain: restore exited 0 (ON_ERROR_STOP) 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: restored index answers a point lookup 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: restored sort_by resolves and applies 15;16;17;18;19 2026-10-16 columnar_vacuum.c: vacuum_sorted sorts on column 1 instead of the resolved sort keys;pgcolumnar--1.0-alpha5.sql: pgcolumnar.options no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip plain: row count survives 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: row data survives (checksum) 15;16;17;18;19 never - +pg_dump_roundtrip pg_dump_roundtrip plain: the projection declaration IS carried 15;16;17;18;19 2026-10-16 columnar_projection.c: drop_projection keeps the projection's declaration;pgcolumnar--1.0-alpha5.sql: pgcolumnar.projection_declaration no longer registered with pg_extension_config_dump +pg_dump_roundtrip pg_dump_roundtrip the base table has a projection to preserve 15;16;17;18;19 never - prepared_across_sessions prepared_across_sessions ROLLBACK PREPARED of a DELETE leaves every row as it was 15;16;17;18;19 never - prepared_across_sessions prepared_across_sessions ROLLBACK PREPARED of an UPDATE leaves every row as it was, none duplicated 15;16;17;18;19 never - prepared_across_sessions prepared_across_sessions a prepared insert is invisible to another session 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 05119c5ab..94865b834 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 213 +suites_not_covered 212 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3031,4 +3031,20 @@ suites_not_covered 213 # never 2722 + 32 = 2754 # dated 998 + 3 = 1001 # 2754 + 1001 == 3755 -checks_never_observed_red 2754 +# +# #1341, pg_dump_roundtrip seeded, with its twin (OffgridwithJD): 2754 -> 2778, and +# suites_not_covered 213 -> 212. The suite had no rows. It enters with all 40 checks +# from five-major runs (40/40 on each of 15-19). 16 are dated by five mutations: the +# options and the projection-declaration catalogs no longer registered with +# pg_extension_config_dump, rebuild_projections() skipping a missing projection, +# drop_projection keeping its declaration, and vacuum_sorted sorting on column 1 +# (the strengthened sort arm). Twenty-four arrive never: the dump and restore exit +# codes and the dump's columnar AM, the access method, the row count and checksum, +# the index premise and its lookup, the pinned loss of projection storage, the +# inversion premise, the second rebuild's no-op, and the source's own premises. +# +# ledger rows 3755 + 40 = 3795 +# never 2754 + 24 = 2778 +# dated 1001 + 16 = 1017 +# 2778 + 1017 == 3795 +checks_never_observed_red 2778 From 58313d361535d3f7ad52443e3427dfb25eb70d59 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:08 +0000 Subject: [PATCH 16/36] test: pg_upgrade gets its pytest twin, and its index arm takes the index (#1341) The shell's "the index still answers the same range" never touched the index: right after pg_upgrade there are no planner statistics, the planner took the columnar scan for the range even with seqscan off, and the arm passed with c_id dropped after the upgrade. The arm and a new premise now run with the custom scan off, and the premise asserts the plan scans c_id. test_pg_upgrade.py carries all 16 checks by the shell's name. Like the shell it needs two majors: --pg-config is the new one, PGC_UPGRADE_FROM_PG_CONFIG the old one, whose extension it builds from a copy of the tree. The matrix runs one major at a time, so there it declines, listed for 15-19 in expected_unrunnable.txt. cluster_tests 680 -> 681, TESTS.md section 151. The suite has its own check() and records nothing, so no ledger rows. Co-Authored-By: Claude Opus 5.5 --- test/pg_upgrade.sh | 14 ++- test/pytest/TESTS.md | 13 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/expected_unrunnable.txt | 9 ++ test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_pg_upgrade.py | 165 ++++++++++++++++++++++++++++ 6 files changed, 204 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_pg_upgrade.py diff --git a/test/pg_upgrade.sh b/test/pg_upgrade.sh index bb4e5c226..ebf7fdcea 100755 --- a/test/pg_upgrade.sh +++ b/test/pg_upgrade.sh @@ -231,8 +231,20 @@ check "per-table options preserved" \ # Compared against the value captured on the OLD cluster, not against another # query on the new one -- the first version of this line compared the new answer # with itself and would have passed whatever the index did. +# The plan premise is what makes the next arm about the INDEX (#1341): the columnar +# scan is not disabled by enable_seqscan, and with the index dropped after the +# upgrade the same range count still came back right. +# Right after pg_upgrade there are no planner statistics, and the planner then took +# the columnar scan for this range even with seqscan off: measured, the arm below +# never touched the index. So the custom scan is turned off for the premise and the +# lookup alike, and any index scan of c_id (a bitmap one too) counts. +IDXOFF="SET enable_seqscan=off; SET pgcolumnar.enable_custom_scan=off;" +up_plan="$(nq "$IDXOFF EXPLAIN (COSTS off) SELECT count(*) FROM c WHERE id BETWEEN 100 AND 200")" +echo "-- range plan after the upgrade:"; sed 's/^/ /' <<<"$up_plan" +check "premise: the range lookup takes the upgraded index" \ + "$(grep -cE 'Index (Only )?Scan (using|on) c_id' <<<"$up_plan")" "1" check "the index still answers the same range" \ - "$(nq "SET enable_seqscan=off; SELECT count(*) FROM c WHERE id BETWEEN 100 AND 200" | tail -1)" \ + "$(nq "$IDXOFF SELECT count(*) FROM c WHERE id BETWEEN 100 AND 200" | tail -1)" \ "$old_range" # The upgraded relation must still be writable, not merely readable: a table that diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index cb4ff342d..610c707b0 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -196,6 +196,7 @@ behaviour, the source of that number is named. - [148. test_native_decode_gating.py: an unprunable qual gates payload decode per vector](#148-test_native_decode_gatingpy-an-unprunable-qual-gates-payload-decode-per-vector) - [149. test_native_decode_gate_width.py: decode gating runs only on a projection wide enough to pay](#149-test_native_decode_gate_widthpy-decode-gating-runs-only-on-a-projection-wide-enough-to-pay) - [150. test_pg_dump_roundtrip.py: a columnar table survives pg_dump and restore](#150-test_pg_dump_roundtrippy-a-columnar-table-survives-pg_dump-and-restore) +- [151. test_pg_upgrade.py: a columnar table survives pg_upgrade](#151-test_pg_upgradepy-a-columnar-table-survives-pg_upgrade) ## 1. How to read a test in here @@ -7477,3 +7478,15 @@ The pytest twin of `test/pg_dump_roundtrip.sh` (ported for #1341), every check b | `test_dropping_a_projection_forgets_its_declaration` | a dropped projection's declaration goes with it, so a rebuild builds nothing | | `test_a_plain_dump_round_trips` | pg_dump and psql exit 0, and the restored database passes every survival arm | | `test_a_custom_archive_round_trips` | the same through `pg_dump -Fc` and `pg_restore --exit-on-error` | + +## 151. test_pg_upgrade.py: a columnar table survives pg_upgrade + +The pytest twin of `test/pg_upgrade.sh` (ported for #1341), every check by the shell's name. A cluster on an older major gets a columnar table with non-default options, deletes and an index; it is upgraded with pg_upgrade into an empty cluster on the new major, and the row count, the content byte for byte, the options, the index and writability must all survive. The index arm runs with the columnar scan off and must take the index: right after pg_upgrade there are no planner statistics, the planner took the columnar scan for the range, and the arm passed with the index dropped. + +Like the shell it needs TWO majors: the new one is `--pg-config`, the old one `PGC_UPGRADE_FROM_PG_CONFIG`, and the test builds and installs the extension for the old one from a copy of the tree. The matrix runs one major at a time, so there it declines, and `expected_unrunnable.txt` lists it for every major. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_pg_upgrade_preserves_a_columnar_table` | pg_upgrade exits 0, and the upgraded table keeps its rows, content, options and index, and takes writes | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 1ffe4fbed..22f9726d0 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -763,4 +763,6 @@ guard_tests 438 # native_decode_gate_width.sh: one test, by collection over the cluster files on this tree. # cluster_tests 677 -> 680 for #1341's test_pg_dump_roundtrip.py, the twin of pg_dump_roundtrip.sh: # three tests, by collection over the cluster files on this tree. -cluster_tests 680 +# cluster_tests 680 -> 681 for #1341's test_pg_upgrade.py, the twin of pg_upgrade.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 681 diff --git a/test/pytest/expected_unrunnable.txt b/test/pytest/expected_unrunnable.txt index 135205390..3259220b0 100644 --- a/test/pytest/expected_unrunnable.txt +++ b/test/pytest/expected_unrunnable.txt @@ -78,3 +78,12 @@ 16 test_inheritance.py::test_a_partitioned_parent_is_refused_and_its_leaf_is_not 15 test_parquet_import.py::test_a_partitioned_parent_is_refused 16 test_parquet_import.py::test_a_partitioned_parent_is_refused + +# Every major: pg_upgrade needs a SECOND, older major, named by +# PGC_UPGRADE_FROM_PG_CONFIG, and the matrix runs one major at a time. A run that +# sets it is an upgrade run, not a matrix run, and this file would refuse the row. +15 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table +16 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table +17 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table +18 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table +19 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 3aa3c0385..8437604a1 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -145,6 +145,7 @@ "partition_rows", "partman_columnar", "pg_dump_roundtrip", + "pg_upgrade", "prepared_across_sessions", "prepared_writes", "projection_drop_column", diff --git a/test/pytest/test_pg_upgrade.py b/test/pytest/test_pg_upgrade.py new file mode 100644 index 000000000..5061ff89d --- /dev/null +++ b/test/pytest/test_pg_upgrade.py @@ -0,0 +1,165 @@ +"""Columnar tables survive pg_upgrade across majors (#239, #1341). The pytest twin of +`test/pg_upgrade.sh`, every check by the shell's name. + +pg_upgrade carries relation files and the pgcolumnar.* catalogs to a new major. A +cluster on the OLD major gets a columnar table with non-default options, deletes and an +index; an empty cluster on the NEW major, the extension installed on both, is upgraded +in place of it; and the rows, their content, the options, the index and writability +must all survive. The index arm runs with the columnar scan off: right after pg_upgrade +there are no planner statistics, and the planner took the columnar scan for the range, +so without that the arm never touched the index. + +TWO MAJORS AT ONCE, like the shell, which is why neither runs in the matrix: the new +major is --pg-config and the old one is PGC_UPGRADE_FROM_PG_CONFIG. Without it, or +with an old major not below the new one, the test declines. +""" +import os +import pathlib +import re +import shutil +import subprocess +import tempfile + +import psycopg + +REPO = pathlib.Path(__file__).resolve().parents[2] + + +def _cfg(pg_config, flag): + return subprocess.run([pg_config, flag], capture_output=True, text=True, check=True).stdout.strip() + + +def _major(pg_config): + # "PostgreSQL 19rc1" is 19: the leading digits, not every digit + return int(re.match(r"\d+", _cfg(pg_config, "--version").split()[1]).group(0)) + + +def _sh(cmd, cwd=None): + return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True) + + +def _q(port, sql): + """-> the first column of the first row, or '' when the query cannot run.""" + try: + with psycopg.connect(f"host=127.0.0.1 port={port} user=postgres dbname=postgres", autocommit=True, + connect_timeout=10) as c: + row = c.execute(sql).fetchone() + return "" if row is None else row[0] + except psycopg.Error: + return "" + + +def _run(port, *sqls): + with psycopg.connect(f"host=127.0.0.1 port={port} user=postgres dbname=postgres", autocommit=True) as c: + for sql in sqls: + c.execute(sql) + + +def _cluster(bindir, data, port): + """initdb -k (data checksums on both sides, or pg_upgrade refuses the pair) and conf.""" + _sh([f"{bindir}/initdb", "-k", "-D", str(data), "-A", "trust"]) + with open(data / "postgresql.conf", "a") as fh: + fh.write(f"\nport={port}\nlisten_addresses='127.0.0.1'\nshared_preload_libraries='pgcolumnar'\n") + + +def _ctl(bindir, data, action, log=None): + cmd = [f"{bindir}/pg_ctl", "-D", str(data), "-w"] + cmd += (["-l", str(log), "start"] if action == "start" else ["-m", "fast", "stop"]) + return _sh(cmd).returncode + + +def test_pg_upgrade_preserves_a_columnar_table(pgc_cluster, expect): + from pgc_cluster import pick_port + + old_pgc = os.environ.get("PGC_UPGRADE_FROM_PG_CONFIG", "") + new_pgc = pgc_cluster.pg_config + if not old_pgc or not os.access(old_pgc, os.X_OK): + expect.cannot_run("UNMET_PRECONDITION", + "PGC_UPGRADE_FROM_PG_CONFIG names no older major's pg_config, so there is nothing to " + "upgrade from; the matrix runs one major at a time", + name="pg_upgrade exited cleanly") + return + old_maj, new_maj = _major(old_pgc), _major(new_pgc) + if new_maj <= old_maj: + expect.cannot_run("UNMET_PRECONDITION", f"PG{old_maj} -> PG{new_maj} is not an upgrade", + name="pg_upgrade exited cleanly") + return + old_bin, new_bin = _cfg(old_pgc, "--bindir"), _cfg(new_pgc, "--bindir") + + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-upgrade-")) + work.chmod(0o777) + old_data, new_data = work / "old", work / "new" + old_port, new_port = pick_port(17), pick_port(18) + try: + # The extension on the OLD major, built from a copy so the two majors' objects + # never mix; conftest already installed it for the new one. + build = work / "src" + shutil.copytree(REPO, build, ignore=shutil.ignore_patterns(".git", "*.o", "*.so", "*.bc", "__pycache__")) + _sh(["make", "-C", str(build), "clean", f"PG_CONFIG={old_pgc}"]) + built = _sh(["make", "-C", str(build), f"PG_CONFIG={old_pgc}"]).returncode == 0 and \ + _sh(["make", "-C", str(build), "install", f"PG_CONFIG={old_pgc}"]).returncode == 0 + assert built, "the extension did not build and install for the old major" + + _cluster(old_bin, old_data, old_port) + _ctl(old_bin, old_data, "start", work / "old.log") + _run(old_port, "CREATE EXTENSION pgcolumnar", + "CREATE TABLE c (id int, v text, n numeric) USING pgcolumnar", + # non-default options, so the options catalog row is carried, not only the pages + "SELECT pgcolumnar.set_options('c', stripe_row_limit => 4000, chunk_group_row_limit => 800, " + "compression => 'zstd')", + "INSERT INTO c SELECT g, md5(g::text), (g::numeric) / 7 FROM generate_series(1, 50000) g", + "DELETE FROM c WHERE id % 13 = 0", + "CREATE INDEX c_id ON c (id)") + content = "SELECT md5(string_agg(t::text, '' ORDER BY t::text)) FROM c t" + opts = ("SELECT stripe_row_limit || '/' || chunk_group_row_limit || '/' || compression " + "FROM pgcolumnar.options WHERE regclass = 'c'::regclass") + old_hash, old_count, old_opts = _q(old_port, content), _q(old_port, "SELECT count(*) FROM c"), _q(old_port, opts) + old_range = _q(old_port, "SELECT count(*) FROM c WHERE id BETWEEN 100 AND 200") + expect.text("yes" if isinstance(old_count, int) and old_count > 0 else "no", "yes", + "old cluster has rows to preserve") + expect.text(old_opts, "4000/800/zstd", "old cluster recorded non-default options") + _ctl(old_bin, old_data, "stop") + + _cluster(new_bin, new_data, new_port) + _ctl(new_bin, new_data, "start", work / "new-pre.log") + # a pass must not come from the table having been made here + expect.num(_q(new_port, "SELECT count(*) FROM pg_class WHERE relname = 'c'"), 0, + "the new cluster does NOT have the table before the upgrade") + expect.num(int(_q(new_port, "SHOW server_version_num")) // 10000, new_maj, "and is genuinely the new major") + _ctl(new_bin, new_data, "stop") + + up = _sh([f"{new_bin}/pg_upgrade", "-b", old_bin, "-B", new_bin, "-d", str(old_data), "-D", str(new_data), + "-p", str(old_port), "-P", str(new_port), "-U", "postgres"], cwd=work) + expect.num(up.returncode, 0, "pg_upgrade exited cleanly") + _ctl(new_bin, new_data, "start", work / "new.log") + + expect.num(_q(new_port, "SELECT 1"), 1, "the new cluster starts after the upgrade") + expect.num(int(_q(new_port, "SHOW server_version_num") or 0) // 10000, new_maj, "we are on the new major") + expect.num(_q(new_port, "SELECT count(*) FROM pg_extension WHERE extname = 'pgcolumnar'"), 1, + "the extension is present") + expect.text(_q(new_port, "SELECT a.amname::text FROM pg_class c JOIN pg_am a ON a.oid = c.relam " + "WHERE c.relname = 'c'"), "pgcolumnar", "the table is still a columnar table") + expect.num(_q(new_port, "SELECT count(*) FROM c"), old_count, "row count preserved") + expect.text(_q(new_port, content), old_hash, "content preserved byte for byte") + expect.text(_q(new_port, opts), old_opts, "per-table options preserved") + with psycopg.connect(f"host=127.0.0.1 port={new_port} user=postgres dbname=postgres", autocommit=True) as c: + c.execute("SET enable_seqscan = off") + c.execute("SET pgcolumnar.enable_custom_scan = off") + plan = "\n".join(r[0] for r in c.execute( + "EXPLAIN (COSTS off) SELECT count(*) FROM c WHERE id BETWEEN 100 AND 200")) + uses = sum(1 for line in plan.splitlines() + if ("Index Scan" in line or "Index Only Scan" in line) and "c_id" in line) + expect.num(uses, 1, "premise: the range lookup takes the upgraded index") + # compared with the OLD cluster's answer, not with another query on the new one + expect.num(c.execute("SELECT count(*) FROM c WHERE id BETWEEN 100 AND 200").fetchone()[0], old_range, + "the index still answers the same range") + _run(new_port, "INSERT INTO c SELECT g, 'post', 1 FROM generate_series(900001, 900100) g") + expect.num(_q(new_port, "SELECT count(*) FROM c WHERE v = 'post'"), 100, + "the upgraded table still accepts writes") + expect.num(_q(new_port, "SELECT count(*) FROM c"), old_count + 100, + "and reads back the new rows with the old ones") + finally: + for b, d in ((old_bin, old_data), (new_bin, new_data)): + if (d / "postmaster.pid").exists(): + _sh([f"{b}/pg_ctl", "-D", str(d), "-m", "immediate", "-w", "stop"]) + shutil.rmtree(work, ignore_errors=True) From d70e5d6f3c0502d9a6ccd5933fde14c7db8919ee Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:08 +0000 Subject: [PATCH 17/36] test: test_pg_upgrade runs its server tools as postgres under root, and names a refused step (#1341) jdatcmd, reviewing #1461: under root, initdb refused ("cannot be run as root"), _sh discarded the code, and the failure surfaced one step later as a FileNotFoundError for postgresql.conf. initdb, pg_ctl and pg_upgrade now go through _server_tool, which drops to postgres with runuser when euid is 0 (as pgc_cluster._asroot and lib.sh do), chowns the data directories, and raises naming the step and its stderr. initdb passes -U postgres, and the socket directory is the datadir, as pgc_cluster pins it. A failed make of the old major's extension now shows its stderr too. Measured as root (sudo, PG17 -> PG18): 585369d6 fails with that FileNotFoundError, this commit is 16/16. As a normal user: 16/16. Co-Authored-By: Claude Opus 5.5 --- test/pytest/test_pg_upgrade.py | 50 ++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 12 deletions(-) diff --git a/test/pytest/test_pg_upgrade.py b/test/pytest/test_pg_upgrade.py index 5061ff89d..e920ae1b1 100644 --- a/test/pytest/test_pg_upgrade.py +++ b/test/pytest/test_pg_upgrade.py @@ -38,6 +38,23 @@ def _sh(cmd, cwd=None): return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True) +def _server_tool(bindir, argv, own=(), cwd=None, check=True): + """Run initdb, pg_ctl or pg_upgrade, as postgres when we are root: each refuses + root, which is why pgc_cluster._asroot and lib.sh's runuser exist. A refused step + raises naming itself and its stderr, so it cannot surface a step later as a + missing file (jdatcmd, review of #1461). `own` is chowned to postgres first.""" + cmd = [str(pathlib.Path(bindir) / argv[0])] + list(argv[1:]) + if os.geteuid() == 0: + for top in own: + for path in [top, *pathlib.Path(top).rglob("*")]: + shutil.chown(path, user="postgres") + cmd = ["runuser", "-u", "postgres", "--"] + cmd + proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True) + if check and proc.returncode != 0: + raise RuntimeError(f"{argv[0]} failed rc={proc.returncode}: {(proc.stderr or proc.stdout).strip()[-2000:]}") + return proc + + def _q(port, sql): """-> the first column of the first row, or '' when the query cannot run.""" try: @@ -56,16 +73,20 @@ def _run(port, *sqls): def _cluster(bindir, data, port): - """initdb -k (data checksums on both sides, or pg_upgrade refuses the pair) and conf.""" - _sh([f"{bindir}/initdb", "-k", "-D", str(data), "-A", "trust"]) + """initdb -k (data checksums on both sides, or pg_upgrade refuses the pair) and conf. + The socket directory is the datadir, as pgc_cluster pins it: a packaged build's + default is a directory only postgres can write.""" + data.mkdir() + _server_tool(bindir, ["initdb", "-k", "-D", str(data), "-A", "trust", "-U", "postgres"], own=[data]) with open(data / "postgresql.conf", "a") as fh: - fh.write(f"\nport={port}\nlisten_addresses='127.0.0.1'\nshared_preload_libraries='pgcolumnar'\n") + fh.write(f"\nport={port}\nlisten_addresses='127.0.0.1'\nunix_socket_directories='{data}'\n" + "shared_preload_libraries='pgcolumnar'\n") def _ctl(bindir, data, action, log=None): - cmd = [f"{bindir}/pg_ctl", "-D", str(data), "-w"] - cmd += (["-l", str(log), "start"] if action == "start" else ["-m", "fast", "stop"]) - return _sh(cmd).returncode + argv = ["pg_ctl", "-D", str(data), "-w"] + argv += (["-l", str(log), "start"] if action == "start" else ["-m", "fast", "stop"]) + return _server_tool(bindir, argv, own=[data]).returncode def test_pg_upgrade_preserves_a_columnar_table(pgc_cluster, expect): @@ -96,9 +117,10 @@ def test_pg_upgrade_preserves_a_columnar_table(pgc_cluster, expect): build = work / "src" shutil.copytree(REPO, build, ignore=shutil.ignore_patterns(".git", "*.o", "*.so", "*.bc", "__pycache__")) _sh(["make", "-C", str(build), "clean", f"PG_CONFIG={old_pgc}"]) - built = _sh(["make", "-C", str(build), f"PG_CONFIG={old_pgc}"]).returncode == 0 and \ - _sh(["make", "-C", str(build), "install", f"PG_CONFIG={old_pgc}"]).returncode == 0 - assert built, "the extension did not build and install for the old major" + for target in ([], ["install"]): + made = _sh(["make", "-C", str(build), *target, f"PG_CONFIG={old_pgc}"]) + assert made.returncode == 0, \ + f"make {' '.join(target)} for the old major failed: {made.stderr.strip()[-2000:]}" _cluster(old_bin, old_data, old_port) _ctl(old_bin, old_data, "start", work / "old.log") @@ -128,8 +150,12 @@ def test_pg_upgrade_preserves_a_columnar_table(pgc_cluster, expect): expect.num(int(_q(new_port, "SHOW server_version_num")) // 10000, new_maj, "and is genuinely the new major") _ctl(new_bin, new_data, "stop") - up = _sh([f"{new_bin}/pg_upgrade", "-b", old_bin, "-B", new_bin, "-d", str(old_data), "-D", str(new_data), - "-p", str(old_port), "-P", str(new_port), "-U", "postgres"], cwd=work) + # pg_upgrade refuses root too, and writes its logs and socket into cwd + up = _server_tool(new_bin, ["pg_upgrade", "-b", old_bin, "-B", new_bin, "-d", str(old_data), + "-D", str(new_data), "-p", str(old_port), "-P", str(new_port), + "-U", "postgres"], own=[old_data, new_data], cwd=work, check=False) + if up.returncode != 0: + print(f"-- pg_upgrade said:\n{up.stdout[-3000:]}\n{up.stderr[-2000:]}") expect.num(up.returncode, 0, "pg_upgrade exited cleanly") _ctl(new_bin, new_data, "start", work / "new.log") @@ -161,5 +187,5 @@ def test_pg_upgrade_preserves_a_columnar_table(pgc_cluster, expect): finally: for b, d in ((old_bin, old_data), (new_bin, new_data)): if (d / "postmaster.pid").exists(): - _sh([f"{b}/pg_ctl", "-D", str(d), "-m", "immediate", "-w", "stop"]) + _server_tool(b, ["pg_ctl", "-D", str(d), "-m", "immediate", "-w", "stop"], check=False) shutil.rmtree(work, ignore_errors=True) From a25fdffc06876262e5ac36038314ccf36140d44e Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:08 +0000 Subject: [PATCH 18/36] test: replication gets its pytest twin, and its vacuum and compact arms replay a write (#1341) Measured: compact on r retired nothing, because vacuum had already reclaimed every deleted row, so "compact replays identically" compared a state compact never changed. The fixture now gives compact one fully deleted row group, and a new arm asserts it was retired (1|1). "vacuum rewrote r into a new relfilenode on the primary" does the same for vacuum, and a premise pins r2's second batch, without which the module-less replay arms passed on a failed INSERT. Red first: with vacuum returning before it rewrites, the old shell is 41/41 green and the new one reddens the vacuum arm. With compact retiring nothing, the new compact arm is red. test_replication.py carries all 44 checks by the shell's name, on a private primary, a streaming standby and an isolated restore. COMPLETE gains replication, TESTS.md section 152, cluster_tests 681 -> 682. Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 13 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_replication.py | 269 ++++++++++++++++++++++++++++ test/replication.sh | 22 ++- 5 files changed, 307 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_replication.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 610c707b0..9a2e28fd3 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -197,6 +197,7 @@ behaviour, the source of that number is named. - [149. test_native_decode_gate_width.py: decode gating runs only on a projection wide enough to pay](#149-test_native_decode_gate_widthpy-decode-gating-runs-only-on-a-projection-wide-enough-to-pay) - [150. test_pg_dump_roundtrip.py: a columnar table survives pg_dump and restore](#150-test_pg_dump_roundtrippy-a-columnar-table-survives-pg_dump-and-restore) - [151. test_pg_upgrade.py: a columnar table survives pg_upgrade](#151-test_pg_upgradepy-a-columnar-table-survives-pg_upgrade) +- [152. test_replication.py: physical replication and standby redo](#152-test_replicationpy-physical-replication-and-standby-redo) ## 1. How to read a test in here @@ -7490,3 +7491,15 @@ Like the shell it needs TWO majors: the new one is `--pg-config`, the old one `P | test | what it holds | | --- | --- | | `test_pg_upgrade_preserves_a_columnar_table` | pg_upgrade exits 0, and the upgraded table keeps its rows, content, options and index, and takes writes | + +## 152. test_replication.py: physical replication and standby redo + +The pytest twin of `test/replication.sh` (ported for #1341), every check by the shell's name. A private primary gets a columnar table and a projection. pg_basebackup makes a streaming standby of it, and a second backup is brought up alone with no upstream, so what it holds came from the backup bytes. Inserts, deletes, updates, vacuum, compact and an SMGR end truncation are replayed and compared with the primary. A standby running without the module must still replay, because every record the extension emits is a core type. + +Each replay arm is about a write the primary made. Vacuum must have rewritten the table into a new relfilenode. Compact must have retired a row group; after vacuum it had none to retire until the fixture gave it one. The truncate must have shrunk the primary. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_a_streaming_standby_and_a_restore_replay_columnar_tables` | the standby and the isolated restore hold the primary's rows and projections, every write path replays identically, and a module-less standby replays | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 22f9726d0..10ac37654 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -765,4 +765,6 @@ guard_tests 438 # three tests, by collection over the cluster files on this tree. # cluster_tests 680 -> 681 for #1341's test_pg_upgrade.py, the twin of pg_upgrade.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 681 +# cluster_tests 681 -> 682 for #1341's test_replication.py, the twin of replication.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 682 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 8437604a1..bfd786dfe 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -157,6 +157,7 @@ "projections", "recovery", "recovery_minimal_wal", + "replication", "rewrite_storage_oid", "rls_direct_storage", "rls_query_paths", diff --git a/test/pytest/test_replication.py b/test/pytest/test_replication.py new file mode 100644 index 000000000..7fedd6f21 --- /dev/null +++ b/test/pytest/test_replication.py @@ -0,0 +1,269 @@ +"""Physical replication and standby redo of columnar tables (#241, #1341). The pytest +twin of `test/replication.sh`, every check by the shell's name. + +Every WAL record the extension emits is a core type: data pages go through +log_newpage and the one direct insert is an SMGR truncate. So a standby replays +columnar WAL with core redo and needs the module only to READ a columnar table. A +private primary gets a columnar table and a projection, pg_basebackup makes a +streaming standby of it, and a second backup is brought up on its own with no +upstream at all, so what it holds came from the backup bytes alone. Then inserts, +deletes, updates, vacuum, compact and an end truncation are replayed and compared +with the primary, and a standby running WITHOUT the module must still replay. + +Each replay arm is about a write the primary made. Vacuum must have rewritten the +table into a new relfilenode, compact must have retired a row group (measured: after +vacuum it had nothing to retire), and the truncate must have shrunk the primary, +or the standby would be compared against a state nothing changed. + +PRIVATE CLUSTERS, three of them: the standby and the restore need a primary they can +back up, with end truncation on for its database. +""" +import pathlib +import shutil +import subprocess +import tempfile +import time + +import psycopg + +from pgc_vacuity import query_error + +PROJ_SQL = ("SELECT count(*) FROM pgcolumnar.projection " + "WHERE storage_id = pgcolumnar.get_storage_id('r') AND projection_id > 0") +RG_SQL = "SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('r')" +PROJ_HASH = "SELECT md5(string_agg(x, '' ORDER BY x)) FROM pgcolumnar.read_projection('r','{p}') x" + + +def _q(cluster, sql): + """-> the first column of the first row, or a distinct failure sentinel.""" + try: + with psycopg.connect(cluster.dsn(), autocommit=True, connect_timeout=5) as c: + row = c.execute(sql).fetchone() + return None if row is None else row[0] + except psycopg.Error as exc: + return query_error(str(exc).splitlines()[0]) + + +def _run(cluster, *sqls): + with psycopg.connect(cluster.dsn(), autocommit=True) as c: + for sql in sqls: + c.execute(sql) + + +def _hash(cluster, table): + return _q(cluster, f"SELECT md5(string_agg(t::text, '' ORDER BY t::text)) FROM {table} t") + + +def _answers(cluster, tries=240): + for _ in range(tries): + if _q(cluster, "SELECT 1") == 1: + return True + time.sleep(0.5) + return False + + +def _caught_up(primary, standby, tries=360): + """Has the standby replayed at least the primary's current LSN? Polled, three + minutes, like the shell: a slow standby is not a broken one.""" + target = _q(primary, "SELECT pg_current_wal_lsn()::text") + for _ in range(tries): + if _q(standby, f"SELECT pg_last_wal_replay_lsn() >= '{target}'::pg_lsn") is True: + return True + time.sleep(0.5) + return False + + +def _basebackup(primary, datadir, *extra): + return subprocess.run([str(pathlib.Path(primary.bindir) / "pg_basebackup"), "-h", "127.0.0.1", + "-p", str(primary.port), "-U", "postgres", "-D", str(datadir), *extra, + "-X", "stream", "-c", "fast"], capture_output=True, text=True) + + +def _conf(datadir, *lines): + """Append settings; a later line wins, so this overrides what the backup copied.""" + with open(datadir / "postgresql.conf", "a") as fh: + fh.write("\n" + "\n".join(lines) + "\n") + + +def test_a_streaming_standby_and_a_restore_replay_columnar_tables(pgc_cluster, expect): + from pgc_cluster import Cluster, pick_port + + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-repl-")) + work.chmod(0o777) + (work / "primary").mkdir() + pri = Cluster(pgc_cluster.pg_config, "repl", work / "primary", pick_port(20)) + sb_dir, rs_dir = work / "standby", work / "restore" + sb = rs = None + try: + pri.initdb() + pri.start() + _run(pri, "CREATE EXTENSION pgcolumnar", + "CREATE TABLE r (id int, v text) USING pgcolumnar", + "INSERT INTO r SELECT g, 'row' || g FROM generate_series(1,20000) g", + # declared BEFORE the backup, so the backup's bytes must contain it + "SELECT pgcolumnar.add_projection('r', 'r_pre', ARRAY['id','v'], ARRAY['id'])") + pri_proj = _q(pri, PROJ_SQL) + expect.num(pri_proj, 1, "the primary has a projection to preserve, before the backup runs") + pri_pre_rows = _q(pri, "SELECT count(*) FROM pgcolumnar.read_projection('r','r_pre')") + expect.num(pri_pre_rows, 20000, "and it reads on the primary") + + # -C -S: a physical slot, so the primary keeps the WAL the standby still needs + _basebackup(pri, sb_dir, "-R", "-C", "-S", "pgc_standby") + expect.text("yes" if (sb_dir / "PG_VERSION").exists() else "no", "yes", + "pg_basebackup produced a data directory") + sb = Cluster(pgc_cluster.pg_config, "repl-sb", sb_dir, pick_port(21)) + _conf(sb_dir, f"port={sb.port}", f"unix_socket_directories='{sb_dir}'") + sb.start() + _answers(sb) + expect.num(_q(sb, "SELECT 1"), 1, "the standby accepts connections") + expect.num(_q(pri, "SELECT count(*) FROM pg_replication_slots " + "WHERE slot_name = 'pgc_standby' AND slot_type = 'physical'"), 1, + "a physical slot exists to hold WAL for the standby") + expect.text(str(_q(pri, "SELECT active FROM pg_replication_slots WHERE slot_name = 'pgc_standby'")), + "True", "and the standby is actually connected through it") + + # the controls: were this the primary, every comparison below would agree + expect.text(str(_q(sb, "SELECT pg_is_in_recovery()")), "True", "the standby is in recovery") + expect.text("yes" if _q(sb, "SHOW data_directory") != _q(pri, "SHOW data_directory") else "no", "yes", + "the standby is a different cluster from the primary") + + assert _caught_up(pri, sb), "the standby did not catch up (sync 1)" + expect.hash(_hash(sb, "r"), _hash(pri, "r"), "the base backup carried the initial rows") + expect.num(_q(sb, PROJ_SQL), pri_proj, "pg_basebackup carried the projection catalog row, unlike pg_dump (#266)") + expect.num(_q(sb, "SELECT count(*) FROM pgcolumnar.read_projection('r','r_pre')"), pri_pre_rows, + "and the projection's storage came across intact and readable") + expect.hash(_q(sb, PROJ_HASH.format(p="r_pre")), _q(pri, PROJ_HASH.format(p="r_pre")), + "with the same content the primary reads") + + # a projection made AFTER the backup reaches the standby by replay + _run(pri, "SELECT pgcolumnar.add_projection('r', 'r_post', ARRAY['id','v'], ARRAY['id'])") + expect.num(_q(pri, PROJ_SQL), 2, "the primary now has a second, post-backup projection") + assert _caught_up(pri, sb), "the standby did not catch up (sync 2)" + expect.num(_q(sb, PROJ_SQL), 2, "streaming replay carried a projection created after the backup") + expect.num(_q(sb, "SELECT count(*) FROM pgcolumnar.read_projection('r','r_post')"), pri_pre_rows, + "and that one reads on the standby too") + + # The same claim with streaming taken out: no -R, so no standby.signal and no + # upstream. What this cluster holds came from the backup bytes alone. + _basebackup(pri, rs_dir) + expect.text("yes" if (rs_dir / "PG_VERSION").exists() else "no", "yes", + "the second base backup produced a data directory") + rs = Cluster(pgc_cluster.pg_config, "repl-rs", rs_dir, pick_port(22)) + _conf(rs_dir, f"port={rs.port}", f"unix_socket_directories='{rs_dir}'") + rs.start() + _answers(rs) + expect.num(_q(rs, "SELECT 1"), 1, "the restored cluster accepts connections") + expect.text(str(_q(rs, "SELECT pg_is_in_recovery()")), "False", + "the restored cluster is not in recovery (it is not following anyone)") + expect.num(_q(rs, "SELECT count(*) FROM pg_stat_wal_receiver"), 0, "and has no upstream configured") + rs_dd = _q(rs, "SHOW data_directory") + expect.text("yes" if rs_dd not in (_q(pri, "SHOW data_directory"), _q(sb, "SHOW data_directory")) else "no", + "yes", "and is a different cluster from the primary and the standby") + # two empty results would agree, so the primary side is pinned first + rs_pri_count = _q(pri, PROJ_SQL) + rs_pri_hash = _q(pri, PROJ_HASH.format(p="r_pre")) + expect.num(rs_pri_count, 2, "the primary still has both projections when the backup is taken") + expect.text("yes" if isinstance(rs_pri_hash, str) and rs_pri_hash else "no", "yes", + "and a non-empty projection to compare against") + expect.num(_q(rs, PROJ_SQL), rs_pri_count, "a physical backup alone preserves the projections, unlike pg_dump (#266)") + expect.hash(_q(rs, PROJ_HASH.format(p="r_pre")), rs_pri_hash, "and its storage, readable with the same content") + expect.num(_q(rs, "SELECT count(*) FROM pgcolumnar.read_projection('r','r_post')"), pri_pre_rows, + "and the later projection is in this backup as well, so it is not stale") + rs.stop() + + # Positive control: with replay paused, a change on the primary must NOT appear. + _q(sb, "SELECT pg_wal_replay_pause()") + _run(pri, "INSERT INTO r SELECT g, 'paused' || g FROM generate_series(90001,90100) g") + time.sleep(1) + expect.num(_q(sb, "SELECT count(*) FROM r WHERE v LIKE 'paused%'"), 0, + "with replay paused the standby does NOT see the new rows") + _q(sb, "SELECT pg_wal_replay_resume()") + assert _caught_up(pri, sb), "the standby did not catch up (sync 3)" + expect.num(_q(sb, "SELECT count(*) FROM r WHERE v LIKE 'paused%'"), 100, "after resuming, the standby sees them") + + # scenario 1: the ordinary write paths + _run(pri, "INSERT INTO r SELECT g, 'more' || g FROM generate_series(20001,40000) g", + "DELETE FROM r WHERE id % 7 = 0", + "UPDATE r SET v = v || '-u' WHERE id % 11 = 0") + assert _caught_up(pri, sb), "the standby did not catch up (sync 4)" + expect.hash(_hash(sb, "r"), _hash(pri, "r"), "insert, delete and update replay identically") + expect.num(_q(sb, "SELECT count(*) FROM r"), _q(pri, "SELECT count(*) FROM r"), "and the row counts agree") + + # scenario 2: each replay arm is about a write the primary made + fn_before = _q(pri, "SELECT pg_relation_filenode('r')") + _run(pri, "SELECT pgcolumnar.vacuum('r')") + fn_after = _q(pri, "SELECT pg_relation_filenode('r')") + expect.text("yes" if isinstance(fn_before, int) and fn_after != fn_before else "no", "yes", + "vacuum rewrote r into a new relfilenode on the primary") + assert _caught_up(pri, sb), "the standby did not catch up (sync 5)" + expect.hash(_hash(sb, "r"), _hash(pri, "r"), "vacuum replays identically") + # a row group of its own (one INSERT is one flush), then all of it deleted + _run(pri, "INSERT INTO r SELECT g, 'doomed' || g FROM generate_series(50001,51000) g", + "DELETE FROM r WHERE v LIKE 'doomed%'") + rg_before = _q(pri, RG_SQL) + retired = _q(pri, "SELECT pgcolumnar.compact('r')") + expect.text(f"{retired}|{rg_before - _q(pri, RG_SQL)}", "1|1", + "compact retired the fully deleted row group on the primary") + assert _caught_up(pri, sb), "the standby did not catch up (sync 6)" + expect.hash(_hash(sb, "r"), _hash(pri, "r"), "compact replays identically") + + # The SMGR truncate, the one direct XLogInsert. End truncation is opt-in, and + # the trailing rows need row groups of their own that compact frees first. + # One statement per call: set_options and compact must commit before the next. + _run(pri, "ALTER DATABASE postgres SET pgcolumnar.enable_end_truncation = on") + _run(pri, "CREATE TABLE t (id int, v text) USING pgcolumnar") + _run(pri, "SELECT pgcolumnar.set_options('t', stripe_row_limit => 1000, chunk_group_row_limit => 1000)") + _run(pri, "INSERT INTO t SELECT g, md5(g::text) FROM generate_series(1,30000) g") + assert _caught_up(pri, sb), "the standby did not catch up (sync 7)" + _run(pri, "DELETE FROM t WHERE id > 15000") + _run(pri, "SELECT pgcolumnar.compact('t')") + pri_before = _q(pri, "SELECT pg_relation_size('t')") + trunc = _q(pri, "SELECT pgcolumnar.truncate('t')") + expect.text("yes" if isinstance(trunc, int) and trunc > 0 else f"no (returned {trunc})", "yes", + "the primary actually truncated (record was emitted)") + pri_after = _q(pri, "SELECT pg_relation_size('t')") + assert _caught_up(pri, sb), "the standby did not catch up (sync 8)" + sb_after = _q(sb, "SELECT pg_relation_size('t')") + expect.text("yes" if isinstance(sb_after, int) and sb_after == pri_after + else f"no (primary {pri_before}->{pri_after}, standby {sb_after})", "yes", + "the standby's main fork shrank to match the primary") + expect.num(_q(sb, "SELECT count(*) FROM t"), _q(pri, "SELECT count(*) FROM t"), + "the truncate did not cost rows on the standby") + expect.hash(_hash(sb, "t"), _hash(pri, "t"), "content still matches after the truncate record") + expect.text(str(_q(sb, "SELECT pg_is_in_recovery()")), "True", + "the standby is still in recovery after the truncate record") + + # Scenario 3: a standby WITHOUT the module still replays, because every record + # is a core type. If this fails, the extension has grown WAL semantics. + _run(pri, "CREATE TABLE r2 (id int, v text) USING pgcolumnar", + "INSERT INTO r2 SELECT g, 'x' || g FROM generate_series(1,5000) g") + sb.stop() + _conf(sb_dir, "shared_preload_libraries=''") + sb.start() + _answers(sb) + expect.num(_q(sb, "SELECT 1"), 1, "a standby without pgcolumnar loaded still starts") + expect.num(_q(sb, "SELECT count(*) FROM pg_extension WHERE extname='pgcolumnar' " + "AND '' = current_setting('shared_preload_libraries')"), 1, + "and it is not running the module") + _run(pri, "INSERT INTO r2 SELECT g, 'y' || g FROM generate_series(5001,9000) g") + expect.num(_q(pri, "SELECT count(*) FROM r2"), 9000, + "premise: the primary wrote r2's second batch while the standby ran without the module") + target = _q(pri, "SELECT pg_current_wal_lsn()::text") + _caught_up(pri, sb, tries=120) + expect.text(str(_q(sb, f"SELECT pg_last_wal_replay_lsn() >= '{target}'::pg_lsn")), "True", + "it replayed columnar WAL past the primary's LSN without the module") + expect.text(str(_q(sb, "SELECT pg_is_in_recovery()")), "True", + "and did not PANIC (still in recovery, still answering)") + + # the bytes are there; reload the module and the same standby reads them + sb.stop() + _conf(sb_dir, "shared_preload_libraries='pgcolumnar'") + sb.start() + _answers(sb) + assert _caught_up(pri, sb), "the standby did not catch up (sync 9)" + expect.hash(_hash(sb, "r2"), _hash(pri, "r2"), "with the module back, the replayed table reads correctly") + finally: + for c in (rs, sb, pri): + if c is not None: + c.stop() + shutil.rmtree(work, ignore_errors=True) diff --git a/test/replication.sh b/test/replication.sh index abd923d67..e1ad10477 100755 --- a/test/replication.sh +++ b/test/replication.sh @@ -458,11 +458,27 @@ check "and the row counts agree" \ # Scenario 2: compaction, and the SMGR truncate redo the issue asked for # --------------------------------------------------------------------------- +# Each replay arm below is about a write the primary actually made (#1341): measured, +# compact here retired nothing, because vacuum had already reclaimed every deleted +# row, so "compact replays identically" compared a state compact never touched. +# vacuum rewrites into a new relfilenode, and compact retires fully deleted row +# groups, so each is asserted to have done that first. +rg_sql="SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('r')" +fn_before="$(q "SELECT pg_relation_filenode('r')")" psql_run "SELECT pgcolumnar.vacuum('r');" >/dev/null 2>&1 +check "vacuum rewrote r into a new relfilenode on the primary" \ + "$([ -n "$fn_before" ] && [ "$(q "SELECT pg_relation_filenode('r')")" != "$fn_before" ] && echo yes || echo no)" "yes" sync_or_fail "sync 5" check "vacuum replays identically" "$(hash_standby r)" "$(hash_primary r)" -psql_run "SELECT pgcolumnar.compact('r');" >/dev/null 2>&1 +# A row group of its own (one INSERT is one flush), then every row of it deleted, +# so compact has exactly one group to retire. +psql_run "INSERT INTO r SELECT g, 'doomed' || g FROM generate_series(50001,51000) g;" >/dev/null 2>&1 +psql_run "DELETE FROM r WHERE v LIKE 'doomed%';" >/dev/null 2>&1 +rg_before="$(q "$rg_sql")" +retired="$(q "SELECT pgcolumnar.compact('r');")" +check "compact retired the fully deleted row group on the primary" \ + "$retired|$(( ${rg_before:-0} - $(q "$rg_sql") ))" "1|1" sync_or_fail "sync 6" check "compact replays identically" "$(hash_standby r)" "$(hash_primary r)" @@ -542,6 +558,10 @@ check "and it is not running the module" \ "1" psql_run "INSERT INTO r2 SELECT g, 'y' || g FROM generate_series(5001,9000) g;" >/dev/null 2>&1 +# The replay arm below is only about columnar WAL if the primary wrote some: a +# failed INSERT left the LSN arm and the final hash comparison green. +check "premise: the primary wrote r2's second batch while the standby ran without the module" \ + "$(q 'SELECT count(*) FROM r2')" "9000" target_lsn="$(q 'SELECT pg_current_wal_lsn()')" for _i in $(seq 1 120); do rep="$(sb_q 'SELECT pg_last_wal_replay_lsn()')" From 3f27e4796e019e21cf3f940fbfc35bc8eb8cd4c1 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:09 +0000 Subject: [PATCH 19/36] test: replication seeds the ledger, 11 of 44 arms dated by four mutations (#1341) The suite had no rows. All 44 checks enter from five-major runs (44/44 on each of 15-19). Dated by: no SMGR truncate record, data and gap pages written without WAL, vacuum returning early, compact retiring nothing. suites_not_covered 212 -> 211; census 3839 rows / 2811 never / 1028 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 44 ++++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 23 +++++++++++++++++-- 2 files changed, 65 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index c0267df50..b0980fd22 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -3065,6 +3065,50 @@ recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar ta recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) recovery_minimal_wal recovery_minimal_wal premise: the cluster runs at wal_level=minimal 15;16;17;18;19 never - +replication replication a physical backup alone preserves the projections, unlike pg_dump (#266) 15;16;17;18;19 never - +replication replication a physical slot exists to hold WAL for the standby 15;16;17;18;19 never - +replication replication a standby without pgcolumnar loaded still starts 15;16;17;18;19 never - +replication replication after resuming, the standby sees them 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication and a non-empty projection to compare against 15;16;17;18;19 never - +replication replication and did not PANIC (still in recovery, still answering) 15;16;17;18;19 never - +replication replication and has no upstream configured 15;16;17;18;19 never - +replication replication and is a different cluster from the primary and the standby 15;16;17;18;19 never - +replication replication and it is not running the module 15;16;17;18;19 never - +replication replication and it reads on the primary 15;16;17;18;19 never - +replication replication and its storage, readable with the same content 15;16;17;18;19 never - +replication replication and that one reads on the standby too 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication and the later projection is in this backup as well, so it is not stale 15;16;17;18;19 never - +replication replication and the projection's storage came across intact and readable 15;16;17;18;19 never - +replication replication and the row counts agree 15;16;17;18;19 never - +replication replication and the standby is actually connected through it 15;16;17;18;19 never - +replication replication compact replays identically 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication compact retired the fully deleted row group on the primary 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact retires no fully deleted row group (retired = 0) +replication replication content still matches after the truncate record 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication insert, delete and update replay identically 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication it replayed columnar WAL past the primary's LSN without the module 15;16;17;18;19 never - +replication replication pg_basebackup carried the projection catalog row, unlike pg_dump (#266) 15;16;17;18;19 never - +replication replication pg_basebackup produced a data directory 15;16;17;18;19 never - +replication replication premise: the primary wrote r2's second batch while the standby ran without the module 15;16;17;18;19 never - +replication replication streaming replay carried a projection created after the backup 15;16;17;18;19 never - +replication replication the base backup carried the initial rows 15;16;17;18;19 never - +replication replication the primary actually truncated (record was emitted) 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact retires no fully deleted row group (retired = 0) +replication replication the primary has a projection to preserve, before the backup runs 15;16;17;18;19 never - +replication replication the primary now has a second, post-backup projection 15;16;17;18;19 never - +replication replication the primary still has both projections when the backup is taken 15;16;17;18;19 never - +replication replication the restored cluster accepts connections 15;16;17;18;19 never - +replication replication the restored cluster is not in recovery (it is not following anyone) 15;16;17;18;19 never - +replication replication the second base backup produced a data directory 15;16;17;18;19 never - +replication replication the standby accepts connections 15;16;17;18;19 never - +replication replication the standby is a different cluster from the primary 15;16;17;18;19 never - +replication replication the standby is in recovery 15;16;17;18;19 never - +replication replication the standby is still in recovery after the truncate record 15;16;17;18;19 never - +replication replication the standby's main fork shrank to match the primary 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarTruncateMainFork emits no XLOG_SMGR_TRUNCATE record (if (false && needs_wal));columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication the truncate did not cost rows on the standby 15;16;17;18;19 never - +replication replication vacuum replays identically 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication vacuum rewrote r into a new relfilenode on the primary 15;16;17;18;19 2026-10-16 columnar_vacuum.c: pgcolumnar_vacuum returns before rewriting +replication replication with replay paused the standby does NOT see the new rows 15;16;17;18;19 never - +replication replication with the module back, the replayed table reads correctly 15;16;17;18;19 2026-10-16 columnar_storage.c: PgColumnarWriteLogicalData logs no data or gap page (if (false) for both RelationNeedsWAL) +replication replication with the same content the primary reads 15;16;17;18;19 never - rewrite_storage_oid rewrite_storage_oid a matview created WITH DATA leaves its storage row pointing at itself 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite repair gate no longer covers CreateTableAsStmt, so a matview created WITH DATA keeps the transient's OID rewrite_storage_oid rewrite_storage_oid a type rewrite leaves the storage row pointing at the live table 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite hook does not retarget storage.relation_oid rewrite_storage_oid rewrite_storage_oid and REFRESH still leaves it pointing at itself 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite repair gate no longer covers RefreshMatViewStmt diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 94865b834..acfdd6d88 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 212 +suites_not_covered 211 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3047,4 +3047,23 @@ suites_not_covered 212 # never 2754 + 24 = 2778 # dated 1001 + 16 = 1017 # 2778 + 1017 == 3795 -checks_never_observed_red 2778 +# +# #1341, replication seeded, with its twin (OffgridwithJD): 2778 -> 2811, and +# suites_not_covered 212 -> 211. The suite had no rows. It enters with all 44 checks +# from five-major runs (44/44 on each of 15-19). 11 are dated by four mutations: the +# SMGR truncate record not emitted (the standby's fork stays long), data and gap pages +# written without WAL (seven replay comparisons and the fork size), vacuum returning +# before it rewrites, and compact retiring nothing (its new arm and the truncate's +# precondition). Thirty-three arrive never: the backup, restore and standby controls +# (the data directories, connections, recovery state, slot and upstream), the +# primary's pins and the r2 premise, the projection arms a bytewise copy carries +# without any columnar WAL, the base backup's rows, the row-count comparisons that +# count(*) answers from the catalog, the paused-replay control, the standby's +# recovery state after the truncate, and the module-less standby's arms, which no +# product mutation can redden while every record is a core type. +# +# ledger rows 3795 + 44 = 3839 +# never 2778 + 33 = 2811 +# dated 1017 + 11 = 1028 +# 2811 + 1028 == 3839 +checks_never_observed_red 2811 From 8b0b5826e21071cde13db6f9346d9e2bee42e610 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:09 +0000 Subject: [PATCH 20/36] test: native_recluster gets its pytest twin, and its index arm can see recluster's visibility-map rule (#1341 item 70) Two measured gaps in "index scan returns each row once": - with the index dropped after the recluster it stayed green, because with only seqscan and the custom scan off the count went through the disabled seqscan. Bitmap scans are off now too, and "premise: the count takes an index-only scan of n_id_uq" asserts the plan. - with recluster's PgColumnarVMClearForRowRange removed the whole suite stayed green: no VM bit was ever set, so every index-only probe fetched from the heap. A VACUUM before the recluster sets them (an arm checks the pre-recluster count answers from the index), and the old entries, still in the index, are then counted twice: 81409 for 40960. test_native_recluster.py carries all 14 checks by the shell's name. COMPLETE gains native_recluster, TESTS.md section 153, cluster_tests 682 -> 683. Co-Authored-By: Claude Opus 5.5 --- test/native_recluster.sh | 32 ++++++- test/pytest/TESTS.md | 13 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_recluster.py | 123 +++++++++++++++++++++++++++ 5 files changed, 169 insertions(+), 4 deletions(-) create mode 100644 test/pytest/test_native_recluster.py diff --git a/test/native_recluster.sh b/test/native_recluster.sh index 8916a8220..2de496798 100755 --- a/test/native_recluster.sh +++ b/test/native_recluster.sh @@ -36,11 +36,22 @@ skipped() { -d "$PGC_DB" -At -c "EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) $1" 2>/dev/null \ | grep 'Columnar Chunk Groups Removed by Filter' | grep -oE '[0-9]+' | head -1 } +# The count runs as an INDEX-ONLY scan of n_id_uq, with bitmap scans off too, and +# idxplan prints that plan (#1341). With only seqscan and the custom scan off, the +# count came back right with the index dropped, through the disabled seqscan. +# An index-only scan is also what can see the visibility-map rule recluster +# follows: the old index entries stay until a vacuum, and an index-only scan +# answers for one without a heap fetch wherever its VM bit is still set. +IDX_GUCS="SET enable_seqscan=off; SET enable_bitmapscan=off; SET pgcolumnar.enable_custom_scan=off;" +IDX_Q="SELECT count(*) FROM n WHERE id BETWEEN 1 AND 40960" idxcount() { + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ + -d "$PGC_DB" -At 2>/dev/null -c "$IDX_GUCS $IDX_Q;" | tail -1 +} +idxplan() { env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ -d "$PGC_DB" -At 2>/dev/null -c \ - "SET enable_seqscan=off; SET pgcolumnar.enable_custom_scan=off; - SELECT count(*) FROM n WHERE id BETWEEN 1 AND 40960;" | tail -1 + "$IDX_GUCS EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, BUFFERS OFF) $IDX_Q;" } BOX="x BETWEEN 20 AND 40 AND y BETWEEN 20 AND 40" @@ -57,6 +68,15 @@ check "the plan under test is a columnar custom scan" \ before="$(skipped "SELECT id FROM n WHERE $BOX")"; before="${before:-0}" +# VACUUM first, so the visibility map is set over the groups recluster replaces: +# without it every VM bit is clear, an index-only scan fetches every row from the +# heap, and recluster's VM clear is never needed. +psql_run "VACUUM n;" +# Most bits, not all: measured, 511 of the 40960 rows still take a heap fetch. +pre_fetches="$(idxplan | grep -oE 'Heap Fetches: [0-9]+' | grep -oE '[0-9]+')" +check "VACUUM set the visibility map, so the index-only count answers from the index before the recluster" \ + "$([ -n "$pre_fetches" ] && [ "$pre_fetches" -lt 20480 ] && echo yes || echo "no (${pre_fetches:-no} heap fetches for 40960 rows)")" "yes" + # Online recluster by (x, y). rec="$(q "SELECT pgcolumnar.recluster('n', 'x', 'y');")" echo " (recluster processed $rec groups; box skip before=$before)" @@ -73,7 +93,13 @@ check "full-table parity after recluster" \ "$(pgc_set_hash 'SELECT id, x, y, payload FROM h ORDER BY id')" check "row count after recluster" "$(q 'SELECT count(*) FROM n;')" "40960" -# Online index maintenance: index scan returns each live row exactly once. +# Online index maintenance: index scan returns each live row exactly once. The old +# entries are still in the index, so a recluster that left their VM bits set would +# count them too. +post_plan="$(idxplan)" +echo "-- index count plan after the recluster:"; sed 's/^/ /' <<<"$post_plan" +check "premise: the count takes an index-only scan of n_id_uq" \ + "$(grep -c 'Index Only Scan using n_id_uq on n' <<<"$post_plan")" "1" check "index scan returns each row once" "$(idxcount)" "40960" # THE SQLSTATE, NOT THE EXIT CODE. This arm was # `psql ... && echo no || echo yes`, which reports "yes" for every failure psql can diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 9a2e28fd3..8a5ebc974 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -198,6 +198,7 @@ behaviour, the source of that number is named. - [150. test_pg_dump_roundtrip.py: a columnar table survives pg_dump and restore](#150-test_pg_dump_roundtrippy-a-columnar-table-survives-pg_dump-and-restore) - [151. test_pg_upgrade.py: a columnar table survives pg_upgrade](#151-test_pg_upgradepy-a-columnar-table-survives-pg_upgrade) - [152. test_replication.py: physical replication and standby redo](#152-test_replicationpy-physical-replication-and-standby-redo) +- [153. test_native_recluster.py: online reclustering under ShareUpdateExclusiveLock](#153-test_native_reclusterpy-online-reclustering-under-shareupdateexclusivelock) ## 1. How to read a test in here @@ -7503,3 +7504,15 @@ Each replay arm is about a write the primary made. Vacuum must have rewritten th | test | what it holds | | --- | --- | | `test_a_streaming_standby_and_a_restore_replay_columnar_tables` | the standby and the isolated restore hold the primary's rows and projections, every write path replays identically, and a module-less standby replays | + +## 153. test_native_recluster.py: online reclustering under ShareUpdateExclusiveLock + +The pytest twin of `test/native_recluster.sh` (ported for #1341, item 70), every check by the shell's name. `pgcolumnar.recluster()` rewrites a table's live rows in Z-order online. It writes them back as fresh row groups, inserts their index entries, clears the visibility map over the row numbers it retires, and retires the old groups. A 2D box skips more groups afterwards, results match a heap mirror, the unique index still enforces by SQLSTATE 23505, and the lock is ShareUpdateExclusiveLock, never AccessExclusiveLock. + +The index arm counts through an index-only scan of the unique index, after a VACUUM has set the visibility map, so a recluster that left the old entries' VM bits set counts them twice. The plan is asserted too: with only seqscan and the custom scan off, the count came back right with the index dropped. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_online_recluster` | recluster tightens 2D skipping, keeps the rows, keeps the index exact through an index-only scan and the unique constraint enforced, and holds only ShareUpdateExclusiveLock | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 10ac37654..5a3f2d111 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -767,4 +767,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 681 -> 682 for #1341's test_replication.py, the twin of replication.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 682 +# cluster_tests 682 -> 683 for #1341's test_native_recluster.py, the twin of native_recluster.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 683 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index bfd786dfe..56f07da0a 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -134,6 +134,7 @@ "native_projection", "native_reclaim_cycles", "native_reclaim_frag", + "native_recluster", "native_vacuum_race", "objstore_allowlist", "objstore_credentials", diff --git a/test/pytest/test_native_recluster.py b/test/pytest/test_native_recluster.py new file mode 100644 index 000000000..8be24ed4a --- /dev/null +++ b/test/pytest/test_native_recluster.py @@ -0,0 +1,123 @@ +"""Online reclustering re-establishes Z-order clustering under ShareUpdateExclusiveLock +(Phase F3c, #1341). The pytest twin of `test/native_recluster.sh`, every check by the +shell's name. + +pgcolumnar.recluster() rewrites a table's live rows in Z-order over the given columns +while reads and writes go on: it writes them back as fresh row groups, inserts their +index entries online, clears the visibility map over the row numbers it retires, and +retires the old groups. So a small 2D box skips far more groups afterwards, results +match a heap mirror, the unique index still enforces, and the lock is never +AccessExclusiveLock. + +The index arm counts through an INDEX-ONLY scan of the unique index, after a VACUUM +has set the visibility map. The old index entries stay until a vacuum, so a recluster +that left their VM bits set would count them as well; and the plan is asserted, +because with only seqscan and the custom scan off the count came back right with the +index dropped. +""" +import psycopg + +from pgc_vacuity import _plan_nodes + +ROWS = 40960 +GEN = (f"SELECT g, ((g::bigint * 7919) % 200)::int AS x, ((g::bigint * 104729) % 200)::int AS y, " + f"'p' || (g % 97) AS payload FROM generate_series(1, {ROWS}) g") +BOX = "x BETWEEN 20 AND 40 AND y BETWEEN 20 AND 40" +IDX_GUCS = ("SET enable_seqscan = off", "SET enable_bitmapscan = off", "SET pgcolumnar.enable_custom_scan = off") +IDX_Q = f"SELECT count(*) FROM n WHERE id BETWEEN 1 AND {ROWS}" + + +def _one(cur, sql): + cur.execute(sql) + return cur.fetchone()[0] + + +def _rows(cur, sql): + cur.execute(sql) + return cur.fetchall() + + +def _plan(cur, sql): + cur.execute(f"EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, FORMAT JSON) {sql}") + return cur.fetchone()[0] + + +def _skipped(cur, sql): + """-> 'Columnar Chunk Groups Removed by Filter' of the plan, 0 when no node reports it.""" + return next((node["Columnar Chunk Groups Removed by Filter"] for node in _plan_nodes(_plan(cur, sql)) + if "Columnar Chunk Groups Removed by Filter" in node), 0) + + +def _index_only(cur): + """-> (the index-only count's plan nodes, its count), the scan forced to the index.""" + for guc in IDX_GUCS: + cur.execute(guc) + try: + return list(_plan_nodes(_plan(cur, IDX_Q))), _one(cur, IDX_Q) + finally: + for guc in ("enable_seqscan", "enable_bitmapscan", "pgcolumnar.enable_custom_scan"): + cur.execute(f"RESET {guc}") + + +def test_online_recluster(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE h (id int, x int, y int, payload text)") + cur.execute("CREATE TABLE n (id int, x int, y int, payload text) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('n', stripe_row_limit => 2048, chunk_group_row_limit => 1024)") + cur.execute(f"INSERT INTO h {GEN}") + cur.execute(f"INSERT INTO n {GEN}") + cur.execute("CREATE UNIQUE INDEX n_id_uq ON n (id)") + + expect.num(_one(cur, "SELECT count(*) FROM n"), ROWS, "row count") + expect.row_set(_rows(cur, f"SELECT id, x, y, payload FROM n WHERE {BOX}"), + _rows(cur, f"SELECT id, x, y, payload FROM h WHERE {BOX}"), "box parity (pre-recluster)") + # the skip counts below come from the columnar custom scan and nothing else + expect.plan_marker(_plan(cur, f"SELECT id FROM n WHERE {BOX}"), "Columnar Projected Columns", + name="the plan under test is a columnar custom scan") + before = _skipped(cur, f"SELECT id FROM n WHERE {BOX}") + + # VACUUM first, so the VM is set over the groups recluster replaces. Most + # bits, not all: measured, 511 of the 40960 rows still take a heap fetch. + cur.execute("VACUUM n") + nodes, _ = _index_only(cur) + fetches = next((node["Heap Fetches"] for node in nodes if "Heap Fetches" in node), None) + expect.text("yes" if isinstance(fetches, int) and fetches < ROWS // 2 + else f"no ({fetches} heap fetches for {ROWS} rows)", "yes", + "VACUUM set the visibility map, so the index-only count answers from the index before the recluster") + + rec = _one(cur, "SELECT pgcolumnar.recluster('n', 'x', 'y')") + print(f"-- recluster processed {rec} groups; box skip before={before}") + expect.text("yes" if rec > 0 else "no", "yes", "reclustered the groups") + after = _skipped(cur, f"SELECT id FROM n WHERE {BOX}") + print(f"-- box groups skipped: before={before} after={after}") + expect.text("yes" if after > before else f"no ({before} -> {after})", "yes", + "reclustering increases 2D skipping") + expect.text("yes" if after >= 10 else f"no ({after})", "yes", "reclustering skips most groups") + + expect.row_set(_rows(cur, "SELECT id, x, y, payload FROM n"), _rows(cur, "SELECT id, x, y, payload FROM h"), + "full-table parity after recluster") + expect.num(_one(cur, "SELECT count(*) FROM n"), ROWS, "row count after recluster") + + # the old entries are still in the index, so a recluster that left their VM + # bits set would count them too + nodes, count = _index_only(cur) + expect.num(sum(1 for node in nodes if node.get("Node Type") == "Index Only Scan" + and node.get("Index Name") == "n_id_uq"), 1, + "premise: the count takes an index-only scan of n_id_uq") + expect.num(count, ROWS, "index scan returns each row once") + try: + cur.execute("INSERT INTO n VALUES (1, 0, 0, 'x')") + dup = None + except psycopg.Error as exc: + dup = exc + expect.sqlstate(dup, "23505", "unique still enforced, by SQLSTATE 23505 and not by any failure") + + # the lock level, read inside the transaction that holds it + cur.execute("BEGIN") + cur.execute("SELECT pgcolumnar.recluster('n', 'x', 'y')") + locks = _rows(cur, "SELECT count(*) FILTER (WHERE mode = 'ShareUpdateExclusiveLock'), " + "count(*) FILTER (WHERE mode = 'AccessExclusiveLock') " + "FROM pg_locks WHERE relation = 'n'::regclass AND locktype = 'relation'")[0] + cur.execute("ROLLBACK") + expect.num(locks[0], 1, "recluster holds ShareUpdateExclusiveLock") + expect.num(locks[1], 0, "recluster holds no AccessExclusiveLock") From 1ef6ec67659f710807769872d3018118a7f6ae3b Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:09 +0000 Subject: [PATCH 21/36] test: native_recluster seeds the ledger, 8 of 14 arms dated by five mutations (#1341 item 70) The suite had no rows. All 14 checks enter from five-major runs (14/14 on each of 15-19). Dated by: no index entry for a rewritten row, no old group retired, the VM left set over the retired row numbers, AccessExclusiveLock, and a sort key of no columns. suites_not_covered 211 -> 210; census 3853 rows / 2817 never / 1036 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 14 ++++++++++++++ test/check_ledger_budget.txt | 20 ++++++++++++++++++-- 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index b0980fd22..adda83b3e 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2611,6 +2611,20 @@ native_reclaim native_reclaim repeated recluster reuses space (file plateaus) 15 native_reclaim native_reclaim reused-block data matches heap mirror 15;16;17;18;19 never - native_reclaim native_reclaim row count correct after cycle 15;16;17;18;19 never - native_reclaim native_reclaim ten row groups, every one half deleted 15;16;17;18;19 2026-10-14 the write state ignores stripe_row_limit +native_recluster native_recluster VACUUM set the visibility map, so the index-only count answers from the index before the recluster 15;16;17;18;19 never - +native_recluster native_recluster box parity (pre-recluster) 15;16;17;18;19 never - +native_recluster native_recluster full-table parity after recluster 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online retires none of the old groups (PgColumnarRetireGroup removed) +native_recluster native_recluster index scan returns each row once 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online inserts no index entry for a rewritten row (PgColumnarIndexInsertRow removed);columnar_vacuum.c: recluster_online leaves the VM set over the retired row numbers (PgColumnarVMClearForRowRange removed);columnar_vacuum.c: recluster_online retires none of the old groups (PgColumnarRetireGroup removed) +native_recluster native_recluster premise: the count takes an index-only scan of n_id_uq 15;16;17;18;19 never - +native_recluster native_recluster recluster holds ShareUpdateExclusiveLock 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster takes AccessExclusiveLock, not ShareUpdateExclusiveLock +native_recluster native_recluster recluster holds no AccessExclusiveLock 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster takes AccessExclusiveLock, not ShareUpdateExclusiveLock +native_recluster native_recluster reclustered the groups 15;16;17;18;19 never - +native_recluster native_recluster reclustering increases 2D skipping 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online sorts on a key of no columns, so the rows keep their order +native_recluster native_recluster reclustering skips most groups 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online sorts on a key of no columns, so the rows keep their order +native_recluster native_recluster row count 15;16;17;18;19 never - +native_recluster native_recluster row count after recluster 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online retires none of the old groups (PgColumnarRetireGroup removed) +native_recluster native_recluster the plan under test is a columnar custom scan 15;16;17;18;19 never - +native_recluster native_recluster unique still enforced, by SQLSTATE 23505 and not by any failure 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online inserts no index entry for a rewritten row (PgColumnarIndexInsertRow removed) objstore_allowlist objstore_allowlist a multi-entry list matches any entry 15;16;17;18;19 never - objstore_allowlist objstore_allowlist a non-superuser cannot widen the list (42501) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it objstore_allowlist objstore_allowlist a superuser session SET works (admin override) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index acfdd6d88..6f35318bf 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 211 +suites_not_covered 210 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3066,4 +3066,20 @@ suites_not_covered 211 # never 2778 + 33 = 2811 # dated 1017 + 11 = 1028 # 2811 + 1028 == 3839 -checks_never_observed_red 2811 +# +# #1341 item 70, native_recluster seeded, with its twin (OffgridwithJD): 2811 -> +# 2817, and suites_not_covered 211 -> 210. The suite had no rows. It enters with all +# 14 checks from five-major runs (14/14 on each of 15-19). 8 are dated by five +# mutations of recluster_online: no index entry for a rewritten row, no old group +# retired, the visibility map left set over the retired row numbers (the index arm, +# now an index-only count after a VACUUM), AccessExclusiveLock for the lazy lock, and +# a sort key of no columns (the two skipping arms). Six arrive never: the pre-recluster +# count and box parity, the columnar-scan and index-only-plan premises, the VACUUM's +# visibility-map arm, and "reclustered the groups", whose count none of these +# mutations brings to zero. +# +# ledger rows 3839 + 14 = 3853 +# never 2811 + 6 = 2817 +# dated 1028 + 8 = 1036 +# 2817 + 1036 == 3853 +checks_never_observed_red 2817 From ad2fd2aa299887dbe84666b0c58daaef035545f6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:09 +0000 Subject: [PATCH 22/36] test: alter_am_cleanup gets its pytest twin, and each arm reddens for its own reason (#1341 item 70) Three attribution faults, measured under mutations of the drop hook: - every cleanup arm compared with the suite's starting snapshot, so one leak reddened every arm after it: with the cleanup gated on the columnar AM, the never-columnar heap arm went red on aac_t's rows. Each now compares with the snapshot taken just before its own table. - REFRESH MATERIALIZED VIEW reused the CREATE arm's matview and went red as 'relation "rwm" does not exist'. It now refreshes a matview of its own, made WITH NO DATA (a premise), and reddens on the hook's own error. - "premise: and took pgcolumnar.options with it" went red when the member skip was removed, because DROP EXTENSION aborted: a product change, so an arm (#1236). test_alter_am_cleanup.py carries all 47 checks by the shell's name, in four tests on databases of their own. COMPLETE gains alter_am_cleanup, TESTS.md section 154, cluster_tests 683 -> 687. Co-Authored-By: Claude Opus 5.5 --- test/alter_am_cleanup.sh | 28 +++- test/pytest/TESTS.md | 16 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_alter_am_cleanup.py | 199 +++++++++++++++++++++++++++ test/pytest/test_compare_to_bash.py | 1 + 5 files changed, 241 insertions(+), 7 deletions(-) create mode 100644 test/pytest/test_alter_am_cleanup.py diff --git a/test/alter_am_cleanup.sh b/test/alter_am_cleanup.sh index a79556f8a..dc99c2b71 100755 --- a/test/alter_am_cleanup.sh +++ b/test/alter_am_cleanup.sh @@ -55,7 +55,13 @@ psql_run "DROP TABLE aac_t;" check "DROP after SET ACCESS METHOD heap leaves no options or declarations" \ "$(snapshot)" "$base" +# Each cleanup arm below compares with the snapshot taken just before ITS table +# was made, not with the suite's starting one (#1341). Against $base, one leak +# above reddened every arm after it: measured, with the cleanup gated on the +# columnar AM, the never-columnar heap arm went red on aac_t's leftover rows. +# # Round-trip: convert away and back, then DROP a still-columnar table. +pre_rt="$(snapshot)" psql_run "CREATE TABLE aac_rt (id int) USING pgcolumnar;" psql_run "SELECT pgcolumnar.set_options('aac_rt', stripe_row_limit => 8000);" psql_run "ALTER TABLE aac_rt SET ACCESS METHOD heap;" @@ -64,15 +70,16 @@ check "options survive a heap round-trip" \ "$(q "SELECT stripe_row_limit FROM pgcolumnar.options WHERE regclass = 'aac_rt'::regclass")" "8000" psql_run "DROP TABLE aac_rt;" -check "a round-tripped table still cleans up on DROP" "$(snapshot)" "$base" +check "a round-tripped table still cleans up on DROP" "$(snapshot)" "$pre_rt" # Control: a never-columnar heap must not be charged for this either. +pre_h="$(snapshot)" psql_run "CREATE TABLE aac_h (id int);" psql_run "DROP TABLE aac_h;" # kept alive for the VACUUM FULL control further down psql_run "CREATE TABLE aac_h_ctl (id int);" check "dropping a heap table that was never columnar leaves the catalogs alone" \ - "$(snapshot)" "$base" + "$(snapshot)" "$pre_h" # ---- the hook is armed in EVERY database, not only where the extension is ---- # @@ -136,8 +143,12 @@ check "ALTER COLUMN TYPE succeeds without the extension" \ "$(aac_dbrun aac_nocx 'ALTER TABLE rw ALTER COLUMN t TYPE varchar(64);')" "rc=0" check "CREATE MATERIALIZED VIEW succeeds without the extension" \ "$(aac_dbrun aac_nocx 'CREATE MATERIALIZED VIEW rwm AS SELECT * FROM rw;')" "rc=0" +# REFRESH gets a matview of its own, made WITH NO DATA, so it does not inherit the +# CREATE arm's failure: measured, it went red as 'relation "rwm" does not exist'. +check "premise: a matview to refresh exists, made WITH NO DATA" \ + "$(aac_dbrun aac_nocx 'CREATE MATERIALIZED VIEW rwr AS SELECT * FROM rw WITH NO DATA;')" "rc=0" check "and REFRESH MATERIALIZED VIEW does too, without the extension" \ - "$(aac_dbrun aac_nocx 'REFRESH MATERIALIZED VIEW rwm;')" "rc=0" + "$(aac_dbrun aac_nocx 'REFRESH MATERIALIZED VIEW rwr;')" "rc=0" check "TRUNCATE succeeds without the extension" \ "$(aac_dbrun aac_nocx 'TRUNCATE rw;')" "rc=0" @@ -153,6 +164,7 @@ check "control: VACUUM FULL still succeeds where the extension IS installed" \ # different set: DROP EXTENSION drops those as ordinary relations, and opening # options while options is itself being dropped would fail. +pre_in="$(snapshot)" psql_run "CREATE TABLE pgcolumnar.aac_inschema (id int) USING pgcolumnar;" psql_run "SELECT pgcolumnar.set_options('pgcolumnar.aac_inschema', stripe_row_limit => 7000);" check "premise: a table in the extension's schema recorded its options" \ @@ -160,7 +172,7 @@ check "premise: a table in the extension's schema recorded its options" \ WHERE regclass = 'pgcolumnar.aac_inschema'::regclass")" "1" psql_run "DROP TABLE pgcolumnar.aac_inschema;" check "dropping a table that lives in the extension's schema still cleans up" \ - "$(snapshot)" "$base" + "$(snapshot)" "$pre_in" # ---- a relation that predates CREATE EXTENSION is not an extension member ---- # @@ -226,7 +238,9 @@ check "DROP EXTENSION succeeds with those catalogs populated" \ check "premise: DROP EXTENSION left the schema behind" \ "$(aac_dbq aac_dropx "SELECT count(*) FROM pg_namespace WHERE nspname='pgcolumnar'")" "1" -check "premise: and took pgcolumnar.options with it" \ +# An arm, not a premise (#1236): a hook that opened options while DROP EXTENSION +# was dropping it aborted the DROP, and this read the options table still there. +check "and took pgcolumnar.options with it" \ "$(aac_dbq aac_dropx "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'pgcolumnar' AND c.relname = 'options'")" "0" @@ -248,8 +262,10 @@ check "ALTER COLUMN TYPE succeeds after DROP EXTENSION" \ "$(aac_dbrun aac_dropx 'ALTER TABLE dxr ALTER COLUMN t TYPE varchar(64);')" "rc=0" check "CREATE MATERIALIZED VIEW succeeds after DROP EXTENSION" \ "$(aac_dbrun aac_dropx 'CREATE MATERIALIZED VIEW dxm AS SELECT * FROM dxr;')" "rc=0" +check "premise: a matview to refresh exists after DROP EXTENSION, made WITH NO DATA" \ + "$(aac_dbrun aac_dropx 'CREATE MATERIALIZED VIEW dxr_m AS SELECT * FROM dxr WITH NO DATA;')" "rc=0" check "and REFRESH MATERIALIZED VIEW does too, after DROP EXTENSION" \ - "$(aac_dbrun aac_dropx 'REFRESH MATERIALIZED VIEW dxm;')" "rc=0" + "$(aac_dbrun aac_dropx 'REFRESH MATERIALIZED VIEW dxr_m;')" "rc=0" check "TRUNCATE succeeds after DROP EXTENSION" \ "$(aac_dbrun aac_dropx 'TRUNCATE dxr;')" "rc=0" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 8a5ebc974..f30f0487f 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -199,6 +199,7 @@ behaviour, the source of that number is named. - [151. test_pg_upgrade.py: a columnar table survives pg_upgrade](#151-test_pg_upgradepy-a-columnar-table-survives-pg_upgrade) - [152. test_replication.py: physical replication and standby redo](#152-test_replicationpy-physical-replication-and-standby-redo) - [153. test_native_recluster.py: online reclustering under ShareUpdateExclusiveLock](#153-test_native_reclusterpy-online-reclustering-under-shareupdateexclusivelock) +- [154. test_alter_am_cleanup.py: DROP after SET ACCESS METHOD heap cleans up](#154-test_alter_am_cleanuppy-drop-after-set-access-method-heap-cleans-up) ## 1. How to read a test in here @@ -7516,3 +7517,18 @@ The index arm counts through an index-only scan of the unique index, after a VAC | test | what it holds | | --- | --- | | `test_online_recluster` | recluster tightens 2D skipping, keeps the rows, keeps the index exact through an index-only scan and the unique constraint enforced, and holds only ShareUpdateExclusiveLock | + +## 154. test_alter_am_cleanup.py: DROP after SET ACCESS METHOD heap cleans up + +The pytest twin of `test/alter_am_cleanup.sh` (ported for #1341, item 70), every check by the shell's name. `pgcolumnar.options` and `projection_declaration` are keyed by relid, so a table converted to heap still has rows there, and the drop hook must clear them although the table is heap by then (#304). The hook is armed in every database, because the library is preloaded. Where the extension was never installed, or has been dropped, it must not open a catalog that isn't there: an ERROR in an object-access hook aborts the statement, and every table rewrite drops a transient relation through it. It gates on `pgcolumnar.options` resolving, not on the schema, and skips only the extension's own members. + +Each cleanup arm compares the catalogs with their state just before its own table was made, so one leak can't redden the arms after it. The REFRESH arm has a matview of its own, made WITH NO DATA, so it can't inherit the CREATE arm's failure. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_drop_after_converting_to_heap_cleans_up` | a converted, a round-tripped and an in-schema table each leave no relid-keyed rows on DROP, and a never-columnar heap is left alone | +| `test_the_hook_is_harmless_where_the_extension_never_was` | DROP, temp tables and every rewrite path succeed in a database without the extension | +| `test_a_relation_older_than_the_extension_is_cleaned_up` | a table that predates CREATE EXTENSION isn't a member, so its options row goes on DROP | +| `test_the_gate_is_on_the_options_table_not_the_schema` | DROP EXTENSION succeeds with populated catalogs, and afterwards, with only the schema left, every path still works | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 5a3f2d111..ad6f91ada 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -769,4 +769,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 682 -> 683 for #1341's test_native_recluster.py, the twin of native_recluster.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 683 +# cluster_tests 683 -> 687 for #1341's test_alter_am_cleanup.py, the twin of alter_am_cleanup.sh: +# four tests, by collection over the cluster files on this tree. +cluster_tests 687 diff --git a/test/pytest/test_alter_am_cleanup.py b/test/pytest/test_alter_am_cleanup.py new file mode 100644 index 000000000..fad5a0d66 --- /dev/null +++ b/test/pytest/test_alter_am_cleanup.py @@ -0,0 +1,199 @@ +"""DROP TABLE after SET ACCESS METHOD heap still takes the relid-keyed catalogs with it, +and the drop hook is harmless where the extension is absent (#304, #1341 item 70). The +pytest twin of `test/alter_am_cleanup.sh`, every check by the shell's name. + +pgcolumnar.options and projection_declaration are keyed by relid, so a table converted +away from columnar still has rows there, and the drop hook must clear them although the +table is heap by then. The hook is armed in every database, because the library is +preloaded: where the extension was never installed, or has been dropped, it must not +open a catalog that is not there, since an ERROR in an object-access hook aborts the +statement, and every table rewrite drops a transient relation through it. It gates on +pgcolumnar.options resolving, not the schema (DROP EXTENSION leaves the schema), and +skips only the extension's own members, not everything in its schema. + +Each cleanup arm compares the catalogs with their state just before its own table was +made, so one leak cannot redden the arms after it. Each statement against the other +databases is its own arm, and the REFRESH arm has a matview of its own made WITH NO +DATA, so neither inherits another's failure. Four databases, each this file's own. +""" +import psycopg +import pytest + +SNAPSHOT = ("SELECT (SELECT count(*) FROM pgcolumnar.storage) || '/' || " + "(SELECT count(*) FROM pgcolumnar.projection) || '/' || " + "(SELECT count(*) FROM pgcolumnar.row_group) || '/' || " + "(SELECT count(*) FROM pgcolumnar.options) || '/' || " + "(SELECT count(*) FROM pgcolumnar.projection_declaration)") +EXT_COUNT = "SELECT count(*) FROM pg_extension WHERE extname = 'pgcolumnar'" + + +def _admin(cluster, sql): + with psycopg.connect(cluster.dsn(), autocommit=True) as c: + c.execute(sql) + + +def _q(cluster, db, sql): + """-> the first column as text, or '' when the query cannot run (the shell's 2>/dev/null).""" + try: + with psycopg.connect(cluster.dsn(dbname=db), autocommit=True) as c: + return str(c.execute(sql).fetchone()[0]) + except psycopg.Error: + return "" + + +def _dbrun(cluster, db, *sqls): + """-> 'rc=0', or 'rc=1 ' and the first error, so a red names the statement and how.""" + try: + with psycopg.connect(cluster.dsn(dbname=db), autocommit=True) as c: + for sql in sqls: + c.execute(sql) + return "rc=0" + except psycopg.Error as exc: + return f"rc=1 ERROR: {exc.diag.message_primary or exc}"[:132] + + +@pytest.fixture +def dbs(pgc_cluster): + """-> a factory for fresh databases this test owns, dropped afterwards.""" + made = [] + + def make(name, extension): + _admin(pgc_cluster, f"DROP DATABASE IF EXISTS {name} WITH (FORCE)") + _admin(pgc_cluster, f"CREATE DATABASE {name}") + made.append(name) + if extension: + _dbrun(pgc_cluster, name, "CREATE EXTENSION pgcolumnar") + return name + + yield make + for name in made: + _admin(pgc_cluster, f"DROP DATABASE IF EXISTS {name} WITH (FORCE)") + + +def test_drop_after_converting_to_heap_cleans_up(pgc_cluster, dbs, expect): + db = dbs("aac_main", True) + snap = lambda: _q(pgc_cluster, db, SNAPSHOT) # noqa: E731 + run = lambda *s: _dbrun(pgc_cluster, db, *s) # noqa: E731 + base = snap() + run("CREATE TABLE aac_t (id int, a int, b text) USING pgcolumnar", + "SELECT pgcolumnar.set_options('aac_t', stripe_row_limit => 5000, sort_by => ARRAY['id'])", + "SELECT pgcolumnar.add_projection('aac_t','p1',ARRAY['a','b'],ARRAY['a'])", + "INSERT INTO aac_t SELECT g, g, 'x' FROM generate_series(1,200) g") + expect.text("yes" if snap() != base else "no", "yes", "premise: options and a projection declaration were recorded") + run("ALTER TABLE aac_t SET ACCESS METHOD heap") + expect.text(_q(pgc_cluster, db, "SELECT am.amname FROM pg_class c JOIN pg_am am ON am.oid = c.relam " + "WHERE c.relname = 'aac_t'"), "heap", "the table is heap after the conversion") + # a round trip back to columnar should still see the declared options + expect.text(_q(pgc_cluster, db, "SELECT stripe_row_limit FROM pgcolumnar.options WHERE regclass = 'aac_t'::regclass"), + "5000", "converting to heap does not drop the declared options") + run("DROP TABLE aac_t") + expect.text(snap(), base, "DROP after SET ACCESS METHOD heap leaves no options or declarations") + + pre_rt = snap() + run("CREATE TABLE aac_rt (id int) USING pgcolumnar", + "SELECT pgcolumnar.set_options('aac_rt', stripe_row_limit => 8000)", + "ALTER TABLE aac_rt SET ACCESS METHOD heap", + "ALTER TABLE aac_rt SET ACCESS METHOD pgcolumnar") + expect.text(_q(pgc_cluster, db, "SELECT stripe_row_limit FROM pgcolumnar.options WHERE regclass = 'aac_rt'::regclass"), + "8000", "options survive a heap round-trip") + run("DROP TABLE aac_rt") + expect.text(snap(), pre_rt, "a round-tripped table still cleans up on DROP") + + pre_h = snap() + run("CREATE TABLE aac_h (id int)", "DROP TABLE aac_h") + expect.text(snap(), pre_h, "dropping a heap table that was never columnar leaves the catalogs alone") + + run("CREATE TABLE aac_h_ctl (id int)") + expect.text(run("VACUUM FULL aac_h_ctl"), "rc=0", "control: VACUUM FULL still succeeds where the extension IS installed") + + # a table in the extension's schema is ours to clean, not ours to skip + pre_in = snap() + run("CREATE TABLE pgcolumnar.aac_inschema (id int) USING pgcolumnar", + "SELECT pgcolumnar.set_options('pgcolumnar.aac_inschema', stripe_row_limit => 7000)") + expect.text(_q(pgc_cluster, db, "SELECT count(*) FROM pgcolumnar.options " + "WHERE regclass = 'pgcolumnar.aac_inschema'::regclass"), "1", + "premise: a table in the extension's schema recorded its options") + run("DROP TABLE pgcolumnar.aac_inschema") + expect.text(snap(), pre_in, "dropping a table that lives in the extension's schema still cleans up") + + +def test_the_hook_is_harmless_where_the_extension_never_was(pgc_cluster, dbs, expect): + own = dbs("aac_own", True) + nocx = dbs("aac_nocx", False) + run = lambda *s: _dbrun(pgc_cluster, nocx, *s) # noqa: E731 + expect.text(_q(pgc_cluster, nocx, EXT_COUNT), "0", "premise: the extension is absent in the second database") + expect.text(_q(pgc_cluster, own, EXT_COUNT), "1", "premise: and installed in this suite's own database") + expect.text(run("CREATE TABLE nx (i int primary key, t text)"), "rc=0", + "a plain table is created in a database without the extension") + expect.text(run("DROP TABLE nx"), "rc=0", "and dropped there, without the extension") + expect.text(run("CREATE TEMP TABLE nxt (i int)", "DROP TABLE nxt"), "rc=0", + "an explicit DROP of a temp table succeeds there") + expect.text(run("BEGIN", "CREATE TEMP TABLE nxo (i int) ON COMMIT DROP", "COMMIT"), "rc=0", + "and ON COMMIT DROP commits there") + # the rewrite paths, each dropping a transient relation through the hook + expect.text(run("CREATE TABLE rw (i int primary key, t text)", + "INSERT INTO rw SELECT g, g::text FROM generate_series(1,100) g"), "rc=0", + "premise: a table to rewrite exists in that database") + expect.text(run("VACUUM FULL rw"), "rc=0", "VACUUM FULL succeeds without the extension") + expect.text(run("CLUSTER rw USING rw_pkey"), "rc=0", "CLUSTER succeeds without the extension") + expect.text(run("ALTER TABLE rw ALTER COLUMN t TYPE varchar(64)"), "rc=0", + "ALTER COLUMN TYPE succeeds without the extension") + expect.text(run("CREATE MATERIALIZED VIEW rwm AS SELECT * FROM rw"), "rc=0", + "CREATE MATERIALIZED VIEW succeeds without the extension") + expect.text(run("CREATE MATERIALIZED VIEW rwr AS SELECT * FROM rw WITH NO DATA"), "rc=0", + "premise: a matview to refresh exists, made WITH NO DATA") + expect.text(run("REFRESH MATERIALIZED VIEW rwr"), "rc=0", + "and REFRESH MATERIALIZED VIEW does too, without the extension") + expect.text(run("TRUNCATE rw"), "rc=0", "TRUNCATE succeeds without the extension") + + +def test_a_relation_older_than_the_extension_is_cleaned_up(pgc_cluster, dbs, expect): + pre = dbs("aac_pre", False) + run = lambda *s: _dbrun(pgc_cluster, pre, *s) # noqa: E731 + expect.text(run("CREATE TABLE early (id int)"), "rc=0", "premise: a table exists before the extension does") + expect.text(run("CREATE EXTENSION pgcolumnar"), "rc=0", "premise: and the extension is installed after it") + expect.text(run("ALTER TABLE early SET ACCESS METHOD pgcolumnar", + "SELECT pgcolumnar.set_options('early', stripe_row_limit => 6000)"), "rc=0", + "premise: the pre-existing table converts to columnar and records options") + expect.text(_q(pgc_cluster, pre, "SELECT count(*) FROM pgcolumnar.options WHERE regclass = 'early'::regclass"), + "1", "premise: its options row is really there") + expect.text(run("ALTER TABLE early SET ACCESS METHOD heap", "DROP TABLE early"), "rc=0", + "and dropping it after SET ACCESS METHOD heap leaves no orphan") + expect.text(_q(pgc_cluster, pre, "SELECT count(*) FROM pgcolumnar.options"), "0", + "the pre-extension table's options row is gone") + + +def test_the_gate_is_on_the_options_table_not_the_schema(pgc_cluster, dbs, expect): + dx = dbs("aac_dropx", False) + run = lambda *s: _dbrun(pgc_cluster, dx, *s) # noqa: E731 + expect.text(run("CREATE EXTENSION pgcolumnar"), "rc=0", "premise: the extension installs in a third database") + expect.text(run("CREATE TABLE dx (id int, a int, b text) USING pgcolumnar", + "SELECT pgcolumnar.set_options('dx', stripe_row_limit => 5000)", + "SELECT pgcolumnar.add_projection('dx','p1',ARRAY['a','b'],ARRAY['a'])"), "rc=0", + "premise: with a columnar table carrying both relid-keyed catalog rows") + expect.text(_q(pgc_cluster, dx, "SELECT (SELECT count(*) FROM pgcolumnar.options) || '/' || " + "(SELECT count(*) FROM pgcolumnar.projection_declaration)"), "1/1", + "premise: those rows are really there before the extension goes") + # the member skip is what stops the hook opening a sibling catalog DROP EXTENSION removed + expect.text(run("DROP EXTENSION pgcolumnar CASCADE"), "rc=0", "DROP EXTENSION succeeds with those catalogs populated") + expect.text(_q(pgc_cluster, dx, "SELECT count(*) FROM pg_namespace WHERE nspname = 'pgcolumnar'"), "1", + "premise: DROP EXTENSION left the schema behind") + expect.text(_q(pgc_cluster, dx, "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + "WHERE n.nspname = 'pgcolumnar' AND c.relname = 'options'"), "0", + "and took pgcolumnar.options with it") + expect.text(run("CREATE TABLE dxh (i int primary key, t text)"), "rc=0", "a plain table is created after DROP EXTENSION") + expect.text(run("DROP TABLE dxh"), "rc=0", "and dropped there, after DROP EXTENSION") + expect.text(run("CREATE TEMP TABLE dxt (i int)", "DROP TABLE dxt"), "rc=0", + "an explicit DROP of a temp table succeeds after DROP EXTENSION") + expect.text(run("CREATE TABLE dxr (i int primary key, t text)", + "INSERT INTO dxr SELECT g, g::text FROM generate_series(1,100) g"), "rc=0", + "premise: a table to rewrite exists after DROP EXTENSION") + expect.text(run("VACUUM FULL dxr"), "rc=0", "VACUUM FULL succeeds after DROP EXTENSION") + expect.text(run("CLUSTER dxr USING dxr_pkey"), "rc=0", "CLUSTER succeeds after DROP EXTENSION") + expect.text(run("ALTER TABLE dxr ALTER COLUMN t TYPE varchar(64)"), "rc=0", "ALTER COLUMN TYPE succeeds after DROP EXTENSION") + expect.text(run("CREATE MATERIALIZED VIEW dxm AS SELECT * FROM dxr"), "rc=0", + "CREATE MATERIALIZED VIEW succeeds after DROP EXTENSION") + expect.text(run("CREATE MATERIALIZED VIEW dxr_m AS SELECT * FROM dxr WITH NO DATA"), "rc=0", + "premise: a matview to refresh exists after DROP EXTENSION, made WITH NO DATA") + expect.text(run("REFRESH MATERIALIZED VIEW dxr_m"), "rc=0", "and REFRESH MATERIALIZED VIEW does too, after DROP EXTENSION") + expect.text(run("TRUNCATE dxr"), "rc=0", "TRUNCATE succeeds after DROP EXTENSION") diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 56f07da0a..6e2abfa1c 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -89,6 +89,7 @@ "alter_table_scans", "fdw_plan_privilege", "am_contracts", + "alter_am_cleanup", "analyze_differential", "analyze_function", "analyze_reltuples", From 62c80a430086f09b02c08f151f47b12965008efa Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:01:10 +0000 Subject: [PATCH 23/36] test: alter_am_cleanup seeds the ledger, 21 of 47 arms dated by seven mutations of the drop hook (#1341 item 70) The suite had no rows. All 47 checks enter from five-major runs (47/47 on each of 15-19). suites_not_covered 210 -> 209; census 3900 rows / 2843 never / 1057 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 47 ++++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 21 ++++++++++++++-- 2 files changed, 66 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index adda83b3e..50ba93a33 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -15,6 +15,53 @@ advisory_lock_class advisory_lock_class row_lock_buckets is fixed at server star advisory_lock_class advisory_lock_class the lock an insert takes is not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h) advisory_lock_class advisory_lock_class the locks an UPDATE takes are not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h) advisory_lock_class advisory_lock_class with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land 15;16;17;18;19 2026-10-15 the storage-row creation lock skipped whenever bulk_parallel_writer is on, not only once the row is committed (src/columnar_metadata.c) +alter_am_cleanup alter_am_cleanup ALTER COLUMN TYPE succeeds after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup ALTER COLUMN TYPE succeeds without the extension 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup CLUSTER succeeds after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup CLUSTER succeeds without the extension 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup CREATE MATERIALIZED VIEW succeeds after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup CREATE MATERIALIZED VIEW succeeds without the extension 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup DROP EXTENSION succeeds with those catalogs populated 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook does not skip the extension's own members +alter_am_cleanup alter_am_cleanup DROP after SET ACCESS METHOD heap leaves no options or declarations 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook clears relid-keyed rows only for a table still on the columnar AM (the #304 shape);columnar_tableam.c: the drop hook does not delete projection declarations;columnar_tableam.c: the drop hook does not delete the options row +alter_am_cleanup alter_am_cleanup TRUNCATE succeeds after DROP EXTENSION 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup TRUNCATE succeeds without the extension 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup VACUUM FULL succeeds after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup VACUUM FULL succeeds without the extension 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup a plain table is created after DROP EXTENSION 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup a plain table is created in a database without the extension 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup a round-tripped table still cleans up on DROP 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook does not delete the options row +alter_am_cleanup alter_am_cleanup an explicit DROP of a temp table succeeds after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup an explicit DROP of a temp table succeeds there 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup and ON COMMIT DROP commits there 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup and REFRESH MATERIALIZED VIEW does too, after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup and REFRESH MATERIALIZED VIEW does too, without the extension 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup and dropped there, after DROP EXTENSION 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook gates on the pgcolumnar schema resolving, not pgcolumnar.options;columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup and dropped there, without the extension 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook opens the relid-keyed catalogs with no gate at all +alter_am_cleanup alter_am_cleanup and dropping it after SET ACCESS METHOD heap leaves no orphan 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup and took pgcolumnar.options with it 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook does not skip the extension's own members +alter_am_cleanup alter_am_cleanup control: VACUUM FULL still succeeds where the extension IS installed 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup converting to heap does not drop the declared options 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup dropping a heap table that was never columnar leaves the catalogs alone 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup dropping a table that lives in the extension's schema still cleans up 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook does not delete the options row;columnar_tableam.c: the drop hook skips everything in the pgcolumnar schema, not only the extension's members +alter_am_cleanup alter_am_cleanup options survive a heap round-trip 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: DROP EXTENSION left the schema behind 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: a matview to refresh exists after DROP EXTENSION, made WITH NO DATA 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: a matview to refresh exists, made WITH NO DATA 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: a table exists before the extension does 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: a table in the extension's schema recorded its options 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: a table to rewrite exists after DROP EXTENSION 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: a table to rewrite exists in that database 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: and installed in this suite's own database 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: and the extension is installed after it 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: its options row is really there 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: options and a projection declaration were recorded 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: the extension installs in a third database 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: the extension is absent in the second database 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: the pre-existing table converts to columnar and records options 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: those rows are really there before the extension goes 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup premise: with a columnar table carrying both relid-keyed catalog rows 15;16;17;18;19 never - +alter_am_cleanup alter_am_cleanup the pre-extension table's options row is gone 15;16;17;18;19 2026-10-16 columnar_tableam.c: the drop hook clears relid-keyed rows only for a table still on the columnar AM (the #304 shape);columnar_tableam.c: the drop hook does not delete the options row +alter_am_cleanup alter_am_cleanup the table is heap after the conversion 15;16;17;18;19 never - alter_table_scans alter_table_scans ADD COLUMN with a volatile default rewrites the table, as on heap 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c) alter_table_scans alter_table_scans ADD PRIMARY KEY is refused over a duplicate (23505), as on heap 15;16;17;18;19 never - alter_table_scans alter_table_scans ADD PRIMARY KEY succeeds once the duplicate is deleted, as on heap 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 6f35318bf..0f2e7538f 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 210 +suites_not_covered 209 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3082,4 +3082,21 @@ suites_not_covered 210 # never 2811 + 6 = 2817 # dated 1028 + 8 = 1036 # 2817 + 1036 == 3853 -checks_never_observed_red 2817 +# +# #1341 item 70, alter_am_cleanup seeded, with its twin (OffgridwithJD): 2817 -> +# 2843, and suites_not_covered 210 -> 209. The suite had no rows. It enters with all +# 47 checks from five-major runs (47/47 on each of 15-19). 21 are dated by seven +# mutations of the drop hook: the cleanup gated on the columnar AM (#304's shape), a +# gate on the schema instead of pgcolumnar.options, no gate at all, a skip of the +# whole schema instead of the members, no member skip, and no delete of the +# declarations or of the options row. Twenty-six arrive never: the sixteen premises, the +# conversion and round-trip reads that precede each drop, the CREATE TABLE arms and the +# control, the never-columnar heap's arm, the pre-extension table's DROP (rc=0 either +# way; its options-row arm is the one dated), and the two TRUNCATE arms, since TRUNCATE +# drops no relation through the hook and no gate mutation reaches it. +# +# ledger rows 3853 + 47 = 3900 +# never 2817 + 26 = 2843 +# dated 1036 + 21 = 1057 +# 2843 + 1057 == 3900 +checks_never_observed_red 2843 From fd5c9cba8f71753c6c87e6f2831556ce7ed6cb03 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:48 +0000 Subject: [PATCH 24/36] test: native_reclaim gets its pytest twin, and its reuse arms can see reuse turned off (#1341 item 70) With an online compaction never reserving from a freed range, the shell stayed 36/36 green. Measured why: - the four "repeated" recluster('n', 'id') calls each returned 0: recluster self-gates on a table already in that key's order, so they wrote nothing and the file plateaued with reuse disabled. They now alternate v and id, and a new arm requires each to rewrite. Without reuse the file then grows 311296 -> 901120, and the plateau arm reddens. - "free space is being reused (bounded, not growing per cycle)" asserted free_space > 0, the arm before it again. It now compares with the count after the first cycle, and reddens as 1 -> 5 rows. test_native_reclaim.py carries all 37 checks by the shell's name in three tests; maintenance_due's eight refusals are a loop table, so the loop-reader guard's file list gains native_reclaim. COMPLETE gains native_reclaim, TESTS.md section 155, cluster_tests 687 -> 690. Co-Authored-By: Claude Opus 5.5 --- test/native_reclaim.sh | 19 +++- test/pytest/TESTS.md | 15 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 3 +- test/pytest/test_native_reclaim.py | 150 ++++++++++++++++++++++++++++ 5 files changed, 186 insertions(+), 5 deletions(-) create mode 100644 test/pytest/test_native_reclaim.py diff --git a/test/native_reclaim.sh b/test/native_reclaim.sh index e9567c3d2..523ca4639 100755 --- a/test/native_reclaim.sh +++ b/test/native_reclaim.sh @@ -66,6 +66,7 @@ check "no free space yet" "$(freerows)" "0" # With no reusable space yet it appends, so the file grows and free_space fills. psql_run "SELECT pgcolumnar.recluster('n', 'id');" size2="$(fsize)" +free2="$(freerows)" check "free space recorded after first recluster" \ "$([ "$(freerows)" -gt 0 ] && echo yes || echo no)" "yes" check "data intact after first recluster" \ @@ -74,16 +75,28 @@ check "data intact after first recluster" \ # Repeated reclusters must REUSE the previous cycle's freed space (its freeing # transaction has committed and the horizon has passed it), so the file plateaus # instead of growing every cycle. Without reuse it would grow by ~size1 each time. -for i in 1 2 3 4; do - psql_run "SELECT pgcolumnar.recluster('n', 'id');" +# +# The key ALTERNATES (#1341). Measured: four more recluster('n', 'id') calls each +# returned 0, because recluster self-gates on a table already in that key's order, +# so they wrote nothing and the file "plateaued" with reuse disabled too. With v and +# id in turn every call rewrites all six groups; without reuse the file then grows +# 311296 -> 901120 and free_space gains a row per cycle. +recs="" +for key in v id v id; do + recs="$recs$(q "SELECT pgcolumnar.recluster('n', '$key');")|" done size3="$(fsize)" +check "each repeated recluster rewrote the table, so none was skipped as already ordered" \ + "$(awk -F'|' '{ ok = (NF == 5); for (i = 1; i <= 4; i++) if ($i + 0 <= 0) ok = 0; print ok ? "yes" : "no (" $0 ")" }' <<<"$recs")" "yes" echo " (file size: initial=$size1 after-1-recluster=$size2 after-5-reclusters=$size3; free_space rows=$(freerows))" check "repeated recluster reuses space (file plateaus)" \ "$([ "$size3" -le "$((size2 + size1))" ] && echo yes || echo no)" "yes" +# Against the count after the first cycle, not "> 0" (#1341): that was the arm +# above again, and held with nothing reused. +free3="$(freerows)" check "free space is being reused (bounded, not growing per cycle)" \ - "$([ "$(freerows)" -gt 0 ] && echo yes || echo no)" "yes" + "$([ -n "$free3" ] && [ "$free3" -gt 0 ] && [ "$free3" -le "$free2" ] && echo yes || echo "no ($free2 -> $free3 rows)")" "yes" # The reused blocks hold correct data. check "reused-block data matches heap mirror" \ diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index f30f0487f..9cadc874d 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -200,6 +200,7 @@ behaviour, the source of that number is named. - [152. test_replication.py: physical replication and standby redo](#152-test_replicationpy-physical-replication-and-standby-redo) - [153. test_native_recluster.py: online reclustering under ShareUpdateExclusiveLock](#153-test_native_reclusterpy-online-reclustering-under-shareupdateexclusivelock) - [154. test_alter_am_cleanup.py: DROP after SET ACCESS METHOD heap cleans up](#154-test_alter_am_cleanuppy-drop-after-set-access-method-heap-cleans-up) +- [155. test_native_reclaim.py: physical page reclaim and the compaction threshold guards](#155-test_native_reclaimpy-physical-page-reclaim-and-the-compaction-threshold-guards) ## 1. How to read a test in here @@ -7532,3 +7533,17 @@ Each cleanup arm compares the catalogs with their state just before its own tabl | `test_the_hook_is_harmless_where_the_extension_never_was` | DROP, temp tables and every rewrite path succeed in a database without the extension | | `test_a_relation_older_than_the_extension_is_cleaned_up` | a table that predates CREATE EXTENSION isn't a member, so its options row goes on DROP | | `test_the_gate_is_on_the_options_table_not_the_schema` | DROP EXTENSION succeeds with populated catalogs, and afterwards, with only the schema left, every path still works | + +## 155. test_native_reclaim.py: physical page reclaim and the compaction threshold guards + +The pytest twin of `test/native_reclaim.sh` (ported for #1341, item 70), every check by the shell's name. Retiring a row group records its byte range in `pgcolumnar.free_space`, and a later online compaction reserves from a freed range instead of growing the file. So repeated reclusters plateau the file, free_space stays bounded, and the reused blocks read back the heap mirror's rows. The reclusters alternate their key, because recluster self-gates on a table already in that key's order: four more `recluster('n', 'id')` calls each returned 0, and the file "plateaued" with reuse disabled. + +`compact_rewrite()` and `maintenance_due()` refuse NaN, negative, above-1 and (for maintenance_due) NULL thresholds with 22023, and accept the boundaries 0.0 and 1.0. Controls pin that other failures carry other SQLSTATEs. `max_groups` caps one `compact_rewrite` call, which returns what it rewrote. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_repeated_recluster_reuses_space_and_compact_rewrite_is_guarded` | every repeated recluster rewrites, the file plateaus, free_space doesn't grow, reused blocks read back correctly, and compact_rewrite's threshold guard holds in both directions | +| `test_maintenance_due_validates_both_thresholds` | each threshold is refused out of range or NULL by 22023 and accepted on its boundaries, and the verdict follows the deleted fraction | +| `test_max_groups_caps_one_compact_rewrite` | a capped call rewrites three of ten half-deleted groups and an uncapped one the other seven, with every live row kept | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index ad6f91ada..9016daaac 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -771,4 +771,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 683 -> 687 for #1341's test_alter_am_cleanup.py, the twin of alter_am_cleanup.sh: # four tests, by collection over the cluster files on this tree. -cluster_tests 687 +# cluster_tests 687 -> 690 for #1341's test_native_reclaim.py, the twin of native_reclaim.sh: +# three tests, by collection over the cluster files on this tree. +cluster_tests 690 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 6e2abfa1c..9f899e3d6 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -133,6 +133,7 @@ "native_ownership", "native_parquet_dict_oob", "native_projection", + "native_reclaim", "native_reclaim_cycles", "native_reclaim_frag", "native_recluster", @@ -401,7 +402,7 @@ def test_the_loop_reader_invents_nothing_in_this_corpus(expect): "came from") expect.text(", ".join(files), "build_refusal, differential, join_runtime_filter, " - "native_vacuum_race, sorted_pathkeys", + "native_reclaim, native_vacuum_race, sorted_pathkeys", "and it is these files, so another appearing is a diff a reviewer " "sees rather than a number that moved") diff --git a/test/pytest/test_native_reclaim.py b/test/pytest/test_native_reclaim.py new file mode 100644 index 000000000..290c8f632 --- /dev/null +++ b/test/pytest/test_native_reclaim.py @@ -0,0 +1,150 @@ +"""Physical page reclaim: repeated online compaction reuses freed space, and the +compaction thresholds are guarded in both directions (Phase F, #860, #1341 item 70). The +pytest twin of `test/native_reclaim.sh`, every check by the shell's name. + +Retiring a row group records its byte range in pgcolumnar.free_space, and a later +online compaction reserves from a freed range instead of growing the file. So repeated +reclusters plateau the file, free_space stays bounded, and the reused blocks read back +the heap mirror's rows. The reclusters ALTERNATE their key: recluster self-gates on a +table already in that key's order, so four more recluster('n', 'id') calls each +returned 0 and the file "plateaued" with reuse disabled; and free_space is compared with +its count after the first cycle, since "> 0" held with nothing reused. + +compact_rewrite() and maintenance_due() refuse NaN, negative, above-1 and (for +maintenance_due) NULL thresholds with 22023 and accept the boundaries 0.0 and 1.0, with +controls pinning that other failures carry other SQLSTATEs; and max_groups caps one +compact_rewrite call, which returns what it rewrote. +""" +import psycopg + +GEN = "SELECT g, g AS v, md5(g::text) AS payload FROM generate_series(1, 6000) g" +FREE = "SELECT count(*) FROM pgcolumnar.free_space WHERE storage_id = pgcolumnar.get_storage_id('n')" + + +def _one(cur, sql): + cur.execute(sql) + return cur.fetchone()[0] + + +def _rows(cur, sql): + cur.execute(sql) + return cur.fetchall() + + +def _sqlstate(cur, sql): + """-> 'OK', or the SQLSTATE the statement raised; never 'OK' for a failure.""" + try: + cur.execute(sql) + return "OK" + except psycopg.Error as exc: + return exc.sqlstate or f"ERR_NO_SQLSTATE ({exc})" + + +def test_repeated_recluster_reuses_space_and_compact_rewrite_is_guarded(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE h (id int, v int, payload text)") + cur.execute("CREATE TABLE n (id int, v int, payload text) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('n', stripe_row_limit => 1000, chunk_group_row_limit => 1000)") + cur.execute(f"INSERT INTO h {GEN}") + cur.execute(f"INSERT INTO n {GEN}") + size1 = _one(cur, "SELECT pg_relation_size('n')") + expect.num(_one(cur, "SELECT count(*) FROM n"), 6000, "initial data present") + expect.num(_one(cur, FREE), 0, "no free space yet") + + # the first recluster appends, so the file grows and free_space fills + cur.execute("SELECT pgcolumnar.recluster('n', 'id')") + size2, free2 = _one(cur, "SELECT pg_relation_size('n')"), _one(cur, FREE) + expect.text("yes" if free2 > 0 else "no", "yes", "free space recorded after first recluster") + expect.num(_one(cur, "SELECT count(*) FROM n"), 6000, "data intact after first recluster") + + recs = [_one(cur, f"SELECT pgcolumnar.recluster('n', '{key}')") for key in ("v", "id", "v", "id")] + size3, free3 = _one(cur, "SELECT pg_relation_size('n')"), _one(cur, FREE) + print(f"-- file size: initial={size1} after-1={size2} after-5={size3}; free_space rows {free2} -> {free3}") + expect.text("yes" if all(r > 0 for r in recs) else f"no ({recs})", "yes", + "each repeated recluster rewrote the table, so none was skipped as already ordered") + expect.text("yes" if size3 <= size2 + size1 else f"no ({size2} -> {size3})", "yes", + "repeated recluster reuses space (file plateaus)") + expect.text("yes" if 0 < free3 <= free2 else f"no ({free2} -> {free3} rows)", "yes", + "free space is being reused (bounded, not growing per cycle)") + expect.row_set(_rows(cur, "SELECT id, v, payload FROM n"), _rows(cur, "SELECT id, v, payload FROM h"), + "reused-block data matches heap mirror") + + cur.execute("DELETE FROM h WHERE id % 4 = 0") + cur.execute("DELETE FROM n WHERE id % 4 = 0") + # Controls: each is what an "it errored" assertion cannot tell from the guard. + expect.text(_sqlstate(cur, "SELECT pgcolumnar.no_such_function(1)"), "42883", + "control: a missing function is 42883, not the guard's 22023") + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite('n', 0.5, 1, 'x')"), "42883", + "control: a wrong-arity compact_rewrite is 42883, not 22023") + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite(NULL, 0.5)"), "22004", + "control: a null table name is 22004, not 22023") + expect.text(_sqlstate(cur, "SELECT 1/0"), "22012", "control: division by zero is 22012, not 22023") + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite('n', 'NaN'::float8)"), "22023", + "compact_rewrite rejects a NaN threshold with 22023") + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite('n', 2.0)"), "22023", + "compact_rewrite rejects a threshold above 1 with 22023") + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite('n', -1.0)"), "22023", + "compact_rewrite rejects a negative threshold with 22023") + # the other direction: 0.0 and 1.0 sit on the boundary and are legal + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite('n', 0.0)"), "OK", + "compact_rewrite ACCEPTS the boundary threshold 0.0") + expect.text(_sqlstate(cur, "SELECT pgcolumnar.compact_rewrite('n', 1.0)"), "OK", + "compact_rewrite ACCEPTS the boundary threshold 1.0") + + cur.execute("SELECT pgcolumnar.recluster('n', 'id')") + expect.row_set(_rows(cur, "SELECT id, v, payload FROM n"), _rows(cur, "SELECT id, v, payload FROM h"), + "data correct after delete + compact_rewrite + recluster") + expect.num(_one(cur, "SELECT count(*) FROM n"), _one(cur, "SELECT count(*) FROM h"), "row count correct after cycle") + + +def test_maintenance_due_validates_both_thresholds(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE nd (id int, v int, payload text) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('nd', stripe_row_limit => 1000, chunk_group_row_limit => 1000)") + cur.execute("INSERT INTO nd SELECT g, g, md5(g::text) FROM generate_series(1, 20000) g") + cur.execute("DELETE FROM nd WHERE id % 2 = 0") + # every verdict below is about a table that is 50% deleted + total, deleted = _rows(cur, "SELECT total_rows, deleted_rows FROM pgcolumnar.maintenance_due('nd', 0.2, 0.05)")[0] + expect.num(total, 20000, "premise: nd holds 20000 rows") + expect.num(deleted, 10000, "premise: 10000 of nd's rows are deleted") + due = "SELECT compact_rewrite_due FROM pgcolumnar.maintenance_due('nd', {}, 0.05)" + expect.text(str(_one(cur, due.format(0.2))), "True", "maintenance_due(0.2) on a 50%-deleted table is still due") + expect.text(str(_one(cur, due.format(0.6))), "False", "maintenance_due(0.6) on a 50%-deleted table is not due") + + md = "SELECT * FROM pgcolumnar.maintenance_due('nd', {}, {})" + for args, name in ( + (("'NaN'::float8", "0.05"), "maintenance_due rejects a NaN compact threshold with 22023"), + (("2.0", "0.05"), "maintenance_due rejects a compact threshold above 1 with 22023"), + (("-1.0", "0.05"), "maintenance_due rejects a negative compact threshold with 22023"), + (("NULL::float8", "0.05"), "maintenance_due rejects a NULL compact threshold with 22023"), + (("0.2", "'NaN'::float8"), "maintenance_due rejects a NaN recluster threshold with 22023"), + (("0.2", "2.0"), "maintenance_due rejects a recluster threshold above 1 with 22023"), + (("0.2", "-1.0"), "maintenance_due rejects a negative recluster threshold with 22023"), + (("0.2", "NULL::float8"), "maintenance_due rejects a NULL recluster threshold with 22023"), + ): + expect.text(_sqlstate(cur, md.format(*args)), "22023", name) + # not over-broad: the boundaries are legal on both, and the defaults resolve + expect.text(_sqlstate(cur, md.format("0.0", "0.0")), "OK", "maintenance_due ACCEPTS the boundary thresholds 0.0, 0.0") + expect.text(_sqlstate(cur, md.format("1.0", "1.0")), "OK", "maintenance_due ACCEPTS the boundary thresholds 1.0, 1.0") + expect.text(_sqlstate(cur, "SELECT * FROM pgcolumnar.maintenance_due('nd')"), "OK", + "maintenance_due ACCEPTS its own defaults") + + +def test_max_groups_caps_one_compact_rewrite(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE ncap (id int, v text) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('ncap', stripe_row_limit => 1000)") + cur.execute("INSERT INTO ncap SELECT g, 'v'||g FROM generate_series(1, 10000) g") + cur.execute("DELETE FROM ncap WHERE id % 2 = 0") + # ten groups only if stripe_row_limit is honoured, and each half deleted + groups = _one(cur, "SELECT count(*) || '|' || count(*) FILTER (WHERE g.row_count - coalesce(d.deleted_count, 0) = 500 " + "AND coalesce(d.deleted_count, 0) = 500) FROM pgcolumnar.row_group g " + "LEFT JOIN pgcolumnar.delete_vector d ON d.storage_id = g.storage_id " + "AND d.group_number = g.group_number WHERE g.storage_id = pgcolumnar.get_storage_id('ncap')") + expect.text(f"{groups}|{_one(cur, 'SELECT count(*) FROM ncap')}", "10|10|5000", + "ten row groups, every one half deleted") + calls = [_one(cur, f"SELECT pgcolumnar.compact_rewrite('ncap', 0.1, {cap})") for cap in (3, 0, 0)] + expect.text("|".join(str(c) for c in calls), "3|7|0", + "compact_rewrite with max_groups 3 rewrites three, then an uncapped call the other seven") + expect.text(_one(cur, "SELECT count(*) || '|' || count(DISTINCT id) || '|' || sum(id) FROM ncap"), + "5000|5000|25000000", "and every live row is still there, once") From 8c74c45c8e6c2b5a04230c4d071ee2dbeb7f082e Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:49 +0000 Subject: [PATCH 25/36] test: native_reclaim re-dates 17 arms by sixteen mutations of reuse and the threshold guards (#1341 item 70) One new check enters never; 17 of the suite's never rows are dated from five-major runs (37/37 on each of 15-19). Census 3901 rows / 2828 never / 1073 dated; suites_not_covered unchanged (209). Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 35 ++++++++++++++++++----------------- test/check_ledger_budget.txt | 18 +++++++++++++++++- 2 files changed, 35 insertions(+), 18 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 50ba93a33..e6f1602b3 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2623,11 +2623,11 @@ native_join_vector_agg native_join_vector_agg unique join fold answer equals hea native_join_vector_agg native_join_vector_agg unique join uses core Agg when GUC off 15;16;17;18;19 never - native_join_vector_agg native_join_vector_agg unique join uses vectorized agg when GUC on 15;16;17;18;19 2026-09-12 drop JOINREL fold native_reclaim native_reclaim and every live row is still there, once 15;16;17;18;19 never - -native_reclaim native_reclaim compact_rewrite ACCEPTS the boundary threshold 0.0 15;16;17;18;19 never - -native_reclaim native_reclaim compact_rewrite ACCEPTS the boundary threshold 1.0 15;16;17;18;19 never - -native_reclaim native_reclaim compact_rewrite rejects a NaN threshold with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim compact_rewrite rejects a negative threshold with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim compact_rewrite rejects a threshold above 1 with 22023 15;16;17;18;19 never - +native_reclaim native_reclaim compact_rewrite ACCEPTS the boundary threshold 0.0 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard refuses minFrac <= 0.0 +native_reclaim native_reclaim compact_rewrite ACCEPTS the boundary threshold 1.0 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard refuses minFrac >= 1.0 +native_reclaim native_reclaim compact_rewrite rejects a NaN threshold with 22023 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard drops isnan(minFrac) +native_reclaim native_reclaim compact_rewrite rejects a negative threshold with 22023 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard drops minFrac < 0.0 +native_reclaim native_reclaim compact_rewrite rejects a threshold above 1 with 22023 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard drops minFrac > 1.0 native_reclaim native_reclaim compact_rewrite with max_groups 3 rewrites three, then an uncapped call the other seven 15;16;17;18;19 2026-10-14 compact_rewrite ignoring max_groups;the write state ignores stripe_row_limit native_reclaim native_reclaim control: a missing function is 42883, not the guard's 22023 15;16;17;18;19 never - native_reclaim native_reclaim control: a null table name is 22004, not 22023 15;16;17;18;19 never - @@ -2635,26 +2635,27 @@ native_reclaim native_reclaim control: a wrong-arity compact_rewrite is 42883, n native_reclaim native_reclaim control: division by zero is 22012, not 22023 15;16;17;18;19 never - native_reclaim native_reclaim data correct after delete + compact_rewrite + recluster 15;16;17;18;19 never - native_reclaim native_reclaim data intact after first recluster 15;16;17;18;19 never - -native_reclaim native_reclaim free space is being reused (bounded, not growing per cycle) 15;16;17;18;19 never - +native_reclaim native_reclaim each repeated recluster rewrote the table, so none was skipped as already ordered 15;16;17;18;19 never - +native_reclaim native_reclaim free space is being reused (bounded, not growing per cycle) 15;16;17;18;19 2026-10-16 columnar_write_state.c: an online compaction never reserves from a freed range (PgColumnarAllocateFreeSpace not called) native_reclaim native_reclaim free space recorded after first recluster 15;16;17;18;19 never - native_reclaim native_reclaim initial data present 15;16;17;18;19 never - native_reclaim native_reclaim maintenance_due ACCEPTS its own defaults 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due ACCEPTS the boundary thresholds 0.0, 0.0 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due ACCEPTS the boundary thresholds 1.0, 1.0 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due rejects a NULL compact threshold with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due rejects a NULL recluster threshold with 22023 15;16;17;18;19 never - +native_reclaim native_reclaim maintenance_due ACCEPTS the boundary thresholds 0.0, 0.0 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due refuses compact_due_fraction <= 0.0 +native_reclaim native_reclaim maintenance_due ACCEPTS the boundary thresholds 1.0, 1.0 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due refuses compact_due_fraction >= 1.0 +native_reclaim native_reclaim maintenance_due rejects a NULL compact threshold with 22023 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due accepts a NULL compact_due_fraction +native_reclaim native_reclaim maintenance_due rejects a NULL recluster threshold with 22023 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due accepts a NULL recluster_due_fraction native_reclaim native_reclaim maintenance_due rejects a NaN compact threshold with 22023 15;16;17;18;19 never - native_reclaim native_reclaim maintenance_due rejects a NaN recluster threshold with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due rejects a compact threshold above 1 with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due rejects a negative compact threshold with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due rejects a negative recluster threshold with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due rejects a recluster threshold above 1 with 22023 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due(0.2) on a 50%-deleted table is still due 15;16;17;18;19 never - -native_reclaim native_reclaim maintenance_due(0.6) on a 50%-deleted table is not due 15;16;17;18;19 never - +native_reclaim native_reclaim maintenance_due rejects a compact threshold above 1 with 22023 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due refuses a compact_due_fraction only above 2.5 +native_reclaim native_reclaim maintenance_due rejects a negative compact threshold with 22023 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due refuses a compact_due_fraction only below -1.5 +native_reclaim native_reclaim maintenance_due rejects a negative recluster threshold with 22023 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due refuses a recluster_due_fraction only below -1.5 +native_reclaim native_reclaim maintenance_due rejects a recluster threshold above 1 with 22023 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: maintenance_due refuses a recluster_due_fraction only above 2.5 +native_reclaim native_reclaim maintenance_due(0.2) on a 50%-deleted table is still due 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: compact_rewrite_due is never true +native_reclaim native_reclaim maintenance_due(0.6) on a 50%-deleted table is not due 15;16;17;18;19 2026-10-16 pgcolumnar--1.0-alpha5.sql: compact_rewrite_due is always true native_reclaim native_reclaim no free space yet 15;16;17;18;19 never - native_reclaim native_reclaim premise: 10000 of nd's rows are deleted 15;16;17;18;19 never - native_reclaim native_reclaim premise: nd holds 20000 rows 15;16;17;18;19 never - -native_reclaim native_reclaim repeated recluster reuses space (file plateaus) 15;16;17;18;19 never - +native_reclaim native_reclaim repeated recluster reuses space (file plateaus) 15;16;17;18;19 2026-10-16 columnar_write_state.c: an online compaction never reserves from a freed range (PgColumnarAllocateFreeSpace not called) native_reclaim native_reclaim reused-block data matches heap mirror 15;16;17;18;19 never - native_reclaim native_reclaim row count correct after cycle 15;16;17;18;19 never - native_reclaim native_reclaim ten row groups, every one half deleted 15;16;17;18;19 2026-10-14 the write state ignores stripe_row_limit diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 0f2e7538f..320c24865 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -3099,4 +3099,20 @@ suites_not_covered 209 # never 2817 + 26 = 2843 # dated 1036 + 21 = 1057 # 2843 + 1057 == 3900 -checks_never_observed_red 2843 +# +# #1341 item 70, native_reclaim re-dated, with its twin (OffgridwithJD): 2843 -> +# 2827. The suite had 36 rows, 2 dated. One check is new and arrives never ("each +# repeated recluster rewrote the table ...", which no mutation here brings to zero), +# and 17 are newly dated by sixteen mutations from five-major runs (37/37 on each of +# 15-19): no reservation from a freed range (the plateau and the free_space bound, +# which now compares with the first cycle's count, and whose reclusters now alternate +# their key so the self-gate cannot skip them), compact_rewrite's guard without each +# of its three bounds or strict at either boundary, and maintenance_due with each +# compact and recluster bound loosened or tightened and a due verdict pinned true or +# false. suites_not_covered does not move: the suite was already covered. +# +# ledger rows 3900 + 1 = 3901 +# never 2843 + 1 - 17 = 2827 +# dated 1057 + 17 = 1074 +# 2827 + 1074 == 3901 +checks_never_observed_red 2827 From 7e8d31088df4415e2d4765d7e1724d78e376b562 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:49 +0000 Subject: [PATCH 26/36] test: native_repack gets its pytest twin, and "the supported alternative" must reclaim (#1341) With pgcolumnar.vacuum() returning before it rewrote anything, the shell stayed 19/19 green: its alternative arms asserted only that vacuum() returned and kept the rows. A premise now counts the fixture's deleted rows (6666) when the table is made, before any REPACK can touch it, and "and it reclaimed the deleted rows" requires 0 after vacuum(). test_native_repack.py carries all 21 checks (and the shell's REPACK skip and dead-server names) on a private cluster at wal_level=logical, which REPACK (CONCURRENTLY) needs. Below 19 it declines UNSUPPORTED_MAJOR, listed for 15-18 in expected_unrunnable.txt. The loop-reader guard's file list gains native_repack. COMPLETE gains native_repack, TESTS.md section 156, cluster_tests 690 -> 691. Co-Authored-By: Claude Opus 5.5 --- test/native_repack.sh | 9 ++ test/pytest/TESTS.md | 13 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/expected_unrunnable.txt | 6 ++ test/pytest/test_compare_to_bash.py | 3 +- test/pytest/test_native_repack.py | 136 ++++++++++++++++++++++++++++ 6 files changed, 169 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_native_repack.py diff --git a/test/native_repack.sh b/test/native_repack.sh index 55d349702..515186da7 100755 --- a/test/native_repack.sh +++ b/test/native_repack.sh @@ -78,6 +78,10 @@ hash_before=$(q "SELECT md5(string_agg(id::text||':'||v, ',' ORDER BY id)) FROM opts_before=$(q "SELECT stripe_row_limit || '/' || compression FROM pgcolumnar.options o JOIN pg_class c ON c.oid = o.regclass WHERE c.relname = 'rp'") +dead_sql="SELECT coalesce(sum(deleted_count), 0) FROM pgcolumnar.delete_vector WHERE storage_id = pgcolumnar.get_storage_id('rp')" +check "premise: the table holds deleted rows for the alternative to reclaim" \ + "$(q "$dead_sql")" "$(( ROWS / 3 ))" + err_of() { psql_run "$1" 2>&1 | grep -oE 'ERROR:.*' | head -1; } # ---- every spelling errors, and the message names the command ---------------- @@ -167,8 +171,13 @@ check "the columnar table is unharmed: access method" \ "pgcolumnar" # ---- and the supported route does work --------------------------------------- +# "Works" is that it reclaims, not only that it returns (#1341): with vacuum() +# returning before it rewrote anything, every arm here stayed green. The rows the +# fixture deleted are in storage when it is made (asserted there, before any REPACK +# can touch the table), and gone after. check "pgcolumnar.vacuum is the supported alternative and succeeds" \ "$(psql_run "SELECT pgcolumnar.vacuum('rp');" 2>&1 | grep -c ERROR || true)" "0" +check "and it reclaimed the deleted rows" "$(q "$dead_sql")" "0" check "and it kept every live row" "$(q 'SELECT count(*) FROM rp')" "$live" check "and the content is still identical" \ "$(q "SELECT md5(string_agg(id::text||':'||v, ',' ORDER BY id)) FROM rp")" "$hash_before" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 9cadc874d..5a5015eaf 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -201,6 +201,7 @@ behaviour, the source of that number is named. - [153. test_native_recluster.py: online reclustering under ShareUpdateExclusiveLock](#153-test_native_reclusterpy-online-reclustering-under-shareupdateexclusivelock) - [154. test_alter_am_cleanup.py: DROP after SET ACCESS METHOD heap cleans up](#154-test_alter_am_cleanuppy-drop-after-set-access-method-heap-cleans-up) - [155. test_native_reclaim.py: physical page reclaim and the compaction threshold guards](#155-test_native_reclaimpy-physical-page-reclaim-and-the-compaction-threshold-guards) +- [156. test_native_repack.py: REPACK on a columnar table](#156-test_native_repackpy-repack-on-a-columnar-table) ## 1. How to read a test in here @@ -7547,3 +7548,15 @@ The pytest twin of `test/native_reclaim.sh` (ported for #1341, item 70), every c | `test_repeated_recluster_reuses_space_and_compact_rewrite_is_guarded` | every repeated recluster rewrites, the file plateaus, free_space doesn't grow, reused blocks read back correctly, and compact_rewrite's threshold guard holds in both directions | | `test_maintenance_due_validates_both_thresholds` | each threshold is refused out of range or NULL by 22023 and accepted on its boundaries, and the verdict follows the deleted fraction | | `test_max_groups_caps_one_compact_rewrite` | a capped call rewrites three of ten half-deleted groups and an uncapped one the other seven, with every live row kept | + +## 156. test_native_repack.py: REPACK on a columnar table + +The pytest twin of `test/native_repack.sh` (ported for #1341), every check by the shell's name. PostgreSQL 19 and later only. REPACK replaces CLUSTER and VACUUM FULL in 19 and dispatches through `relation_copy_for_cluster`, which pgColumnar registers as a stub that raises. So every spelling errors, the message names REPACK, and the table keeps its rows, content, options and access method. REPACK (CONCURRENTLY) needs two fixtures: without an identity index core refuses it before the access method is reached, and with one heap succeeds while columnar is refused. + +The alternative must reclaim, not only return. With `vacuum()` returning before it rewrote anything, every arm stayed green, so the deleted rows are counted at fixture time and again after it. The twin runs on a private cluster at `wal_level=logical`, which the concurrent form needs. Below 19 it declines with `UNSUPPORTED_MAJOR`, listed in `expected_unrunnable.txt`. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_repack_on_a_columnar_table` | every REPACK spelling is refused naming REPACK, the concurrent form is refused for a stated reason, the table is unharmed, and `pgcolumnar.vacuum()` reclaims the deleted rows | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 9016daaac..67e3ae5ec 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -773,4 +773,6 @@ guard_tests 438 # four tests, by collection over the cluster files on this tree. # cluster_tests 687 -> 690 for #1341's test_native_reclaim.py, the twin of native_reclaim.sh: # three tests, by collection over the cluster files on this tree. -cluster_tests 690 +# cluster_tests 690 -> 691 for #1341's test_native_repack.py, the twin of native_repack.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 691 diff --git a/test/pytest/expected_unrunnable.txt b/test/pytest/expected_unrunnable.txt index 3259220b0..5d183b579 100644 --- a/test/pytest/expected_unrunnable.txt +++ b/test/pytest/expected_unrunnable.txt @@ -87,3 +87,9 @@ 17 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table 18 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table 19 test_pg_upgrade.py::test_pg_upgrade_preserves_a_columnar_table + +# PostgreSQL 15 to 18: REPACK arrived in 19, so there is no REPACK to refuse. +15 test_native_repack.py::test_repack_on_a_columnar_table +16 test_native_repack.py::test_repack_on_a_columnar_table +17 test_native_repack.py::test_repack_on_a_columnar_table +18 test_native_repack.py::test_repack_on_a_columnar_table diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 9f899e3d6..8821bd0b6 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -137,6 +137,7 @@ "native_reclaim_cycles", "native_reclaim_frag", "native_recluster", + "native_repack", "native_vacuum_race", "objstore_allowlist", "objstore_credentials", @@ -402,7 +403,7 @@ def test_the_loop_reader_invents_nothing_in_this_corpus(expect): "came from") expect.text(", ".join(files), "build_refusal, differential, join_runtime_filter, " - "native_reclaim, native_vacuum_race, sorted_pathkeys", + "native_reclaim, native_repack, native_vacuum_race, sorted_pathkeys", "and it is these files, so another appearing is a diff a reviewer " "sees rather than a number that moved") diff --git a/test/pytest/test_native_repack.py b/test/pytest/test_native_repack.py new file mode 100644 index 000000000..2c6d8731c --- /dev/null +++ b/test/pytest/test_native_repack.py @@ -0,0 +1,136 @@ +"""REPACK on a columnar table is refused, naming what the user typed, and leaves the +table unharmed; pgcolumnar.vacuum() is the alternative that works (#399, #1341). The +pytest twin of `test/native_repack.sh`, every check by the shell's name. PostgreSQL 19 +and later only. + +REPACK replaces CLUSTER and VACUUM FULL in 19 and dispatches through +relation_copy_for_cluster, which pgColumnar registers as a stub that raises. So every +spelling errors, the message names REPACK, and the table keeps its rows, content, +options and access method. REPACK (CONCURRENTLY) needs two fixtures: without an +identity index core refuses it before the access method is reached, and with one heap +succeeds while columnar is refused (by core's heap-only check from 19beta4, by ours +before). The alternative must reclaim, not only return: with vacuum() returning before +it rewrote anything every arm stayed green, so the deleted rows are counted before and +after. + +A PRIVATE CLUSTER at wal_level=logical, which the concurrent form needs and the session +cluster does not have. +""" +import pathlib +import shutil +import tempfile + +import psycopg + +ROWS = 20000 +HASH = "SELECT md5(string_agg(id::text||':'||v, ',' ORDER BY id)) FROM {}" +OPTS = ("SELECT stripe_row_limit || '/' || compression FROM pgcolumnar.options o " + "JOIN pg_class c ON c.oid = o.regclass WHERE c.relname = 'rp'") +DEAD = ("SELECT coalesce(sum(deleted_count), 0) FROM pgcolumnar.delete_vector " + "WHERE storage_id = pgcolumnar.get_storage_id('rp')") + + +def _one(conn, sql): + return conn.execute(sql).fetchone()[0] + + +def _err(conn, sql): + """-> the error's primary message and detail, or '' when the statement succeeded.""" + try: + conn.execute(sql) + return "" + except psycopg.Error as exc: + return f"{exc.diag.message_primary} {exc.diag.message_detail or ''}" + + +def test_repack_on_a_columnar_table(pgc_cluster, expect): + from pgc_cluster import Cluster, pick_port + + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-repack-")) + work.chmod(0o777) + (work / "data").mkdir() + cluster = Cluster(pgc_cluster.pg_config, "repack", work / "data", pick_port(23)) + try: + cluster.initdb() + with open(cluster.datadir / "postgresql.conf", "a") as fh: + fh.write("wal_level=logical\nmax_locks_per_transaction=1024\nmax_wal_senders=10\n" + "max_replication_slots=10\n") + cluster.start() + with psycopg.connect(cluster.dsn(), autocommit=True) as conn: + srv = str(_one(conn, "SHOW server_version_num")) + if not srv.isdigit(): + # a cluster that stopped talking, not an older major (the shell's pgc_fail) + expect.text(srv, "a server_version_num", "the server did not answer 'SHOW server_version_num'") + major = int(srv) // 10000 + if major < 19: + expect.cannot_run("UNSUPPORTED_MAJOR", f"REPACK arrived in PostgreSQL 19; this server is {major}", + name="REPACK") + return + for sql in ("CREATE EXTENSION pgcolumnar", + "CREATE TABLE rp (id int, v text) USING pgcolumnar", + "SELECT pgcolumnar.set_options('rp', stripe_row_limit => 5000, compression => 'zstd')", + f"INSERT INTO rp SELECT g, 'x'||g FROM generate_series(1,{ROWS}) g", + "DELETE FROM rp WHERE id % 3 = 0", + "CREATE INDEX rp_id ON rp (id)", + "CREATE TABLE rp_heap (id int, v text)", + f"INSERT INTO rp_heap SELECT g, 'x'||g FROM generate_series(1,{ROWS}) g", + "CREATE INDEX rp_heap_id ON rp_heap (id)"): + conn.execute(sql) + live, hash_before, opts_before = _one(conn, "SELECT count(*) FROM rp"), _one(conn, HASH.format("rp")), \ + _one(conn, OPTS) + # asserted here, before any REPACK can touch the table + expect.num(_one(conn, DEAD), ROWS // 3, "premise: the table holds deleted rows for the alternative to reclaim") + + # every spelling errors, and the message names the command + e = _err(conn, "REPACK rp") + expect.text("yes" if e else "no (succeeded)", "yes", "REPACK on a columnar table errors (#399)") + expect.text("yes" if "REPACK" in e else f"no ({e})", "yes", + "and the error names REPACK, which is what the user typed (#399)") + for sql, name in (("REPACK rp USING INDEX rp_id", "REPACK ... USING INDEX errors the same way"), + ("REPACK (VERBOSE) rp", "REPACK (VERBOSE) errors the same way"), + ("CLUSTER rp USING rp_id", "CLUSTER errors the same way"), + ("VACUUM FULL rp", "VACUUM FULL errors the same way")): + expect.text("yes" if "not supported" in _err(conn, sql) else "no", "yes", name) + + # REPACK (CONCURRENTLY): without an identity index core refuses it first + nopk = _err(conn, "REPACK (CONCURRENTLY) rp") + expect.text("no (ours: %s)" % nopk if "not supported" in nopk + else "yes" if "cannot execute REPACK" in nopk else f"no ({nopk})", "yes", + "without an identity index it is refused by PostgreSQL, not by us") + for sql in ("CREATE TABLE rp_pk (id int PRIMARY KEY, v text) USING pgcolumnar", + f"INSERT INTO rp_pk SELECT g, 'x'||g FROM generate_series(1,{ROWS}) g", + "CREATE TABLE rp_pk_heap (id int PRIMARY KEY, v text)", + f"INSERT INTO rp_pk_heap SELECT g, 'x'||g FROM generate_series(1,{ROWS}) g"): + conn.execute(sql) + pk_rows, pk_hash = _one(conn, "SELECT count(*) FROM rp_pk"), _one(conn, HASH.format("rp_pk")) + expect.text(_err(conn, "REPACK (CONCURRENTLY) rp_pk_heap") or "succeeded", "succeeded", + "control: with an identity index, REPACK (CONCURRENTLY) succeeds on heap") + # from 19beta4 core refuses a non-heap AM first (#1337); earlier betas reach ours + c_pk = _err(conn, "REPACK (CONCURRENTLY) rp_pk") + expect.text("yes" if ("not supported yet" in c_pk or 'only supported for the "heap" access method' in c_pk) + else f"no ({c_pk or 'succeeded'})", "yes", + "and on a columnar table it is refused: by core's heap-only check from 19beta4, else by ours") + expect.num(_one(conn, "SELECT count(*) FROM rp_pk"), pk_rows, + "the columnar table is undamaged after the refused concurrent repack: rows") + expect.text(_one(conn, HASH.format("rp_pk")), pk_hash, + "the columnar table is undamaged after the refused concurrent repack: content") + + expect.text(_err(conn, "REPACK rp_heap") or "succeeded", "succeeded", + "control: REPACK works on a heap table on this build") + + # a failed rewrite must leave the table untouched + expect.num(_one(conn, "SELECT count(*) FROM rp"), live, "the columnar table is unharmed: row count") + expect.text(_one(conn, HASH.format("rp")), hash_before, "the columnar table is unharmed: content hash") + expect.text(_one(conn, OPTS), opts_before, "the columnar table is unharmed: storage options") + expect.text(_one(conn, "SELECT amname::text FROM pg_am a JOIN pg_class c ON c.relam = a.oid " + "WHERE c.relname = 'rp'"), "pgcolumnar", "the columnar table is unharmed: access method") + + # the supported route reclaims, not only returns: the dead rows counted above go + expect.text(_err(conn, "SELECT pgcolumnar.vacuum('rp')") or "succeeded", "succeeded", + "pgcolumnar.vacuum is the supported alternative and succeeds") + expect.num(_one(conn, DEAD), 0, "and it reclaimed the deleted rows") + expect.num(_one(conn, "SELECT count(*) FROM rp"), live, "and it kept every live row") + expect.text(_one(conn, HASH.format("rp")), hash_before, "and the content is still identical") + finally: + cluster.stop() + shutil.rmtree(work, ignore_errors=True) From 0692d9563c4a4a66d51cbc249a5690607f56e24d Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:49 +0000 Subject: [PATCH 27/36] test: native_repack seeds the ledger, 11 of 22 rows dated by four mutations (#1341) The suite had no rows. 21 checks enter from PostgreSQL 19 and the REPACK skip from 15-18. suites_not_covered 209 -> 208; census 3923 rows / 2838 never / 1085 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 22 ++++++++++++++++++++++ test/check_ledger_budget.txt | 21 +++++++++++++++++++-- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index e6f1602b3..26c12b6ec 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2673,6 +2673,28 @@ native_recluster native_recluster row count 15;16;17;18;19 never - native_recluster native_recluster row count after recluster 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online retires none of the old groups (PgColumnarRetireGroup removed) native_recluster native_recluster the plan under test is a columnar custom scan 15;16;17;18;19 never - native_recluster native_recluster unique still enforced, by SQLSTATE 23505 and not by any failure 15;16;17;18;19 2026-10-16 columnar_vacuum.c: recluster_online inserts no index entry for a rewritten row (PgColumnarIndexInsertRow removed) +native_repack native_repack CLUSTER errors the same way 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing;columnar_tableam.c: the refusal no longer says not supported +native_repack native_repack REPACK 15;16;17;18 never - +native_repack native_repack REPACK (VERBOSE) errors the same way 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing;columnar_tableam.c: the refusal no longer says not supported +native_repack native_repack REPACK ... USING INDEX errors the same way 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing;columnar_tableam.c: the refusal no longer says not supported +native_repack native_repack REPACK on a columnar table errors (#399) 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing +native_repack native_repack VACUUM FULL errors the same way 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing;columnar_tableam.c: the refusal no longer says not supported +native_repack native_repack and it kept every live row 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing +native_repack native_repack and it reclaimed the deleted rows 19 2026-10-16 columnar_vacuum.c: pgcolumnar_vacuum returns before rewriting +native_repack native_repack and on a columnar table it is refused: by core's heap-only check from 19beta4, else by ours 19 never - +native_repack native_repack and the content is still identical 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing +native_repack native_repack and the error names REPACK, which is what the user typed (#399) 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing;columnar_tableam.c: the PG19 refusal names CLUSTER and VACUUM FULL but not REPACK +native_repack native_repack control: REPACK works on a heap table on this build 19 never - +native_repack native_repack control: with an identity index, REPACK (CONCURRENTLY) succeeds on heap 19 never - +native_repack native_repack pgcolumnar.vacuum is the supported alternative and succeeds 19 never - +native_repack native_repack premise: the table holds deleted rows for the alternative to reclaim 19 never - +native_repack native_repack the columnar table is undamaged after the refused concurrent repack: content 19 never - +native_repack native_repack the columnar table is undamaged after the refused concurrent repack: rows 19 never - +native_repack native_repack the columnar table is unharmed: access method 19 never - +native_repack native_repack the columnar table is unharmed: content hash 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing +native_repack native_repack the columnar table is unharmed: row count 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing +native_repack native_repack the columnar table is unharmed: storage options 19 never - +native_repack native_repack without an identity index it is refused by PostgreSQL, not by us 19 never - objstore_allowlist objstore_allowlist a multi-entry list matches any entry 15;16;17;18;19 never - objstore_allowlist objstore_allowlist a non-superuser cannot widen the list (42501) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it objstore_allowlist objstore_allowlist a superuser session SET works (admin override) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 320c24865..9414c2905 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 209 +suites_not_covered 208 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3115,4 +3115,21 @@ suites_not_covered 209 # never 2843 + 1 - 17 = 2827 # dated 1057 + 17 = 1074 # 2827 + 1074 == 3901 -checks_never_observed_red 2827 +# +# #1341, native_repack seeded, with its twin (OffgridwithJD): 2827 -> 2838, and +# suites_not_covered 209 -> 208. The suite had no rows. It enters with 22: its 21 +# checks from PostgreSQL 19, where REPACK exists (21/21), and the REPACK skip that +# 15-18 record. 11 are dated by four mutations: the PG19 refusal not naming REPACK, the +# refusal not saying "not supported", relation_copy_for_cluster returning without +# raising (every spelling succeeds and the table comes back empty), and vacuum() +# returning before it rewrites (the new "and it reclaimed the deleted rows"). Eleven +# arrive never: the skip, the deleted-rows premise, the heap controls, the two +# REPACK (CONCURRENTLY) refusals, which core raises before our callback is reached on +# 19beta4 and later, the concurrent fixture's undamaged arms, the storage-options and +# access-method arms, and vacuum()'s own success. +# +# ledger rows 3901 + 22 = 3923 +# never 2827 + 11 = 2838 +# dated 1074 + 11 = 1085 +# 2838 + 1085 == 3923 +checks_never_observed_red 2838 From abc7c8da145a5643392a40842eed6d87d47ed8a8 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:49 +0000 Subject: [PATCH 28/36] test: native_join_vector_agg gets its pytest twin (#1341) test_native_join_vector_agg.py carries all 13 checks by the shell's name. The shell needed no change: removing the join-filter check, the key and uniqueness checks, or the join-rel path each reddens it, and its ledger rows for those are already dated. COMPLETE gains native_join_vector_agg, TESTS.md section 157, cluster_tests 691 -> 692. Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 11 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_join_vector_agg.py | 113 +++++++++++++++++++++ 4 files changed, 128 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_native_join_vector_agg.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 5a5015eaf..baffd3e2d 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -202,6 +202,7 @@ behaviour, the source of that number is named. - [154. test_alter_am_cleanup.py: DROP after SET ACCESS METHOD heap cleans up](#154-test_alter_am_cleanuppy-drop-after-set-access-method-heap-cleans-up) - [155. test_native_reclaim.py: physical page reclaim and the compaction threshold guards](#155-test_native_reclaimpy-physical-page-reclaim-and-the-compaction-threshold-guards) - [156. test_native_repack.py: REPACK on a columnar table](#156-test_native_repackpy-repack-on-a-columnar-table) +- [157. test_native_join_vector_agg.py: the vectorized aggregate over a unique-key inner join](#157-test_native_join_vector_aggpy-the-vectorized-aggregate-over-a-unique-key-inner-join) ## 1. How to read a test in here @@ -7560,3 +7561,13 @@ The alternative must reclaim, not only return. With `vacuum()` returning before | test | what it holds | | --- | --- | | `test_repack_on_a_columnar_table` | every REPACK spelling is refused naming REPACK, the concurrent form is refused for a stated reason, the table is unharmed, and `pgcolumnar.vacuum()` reclaims the deleted rows | + +## 157. test_native_join_vector_agg.py: the vectorized aggregate over a unique-key inner join + +The pytest twin of `test/native_join_vector_agg.sh` (ported for #1341), every check by the shell's name. A star-schema inner Hash Join onto a unique dimension only filters the fact table, so the ungrouped vectorized aggregate can survive it and still answer exactly (#752). A dimension with duplicate keys, a LEFT join, and a Join Filter besides the hash clause must each refuse the fold, and every answer must equal the heap mirror's. Plans are read with parallel workers off and nested-loop and merge joins disabled, as in the shell. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_join_fold` | the fold takes a unique-key inner join and answers as core does and as heap does; duplicate keys, a LEFT join and an extra join filter each refuse it and answer as heap does | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 67e3ae5ec..d46e6a139 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -775,4 +775,6 @@ guard_tests 438 # three tests, by collection over the cluster files on this tree. # cluster_tests 690 -> 691 for #1341's test_native_repack.py, the twin of native_repack.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 691 +# cluster_tests 691 -> 692 for #1341's test_native_join_vector_agg.py, the twin of native_join_vector_agg.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 692 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 8821bd0b6..aafa2a7d8 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -130,6 +130,7 @@ "native_fetch_coalesce", "native_groupagg", "native_index_fetch_stripe_cost", + "native_join_vector_agg", "native_ownership", "native_parquet_dict_oob", "native_projection", diff --git a/test/pytest/test_native_join_vector_agg.py b/test/pytest/test_native_join_vector_agg.py new file mode 100644 index 000000000..46c10e1ca --- /dev/null +++ b/test/pytest/test_native_join_vector_agg.py @@ -0,0 +1,113 @@ +"""The ungrouped vectorized aggregate survives an inner join onto a UNIQUE dimension, +and refuses every join that is not a plain filter of the fact table (#752, #1341). The +pytest twin of `test/native_join_vector_agg.sh`, every check by the shell's name. + +A star-schema inner Hash Join onto a unique dimension only filters the fact table, so +the fold can drain the dimension's keys and still answer exactly. A dimension with +duplicate keys would multiply fact rows, a LEFT join keeps unmatched ones, and a Join +Filter besides the hash clause is not a membership test: each must refuse the fold, +and every answer must equal the heap mirror's. Plans are read with parallel workers +off and nested-loop and merge joins disabled, so the join is a hash join, as in the +shell. +""" +GUC = "pgcolumnar.enable_ungrouped_vector_agg" +MARKER = "Columnar Vectorized Aggregates" + + +def _exec(conn, *sqls): + with conn.cursor() as cur: + for sql in sqls: + cur.execute(sql) + + +def _row(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return "|".join(str(v) for v in cur.fetchone()) + + +def _with(conn, guc, sql): + """-> the plan's text for 'EXPLAIN ...', else the first row, under `guc`, hash joins only.""" + _exec(conn, "SET max_parallel_workers_per_gather = 0", "SET enable_nestloop = off", + "SET enable_mergejoin = off", f"SET {GUC} = {guc}") + try: + with conn.cursor() as cur: + cur.execute(sql) + if sql.startswith("EXPLAIN"): + return "\n".join(r[0] for r in cur.fetchall()) + return "|".join(str(v) for v in cur.fetchone()) + finally: + _exec(conn, "RESET max_parallel_workers_per_gather", "RESET enable_nestloop", + "RESET enable_mergejoin", f"RESET {GUC}") + + +def _folds(conn, sql): + return _with(conn, "on", f"EXPLAIN (COSTS OFF) {sql}").count(MARKER) + + +def test_the_join_fold(pgc_conn, expect): + conn = pgc_conn + _exec(conn, "CREATE TABLE dim_u(k int PRIMARY KEY)", + "INSERT INTO dim_u SELECT g FROM generate_series(1,25) g", + "CREATE TABLE fact_u(k int, m float8) USING pgcolumnar", + "SELECT pgcolumnar.set_options('fact_u', stripe_row_limit => 1000)", + "INSERT INTO fact_u SELECT 1 + (g % 50), (g % 17)::float8 FROM generate_series(1,5000) g", + "CREATE TABLE heap_u (k int, m float8) USING heap", "INSERT INTO heap_u SELECT * FROM fact_u", + "ANALYZE dim_u", "ANALYZE fact_u") + sqlu = "SELECT count(*), coalesce(sum(fact_u.m)::text,'z') FROM fact_u JOIN dim_u ON fact_u.k = dim_u.k" + expect.num(_folds(conn, sqlu), 1, "unique join uses vectorized agg when GUC on") + expect.num(_with(conn, "off", f"EXPLAIN (COSTS OFF) {sqlu}").count(MARKER), 0, "unique join uses core Agg when GUC off") + expect.text(_with(conn, "on", sqlu), _with(conn, "off", sqlu), "unique join fold answer equals GUC off") + expect.text(_with(conn, "on", sqlu), _row(conn, "SELECT count(*), coalesce(sum(heap_u.m)::text,'z') " + "FROM heap_u JOIN dim_u ON heap_u.k = dim_u.k"), + "unique join fold answer equals heap") + + _exec(conn, "CREATE TABLE dim_d(k int)", "INSERT INTO dim_d SELECT g FROM generate_series(1,50) g", + "INSERT INTO dim_d SELECT g FROM generate_series(1,50) g", + "CREATE TABLE fact_d(k int, m float8) USING pgcolumnar", + "INSERT INTO fact_d SELECT 1 + (g % 50), 1::float8 FROM generate_series(1,100) g", + "CREATE TABLE heap_d (k int, m float8) USING heap", "INSERT INTO heap_d SELECT * FROM fact_d", + "ANALYZE dim_d", "ANALYZE fact_d") + sqld = "SELECT count(*), coalesce(sum(fact_d.m)::text,'z') FROM fact_d JOIN dim_d ON fact_d.k = dim_d.k" + expect.num(_folds(conn, sqld), 0, "duplicate dim keys refuse the join fold") + expect.text(_with(conn, "on", sqld), _row(conn, "SELECT count(*), coalesce(sum(heap_d.m)::text,'z') " + "FROM heap_d JOIN dim_d ON heap_d.k = dim_d.k"), + "duplicate dim keys answer equals heap") + + _exec(conn, "CREATE TABLE dim_l(k int PRIMARY KEY)", "INSERT INTO dim_l VALUES (1)", + "CREATE TABLE fact_l(k int, m float8) USING pgcolumnar", "INSERT INTO fact_l VALUES (1, 1.0), (2, 2.0)", + "CREATE TABLE heap_l (k int, m float8) USING heap", "INSERT INTO heap_l VALUES (1, 1.0), (2, 2.0)", + "ANALYZE dim_l", "ANALYZE fact_l") + sqll = "SELECT count(*), coalesce(sum(dim_l.k)::text,'z') FROM fact_l LEFT JOIN dim_l ON fact_l.k = dim_l.k" + expect.num(_folds(conn, sqll), 0, "LEFT join refuses the join fold") + expect.text(_with(conn, "on", sqll), _row(conn, "SELECT count(*), coalesce(sum(dim_l.k)::text,'z') " + "FROM heap_l LEFT JOIN dim_l ON heap_l.k = dim_l.k"), + "LEFT join answer equals heap") + + # a Join Filter besides the hash clause is not a membership test + _exec(conn, "CREATE TABLE dim_xf(k int PRIMARY KEY, t int)", + "INSERT INTO dim_xf SELECT g, 20 FROM generate_series(1,40) g", + "CREATE TABLE fact_xf(k int, v int) USING pgcolumnar", + "INSERT INTO fact_xf SELECT 1 + (g % 40), g % 40 FROM generate_series(1,2000) g", + "CREATE TABLE heap_xf (k int, v int) USING heap", "INSERT INTO heap_xf SELECT * FROM fact_xf", + "ANALYZE dim_xf", "ANALYZE fact_xf") + sqlxf = ("SELECT coalesce(sum(fact_xf.v)::text,'z') FROM fact_xf JOIN dim_xf " + "ON fact_xf.k = dim_xf.k AND fact_xf.v > dim_xf.t") + expect.num(_folds(conn, sqlxf), 0, "extra join filter refuses the join fold") + expect.text(_with(conn, "on", sqlxf), _with(conn, "off", sqlxf), "extra join filter answer equals GUC off") + expect.text(_with(conn, "on", sqlxf), _row(conn, "SELECT coalesce(sum(heap_xf.v)::text,'z') FROM heap_xf " + "JOIN dim_xf ON heap_xf.k = dim_xf.k AND heap_xf.v > dim_xf.t"), + "extra join filter answer equals heap") + + _exec(conn, "CREATE TABLE dim_xn(k int PRIMARY KEY, t int)", + "INSERT INTO dim_xn SELECT g, 7 FROM generate_series(1,12) g", + "CREATE TABLE fact_xn(k int, v int) USING pgcolumnar", + "INSERT INTO fact_xn SELECT 1 + (g % 12), g % 9 FROM generate_series(1,360) g", + "CREATE TABLE heap_xn (k int, v int) USING heap", "INSERT INTO heap_xn SELECT * FROM fact_xn", + "ANALYZE dim_xn", "ANALYZE fact_xn") + sqlxn = ("SELECT coalesce(sum(fact_xn.v)::text,'z') FROM fact_xn JOIN dim_xn " + "ON fact_xn.k = dim_xn.k AND fact_xn.v <> dim_xn.t") + expect.num(_folds(conn, sqlxn), 0, "inequality join filter refuses the join fold") + expect.text(_with(conn, "on", sqlxn), _row(conn, "SELECT coalesce(sum(heap_xn.v)::text,'z') FROM heap_xn " + "JOIN dim_xn ON heap_xn.k = dim_xn.k AND heap_xn.v <> dim_xn.t"), + "inequality join filter answer equals heap") From a803ee65db19c79d115e3a4ba0ba94dbfb42eeef Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:50 +0000 Subject: [PATCH 29/36] test: native_writer gets its pytest twin (#1341) test_native_writer.py carries all 10 checks by the shell's name. The shell needed no change: each of its catalog arms reddens under a mutation of the writer or of the drop cleanup. COMPLETE gains native_writer, TESTS.md section 158, cluster_tests 692 -> 693. Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 11 +++++++ test/pytest/expected_tests.txt | 4 ++- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_writer.py | 47 +++++++++++++++++++++++++++++ 4 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_native_writer.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index baffd3e2d..49820c4e4 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -203,6 +203,7 @@ behaviour, the source of that number is named. - [155. test_native_reclaim.py: physical page reclaim and the compaction threshold guards](#155-test_native_reclaimpy-physical-page-reclaim-and-the-compaction-threshold-guards) - [156. test_native_repack.py: REPACK on a columnar table](#156-test_native_repackpy-repack-on-a-columnar-table) - [157. test_native_join_vector_agg.py: the vectorized aggregate over a unique-key inner join](#157-test_native_join_vector_aggpy-the-vectorized-aggregate-over-a-unique-key-inner-join) +- [158. test_native_writer.py: the native writer by its catalog rows](#158-test_native_writerpy-the-native-writer-by-its-catalog-rows) ## 1. How to read a test in here @@ -7571,3 +7572,13 @@ The pytest twin of `test/native_join_vector_agg.sh` (ported for #1341), every ch | test | what it holds | | --- | --- | | `test_the_join_fold` | the fold takes a unique-key inner join and answers as core does and as heap does; duplicate keys, a LEFT join and an extra join filter each refuse it and answer as heap does | + +## 158. test_native_writer.py: the native writer by its catalog rows + +The pytest twin of `test/native_writer.sh` (ported for #1341), every check by the shell's name. A columnar table writes the native format, judged by the catalog rows it produces. 5000 rows, a quarter of them NULL, under a 1000-row stripe limit give one storage row at format version 2 with 1024-value vectors, five row groups holding 5000 rows, and ten column chunks counting 10000 values. DROP clears the storage, row-group and column-chunk rows for its storage id. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_native_writer_by_its_catalog_rows` | the writer's storage, row-group and column-chunk rows match the fixture, and DROP deletes all three | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index d46e6a139..f422877cc 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -777,4 +777,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 691 -> 692 for #1341's test_native_join_vector_agg.py, the twin of native_join_vector_agg.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 692 +# cluster_tests 692 -> 693 for #1341's test_native_writer.py, the twin of native_writer.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 693 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index aafa2a7d8..a238f0dfa 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -140,6 +140,7 @@ "native_recluster", "native_repack", "native_vacuum_race", + "native_writer", "objstore_allowlist", "objstore_credentials", "objstore_crlf", diff --git a/test/pytest/test_native_writer.py b/test/pytest/test_native_writer.py new file mode 100644 index 000000000..16e1e01f4 --- /dev/null +++ b/test/pytest/test_native_writer.py @@ -0,0 +1,47 @@ +"""A columnar table writes the native format, judged by the catalog rows it produces, +and DROP clears them (#1341). The pytest twin of `test/native_writer.sh`, every check +by the shell's name. + +5000 rows, a quarter of them NULL in v, under a 1000-row stripe limit: one storage row +at format version 2 with 1024-value vectors, five row groups holding 5000 rows, and ten +column chunks (five groups, two columns) counting 10000 values. Dropping the table +deletes the storage, row-group and column-chunk rows for its storage id. Like the shell +it reads no data back; native_roundtrip does that. +""" + + +def _one(cur, sql): + cur.execute(sql) + return cur.fetchone()[0] + + +def test_the_native_writer_by_its_catalog_rows(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE nw (id int, v text) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('nw', stripe_row_limit => 1000)") + cur.execute("INSERT INTO nw SELECT g, CASE WHEN g % 4 = 0 THEN NULL ELSE 'v'||g END " + "FROM generate_series(1, 5000) g") + sid = _one(cur, "SELECT pgcolumnar.get_storage_id('nw')") + of = lambda sql: _one(cur, sql.format(sid=sid)) # noqa: E731 + expect.num(of("SELECT count(*) FROM pgcolumnar.storage WHERE storage_id = {sid}"), 1, + "native storage row written") + expect.num(of("SELECT format_version FROM pgcolumnar.storage WHERE storage_id = {sid}"), 2, + "native format version recorded") + expect.num(of("SELECT vector_length FROM pgcolumnar.storage WHERE storage_id = {sid}"), 1024, + "native vector length recorded") + expect.num(of("SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = {sid}"), 5, + "native row groups (5000 rows / 1000 limit)") + expect.num(int(of("SELECT sum(row_count) FROM pgcolumnar.row_group WHERE storage_id = {sid}")), 5000, + "native row-group row total") + expect.num(of("SELECT count(*) FROM pgcolumnar.column_chunk WHERE storage_id = {sid}"), 10, + "native column chunks (5 groups x 2 cols)") + expect.num(int(of("SELECT sum(value_count) FROM pgcolumnar.column_chunk WHERE storage_id = {sid}")), 10000, + "native column-chunk value total") + + cur.execute("DROP TABLE nw") + expect.num(of("SELECT count(*) FROM pgcolumnar.storage WHERE storage_id = {sid}"), 0, + "drop clears native storage row") + expect.num(of("SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = {sid}"), 0, + "drop clears native row groups") + expect.num(of("SELECT count(*) FROM pgcolumnar.column_chunk WHERE storage_id = {sid}"), 0, + "drop clears native column chunks") From 79eb5ccef666b7e111f382550caf10680640ae1a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:50 +0000 Subject: [PATCH 30/36] test: native_writer seeds the ledger, 7 of 10 arms dated by six mutations (#1341) The suite had no rows. suites_not_covered 208 -> 207; census 3933 rows / 2841 never / 1092 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 10 ++++++++++ test/check_ledger_budget.txt | 17 +++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 26c12b6ec..729144a38 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2695,6 +2695,16 @@ native_repack native_repack the columnar table is unharmed: content hash 19 2026 native_repack native_repack the columnar table is unharmed: row count 19 2026-10-16 columnar_tableam.c: relation_copy_for_cluster returns without raising, so a rewrite copies nothing native_repack native_repack the columnar table is unharmed: storage options 19 never - native_repack native_repack without an identity index it is refused by PostgreSQL, not by us 19 never - +native_writer native_writer drop clears native column chunks 15;16;17;18;19 2026-10-16 columnar_metadata.c: PgColumnarDeleteMetadata leaves the column_chunk rows +native_writer native_writer drop clears native row groups 15;16;17;18;19 2026-10-16 columnar_metadata.c: PgColumnarDeleteMetadata leaves the row_group rows +native_writer native_writer drop clears native storage row 15;16;17;18;19 2026-10-16 columnar_metadata.c: PgColumnarDeleteMetadata leaves the storage row +native_writer native_writer native column chunks (5 groups x 2 cols) 15;16;17;18;19 2026-10-16 columnar_write_state.c: the writer ignores the table's stripe_row_limit +native_writer native_writer native column-chunk value total 15;16;17;18;19 never - +native_writer native_writer native format version recorded 15;16;17;18;19 2026-10-16 columnar.h: COLUMNAR_NATIVE_FORMAT_VERSION 3 +native_writer native_writer native row groups (5000 rows / 1000 limit) 15;16;17;18;19 2026-10-16 columnar_write_state.c: the writer ignores the table's stripe_row_limit +native_writer native_writer native row-group row total 15;16;17;18;19 never - +native_writer native_writer native storage row written 15;16;17;18;19 never - +native_writer native_writer native vector length recorded 15;16;17;18;19 2026-10-16 columnar.h: COLUMNAR_NATIVE_VECTOR_LENGTH 2048 objstore_allowlist objstore_allowlist a multi-entry list matches any entry 15;16;17;18;19 never - objstore_allowlist objstore_allowlist a non-superuser cannot widen the list (42501) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it objstore_allowlist objstore_allowlist a superuser session SET works (admin override) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 9414c2905..6dc1b2286 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 208 +suites_not_covered 207 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3132,4 +3132,17 @@ suites_not_covered 208 # never 2827 + 11 = 2838 # dated 1074 + 11 = 1085 # 2838 + 1085 == 3923 -checks_never_observed_red 2838 +# +# #1341, native_writer seeded, with its twin (OffgridwithJD): 2838 -> 2841, and +# suites_not_covered 208 -> 207. The suite had no rows. It enters with all 10 checks +# from five-major runs (10/10 on each of 15-19). 7 are dated by six mutations: the +# drop leaving each of the column_chunk, row_group and storage rows, the format +# version and the vector length changed, and the writer ignoring the table's +# stripe_row_limit (the row-group and column-chunk counts). Three arrive never: the +# storage row's existence and the row and value totals, which no mutation here moves. +# +# ledger rows 3923 + 10 = 3933 +# never 2838 + 3 = 2841 +# dated 1085 + 7 = 1092 +# 2841 + 1092 == 3933 +checks_never_observed_red 2841 From 98837cb0059f109e4ca217214e1f2152298885f1 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:50 +0000 Subject: [PATCH 31/36] test: native_batch_fold_projection gets its pytest twin, and its reference cannot be the fold (#1341) The arm "the folded aggregate equals the non-fold aggregate" trusted that FOLDOFF turned the fold off. Measured: with FOLDOFF leaving the fold on and the payload list missing v, the shell stayed 4/4 green, both sides reading the same wrong sum. "premise: the reference runs without the batch fold" now reads the reference's EXPLAIN ANALYZE, and "the non-fold aggregate equals the generator's own answer" checks the reference against a query with no table in it. test_native_batch_fold_projection.py carries all 6 checks by the shell's name. COMPLETE gains native_batch_fold_projection, TESTS.md section 159, cluster_tests 693 -> 694. Co-Authored-By: Claude Opus 5.5 --- test/native_batch_fold_projection.sh | 12 ++++ test/pytest/TESTS.md | 11 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + .../test_native_batch_fold_projection.py | 70 +++++++++++++++++++ 5 files changed, 97 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_native_batch_fold_projection.py diff --git a/test/native_batch_fold_projection.sh b/test/native_batch_fold_projection.sh index 9b7ed8d9a..08ef6f771 100755 --- a/test/native_batch_fold_projection.sh +++ b/test/native_batch_fold_projection.sh @@ -56,6 +56,18 @@ check "the ungrouped batch fold actually ran (Batch Fold: yes)" \ "yes" check "ref is a non-empty result (not a vacuous empty compare)" \ "$(case "$ref" in ''|'') echo no ;; *) echo yes ;; esac)" "yes" +# The reference is only a reference if the fold did not compute it too (#1341): a +# FOLDOFF that left the fold on made the arm below compare the fold with itself, and +# with the payload list missing v both sides read the same wrong sum. So the +# reference's plan is read, and the reference is checked against the generator +# itself, with no table in it at all. +ref_plan="$(env PATH="$PGC_BINDIR:$PATH" psql -h "127.0.0.1" -p "$PGC_PORT" -U postgres -d "$PGC_DB" -qtA \ + -c "$FOLDOFF" -c "EXPLAIN (ANALYZE, COSTS OFF, VERBOSE, TIMING OFF) $INNER" 2>&1)" +check "premise: the reference runs without the batch fold" \ + "$(printf '%s' "$ref_plan" | grep -ciE 'Batch Fold: yes')" "0" +oracle="$(q "SELECT count(*), sum(CASE WHEN g % 50 = 0 THEN NULL ELSE ((g % 13) - 6)::float8 END) + FROM generate_series(1, 400000) g WHERE g % 1000 < 400" | tail -1)" +check "the non-fold aggregate equals the generator's own answer" "$ref" "$oracle" check "the folded aggregate equals the non-fold aggregate (payload columns correct)" "$got" "$ref" check "backend alive" "$(q 'SELECT 1')" "1" pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 49820c4e4..69fb035df 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -204,6 +204,7 @@ behaviour, the source of that number is named. - [156. test_native_repack.py: REPACK on a columnar table](#156-test_native_repackpy-repack-on-a-columnar-table) - [157. test_native_join_vector_agg.py: the vectorized aggregate over a unique-key inner join](#157-test_native_join_vector_aggpy-the-vectorized-aggregate-over-a-unique-key-inner-join) - [158. test_native_writer.py: the native writer by its catalog rows](#158-test_native_writerpy-the-native-writer-by-its-catalog-rows) +- [159. test_native_batch_fold_projection.py: the batch fold's column lists on a wide table](#159-test_native_batch_fold_projectionpy-the-batch-folds-column-lists-on-a-wide-table) ## 1. How to read a test in here @@ -7582,3 +7583,13 @@ The pytest twin of `test/native_writer.sh` (ported for #1341), every check by th | test | what it holds | | --- | --- | | `test_the_native_writer_by_its_catalog_rows` | the writer's storage, row-group and column-chunk rows match the fixture, and DROP deletes all three | + +## 159. test_native_batch_fold_projection.py: the batch fold's column lists on a wide table + +The pytest twin of `test/native_batch_fold_projection.sh` (ported for #1341), every check by the shell's name. The ungrouped batch fold steps over compact key and payload column lists, and a short list would silently return a wrong aggregate (#10). So a 47-column table sums its last column under a selective pushed key, and the folded answer must equal the unfolded one. The fold must really have run, which EXPLAIN ANALYZE shows and a plain EXPLAIN's prediction doesn't. The reference must really be unfolded, or the comparison is of the fold with itself. And the reference must equal the generator's own answer, computed with no table at all. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_batch_fold_gathers_the_right_columns_on_a_wide_table` | the fold runs and sums the right column on a wide table, against an unfolded reference that itself matches the generator | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index f422877cc..f8cdb99a4 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -779,4 +779,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 692 -> 693 for #1341's test_native_writer.py, the twin of native_writer.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 693 +# cluster_tests 693 -> 694 for #1341's test_native_batch_fold_projection.py, the twin of native_batch_fold_projection.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 694 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index a238f0dfa..606190dfd 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -120,6 +120,7 @@ "maintenance_crash", "maintenance_rollback", "native_backend_crash", + "native_batch_fold_projection", "native_chunk_length_bound", "native_decode_gate_width", "native_decode_gating", diff --git a/test/pytest/test_native_batch_fold_projection.py b/test/pytest/test_native_batch_fold_projection.py new file mode 100644 index 000000000..89b7fca13 --- /dev/null +++ b/test/pytest/test_native_batch_fold_projection.py @@ -0,0 +1,70 @@ +"""The ungrouped batch fold gathers exactly the columns it needs on a wide table (#10, +#1341). The pytest twin of `test/native_batch_fold_projection.sh`, every check by the +shell's name. + +The fold's per-row loops step over compact key and payload column lists instead of +masking every attribute, and a list that came out short would silently return a wrong +aggregate. So a 47-column table sums its LAST column, under a selective pushed key so +the deferred-payload path runs, and the folded answer must equal the unfolded one. The +fold must really have run (EXPLAIN ANALYZE's "Batch Fold: yes", not the prediction a +plain EXPLAIN prints); the reference must really be unfolded, or the comparison is of +the fold with itself; and the reference must equal the generator's own answer, which +involves no table. +""" +FILL = ", ".join(f"f{i} bigint" for i in range(44)) +FVALS = ", ".join(f"g*{i + 2}" for i in range(44)) +INNER = "SELECT count(*) n, sum(v) sv FROM t WHERE k < 400" +PAR = ("SET max_parallel_workers_per_gather=4", "SET parallel_setup_cost=0", "SET parallel_tuple_cost=0", + "SET min_parallel_table_scan_size=0", "SET min_parallel_index_scan_size=0") +FOLDOFF = PAR + ("SET pgcolumnar.enable_ungrouped_vector_agg=off", "SET pgcolumnar.enable_group_vectorization=off") +FOLDON = PAR + ("SET pgcolumnar.enable_ungrouped_vector_agg=on", "SET pgcolumnar.enable_parallel_vector_agg=on") +RESETS = ("max_parallel_workers_per_gather", "parallel_setup_cost", "parallel_tuple_cost", + "min_parallel_table_scan_size", "min_parallel_index_scan_size", "pgcolumnar.enable_ungrouped_vector_agg", + "pgcolumnar.enable_group_vectorization", "pgcolumnar.enable_parallel_vector_agg") + + +def _under(cur, gucs, sql): + """-> the rows of `sql` under `gucs`, every setting reset afterwards.""" + for g in gucs: + cur.execute(g) + try: + cur.execute(sql) + return cur.fetchall() + finally: + for g in RESETS: + cur.execute(f"RESET {g}") + + +def _answer(rows): + return "|".join(str(v) for v in rows[0]) + + +def _folded(cur, gucs): + plan = "\n".join(r[0] for r in _under(cur, gucs, f"EXPLAIN (ANALYZE, COSTS OFF, VERBOSE, TIMING OFF) {INNER}")) + return sum(1 for line in plan.splitlines() if "batch fold: yes" in line.lower()), plan + + +def test_the_batch_fold_gathers_the_right_columns_on_a_wide_table(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute(f"CREATE TABLE t (id int, k int, {FILL}, v float8) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('t'::regclass, stripe_row_limit => 40000)") + cur.execute(f"INSERT INTO t SELECT g, g % 1000, {FVALS}, " + "CASE WHEN g % 50 = 0 THEN NULL ELSE ((g % 13) - 6)::float8 END FROM generate_series(1, 400000) g") + cur.execute("ANALYZE t") + + ref = _answer(_under(cur, FOLDOFF, INNER)) + got = _answer(_under(cur, FOLDON, INNER)) + print(f"-- ref=[{ref}] got=[{got}]") + # ANALYZE: a plain EXPLAIN prints the prediction, not whether the fold ran (#1149) + folds, plan = _folded(cur, FOLDON) + expect.text("yes" if folds > 0 else f"no ({plan.lower().count('batch fold')} Batch Fold lines)", "yes", + "the ungrouped batch fold actually ran (Batch Fold: yes)") + expect.text("yes" if ref not in ("", "") else "no", "yes", + "ref is a non-empty result (not a vacuous empty compare)") + expect.num(_folded(cur, FOLDOFF)[0], 0, "premise: the reference runs without the batch fold") + cur.execute("SELECT count(*), sum(CASE WHEN g % 50 = 0 THEN NULL ELSE ((g % 13) - 6)::float8 END) " + "FROM generate_series(1, 400000) g WHERE g % 1000 < 400") + expect.text(ref, _answer(cur.fetchall()), "the non-fold aggregate equals the generator's own answer") + expect.text(got, ref, "the folded aggregate equals the non-fold aggregate (payload columns correct)") + cur.execute("SELECT 1") + expect.num(cur.fetchone()[0], 1, "backend alive") From 80af30a5d5a35588c0629ec47fa7ec7931ec2c17 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:50 +0000 Subject: [PATCH 32/36] test: native_batch_fold_projection seeds the ledger, 2 of 6 arms dated (#1341) The suite had no rows. Dated by the payload list omitting the last column and the fold never taken. suites_not_covered 207 -> 206; census 3939 rows / 2845 never / 1094 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 6 ++++++ test/check_ledger_budget.txt | 18 ++++++++++++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 729144a38..4cd94d9be 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2349,6 +2349,12 @@ native_backend_crash native_backend_crash with-reader: the crash reinitialized t native_backend_crash native_backend_crash with-reader: the killed insert left no partial row group 15;16;17;18;19 never - native_backend_crash native_backend_crash with-reader: the killed insert left no visible rows 15;16;17;18;19 never - native_backend_crash native_backend_crash with-reader: the mirror itself is unchanged 15;16;17;18;19 never - +native_batch_fold_projection native_batch_fold_projection backend alive 15;16;17;18;19 never - +native_batch_fold_projection native_batch_fold_projection premise: the reference runs without the batch fold 15;16;17;18;19 never - +native_batch_fold_projection native_batch_fold_projection ref is a non-empty result (not a vacuous empty compare) 15;16;17;18;19 never - +native_batch_fold_projection native_batch_fold_projection the folded aggregate equals the non-fold aggregate (payload columns correct) 15;16;17;18;19 2026-10-16 columnar_vector.c: the batch fold's payload list omits the table's last column +native_batch_fold_projection native_batch_fold_projection the non-fold aggregate equals the generator's own answer 15;16;17;18;19 never - +native_batch_fold_projection native_batch_fold_projection the ungrouped batch fold actually ran (Batch Fold: yes) 15;16;17;18;19 2026-10-16 columnar_vector.c: the ungrouped aggregate never takes the batch fold (row path instead) native_chunk_length_bound native_chunk_length_bound a sequential scan of the same poisoned chunk is refused (XX001) 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound an index fetch of a chunk whose page_length is 2^32 too large is refused (XX001) 15;16;17;18;19 never - native_chunk_length_bound native_chunk_length_bound backend survived the sequential refusal 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 6dc1b2286..29b6d55a6 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 207 +suites_not_covered 206 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3145,4 +3145,18 @@ suites_not_covered 207 # never 2838 + 3 = 2841 # dated 1085 + 7 = 1092 # 2841 + 1092 == 3933 -checks_never_observed_red 2841 +# +# #1341, native_batch_fold_projection seeded, with its twin (OffgridwithJD): 2841 -> +# 2845, and suites_not_covered 207 -> 206. The suite had no rows. It enters with all +# 6 checks from five-major runs (6/6 on each of 15-19). 2 are dated: the payload list +# omitting the last column (the fold-equals-reference arm) and the fold never taken +# (the fold-ran arm). Four arrive never: the non-empty reference, the new premise that +# the reference ran unfolded, the new arm comparing the reference with the generator's +# own answer, which no product mutation here reaches because the reference runs on the +# row path, and the liveness probe. +# +# ledger rows 3933 + 6 = 3939 +# never 2841 + 4 = 2845 +# dated 1092 + 2 = 1094 +# 2845 + 1094 == 3939 +checks_never_observed_red 2845 From 4d88bb3cab71bb6aba30be043ac0566fe7afd0bb Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:50 +0000 Subject: [PATCH 33/36] test: native_encdesc_golden gets its pytest twin, and a decoding read closes the reader side (#1341) The suite pins the descriptor's bytes and then claims the table "reads back". Measured: with the reader taking rawLen and encLen from each other's offsets, the shell stayed 16/16 green. The bytes on disk were right, and count(*) and sum(a) are answered from the catalog and the zone maps without decoding a page. "every row reads back as the generator wrote it (a decoding read)" compares every row's text with the generator's, and reddens. test_native_encdesc_golden.py carries all 17 checks by the shell's name. COMPLETE gains native_encdesc_golden, TESTS.md section 160, cluster_tests 694 -> 695. Co-Authored-By: Claude Opus 5.5 --- test/native_encdesc_golden.sh | 8 +++ test/pytest/TESTS.md | 11 +++++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_encdesc_golden.py | 59 +++++++++++++++++++++++ 5 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_native_encdesc_golden.py diff --git a/test/native_encdesc_golden.sh b/test/native_encdesc_golden.sh index 5d44845d7..ca737561e 100755 --- a/test/native_encdesc_golden.sh +++ b/test/native_encdesc_golden.sh @@ -94,5 +94,13 @@ check "bigint column b rawLen parses as 2400 (300*8, at offset 11)" "$(le32 1 11 # the whole table still reads back correctly under this descriptor check "the table reads back all rows" "$(q 'SELECT count(*) FROM d;')" "300" check "a full-column read is correct" "$(q 'SELECT sum(a) FROM d;')" "45150" +# A read that DECODES (#1341). count(*) and sum(a) are answered from the catalog and +# the zone maps' sums, so both stayed green with the reader parsing rawLen and encLen +# from each other's offsets, a drift the byte arms above cannot see because the bytes +# on disk are right. Every row's text, against the generator's, needs the pages. +check "every row reads back as the generator wrote it (a decoding read)" \ + "$(q "SELECT md5(string_agg(d::text, ',' ORDER BY a)) FROM d;")" \ + "$(q "SELECT md5(string_agg(x::text, ',' ORDER BY g)) FROM (SELECT g, ROW(g, g*1000000::bigint, 'val'||(g%4)) x + FROM generate_series(1,300) g) s;")" pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 69fb035df..dc571efbe 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -205,6 +205,7 @@ behaviour, the source of that number is named. - [157. test_native_join_vector_agg.py: the vectorized aggregate over a unique-key inner join](#157-test_native_join_vector_aggpy-the-vectorized-aggregate-over-a-unique-key-inner-join) - [158. test_native_writer.py: the native writer by its catalog rows](#158-test_native_writerpy-the-native-writer-by-its-catalog-rows) - [159. test_native_batch_fold_projection.py: the batch fold's column lists on a wide table](#159-test_native_batch_fold_projectionpy-the-batch-folds-column-lists-on-a-wide-table) +- [160. test_native_encdesc_golden.py: the encoding descriptor's wire layout](#160-test_native_encdesc_goldenpy-the-encoding-descriptors-wire-layout) ## 1. How to read a test in here @@ -7593,3 +7594,13 @@ The pytest twin of `test/native_batch_fold_projection.sh` (ported for #1341), ev | test | what it holds | | --- | --- | | `test_the_batch_fold_gathers_the_right_columns_on_a_wide_table` | the fold runs and sums the right column on a wide table, against an unfolded reference that itself matches the generator | + +## 160. test_native_encdesc_golden.py: the encoding descriptor's wire layout + +The pytest twin of `test/native_encdesc_golden.sh` (ported for #1341), every check by the shell's name. The encoding descriptor is the writer-to-reader contract for a column chunk's per-vector encoding, and a deterministic 300-row table pins its version 3 bytes at their offsets: version, flags, vectorCount, each entry's valueCount and rawLen, the length, and the trailer. A read that decodes closes it. `count(*)` and `sum(a)` are answered from the catalog and the zone maps, so only every row's text, compared with the generator's, needs the pages. That's what sees a reader that parses rawLen and encLen from each other's offsets. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_descriptor_layout` | the descriptor's bytes sit at their fixed offsets with the expected values, and every row decodes to what the generator wrote | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index f8cdb99a4..4557036f7 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -781,4 +781,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 693 -> 694 for #1341's test_native_batch_fold_projection.py, the twin of native_batch_fold_projection.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 694 +# cluster_tests 694 -> 695 for #1341's test_native_encdesc_golden.py, the twin of native_encdesc_golden.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 695 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 606190dfd..c284d6051 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -128,6 +128,7 @@ "native_delete_visibility_paths", "native_dict_underfill", "native_dml", + "native_encdesc_golden", "native_fetch_coalesce", "native_groupagg", "native_index_fetch_stripe_cost", diff --git a/test/pytest/test_native_encdesc_golden.py b/test/pytest/test_native_encdesc_golden.py new file mode 100644 index 000000000..f0a1a1c47 --- /dev/null +++ b/test/pytest/test_native_encdesc_golden.py @@ -0,0 +1,59 @@ +"""The native encoding descriptor's wire layout is stable (#1130, #1341). The pytest +twin of `test/native_encdesc_golden.sh`, every check by the shell's name. + +The descriptor is the writer-to-reader contract for a column chunk's per-vector +encoding, laid out as (version 3): header version u8 @0, flags u8 @1, vectorCount u32 +@2; then per vector type u8, valueCount u32, rawLen u32, encLen u32 (13 bytes); then a +sharedTableLen u32 trailer. A deterministic 300-row table pins the bytes at those +offsets, not which encoding was chosen. A read that DECODES closes it: count(*) and +sum(a) are answered from the catalog and the zone maps, so both stayed green with the +reader parsing rawLen and encLen from each other's offsets, and only every row's text, +against the generator's, needs the pages. +""" +import psycopg + + +def _one(cur, sql): + cur.execute(sql) + return cur.fetchone()[0] + + +def test_the_descriptor_layout(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE d (a int, b bigint, c text) USING pgcolumnar") + cur.execute("INSERT INTO d SELECT g, g*1000000::bigint, 'val'||(g%4) FROM generate_series(1,300) g") + sid = _one(cur, "SELECT pgcolumnar.get_storage_id('d')") + chunk = f"FROM pgcolumnar.column_chunk WHERE storage_id = {sid} AND column_index = {{col}}" + + def byte(col, off): + return _one(cur, f"SELECT get_byte(encoding_descriptor, {off}) {chunk.format(col=col)}") + + def le32(col, off): + return int(_one(cur, f"SELECT get_byte(d,{off})+get_byte(d,{off}+1)*256+get_byte(d,{off}+2)*65536" + f"+get_byte(d,{off}+3)*16777216 FROM (SELECT encoding_descriptor d " + f"{chunk.format(col=col)}) x")) + + for col in (0, 1, 2): + expect.num(byte(col, 0), 3, f"column {col} descriptor version byte is 3") + # flags: the fixture has no NULL, so every chunk says it stored no bitmap + for col in (0, 1, 2): + expect.num(byte(col, 1), 1, f"column {col} descriptor flags byte says the bitmap was omitted") + expect.num(le32(0, 2), 1, "vectorCount parses as 1 (at offset 2)") + expect.num(_one(cur, f"SELECT octet_length(encoding_descriptor) {chunk.format(col=0)}"), 23, + "descriptor length is header + 1 entry + empty trailer (23)") + expect.num(le32(0, 19), 0, "trailing sharedTableLen is 0 (u32 at offset 6+13=19)") + for col in (0, 1, 2): + expect.num(le32(col, 7), 300, f"column {col} entry valueCount parses as 300 (at offset 7)") + expect.num(le32(0, 11), 1200, "int column a rawLen parses as 1200 (300*4, at offset 11)") + expect.num(le32(1, 11), 2400, "bigint column b rawLen parses as 2400 (300*8, at offset 11)") + + expect.num(_one(cur, "SELECT count(*) FROM d"), 300, "the table reads back all rows") + expect.num(int(_one(cur, "SELECT sum(a) FROM d")), 45150, "a full-column read is correct") + try: + cur.execute("SELECT md5(string_agg(d::text, ',' ORDER BY a)) FROM d") + got = cur.fetchone()[0] + except psycopg.Error as exc: + got = f"" + expect.text(got, _one(cur, "SELECT md5(string_agg(x::text, ',' ORDER BY g)) FROM (SELECT g, " + "ROW(g, g*1000000::bigint, 'val'||(g%4)) x FROM generate_series(1,300) g) s"), + "every row reads back as the generator wrote it (a decoding read)") From 5b031586703466e9b84f46cf6bfc9c2b28546629 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:51 +0000 Subject: [PATCH 34/36] test: native_encdesc_golden seeds the ledger, 9 of 17 arms dated by four codec mutations (#1341) The suite had no rows. suites_not_covered 206 -> 205; census 3956 rows / 2853 never / 1103 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 17 +++++++++++++++++ test/check_ledger_budget.txt | 19 +++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 4cd94d9be..c27d79f35 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2452,6 +2452,23 @@ native_dml native_dml rows after merge match heap 15;16;17;18;19 2026-10-16 colu native_dml native_dml rows after update match heap 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector native_dml native_dml sum(k) after delete 15;16;17;18;19 never - native_dml native_dml sum(k) after update 15;16;17;18;19 2026-10-16 columnar_tableam.c: the index fetch no longer checks the delete vector +native_encdesc_golden native_encdesc_golden a full-column read is correct 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden bigint column b rawLen parses as 2400 (300*8, at offset 11) 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutEntry writes encLen before rawLen +native_encdesc_golden native_encdesc_golden column 0 descriptor flags byte says the bitmap was omitted 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version;columnar_write_state.c: the writer never sets NO_VALIDITY in the descriptor flags +native_encdesc_golden native_encdesc_golden column 0 descriptor version byte is 3 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version +native_encdesc_golden native_encdesc_golden column 0 entry valueCount parses as 300 (at offset 7) 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden column 1 descriptor flags byte says the bitmap was omitted 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version;columnar_write_state.c: the writer never sets NO_VALIDITY in the descriptor flags +native_encdesc_golden native_encdesc_golden column 1 descriptor version byte is 3 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version +native_encdesc_golden native_encdesc_golden column 1 entry valueCount parses as 300 (at offset 7) 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden column 2 descriptor flags byte says the bitmap was omitted 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version;columnar_write_state.c: the writer never sets NO_VALIDITY in the descriptor flags +native_encdesc_golden native_encdesc_golden column 2 descriptor version byte is 3 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version +native_encdesc_golden native_encdesc_golden column 2 entry valueCount parses as 300 (at offset 7) 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden descriptor length is header + 1 entry + empty trailer (23) 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden every row reads back as the generator wrote it (a decoding read) 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutEntry writes encLen before rawLen;columnar_encdesc.h: PgColumnarEncdescPutHeaderFlags writes flags before version;columnar_encdesc.h: PgColumnarEncdescReadEntry reads rawLen and encLen from each other's offsets;columnar_write_state.c: the writer never sets NO_VALIDITY in the descriptor flags +native_encdesc_golden native_encdesc_golden int column a rawLen parses as 1200 (300*4, at offset 11) 15;16;17;18;19 2026-10-16 columnar_encdesc.h: PgColumnarEncdescPutEntry writes encLen before rawLen +native_encdesc_golden native_encdesc_golden the table reads back all rows 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden trailing sharedTableLen is 0 (u32 at offset 6+13=19) 15;16;17;18;19 never - +native_encdesc_golden native_encdesc_golden vectorCount parses as 1 (at offset 2) 15;16;17;18;19 never - native_fetch_coalesce native_fetch_coalesce a wide index fetch does not pin once per column 15;16;17;18;19 2026-10-13 the coalesced fetch merging no adjacent ranges, so each column is read on its own native_fetch_coalesce native_fetch_coalesce premise: a point lookup uses the index 15;16;17;18;19 never - native_fetch_coalesce native_fetch_coalesce premise: fetching every projected column touched a measurable number of buffers 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 29b6d55a6..4680dcf5d 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 206 +suites_not_covered 205 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3159,4 +3159,19 @@ suites_not_covered 206 # never 2841 + 4 = 2845 # dated 1092 + 2 = 1094 # 2845 + 1094 == 3939 -checks_never_observed_red 2845 +# +# #1341, native_encdesc_golden seeded, with its twin (OffgridwithJD): 2845 -> 2853, +# and suites_not_covered 206 -> 205. The suite had no rows. It enters with all 17 +# checks from five-major runs (17/17 on each of 15-19). 9 are dated by four mutations +# of the descriptor codec: rawLen and encLen written in each other's place, read from +# each other's offsets (the new decoding-read arm, which the byte arms cannot see +# because the bytes on disk are right), NO_VALIDITY never set, and the version and +# flags bytes swapped. Eight arrive never: the vectorCount, valueCount, length and +# trailer arms, which no mutation here moves, and the count and sum read-backs, which +# the catalog and the zone maps answer without decoding a page. +# +# ledger rows 3939 + 17 = 3956 +# never 2845 + 8 = 2853 +# dated 1094 + 9 = 1103 +# 2853 + 1103 == 3956 +checks_never_observed_red 2853 From 75ac07edccf2e6315908740dba3ba0ec6bc7d301 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:51 +0000 Subject: [PATCH 35/36] test: native_parquet_schema gets its pytest twin, and bad magic must be refused by the magic check (#1341) "bad magic rejected (empty At output)" passed on any error. Measured: with the PAR1 magic check skipped, the same 33-byte file is still refused, as a corrupt footer, and the shell stayed 35/35 green. The arm now requires the magic check's own message, "is not a Parquet file (bad magic)". test_native_parquet_schema.py carries all 35 checks by the shell's name in two tests; the six that need pyarrow to write their fixture decline through require_module, as the shell's skip them. COMPLETE gains native_parquet_schema, TESTS.md section 161, cluster_tests 695 -> 697. Co-Authored-By: Claude Opus 5.5 --- test/native_parquet_schema.sh | 10 ++- test/pytest/TESTS.md | 12 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_native_parquet_schema.py | 92 +++++++++++++++++++++++ 5 files changed, 116 insertions(+), 3 deletions(-) create mode 100644 test/pytest/test_native_parquet_schema.py diff --git a/test/native_parquet_schema.sh b/test/native_parquet_schema.sh index 6ec9de4d0..ab4ef19ac 100755 --- a/test/native_parquet_schema.sh +++ b/test/native_parquet_schema.sh @@ -81,8 +81,14 @@ notpq="$PGC_WORKDIR/not.parquet" psql_run "SELECT 1;" >/dev/null # noop to keep ordering readable printf 'this is not a parquet file at all' > "$notpq" chmod 644 "$notpq" -err="$(q "SELECT count(*) FROM pgcolumnar.parquet_schema('$notpq');" 2>&1)" -check "bad magic rejected (empty At output)" "$err" "" +# By the magic check, not by whatever fails next (#1341). This arm was "the query +# printed nothing", which any error satisfies: measured, with the magic check gone +# the 33-byte file is still refused, as a corrupt footer, and the arm passed. The +# message is the magic check's own. +err="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \ + -c "SELECT count(*) FROM pgcolumnar.parquet_schema('$notpq');" 2>&1 | grep -m1 -oE 'ERROR:.*')" +check "bad magic is rejected by the magic check" \ + "$(case "$err" in *"is not a Parquet file (bad magic)"*) echo yes ;; *) echo "no (${err:-no error})" ;; esac)" "yes" # WHY NOT `pgc_skip`, which 25 pyarrow suites use (#1215). It is terminal, and # that is right for a suite with nothing else to do. Measured on pg18a, this diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index dc571efbe..407082489 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -206,6 +206,7 @@ behaviour, the source of that number is named. - [158. test_native_writer.py: the native writer by its catalog rows](#158-test_native_writerpy-the-native-writer-by-its-catalog-rows) - [159. test_native_batch_fold_projection.py: the batch fold's column lists on a wide table](#159-test_native_batch_fold_projectionpy-the-batch-folds-column-lists-on-a-wide-table) - [160. test_native_encdesc_golden.py: the encoding descriptor's wire layout](#160-test_native_encdesc_goldenpy-the-encoding-descriptors-wire-layout) +- [161. test_native_parquet_schema.py: parquet_schema round trip](#161-test_native_parquet_schemapy-parquet_schema-round-trip) ## 1. How to read a test in here @@ -7604,3 +7605,14 @@ The pytest twin of `test/native_encdesc_golden.sh` (ported for #1341), every che | test | what it holds | | --- | --- | | `test_the_descriptor_layout` | the descriptor's bytes sit at their fixed offsets with the expected values, and every row decodes to what the generator wrote | + +## 161. test_native_parquet_schema.py: parquet_schema round trip + +The pytest twin of `test/native_parquet_schema.sh` (ported for #1341), every check by the shell's name. `pgcolumnar.parquet_schema(path)` reports a Parquet file's leaf columns with the PostgreSQL type each maps to. A columnar table covering every exported type is exported, and each source type must come back, every column nullable. A file that isn't Parquet must be refused by the magic check itself: any error was the old arm, and with the magic check gone the same file is still refused, as a corrupt footer. pyarrow supplies what the exporter can't: a REQUIRED column, and field ids out of order, with absence reported as NULL. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_exported_schema_round_trips` | every exported type and its nullability come back from `parquet_schema`, and bad magic is refused by the magic check | +| `test_required_columns_and_field_ids` | a pyarrow REQUIRED column reports nullable=f, and field ids report as written or as NULL when absent | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 4557036f7..91cee63bc 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -783,4 +783,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 694 -> 695 for #1341's test_native_encdesc_golden.py, the twin of native_encdesc_golden.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 695 +# cluster_tests 695 -> 697 for #1341's test_native_parquet_schema.py, the twin of native_parquet_schema.sh: +# two tests, by collection over the cluster files on this tree. +cluster_tests 697 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index c284d6051..9c14db8aa 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -135,6 +135,7 @@ "native_join_vector_agg", "native_ownership", "native_parquet_dict_oob", + "native_parquet_schema", "native_projection", "native_reclaim", "native_reclaim_cycles", diff --git a/test/pytest/test_native_parquet_schema.py b/test/pytest/test_native_parquet_schema.py new file mode 100644 index 000000000..3e6d60ec3 --- /dev/null +++ b/test/pytest/test_native_parquet_schema.py @@ -0,0 +1,92 @@ +"""pgcolumnar.parquet_schema(path) reports a Parquet file's leaf columns with the +PostgreSQL type each maps to (Phase G, #388, #1341). The pytest twin of +`test/native_parquet_schema.sh`, every check by the shell's name. + +The round trip: a columnar table covering every type the exporter emits is exported, +and parquet_schema must report each source type back, every column nullable (the +exporter writes OPTIONAL only). A file that is not Parquet is refused by the magic +check itself: "any error" was the old arm, and with the magic check gone the same file +is still refused, as a corrupt footer. pyarrow, the independent writer, supplies what +the exporter cannot: a REQUIRED column (nullable=f), and field ids out of order and +disjoint from positions, with absence reported as NULL, since 0 is a legal id. +""" +import pathlib + +import psycopg + + +def _one(cur, sql, params=None): + cur.execute(sql, params) + return cur.fetchone()[0] + + +def _schema(cur, path, col, name): + return str(_one(cur, f"SELECT {col} FROM pgcolumnar.parquet_schema(%s) WHERE column_name = %s", (path, name))) + + +def test_the_exported_schema_round_trips(pgc_conn, server_dir, expect): + # every type the exporter emits, with the type parquet_schema must report back + types = (("c_i2", "smallint"), ("c_i4", "integer"), ("c_i8", "bigint"), ("c_f4", "real"), + ("c_f8", "double precision"), ("c_bool", "boolean"), ("c_text", "text"), ("c_bytea", "bytea"), + ("c_date", "date"), ("c_time", "time without time zone"), ("c_ts", "timestamp without time zone"), + ("c_uuid", "uuid"), ("c_num", "numeric(10,2)")) + path = str(server_dir / "schema.parquet") + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE n (c_i2 int2, c_i4 int4, c_i8 int8, c_f4 float4, c_f8 float8, c_bool bool, " + "c_text text, c_bytea bytea, c_date date, c_time time, c_ts timestamp, c_uuid uuid, " + "c_num numeric(10,2)) USING pgcolumnar") + cur.execute("INSERT INTO n VALUES (2::int2, 3, 4::int8, 1.5::float4, 2.5::float8, true, 'hi', " + "'\\xdeadbeef'::bytea, '2020-01-01'::date, '12:34:56'::time, '2020-01-01 12:34:56'::timestamp, " + "'11111111-1111-1111-1111-111111111111'::uuid, 123.45::numeric(10,2)), " + "(NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)") + expect.num(_one(cur, "SELECT pgcolumnar.export_parquet('n', %s)", (path,)), 2, "export_parquet rows written") + expect.num(_one(cur, "SELECT count(*) FROM pgcolumnar.parquet_schema(%s)", (path,)), 13, + "parquet_schema column count") + for name, want in types: + expect.text(_schema(cur, path, "data_type", name), want, f"type of {name}") + expect.text(_schema(cur, path, "nullable", name), "True", f"nullable of {name}") + + notpq = server_dir / "not.parquet" + notpq.write_text("this is not a parquet file at all") + notpq.chmod(0o644) + try: + cur.execute("SELECT count(*) FROM pgcolumnar.parquet_schema(%s)", (str(notpq),)) + err = f"no error ({cur.fetchone()[0]})" + except psycopg.Error as exc: + err = exc.diag.message_primary or str(exc) + # by the magic check, not by whatever fails next + expect.text("yes" if "is not a Parquet file (bad magic)" in err else f"no ({err})", "yes", + "bad magic is rejected by the magic check") + + +def test_required_columns_and_field_ids(pgc_conn, server_dir, expect): + pa = expect.require_module("pyarrow", name="REQUIRED column reports nullable=f") + if pa is None: + return + pq = expect.require_module("pyarrow.parquet", name="REQUIRED column reports nullable=f") + + req = server_dir / "required.parquet" + schema = pa.schema([pa.field("req_i", pa.int32(), nullable=False), pa.field("opt_s", pa.string(), nullable=True)]) + pq.write_table(pa.table({"req_i": pa.array([1, 2, 3], pa.int32()), + "opt_s": pa.array(["a", None, "c"], pa.string())}, schema=schema), str(req)) + fid = server_dir / "fieldids.parquet" + schema = pa.schema([pa.field("alpha", pa.int32(), metadata={b"PARQUET:field_id": b"7"}), + pa.field("beta", pa.string(), metadata={b"PARQUET:field_id": b"3"}), + pa.field("gamma", pa.float64(), metadata={b"PARQUET:field_id": b"12"})]) + pq.write_table(pa.table({"alpha": pa.array([1], pa.int32()), "beta": pa.array(["x"], pa.string()), + "gamma": pa.array([1.5], pa.float64())}, schema=schema), str(fid)) + for p in (req, fid): + p.chmod(0o644) + with pgc_conn.cursor() as cur: + expect.text(_schema(cur, str(req), "nullable", "req_i"), "False", "REQUIRED column reports nullable=f") + expect.text(_schema(cur, str(req), "nullable", "opt_s"), "True", "OPTIONAL column reports nullable=t") + expect.text(_schema(cur, str(req), "data_type", "req_i"), "integer", "REQUIRED int32 maps to integer") + expect.text("yes" if pathlib.Path(fid).stat().st_size > 0 else "no", "yes", + "premise: pyarrow wrote the field-id file") + # ids out of order and disjoint from positions, so a reader inventing them from position cannot pass + expect.text(_one(cur, "SELECT string_agg(field_id::text, ',' ORDER BY ord) FROM (SELECT field_id, " + "row_number() OVER () AS ord FROM pgcolumnar.parquet_schema(%s)) s", (str(fid),)), + "7,3,12", "field ids report as written, in schema order") + expect.text(_one(cur, "SELECT count(*) FILTER (WHERE field_id IS NULL) || '/' || count(*) " + "FROM pgcolumnar.parquet_schema(%s)", (str(req),)), + "2/2", "a file without field ids reports NULL, not zero") From b20dbb24a2f6fa9600ed0d5ae80d27acba86cc4d Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 16 Oct 2026 05:12:51 +0000 Subject: [PATCH 36/36] test: native_parquet_schema seeds the ledger, 5 of 35 arms dated by five reader mutations (#1341) The suite had no rows. suites_not_covered 205 -> 204; census 3991 rows / 2883 never / 1108 dated. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 35 +++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 20 ++++++++++++++++++-- 2 files changed, 53 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index c27d79f35..8bdda7a4b 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2645,6 +2645,41 @@ native_join_vector_agg native_join_vector_agg unique join fold answer equals GUC native_join_vector_agg native_join_vector_agg unique join fold answer equals heap 15;16;17;18;19 never - native_join_vector_agg native_join_vector_agg unique join uses core Agg when GUC off 15;16;17;18;19 never - native_join_vector_agg native_join_vector_agg unique join uses vectorized agg when GUC on 15;16;17;18;19 2026-09-12 drop JOINREL fold +native_parquet_schema native_parquet_schema OPTIONAL column reports nullable=t 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema REQUIRED column reports nullable=f 15;16;17;18;19 2026-10-16 columnar_parquet_reader.c: parquet_schema reports every column nullable +native_parquet_schema native_parquet_schema REQUIRED int32 maps to integer 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema a file without field ids reports NULL, not zero 15;16;17;18;19 2026-10-16 columnar_parquet_reader.c: parquet_schema reports an absent field id as -1, not NULL +native_parquet_schema native_parquet_schema bad magic is rejected by the magic check 15;16;17;18;19 2026-10-16 columnar_parquet_reader.c: the reader skips the PAR1 magic check +native_parquet_schema native_parquet_schema export_parquet rows written 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema field ids report as written, in schema order 15;16;17;18;19 2026-10-16 columnar_parquet_reader.c: parquet_schema reports the position as the field id +native_parquet_schema native_parquet_schema nullable of c_bool 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_bytea 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_date 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_f4 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_f8 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_i2 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_i4 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_i8 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_num 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_text 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_time 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_ts 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema nullable of c_uuid 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema parquet_schema column count 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema premise: pyarrow wrote the field-id file 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_bool 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_bytea 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_date 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_f4 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_f8 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_i2 15;16;17;18;19 2026-10-16 columnar_parquet_reader.c: INT_8 and INT_16 annotations map to integer +native_parquet_schema native_parquet_schema type of c_i4 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_i8 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_num 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_text 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_time 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_ts 15;16;17;18;19 never - +native_parquet_schema native_parquet_schema type of c_uuid 15;16;17;18;19 never - native_reclaim native_reclaim and every live row is still there, once 15;16;17;18;19 never - native_reclaim native_reclaim compact_rewrite ACCEPTS the boundary threshold 0.0 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard refuses minFrac <= 0.0 native_reclaim native_reclaim compact_rewrite ACCEPTS the boundary threshold 1.0 15;16;17;18;19 2026-10-16 columnar_vacuum.c: compact_rewrite's guard refuses minFrac >= 1.0 diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 4680dcf5d..da5208f7a 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 205 +suites_not_covered 204 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -3174,4 +3174,20 @@ suites_not_covered 205 # never 2845 + 8 = 2853 # dated 1094 + 9 = 1103 # 2853 + 1103 == 3956 -checks_never_observed_red 2853 +# +# #1341, native_parquet_schema seeded, with its twin (OffgridwithJD): 2853 -> 2883, +# and suites_not_covered 205 -> 204. The suite had no rows. It enters with all 35 +# checks from five-major runs (35/35 on each of 15-19). 5 are dated by five mutations +# of the Parquet reader: the PAR1 magic check skipped (the bad-magic arm, which now +# requires the magic check's own message, since the same file is refused as a corrupt +# footer without it), INT_16 mapped to integer, every column reported nullable, the +# position reported as the field id, and an absent field id reported as -1. Thirty +# arrive never: the other twelve type arms and the thirteen nullable arms, which no +# single mapping mutation here moves, the export and column counts, and the pyarrow +# fixture's premise and its OPTIONAL and int32 arms. +# +# ledger rows 3956 + 35 = 3991 +# never 2853 + 30 = 2883 +# dated 1103 + 5 = 1108 +# 2883 + 1108 == 3991 +checks_never_observed_red 2883