From d0ca319fb50b474ac5f143f4edaef2a69eb18c08 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 19:16:33 +0000 Subject: [PATCH 01/41] test: the ledger refuses a mutation split on ';', and the 12 split rows are reworded The mutation column is a ';'-separated set, written back sorted. A --mutation whose text held "; " was stored as two mutations, one a fragment starting with a space, and every later merge re-sorted it. #1427's recovery_minimal_wal text did this to 8 rows, and #1423, #1424 and #1426 each carried them as a change of their own. Four older harness_selftest rows were split the same way. - merge refuses a --mutation holding ';' (rc 2, ledger untouched). - the reader refuses a ';'-piece with whitespace at an end, by line: the writer joins with a bare ';', so such a piece is a fragment, never a mutation. - the 12 rows are reworded without ';', from their original text in history. Only field 6 changes; the census stays 3007 / 2282 / 725. Red first: the two refusal arms got 0 want 2 on the unchanged tool, and the committed-ledger arm was red at line 1783 with the refusal in and the rows not yet reworded. guard_tests 435 -> 438 by collection. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 24 +++++++-------- test/pgc_ledger.py | 16 ++++++++++ test/pytest/TESTS.md | 20 ++++++++++++ test/pytest/expected_tests.txt | 5 ++- test/pytest/test_mutation_ledger.py | 48 +++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+), 13 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 734b4a672..694fb5d6e 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1805,8 +1805,8 @@ harness_selftest 540-an-arm-must-carry-its-measurement a new arm that cannot say harness_selftest 540-an-arm-must-carry-its-measurement a repaired arm must have its row removed, so the list only shrinks 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: a determinate or carrying arm is not swept up 15;16;17;18;19 2026-10-12 part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: a determinate or carrying awk arm is not swept up 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's awk_determinate treats a comparison against > 0 as indeterminate (return 1 made return 0) -harness_selftest 540-an-arm-must-carry-its-measurement control: the carve-outs hold and the innocent arm above the helper is not named 15;16;17;18;19 2026-09-24 else echo 0; fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent); then echo 1;the shell if/then/else spelling removed from the matcher, so `$(if [ C ] -harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a shell if/then/else that discards its operand 15;16;17;18;19 2026-09-24 else echo 0; fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent); then echo 1;the shell if/then/else spelling removed from the matcher, so `$(if [ C ] +harness_selftest 540-an-arm-must-carry-its-measurement control: the carve-outs hold and the innocent arm above the helper is not named 15;16;17;18;19 2026-09-24 the shell if/then/else spelling removed from the matcher, so a one-line `$(if [ C ]... then echo 1... else echo 0... fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent) +harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a shell if/then/else that discards its operand 15;16;17;18;19 2026-09-24 the shell if/then/else spelling removed from the matcher, so a one-line `$(if [ C ]... then echo 1... else echo 0... fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent) harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a threshold arm and a floor arm 15;16;17;18;19 2026-10-12 part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees an awk verdict that discards its operands 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's awk_determinate treats a comparison against > 0 as indeterminate (return 1 made return 0) harness_selftest 540-an-arm-must-carry-its-measurement premise: the comment fixture quotes a whole recorder call, not a fragment 15;16;17;18;19 never - @@ -1852,10 +1852,10 @@ harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary harness_selftest 580-a-hand-rebuild-must-record a PostgreSQL symbol is NOT exempt, so the check still has a subject 15;16;17;18;19 2026-10-12 pgc_symbol_is_toolchain deleted from the real lib.sh;the symbol exemption matches every name, so the check has no subject left and a genuinely mislinked major would resolve clean harness_selftest 580-a-hand-rebuild-must-record a log that is not there also produces a diagnosis 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason -harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it, driven alone with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp harness_selftest 580-a-hand-rebuild-must-record an empty log still produces a diagnosis, because silence is the bug 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all -harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it, driven alone with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record and it distinguishes missing from empty, so the two causes do not merge 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record and it names the log as empty rather than printing a bare banner 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record and it never reaches the pg_config check it would otherwise fail on 15;16;17;18;19 2026-09-25 rebuild.sh's fail-closed assertion is removed, so a missing exemption function leaves the undefined list empty and the check reports success @@ -2609,14 +2609,14 @@ range_pruning range_pruning premise: the range type collates differently from it range_pruning range_pruning so does a containment probe 15;16;17;18;19 never - range_pruning range_pruning the overlap qual is pushed down, where it used to be dropped 15;16;17;18;19 never - range_pruning range_pruning the same for a containment probe far to its right 15;16;17;18;19 never - -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) recovery_minimal_wal recovery_minimal_wal premise: the cluster runs at wal_level=minimal 15;16;17;18;19 never - rewrite_storage_oid rewrite_storage_oid a matview created WITH DATA leaves its storage row pointing at itself 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite repair gate no longer covers CreateTableAsStmt, so a matview created WITH DATA keeps the transient's OID rewrite_storage_oid rewrite_storage_oid a type rewrite leaves the storage row pointing at the live table 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite hook does not retarget storage.relation_oid diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 1d880f395..83c6666dc 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -312,6 +312,16 @@ def read_ledger(path): f"{path}:{n}: major {m!r} is neither a number nor " f"{MAJOR_UNKNOWN!r}, so this row names no version it applies to") muts = set() if f[5] == NONE else {m for m in f[5].split(";") if m} + # A PIECE WITH WHITESPACE AT AN END IS A FRAGMENT, not a mutation. The writer + # joins the set with a bare ";", so it never produces one; a text that held + # "; " does, when it is split on the ";" (12 rows on the shipped ledger, 8 of + # them from one recovery_minimal_wal mutation, re-sorted by every later merge). + for m in sorted(muts): + if m != m.strip(): + raise LedgerError( + f"{path}:{n}: mutation {m!r} has whitespace at an end, so it is a " + f"fragment of one mutation's text split on ';'. Reword that mutation " + f"without ';' and keep the row's other mutations as they are.") key = (f[0], f[1], f[2]) # A REPEATED KEY, REFUSED RATHER THAN OVERWRITTEN. This was # `rows[key] = [...]`, so a duplicated row collapsed silently and the LAST line @@ -530,6 +540,12 @@ def cmd_merge(args): "--target names what a mutation killed, so it needs --mutation to say " "which mutation. On its own it attributes nothing.") + if args.mutation and ";" in args.mutation: + raise LedgerError( + f"--mutation {args.mutation!r} holds ';', which separates mutations in the " + f"ledger's mutation column, so it would be stored as several. Reword it " + f"without ';' (', and' reads the same).") + if args.mutation: failed = sorted({key for _p, seen in runs for key, vs in seen.items() if any(v == "FAIL" for v, _m in vs)}) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 4425ba7e6..ed83fde0e 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2445,6 +2445,26 @@ Last-write-wins records the most recent attack rather than the catalogue the col exists to become. One `--mutation` copied across several logs attributes a deliberate change to failures it had nothing to do with, and is refused. +### `test_a_mutation_named_with_a_semicolon_is_refused_rather_than_split` + +The mutation column is a `;`-separated set, written back sorted, so a `--mutation` +whose text held `; ` was stored as two mutations, one a fragment starting with a space. +Every later merge re-sorted it, and three branches in a row carried the same eight +recovery_minimal_wal rows as a change of their own. `merge` now refuses such a text with +rc 2 and leaves the ledger as it was; the same text with `, and` is recorded whole. + +### `test_a_ledger_row_whose_mutation_holds_a_fragment_is_refused` + +The writer joins the set with a bare `;`, so a piece with whitespace at an end is a +fragment of a split sentence, never a mutation it wrote. The reader refuses the ledger +by line. Two whole mutations separated by a bare `;` still read. + +### `test_the_committed_ledger_names_no_mutation_fragment` + +The committed ledger, read by the same reader. Red on the tree before this change at +line 1783, the first of the 12 rows it rewords: 8 from #1427's recovery_minimal_wal +mutation and 4 older harness_selftest rows. + ### `test_a_mutation_with_two_targets_can_be_recorded_when_both_are_named` A mutation with two GENUINE targets is ordinary: reverting the diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 783fc01c6..8e812d734 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -285,7 +285,10 @@ # tests, the twin of selftest part 640, by collection over the NO_CLUSTER files. # guard_tests 433 -> 435 for #1420's two arms in test_writes_wrote_rows.py (a setting # on a watched connection or cursor is refused), by collection over NO_CLUSTER. -guard_tests 435 +# guard_tests 435 -> 438 for three arms in test_mutation_ledger.py: a mutation text holding +# ';' is refused at merge, a ledger piece with whitespace at an end is refused, and the +# committed ledger holds none, by collection over NO_CLUSTER. +guard_tests 438 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index a06e83064..07650ec7f 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -125,6 +125,54 @@ def test_the_mutation_column_accumulates_rather_than_overwriting(tmp_path, expec "one mutation cannot be attributed across several runs at once") +def test_a_mutation_named_with_a_semicolon_is_refused_rather_than_split(tmp_path, expect): + """The mutation column is a ";"-separated SET, written back sorted. A `--mutation` + whose text held "; " was stored as two "mutations", one a sentence fragment that + starts with a space, and the next merge sorted the fragment first. Measured on the + shipped ledger: 12 rows, 8 of them from one recovery_minimal_wal mutation (#1427), + re-sorted by three later branches until each looked like a change of its own. + """ + ledger = _w(tmp_path, "l.tsv", "") + red = _w(tmp_path, "r.log", RED) + out, rc = _run("merge", "--ledger", ledger, "--date", "2026-10-15", + "--mutation", "the sync not registered; this build then asserts", red) + expect.num(rc, 2, "a mutation whose text holds ';' is refused, not split in two") + expect.num(out.count("separates mutations"), 1, "and the refusal says why") + expect.num(len(_rows(ledger)), 0, "and the ledger is left as it was") + + out, rc = _run("merge", "--ledger", ledger, "--date", "2026-10-15", + "--mutation", "the sync not registered, and this build then asserts", red) + expect.num(rc, 0, f"control: the same text without ';' is recorded: {out.strip()}") + expect.text({r[2]: r[5] for r in _rows(ledger)}["first check"], + "the sync not registered, and this build then asserts", + "and it is recorded whole, as one mutation") + + +def test_a_ledger_row_whose_mutation_holds_a_fragment_is_refused(tmp_path, expect): + """A ";"-piece with whitespace at either end cannot be a mutation the writer + produced: it joins the set with a bare ";". It is the fragment of a split sentence, + so the ledger is refused by line rather than read as two mutations.""" + green = _w(tmp_path, "g.log", GREEN) + split = _w(tmp_path, "split.tsv", + "demo\tpart1\tfirst check\t18\t2026-10-15\tthe sync not registered; this build asserts\n") + out, rc = _run("merge", "--ledger", split, green) + expect.num(rc, 2, "a mutation piece that starts with a space is refused") + expect.num(out.count("split.tsv:1:"), 1, "and the refusal names the line") + + two = _w(tmp_path, "two.tsv", + "demo\tpart1\tfirst check\t18\t2026-10-15\tbloom neutered;SAOP 128 -> 0\n") + out, rc = _run("merge", "--ledger", two, green) + expect.num(rc, 0, f"control: two whole mutations separated by a bare ';' still read: {out.strip()}") + + +def test_the_committed_ledger_names_no_mutation_fragment(tmp_path, expect): + """The committed ledger read by the same reader, so a split row on main is an + integrity failure of the file every gate reads, not a cosmetic one.""" + copy = _w(tmp_path, "ledger.tsv", (REPO / "test" / "check_ledger.tsv").read_text()) + out, rc = _run("merge", "--ledger", copy, _w(tmp_path, "g.log", GREEN)) + expect.num(rc, 0, f"the committed ledger reads with no fragment in it: {out.strip()[:400]}") + + def test_a_mutation_with_two_targets_can_be_recorded_when_both_are_named(tmp_path, expect): """#1014. A mutation with TWO GENUINE TARGETS is ordinary, not exotic. From 05fe29e4d579126e7b067831bd97865557132bfa Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 00:48:54 +0000 Subject: [PATCH 02/41] test: recluster recomputes the column extent, in both twins (#1341 item 75) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/column_extent.sh | 15 +++++++++++++++ test/pytest/test_column_extent.py | 16 ++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/test/column_extent.sh b/test/column_extent.sh index bba291034..066a15d78 100755 --- a/test/column_extent.sh +++ b/test/column_extent.sh @@ -213,6 +213,21 @@ q "CREATE TABLE vr (ts timestamptz, v int) USING pgcolumnar; q "VACUUM vr;" >/dev/null check "after VACUUM retires a fully deleted group the extent is the live rows'" "$(ext vr ts)" "$(truth vr ts)" +# recluster rewrites every group too, and the changelog names it with the three +# above (#1341 item 75): the rows a DELETE removed are not written back, so its +# extent must be the live rows', not wider by rows no longer stored. The deleted +# rows hold the minimum, so a stale extent keeps it. +q "CREATE TABLE rc (ts timestamptz, v int) USING pgcolumnar; + SELECT pgcolumnar.set_options('rc', stripe_row_limit => 1000, chunk_group_row_limit => 1000); + INSERT INTO rc SELECT '2026-01-01'::timestamptz + g * interval '1 min', (g * 7919) % 3001 FROM generate_series(1, 3000) g; + DELETE FROM rc WHERE ts < '2026-01-01'::timestamptz + interval '400 min';" >/dev/null +rc_before="$(ext rc ts)" +reclustered="$(q "SELECT pgcolumnar.recluster('rc', 'ts', 'v');")" +check "recluster rewrites the table's row groups" "$([ "${reclustered:-0}" -ge 1 ] && echo yes || echo "no ($reclustered)")" "yes" +check "after recluster the extent is the live rows'" "$(ext rc ts)" "$(truth rc ts)" +check "and it did move: the deleted rows held the old minimum" \ + "$([ -n "$rc_before" ] && [ "$rc_before" != "$(ext rc ts)" ] && echo yes || echo "no ($rc_before)")" "yes" + # ---- no complete extent where the zone maps cannot give one ---------------------- q "CREATE TABLE ac (ts timestamptz) USING pgcolumnar; SELECT pgcolumnar.set_options('ac', stripe_row_limit => 1000, chunk_group_row_limit => 1000); diff --git a/test/pytest/test_column_extent.py b/test/pytest/test_column_extent.py index add40cc65..f0074520a 100644 --- a/test/pytest/test_column_extent.py +++ b/test/pytest/test_column_extent.py @@ -247,6 +247,22 @@ def test_a_shrink_is_recomputed_never_guessed(pgc_own_db, expect): expect.text(_ext(conn, "vr", "ts"), _truth(conn, "vr", "ts"), "after VACUUM retires a fully deleted group the extent is the live rows'") + # recluster, named by the changelog with the three above (#1341 item 75): the + # deleted rows are not written back, and they held the minimum. + _run(conn, "CREATE TABLE rc (ts timestamptz, v int) USING pgcolumnar") + _small_groups(conn, "rc") + _run(conn, "INSERT INTO rc SELECT '2025-05-05'::timestamptz + g * interval '3 min', (g * 104729) % 2711" + " FROM generate_series(1, 3600) g", + "DELETE FROM rc WHERE ts < '2025-05-05'::timestamptz + interval '1500 min'") + before = _ext(conn, "rc", "ts") + reclustered = _one(conn, "SELECT pgcolumnar.recluster('rc', 'ts', 'v')") + expect.text("yes" if (reclustered or 0) >= 1 else f"no ({reclustered})", "yes", + "recluster rewrites the table's row groups") + after = _ext(conn, "rc", "ts") + expect.text(after, _truth(conn, "rc", "ts"), "after recluster the extent is the live rows'") + expect.text("yes" if before and before != after else f"no ({before})", "yes", + "and it did move: the deleted rows held the old minimum") + def test_no_complete_extent_where_the_zone_maps_cannot_give_one(pgc_own_db, expect): conn = pgc_own_db From 639fc193b96d885c0603bf36d71d2c11e6dbd96f Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 00:51:14 +0000 Subject: [PATCH 03/41] test: the recluster extent arms enter the ledger, dated by the retirement-mark mutation (#1341 item 75) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 11 +++++++---- test/check_ledger_budget.txt | 14 +++++++++++++- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 694fb5d6e..54b8eaed2 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -393,11 +393,13 @@ column_extent column_extent a transaction's own flushed groups are inside the ex column_extent column_extent a type with no ordering gets no extent 15;16;17;18;19 2026-10-13 a recompute that skips values without an ordering;pgcolumnar.column_extent returns a row with no complete extent column_extent column_extent a writer that finds no complete extent recomputes one over every group 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent a writer that rolls back leaves the extent as the data has it 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit;no extent rows written at PRE_PREPARE;the writer's fold keeps the largest minimum -column_extent column_extent after VACUUM retires a fully deleted group the extent is the live rows' 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) -column_extent column_extent after compact_rewrite the extent is the live rows' 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;compact_rewrite returning 0 before rewriting anything;no extent row is ever written (PgColumnarExtentPreCommit returns at once) -column_extent column_extent after expire the extent is recomputed to the data that is left 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after VACUUM retires a fully deleted group the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after compact_rewrite the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;compact_rewrite returning 0 before rewriting anything;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after expire the extent is recomputed to the data that is left 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after recluster the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute column_extent column_extent and a plan made in that transaction estimates as the zone maps do 15;16;17;18;19 2026-10-13 the extent a transaction reads leaves out its own pending fold (the tree before the fix) -column_extent column_extent and it did move: the retired groups held the old minimum 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent and it did move: the deleted rows held the old minimum 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute +column_extent column_extent and it did move: the retired groups held the old minimum 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent and the estimate is the one the zone maps give 15;16;17;18;19 2026-10-13 the planner takes the extent's minimum as its upper bound too;the planner's zone-map fallback removed: with no complete extent it reads nothing and bounds nothing column_extent column_extent and the extent covers what they loaded 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit;no extent rows written at PRE_PREPARE;the writer's fold keeps the largest minimum column_extent column_extent and the extent still covers all fifty 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit @@ -414,6 +416,7 @@ column_extent column_extent planning a bucketed GROUP BY scans zone_map not at a column_extent column_extent premise: the extent function exists 15;16;17;18;19 never - column_extent column_extent premise: the transaction's plans gave two estimates 15;16;17;18;19 never - column_extent column_extent premise: the typed fixture spans several row groups 15;16;17;18;19 never - +column_extent column_extent recluster rewrites the table's row groups 15;16;17;18;19 never - column_extent column_extent the extent catalog is not readable by an ordinary role (it holds column values) 15;16;17;18;19 2026-10-13 the column_extent catalog granted to PUBLIC column_extent column_extent the extent of ty.d is the data's min and max, in the type's order 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent the extent of ty.i is the data's min and max, in the type's order 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index cea363447..e8678ba9d 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2533,4 +2533,16 @@ suites_not_covered 230 # never 2344 + 1 = 2345 # dated 767 + 3 = 770 # 2345 + 770 == 3115 -checks_never_observed_red 2345 +# +# #1341 item 75, the changelog names recluster with expire, compact_rewrite and +# VACUUM as recomputing the column extent, and only the other three had arms: +# 2345 -> 2346. column_extent gains 3 checks from five-major runs (42/42 on each +# of 15-19). Two are dated on PG17 by the existing mutation "PgColumnarRetireGroup +# without marking the storage for a recompute", which also re-dates the four +# arms it already held. "recluster rewrites the table's row groups" arrives never. +# +# ledger rows 3115 + 3 = 3118 +# never 2345 + 1 = 2346 +# dated 770 + 2 = 772 +# 2346 + 772 == 3118 +checks_never_observed_red 2346 From 14f6aeb94c6e29ee04673f74e5cb809b3fa81975 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 16:05:52 +0000 Subject: [PATCH 04/41] test: tier_partition and partition_create_ahead run on the caller's own privileges, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/partition_lifecycle.sh | 31 ++++++++++++++ test/pytest/TESTS.md | 2 + test/pytest/expected_tests.txt | 4 +- test/pytest/test_partition_lifecycle.py | 38 +++++++++++++++++ test/pytest/test_tier_partition.py | 54 +++++++++++++++++++++++++ test/tier_partition.sh | 44 ++++++++++++++++++++ 6 files changed, 172 insertions(+), 1 deletion(-) diff --git a/test/partition_lifecycle.sh b/test/partition_lifecycle.sh index 9c649e886..5eb52be5f 100755 --- a/test/partition_lifecycle.sh +++ b/test/partition_lifecycle.sh @@ -241,6 +241,37 @@ check "a parent with a DEFAULT partition is refused" \ check "and nothing was created for it" \ "$(kids td)|$(q "SELECT count(*) FROM pg_proc WHERE proname = 'partition_create_ahead';")" "1|1" +# ---- the caller's own privileges (#1341) ------------------------------------- +# +# partition_create_ahead is not SECURITY DEFINER, so its CREATE TABLE ... PARTITION OF +# runs as the caller, and core refuses a role that does not own the parent. Made +# SECURITY DEFINER, it would run as the extension's owner and let any caller add +# partitions, and the access-exclusive lock that comes with them, to a table it does +# not own. Both roles hold USAGE and CREATE on the schema, so ownership is all that +# separates them. +for r in pl_own pl_peer; do + psql_run "DROP ROLE IF EXISTS $r; CREATE ROLE $r LOGIN NOSUPERUSER; + GRANT USAGE, CREATE ON SCHEMA public TO $r; GRANT USAGE ON SCHEMA pgcolumnar TO $r;" +done +as_role() { # as_role ROLE SQL -> the single value SQL returns as ROLE, or the SQLSTATE it fails with + local out + out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" -d "$PGC_DB" \ + -X -At -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$2" 2>&1) + case "$out" in + *ERROR:*|*FATAL:*) sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' <<<"$out" | head -1 ;; + *) printf '%s\n' "$out" | tail -1 ;; + esac +} +q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own;" >/dev/null +check "premise: pl_own owns the parent, and neither role is a superuser" \ + "$(q "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass;")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ + "pl_own|0" +check "premise: pl_peer can open a session" "$(as_role pl_peer "SELECT 1")" "1" +check "a role that does not own the parent is refused by its own privileges (42501), and creates nothing" \ + "$(as_role pl_peer "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "42501|0" +check "a non-superuser owner creates partitions ahead on its own table" \ + "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "3|3" + # ==== the drop half: partition_drop_plan (#1297) ================================ # # partition_drop_plan(parent regclass, cutoff timestamptz) -> SETOF text diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index ed83fde0e..effcdd57c 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6643,6 +6643,7 @@ Removal proofs, each red in both twins: the prototype's zone handling; `CREATE T | --- | --- | | `test_partition_create_ahead` | from nothing it reaches now plus the lead, columnar, contiguous and one width wide; a second call creates nothing, and asking further ahead creates only the difference; rows land in columnar partitions; the parent is never held ACCESS EXCLUSIVE; west of UTC, for both key types, partitions continue a hand-made one contiguously and a row at every boundary routes; east of UTC, a timestamp parent covers the local now and starts on the width's boundary; inputs that broke the first version (a generated column, a 63-character name, more than 1000 partitions, a MAXVALUE bound) are handled or refused before any DDL; a DEFAULT partition is refused and nothing is created | | `test_partition_drop_plan` | the drop half: `partition_drop_plan` lists `DETACH ... CONCURRENTLY` and `DROP` for the partitions wholly before the cutoff, in range order, from partitions created newest first. A cutoff on an upper bound includes the partition, and one before them all plans nothing. Running the plan, each statement top-level, leaves the rest with their rows and drops the storage rows. A `timestamp` key with a `timestamp` cutoff plans the same in Tokyo and Denver, and a cutoff of the other type is refused (`42804`). MINVALUE is listed once past, and MAXVALUE never. Quoted schema and partition names run as returned. A sub-partitioned partition is planned whole and dropped with its children. A tiered partition is planned with `DROP FOREIGN TABLE`, and its Parquet file stays. A partition whose name holds the drop block's dollar-quote tag drops as planned. A detach cancelled in its second phase is planned as `FINALIZE` and dropped. The DEFAULT and non-timestamp parents are refused (`55000`) | +| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates its partitions ahead | ## 94. test_approx_aggregates.py: hll and tdigest over columnar tables, the compatibility statement @@ -6791,6 +6792,7 @@ On a private database, because foreign servers are per database. A local path st | `test_tier_partition_moves_a_partition_exactly_once` | the move returns the rows moved; the parent's answers are the same; the cold range is a foreign partition with the same name and bound, reading the exported file; no pgcolumnar catalog rows are left; pruning keeps the ranges apart; an insert into the tiered range is refused; five refusals move nothing; the parent is held ACCESS EXCLUSIVE until commit, and a rollback leaves the rows local; a write in flight is exported; a fault after each of the five steps leaves the rows exactly once, still local | | `test_tier_partition_refuses_what_it_cannot_move_and_survives_concurrency` | at each isolation level a row committed between the snapshot and the lock survives, and only READ COMMITTED moves; a UNIQUE parent and four kinds of dependent are refused with no file left, and the partition moves once they are gone; grants and comment carry over; a racing insert leaves every row once; lock_wait times a held parent out with nothing attached; the caller's lock_timeout survives; a retry over a failed attempt's file moves the partition | | `test_tier_partition_moves_a_virtual_generated_column` | PostgreSQL 18+: a partition whose parent has a virtual generated column moves and reads back the same; below 18, `UNSUPPORTED_MAJOR`, listed in `expected_unrunnable.txt` | +| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501`, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers its own partition | ## 101. test_prepared_writes.py: a write prepared and finished in one session leaves the backend alive diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 8e812d734..cb3b19a84 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -705,4 +705,6 @@ guard_tests 438 # cluster_tests 586 -> 587 for #1341's test_serializable.py, one test, by collection # cluster_tests 587 -> 588 for #1341's daemon TimeZone test in test_autovacuum_expire.py, # by collection over the cluster files on this tree. -cluster_tests 588 +# cluster_tests 588 -> 590 for #1341's caller-privilege tests in test_tier_partition.py and +# test_partition_lifecycle.py, one each, by collection over the cluster files on this tree. +cluster_tests 590 diff --git a/test/pytest/test_partition_lifecycle.py b/test/pytest/test_partition_lifecycle.py index c7bcad322..814a912c2 100644 --- a/test/pytest/test_partition_lifecycle.py +++ b/test/pytest/test_partition_lifecycle.py @@ -595,3 +595,41 @@ def name(t): for t in ("xnl", "xls", "xr2")), "|".join(f"55000 {name(t)} is not RANGE-partitioned on one column" for t in ("xnl", "xls", "xr2")), "a drop plan for a parent not RANGE-partitioned on one column is refused, LIST and two-column RANGE included") + + +def test_partition_create_ahead_runs_on_the_callers_own_privileges(pgc_conn, pgc_cluster, expect): + """partition_create_ahead is not SECURITY DEFINER, so its CREATE TABLE ... PARTITION + OF runs as the caller and core refuses a role that does not own the parent (#1341). + Made SECURITY DEFINER, it would run as the extension's owner and let any caller add + partitions to a table it does not own. Both roles hold USAGE and CREATE on the + schema, so ownership is all that separates them.""" + conn = pgc_conn + schema = _one(conn, "SELECT current_schema()") + with conn.cursor() as cur: + for r in ("pl_own_py", "pl_peer_py"): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} LOGIN NOSUPERUSER; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f'GRANT USAGE, CREATE ON SCHEMA "{schema}" TO {r}') + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + cur.execute("CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") + cur.execute("ALTER TABLE pv OWNER TO pl_own_py") + + def as_role(role, sql): + try: + with psycopg.connect(f"host=127.0.0.1 port={pgc_cluster.port} user={role} dbname=postgres", + autocommit=True) as c: + c.execute(f'SET search_path TO "{schema}", pgcolumnar, public') + return str(c.execute(sql).fetchone()[0]) + except psycopg.Error as e: + return e.sqlstate or "?" + + ahead_pv = "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)" + owner = _one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass") + supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own_py', 'pl_peer_py') AND rolsuper") + expect.text(f"{owner}|{supers}", "pl_own_py|0", + "premise: pl_own owns the parent, and neither role is a superuser") + expect.text(as_role("pl_peer_py", "SELECT 1"), "1", "premise: pl_peer can open a session") + expect.text(f"{as_role('pl_peer_py', ahead_pv)}|{_kids(conn, 'pv')}", + "42501|0", "a role that does not own the parent is refused by its own privileges (42501), and creates nothing") + expect.text(f"{as_role('pl_own_py', ahead_pv)}|{_kids(conn, 'pv')}", + "3|3", "a non-superuser owner creates partitions ahead on its own table") diff --git a/test/pytest/test_tier_partition.py b/test/pytest/test_tier_partition.py index ad306382a..b994d3480 100644 --- a/test/pytest/test_tier_partition.py +++ b/test/pytest/test_tier_partition.py @@ -530,3 +530,57 @@ def test_tier_partition_moves_a_virtual_generated_column(pgc_own_db, server_dir, kind = _one(conn, "SELECT relkind::text FROM pg_class WHERE oid = 'cv_mar'::regclass") expect.text(f"{moved}|{kind}|{'same' if before and before == _one(conn, digest) else 'differs'}", "71|f|same", "a virtual generated column moves and reads back the same") + + +def test_tier_partition_runs_on_the_callers_own_privileges(pgc_own_db, pgc_cluster, server_dir, expect): + """tier_partition is not SECURITY DEFINER, so a role that does not own the table is + refused by its own privileges (#1341). Made SECURITY DEFINER, it would run as the + extension's owner and let any caller replace a partition it does not own. Both roles + hold every other privilege the call needs, so ownership is all that separates them.""" + conn = pgc_own_db + db = _one(conn, "SELECT current_database()") + with conn.cursor() as cur: + cur.execute("SET timezone = 'UTC'") + cur.execute("CREATE SERVER tq FOREIGN DATA WRAPPER pgcolumnar_parquet") + for r in ("tier_own_py", "tier_peer_py"): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} LOGIN NOSUPERUSER; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f"GRANT pg_write_server_files TO {r}") + cur.execute(f"GRANT USAGE ON FOREIGN SERVER tq TO {r}") + cur.execute(f"GRANT USAGE, CREATE ON SCHEMA public TO {r}") + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + _build(conn, "po") + with conn.cursor() as cur: + for t in ("po", "po_mar", "po_apr"): + cur.execute(f"ALTER TABLE {t} OWNER TO tier_own_py") + cur.execute("GRANT SELECT ON po, po_mar TO tier_peer_py") + before = _digest(conn, "po") + + def as_role(role, sql): + try: + with psycopg.connect(f"host=127.0.0.1 port={pgc_cluster.port} user={role} dbname={db}", + autocommit=True) as c: + c.execute("SET timezone = 'UTC'") + c.execute(sql) + return "ok" + except psycopg.Error as e: + return e.sqlstate or "?" + + owners = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" + " WHERE relname IN ('po', 'po_mar', 'po_apr')") + supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own_py', 'tier_peer_py') AND rolsuper") + expect.text(f"{owners}|{supers}", "tier_own_py|0", + "premise: tier_own owns the parent and its partitions, and neither role is a superuser") + expect.text(as_role("tier_peer_py", "SELECT count(*) FROM po_mar"), "ok", + "premise: tier_peer can open a session and read the partition") + peer_path = server_dir / "po_peer.parquet" + expect.text(as_role("tier_peer_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{peer_path}')"), + "42501", "a role that does not own the table is refused by its own privileges (42501)") + expect.text(f"{_digest(conn, 'po')}|{_mar_shape(conn, 'po')}", f"{before}|r:pgcolumnar:{BOUND}", + "and the refused call leaves the partition local and attached, every row once") + expect.num(_one(conn, "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid" + " WHERE c.relname LIKE 'po\\_%'"), 0, "with no foreign table behind it") + print(f"-- the refused call's export file: {'written' if peer_path.exists() else 'absent'}") + expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{server_dir / 'po_own.parquet'}')") + + f"|{_mar_shape(conn, 'po')}", f"ok|f:-:{BOUND}", + "a non-superuser owner tiers its own partition") diff --git a/test/tier_partition.sh b/test/tier_partition.sh index 03b3540d8..e543a5e3a 100755 --- a/test/tier_partition.sh +++ b/test/tier_partition.sh @@ -399,6 +399,50 @@ else "PostgreSQL $PGC_MAJOR has no virtual generated columns" fi +# ---- the caller's own privileges (#1341) ------------------------------------- +# +# tier_partition is not SECURITY DEFINER: every step runs as the caller, so the +# owner's ALTER TABLE ... DETACH is what stops a role that does not own the table. +# Made SECURITY DEFINER, it would run as the extension's owner and let any caller +# detach, drop and replace a partition it does not own. Two roles, each holding +# every other privilege the call needs (the server-file role for the export, USAGE +# on the server and CREATE on the schema for the foreign table, and SELECT on the +# partition), so ownership is the only thing between them. +for r in tier_own tier_peer; do + psql_run "DROP ROLE IF EXISTS $r; CREATE ROLE $r LOGIN NOSUPERUSER; + GRANT pg_write_server_files TO $r; GRANT USAGE ON FOREIGN SERVER tier_pq TO $r; + GRANT USAGE, CREATE ON SCHEMA public TO $r; GRANT USAGE ON SCHEMA pgcolumnar TO $r;" +done +sqlstate_as() { # sqlstate_as ROLE SQL -> the SQLSTATE SQL fails with as ROLE, or ok + local out + out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" -d "$PGC_DB" \ + -X -At -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$2" 2>&1 >/dev/null) + [ -z "$out" ] && { echo ok; return; } + echo "$out" | sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' | head -1 +} +build po +psql_run "ALTER TABLE po OWNER TO tier_own; ALTER TABLE po_jan OWNER TO tier_own; + ALTER TABLE po_feb OWNER TO tier_own; GRANT SELECT ON po, po_jan TO tier_peer;" +pb="$(digest po)" +check "premise: tier_own owns the parent and its partitions, and neither role is a superuser" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ + "tier_own|0" +check "premise: tier_peer can open a session and read the partition" \ + "$(sqlstate_as tier_peer "SELECT count(*) FROM po_jan")" "ok" +check "a role that does not own the table is refused by its own privileges (42501)" \ + "$(sqlstate_as tier_peer "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_peer.parquet')")" "42501" +check "and the refused call leaves the partition local and attached, every row once" \ + "$(digest po)|$(jan_shape po)" "$pb|r:pgcolumnar:$JAN_BOUND" +check_num "with no foreign table behind it" \ + "$(q "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid WHERE c.relname LIKE 'po\\_%';")" "0" +# Measured, not asserted: the export runs before the DETACH that refuses, so the +# file is written first. A role holding the server-file role and SELECT can write +# that file with export_parquet directly, so this grants nothing new. +echo "-- the refused call's export file: $([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" +check "a non-superuser owner tiers its own partition" \ + "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_own.parquet')")|$(jan_shape po)" \ + "ok|f:-:$JAN_BOUND" + # ---- a failure after each step ---------------------------------------------- for step in export detach drop create attach; do build f From a4b0ac6be4f89106e11baeaaaf32d6c49ef0871e Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 16:15:14 +0000 Subject: [PATCH 05/41] test: the caller-privilege owner controls get tables of their own, and the refused tier writes no file, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/partition_lifecycle.sh | 9 ++++++--- test/pytest/TESTS.md | 4 ++-- test/pytest/test_partition_lifecycle.py | 13 +++++++++---- test/pytest/test_tier_partition.py | 20 ++++++++++++------- test/tier_partition.sh | 26 +++++++++++++++---------- 5 files changed, 46 insertions(+), 26 deletions(-) diff --git a/test/partition_lifecycle.sh b/test/partition_lifecycle.sh index 5eb52be5f..c8c89a1d9 100755 --- a/test/partition_lifecycle.sh +++ b/test/partition_lifecycle.sh @@ -262,15 +262,18 @@ as_role() { # as_role ROLE SQL -> the single value SQL returns as ROLE, or the S *) printf '%s\n' "$out" | tail -1 ;; esac } -q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own;" >/dev/null +# The owner's control has a parent of its own (pw2), so a refusal that let the peer +# through cannot make the control fail for the peer's reason. +q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own; + CREATE TABLE pw2 (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pw2 OWNER TO pl_own;" >/dev/null check "premise: pl_own owns the parent, and neither role is a superuser" \ - "$(q "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass;")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE oid IN ('pv'::regclass, 'pw2'::regclass);")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ "pl_own|0" check "premise: pl_peer can open a session" "$(as_role pl_peer "SELECT 1")" "1" check "a role that does not own the parent is refused by its own privileges (42501), and creates nothing" \ "$(as_role pl_peer "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "42501|0" check "a non-superuser owner creates partitions ahead on its own table" \ - "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "3|3" + "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pw2', '1 day', 2)")|$(kids pw2)" "3|3" # ==== the drop half: partition_drop_plan (#1297) ================================ # diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index effcdd57c..bb1d13ebb 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6643,7 +6643,7 @@ Removal proofs, each red in both twins: the prototype's zone handling; `CREATE T | --- | --- | | `test_partition_create_ahead` | from nothing it reaches now plus the lead, columnar, contiguous and one width wide; a second call creates nothing, and asking further ahead creates only the difference; rows land in columnar partitions; the parent is never held ACCESS EXCLUSIVE; west of UTC, for both key types, partitions continue a hand-made one contiguously and a row at every boundary routes; east of UTC, a timestamp parent covers the local now and starts on the width's boundary; inputs that broke the first version (a generated column, a 63-character name, more than 1000 partitions, a MAXVALUE bound) are handled or refused before any DDL; a DEFAULT partition is refused and nothing is created | | `test_partition_drop_plan` | the drop half: `partition_drop_plan` lists `DETACH ... CONCURRENTLY` and `DROP` for the partitions wholly before the cutoff, in range order, from partitions created newest first. A cutoff on an upper bound includes the partition, and one before them all plans nothing. Running the plan, each statement top-level, leaves the rest with their rows and drops the storage rows. A `timestamp` key with a `timestamp` cutoff plans the same in Tokyo and Denver, and a cutoff of the other type is refused (`42804`). MINVALUE is listed once past, and MAXVALUE never. Quoted schema and partition names run as returned. A sub-partitioned partition is planned whole and dropped with its children. A tiered partition is planned with `DROP FOREIGN TABLE`, and its Parquet file stays. A partition whose name holds the drop block's dollar-quote tag drops as planned. A detach cancelled in its second phase is planned as `FINALIZE` and dropped. The DEFAULT and non-timestamp parents are refused (`55000`) | -| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates its partitions ahead | +| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates partitions ahead on a parent of its own | ## 94. test_approx_aggregates.py: hll and tdigest over columnar tables, the compatibility statement @@ -6792,7 +6792,7 @@ On a private database, because foreign servers are per database. A local path st | `test_tier_partition_moves_a_partition_exactly_once` | the move returns the rows moved; the parent's answers are the same; the cold range is a foreign partition with the same name and bound, reading the exported file; no pgcolumnar catalog rows are left; pruning keeps the ranges apart; an insert into the tiered range is refused; five refusals move nothing; the parent is held ACCESS EXCLUSIVE until commit, and a rollback leaves the rows local; a write in flight is exported; a fault after each of the five steps leaves the rows exactly once, still local | | `test_tier_partition_refuses_what_it_cannot_move_and_survives_concurrency` | at each isolation level a row committed between the snapshot and the lock survives, and only READ COMMITTED moves; a UNIQUE parent and four kinds of dependent are refused with no file left, and the partition moves once they are gone; grants and comment carry over; a racing insert leaves every row once; lock_wait times a held parent out with nothing attached; the caller's lock_timeout survives; a retry over a failed attempt's file moves the partition | | `test_tier_partition_moves_a_virtual_generated_column` | PostgreSQL 18+: a partition whose parent has a virtual generated column moves and reads back the same; below 18, `UNSUPPORTED_MAJOR`, listed in `expected_unrunnable.txt` | -| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501`, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers its own partition | +| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501` before any file is written, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers a partition of its own table | ## 101. test_prepared_writes.py: a write prepared and finished in one session leaves the backend alive diff --git a/test/pytest/test_partition_lifecycle.py b/test/pytest/test_partition_lifecycle.py index 814a912c2..67791e7c0 100644 --- a/test/pytest/test_partition_lifecycle.py +++ b/test/pytest/test_partition_lifecycle.py @@ -611,8 +611,11 @@ def test_partition_create_ahead_runs_on_the_callers_own_privileges(pgc_conn, pgc "EXCEPTION WHEN duplicate_object THEN NULL; END $$") cur.execute(f'GRANT USAGE, CREATE ON SCHEMA "{schema}" TO {r}') cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") - cur.execute("CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") - cur.execute("ALTER TABLE pv OWNER TO pl_own_py") + # The owner's control has a parent of its own (pw2), so a refusal that let the + # peer through cannot make the control fail for the peer's reason. + for t in ("pv", "pw2"): + cur.execute(f"CREATE TABLE {t} (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") + cur.execute(f"ALTER TABLE {t} OWNER TO pl_own_py") def as_role(role, sql): try: @@ -624,12 +627,14 @@ def as_role(role, sql): return e.sqlstate or "?" ahead_pv = "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)" - owner = _one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass") + owner = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" + " WHERE oid IN ('pv'::regclass, 'pw2'::regclass)") supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own_py', 'pl_peer_py') AND rolsuper") expect.text(f"{owner}|{supers}", "pl_own_py|0", "premise: pl_own owns the parent, and neither role is a superuser") expect.text(as_role("pl_peer_py", "SELECT 1"), "1", "premise: pl_peer can open a session") expect.text(f"{as_role('pl_peer_py', ahead_pv)}|{_kids(conn, 'pv')}", "42501|0", "a role that does not own the parent is refused by its own privileges (42501), and creates nothing") - expect.text(f"{as_role('pl_own_py', ahead_pv)}|{_kids(conn, 'pv')}", + ahead_pw2 = "SELECT pgcolumnar.partition_create_ahead('pw2', '1 day', 2)" + expect.text(f"{as_role('pl_own_py', ahead_pw2)}|{_kids(conn, 'pw2')}", "3|3", "a non-superuser owner creates partitions ahead on its own table") diff --git a/test/pytest/test_tier_partition.py b/test/pytest/test_tier_partition.py index b994d3480..929b99a17 100644 --- a/test/pytest/test_tier_partition.py +++ b/test/pytest/test_tier_partition.py @@ -549,10 +549,14 @@ def test_tier_partition_runs_on_the_callers_own_privileges(pgc_own_db, pgc_clust cur.execute(f"GRANT USAGE ON FOREIGN SERVER tq TO {r}") cur.execute(f"GRANT USAGE, CREATE ON SCHEMA public TO {r}") cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") - _build(conn, "po") + # The owner's control has a table of its own (pw), so a refusal that let the peer + # through cannot make the control fail for the peer's reason. + for t in ("po", "pw"): + _build(conn, t) + with conn.cursor() as cur: + for r in (t, f"{t}_mar", f"{t}_apr"): + cur.execute(f"ALTER TABLE {r} OWNER TO tier_own_py") with conn.cursor() as cur: - for t in ("po", "po_mar", "po_apr"): - cur.execute(f"ALTER TABLE {t} OWNER TO tier_own_py") cur.execute("GRANT SELECT ON po, po_mar TO tier_peer_py") before = _digest(conn, "po") @@ -567,7 +571,7 @@ def as_role(role, sql): return e.sqlstate or "?" owners = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" - " WHERE relname IN ('po', 'po_mar', 'po_apr')") + " WHERE relname IN ('po', 'po_mar', 'po_apr', 'pw', 'pw_mar', 'pw_apr')") supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own_py', 'tier_peer_py') AND rolsuper") expect.text(f"{owners}|{supers}", "tier_own_py|0", "premise: tier_own owns the parent and its partitions, and neither role is a superuser") @@ -580,7 +584,9 @@ def as_role(role, sql): "and the refused call leaves the partition local and attached, every row once") expect.num(_one(conn, "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid" " WHERE c.relname LIKE 'po\\_%'"), 0, "with no foreign table behind it") - print(f"-- the refused call's export file: {'written' if peer_path.exists() else 'absent'}") - expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{server_dir / 'po_own.parquet'}')") - + f"|{_mar_shape(conn, 'po')}", f"ok|f:-:{BOUND}", + # The LOCK ... IN SHARE MODE ahead of the export is the first statement to need + # more than SELECT, so the refusal comes before any file is written. + expect.text("written" if peer_path.exists() else "absent", "absent", "and the refused call wrote no export file") + expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('pw_mar', 'tq', '{server_dir / 'pw_own.parquet'}')") + + f"|{_mar_shape(conn, 'pw')}", f"ok|f:-:{BOUND}", "a non-superuser owner tiers its own partition") diff --git a/test/tier_partition.sh b/test/tier_partition.sh index e543a5e3a..186370e20 100755 --- a/test/tier_partition.sh +++ b/test/tier_partition.sh @@ -402,7 +402,8 @@ fi # ---- the caller's own privileges (#1341) ------------------------------------- # # tier_partition is not SECURITY DEFINER: every step runs as the caller, so the -# owner's ALTER TABLE ... DETACH is what stops a role that does not own the table. +# owner-only steps (its LOCK ... IN SHARE MODE first) stop a role that does not own +# the table. # Made SECURITY DEFINER, it would run as the extension's owner and let any caller # detach, drop and replace a partition it does not own. Two roles, each holding # every other privilege the call needs (the server-file role for the export, USAGE @@ -420,12 +421,17 @@ sqlstate_as() { # sqlstate_as ROLE SQL -> the SQLSTATE SQL fails with as ROLE, o [ -z "$out" ] && { echo ok; return; } echo "$out" | sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' | head -1 } -build po -psql_run "ALTER TABLE po OWNER TO tier_own; ALTER TABLE po_jan OWNER TO tier_own; - ALTER TABLE po_feb OWNER TO tier_own; GRANT SELECT ON po, po_jan TO tier_peer;" +# The owner's control has a table of its own (pw), so a refusal that let the peer +# through cannot make the control fail for the peer's reason. +for t in po pw; do + build $t + psql_run "ALTER TABLE $t OWNER TO tier_own; ALTER TABLE ${t}_jan OWNER TO tier_own; + ALTER TABLE ${t}_feb OWNER TO tier_own;" +done +psql_run "GRANT SELECT ON po, po_jan TO tier_peer;" pb="$(digest po)" check "premise: tier_own owns the parent and its partitions, and neither role is a superuser" \ - "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb', 'pw', 'pw_jan', 'pw_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ "tier_own|0" check "premise: tier_peer can open a session and read the partition" \ "$(sqlstate_as tier_peer "SELECT count(*) FROM po_jan")" "ok" @@ -435,12 +441,12 @@ check "and the refused call leaves the partition local and attached, every row o "$(digest po)|$(jan_shape po)" "$pb|r:pgcolumnar:$JAN_BOUND" check_num "with no foreign table behind it" \ "$(q "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid WHERE c.relname LIKE 'po\\_%';")" "0" -# Measured, not asserted: the export runs before the DETACH that refuses, so the -# file is written first. A role holding the server-file role and SELECT can write -# that file with export_parquet directly, so this grants nothing new. -echo "-- the refused call's export file: $([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" +# The first statement to need more than SELECT is the LOCK ... IN SHARE MODE ahead +# of the export, so the refusal comes before any file is written (measured on 15-19). +check "and the refused call wrote no export file" \ + "$([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" "absent" check "a non-superuser owner tiers its own partition" \ - "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_own.parquet')")|$(jan_shape po)" \ + "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('pw_jan', 'tier_pq', '$W/pw_own.parquet')")|$(jan_shape pw)" \ "ok|f:-:$JAN_BOUND" # ---- a failure after each step ---------------------------------------------- From bfa284c960061776b15ec263cae2937e03835cb6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 16:22:33 +0000 Subject: [PATCH 06/41] test: the caller-privilege arms enter the ledger, five dated by SECURITY DEFINER (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 11 +++++++++++ test/check_ledger_budget.txt | 16 +++++++++++++++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 54b8eaed2..098b45e26 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2399,6 +2399,7 @@ partition_lifecycle partition_lifecycle a cutoff on a partition's upper bound in partition_lifecycle partition_lifecycle a drop plan for a parent not RANGE-partitioned on one column is refused, LIST and two-column RANGE included 15;16;17;18;19 2026-10-14 partition_drop_candidates refusing only a parent with no partition key, not LIST or a multi-column RANGE;partition_drop_candidates without its RANGE-on-one-column check partition_lifecycle partition_lifecycle a drop plan for a parent not partitioned on a timestamp is refused 15;16;17;18;19 2026-10-14 a parent partitioned on a non-timestamp key planned instead of refused;partition_drop_candidates without its timestamp-key check partition_lifecycle partition_lifecycle a drop plan for a parent with a DEFAULT partition is refused 15;16;17;18;19 2026-10-13 a parent with a DEFAULT partition planned instead of refused +partition_lifecycle partition_lifecycle a non-superuser owner creates partitions ahead on its own table 15;16;17;18;19 never - partition_lifecycle partition_lifecycle a parent partitioned on a key that is not a timestamp is refused before any DDL 15;16;17;18;19 2026-10-13 partition_create_ahead without its timestamp key check partition_lifecycle partition_lifecycle a parent that is not RANGE-partitioned on one column is refused before any DDL 15;16;17;18;19 2026-10-13 partition_create_ahead without its RANGE-on-one-column check partition_lifecycle partition_lifecycle a parent with a 63-byte name gets partitions whose names keep their dates 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it;partition_create_ahead stopping one width short of now plus the lead;partition_create_ahead without shortening a long parent name;partition_create_ahead's name bound one byte too long @@ -2410,6 +2411,7 @@ partition_lifecycle partition_lifecycle a partition named with the drop block's partition_lifecycle partition_lifecycle a partition that is itself partitioned is planned whole 15;16;17;18;19 2026-10-13 a partition that is itself partitioned left out of the plan;a partition whose upper bound equals the cutoff left out;a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY partition_lifecycle partition_lifecycle a pending detach is finalised, not started again 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;a pending detach planned as a new concurrent detach, not FINALIZE;a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing partition_lifecycle partition_lifecycle a plan for names that need quoting runs as returned 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;names in the plan with their quotes removed;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing +partition_lifecycle partition_lifecycle a role that does not own the parent is refused by its own privileges (42501), and creates nothing 15;16;17;18;19 2026-10-14 partition_create_ahead made SECURITY DEFINER partition_lifecycle partition_lifecycle a second call creates nothing 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it;partition_create_ahead starting from now, not the highest upper bound partition_lifecycle partition_lifecycle a second plan for the same cutoff is empty, and the partitions it kept are still attached 15;16;17;18;19 2026-10-13 the cutoff filter removed: every partition planned;the lower bound read where the upper is meant partition_lifecycle partition_lifecycle a tiered partition is planned with DROP FOREIGN TABLE 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;a plain DROP, not refusing while the partition is still attached;a tiered partition planned with DROP TABLE;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY @@ -2441,6 +2443,8 @@ partition_lifecycle partition_lifecycle premise: a partition is named with the d partition_lifecycle partition_lifecycle premise: a timestamp parent continued from a hand-made partition, in a session west of UTC 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: a timestamp parent was given partitions from a session east of UTC 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: a timestamptz parent continued from a hand-made partition, in a session west of UTC 15;16;17;18;19 never - +partition_lifecycle partition_lifecycle premise: pl_own owns the parent, and neither role is a superuser 15;16;17;18;19 never - +partition_lifecycle partition_lifecycle premise: pl_peer can open a session 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the January partition was tiered to Parquet 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the call inside the transaction created partitions 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the drop fixture has several partitions, each holding rows 15;16;17;18;19 never - @@ -2884,6 +2888,7 @@ tier_partition tier_partition a failure after drop leaves the rows exactly once, tier_partition tier_partition a failure after export leaves the rows exactly once, still local 15;16;17;18;19 never - tier_partition tier_partition a fault step that is not one of tier_partition's is refused 15;16;17;18;19 2026-10-13 the tier_fault_after check hook accepting any value tier_partition tier_partition a grant to a role whose name needs quoting is carried to the foreign partition 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition not carrying the grants;tier_partition returning 0 rows moved +tier_partition tier_partition a non-superuser owner tiers its own partition 15;16;17;18;19 never - tier_partition tier_partition a parent held by a long reader times the move out, as lock_wait says 15;16;17;18;19 2026-10-13 tier_partition without the lock_wait bound tier_partition tier_partition a parent with a UNIQUE index is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without the UNIQUE-parent refusal tier_partition tier_partition a partition a view depends on is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without the dependents check @@ -2896,6 +2901,7 @@ tier_partition tier_partition a partition with a trigger of its own is refused b tier_partition tier_partition a query on the other range scans only the local partition, beside a foreign one 15;16;17;18;19 never - tier_partition tier_partition a query on the tiered range scans only the foreign partition 15;16;17;18;19 never - tier_partition tier_partition a retry over the failed attempt's file moves the partition, exactly once 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition refusing every existing path (the first version's rule);tier_partition returning 0 rows moved +tier_partition tier_partition a role that does not own the table is refused by its own privileges (42501) 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition a rollback after the whole call leaves the rows local, exactly once 15;16;17;18;19 never - tier_partition tier_partition a server of another wrapper is refused 15;16;17;18;19 2026-10-14 tier_partition without its wrong-wrapper branch, so the iceberg wrapper refuses only after the export;tier_partition without the wrapper check tier_partition tier_partition a server that does not exist is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without its missing-server branch, so core refuses only after the export @@ -2913,6 +2919,8 @@ tier_partition tier_partition and no foreign table behind it after detach 15;16; tier_partition tier_partition and no foreign table behind it after drop 15;16;17;18;19 never - tier_partition tier_partition and no foreign table behind it after export 15;16;17;18;19 never - tier_partition tier_partition and the refusals moved nothing 15;16;17;18;19 2026-10-13 tier_partition without the path refusal +tier_partition tier_partition and the refused call leaves the partition local and attached, every row once 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER +tier_partition tier_partition and the refused call wrote no export file 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition and the timed-out move leaves the rows local, with nothing attached 15;16;17;18;19 2026-10-13 tier_partition without the lock_wait bound tier_partition tier_partition at read committed a row committed between the caller's snapshot and the lock survives 15;16;17;18;19 never - tier_partition tier_partition at read committed the move goes ahead and exports that row 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition returning 0 rows moved @@ -2937,6 +2945,8 @@ tier_partition tier_partition premise: the first attempt failed after export and tier_partition tier_partition premise: the move started while the insert was in flight, and waited for it 15;16;17;18;19 never - tier_partition tier_partition premise: the parent holds every row 15;16;17;18;19 never - tier_partition tier_partition premise: the storage id was read before the move 15;16;17;18;19 never - +tier_partition tier_partition premise: tier_own owns the parent and its partitions, and neither role is a superuser 15;16;17;18;19 never - +tier_partition tier_partition premise: tier_peer can open a session and read the partition 15;16;17;18;19 never - tier_partition tier_partition the answers over the parent are the same after the move 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix tier_partition tier_partition the call holds ACCESS EXCLUSIVE on the parent until commit, as the docs say 15;16;17;18;19 never - tier_partition tier_partition the cold partition is now a foreign partition with its own name and bound 15;16;17;18;19 2026-10-13 tier_partition attaching with the upper bound shifted a minute;tier_partition naming the foreign table with a suffix @@ -2947,6 +2957,7 @@ tier_partition tier_partition the local partition leaves no pgcolumnar storage r tier_partition tier_partition the move ran inside the caller's transaction 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved tier_partition tier_partition the other partition is still local after the refusals 15;16;17;18;19 2026-10-13 tier_partition without the path refusal tier_partition tier_partition tier_partition moves the cold partition and returns the rows it moved 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved +tier_partition tier_partition with no foreign table behind it 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition with the dependents gone, the same partition moves 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved;tier_partition without the dependents check timeseries_recipes timeseries_recipes premise: the readings table is columnar 15;16;17;18;19 never - timeseries_recipes timeseries_recipes premise: the reference has 24 hours, 6 of them empty and one all-NULL 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index e8678ba9d..7f5b72915 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2545,4 +2545,18 @@ suites_not_covered 230 # never 2345 + 1 = 2346 # dated 770 + 2 = 772 # 2346 + 772 == 3118 -checks_never_observed_red 2346 +# +# #1341, tier_partition and partition_create_ahead on the caller's own privileges +# (OffgridwithJD): 2346 -> 2352. Eleven new checks, all in already-covered suites. +# Five are dated, by each function made SECURITY DEFINER: tier_partition's refusal, +# the partition left local, no foreign table and no export file (four), and +# partition_create_ahead's refusal (one). The two owner controls run on tables of +# their own, so neither mutation reddens them; they and four premises arrive never. +# Five-major runs: tier_partition 79/79 (80 on 18-19, which run the virtual column) +# and partition_lifecycle 64/64 on 15-19. +# +# ledger rows 3118 + 11 = 3129 +# never 2346 + 6 = 2352 +# dated 772 + 5 = 777 +# 2352 + 777 == 3129 +checks_never_observed_red 2352 From 278615052b5fe87a4c49689835b87230ffaff984 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:13:58 +0000 Subject: [PATCH 07/41] test: rls_direct_storage's owner-export arm measures the export, read back; both its branches echoed "ran" (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/rls_direct_storage.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 355bda4b9..8c301443f 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -166,8 +166,11 @@ check "and still exportable by it" \ owner_rows="$(q 'SELECT count(*) FROM rls_t;')" check "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)" \ "$(q "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1');" 2>&1 | head -1)" "$owner_rows" -check "and the owner can still export it" \ - "$([ -n "$(q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" 2>&1)" ] && echo ran || echo ran)" "ran" +# Measured as the rows the export holds, read back: both branches of the old test +# here echoed "ran", so a refused owner export passed it. +q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" >/dev/null 2>&1 +check_num "and the owner can still export it" \ + "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo absent)" "$owner_rows" # ---- FORCE ROW LEVEL SECURITY reaches the owner too ------------------------- # From cd0f18dcc933d00cd6ebd350956ebdc825eac7c2 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:13:58 +0000 Subject: [PATCH 08/41] test: rls_direct_storage gets its pytest twin, every check by the shell's name (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/pytest/TESTS.md | 11 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_rls_direct_storage.py | 187 +++++++++++++++++++++++++ 4 files changed, 202 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_rls_direct_storage.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index bb1d13ebb..648373c3e 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -170,6 +170,7 @@ behaviour, the source of that number is named. - [122. test_prepared_across_sessions.py: a prepared transaction, seen from other sessions and across a crash](#122-test_prepared_across_sessionspy-a-prepared-transaction-seen-from-other-sessions-and-across-a-crash) - [123. test_maintenance_crash.py: a crash part-way through maintenance loses nothing](#123-test_maintenance_crashpy-a-crash-part-way-through-maintenance-loses-nothing) - [124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table](#124-test_serializablepy-serializable-refuses-a-write-skew-on-a-columnar-table) +- [125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts](#125-test_rls_direct_storagepy-the-direct-storage-paths-refuse-a-caller-a-policy-restricts) ## 1. How to read a test in here @@ -7100,3 +7101,13 @@ The pytest twin of `test/serializable.sh` (#1341), every check by the shell's na | test | what it holds | | --- | --- | | `test_serializable_refuses_a_write_skew` | the catalog SIRead locks, the four write skews refused by SSI with 40001 against heap, and the REPEATABLE READ controls | + +## 125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts + +The pytest twin of `test/rls_direct_storage.sh` (#563, ported for #1341), every check by the shell's name. Row-level security is applied by the rewriter, and these functions read and write storage with no query to rewrite, so each refuses a relation whose policies apply to the caller. The caller HOLDS SELECT, INSERT, EXECUTE and the server-file roles, so the RLS check is the only refuser left. Eight entry points are refused and nothing was written or imported; a role with no privilege is told "permission denied" and not that RLS exists; a table with no policy, the owner without FORCE, and the owner's export (read back, row for row) stay allowed; and under FORCE a non-superuser owner is refused too. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_direct_storage_paths_refuse_a_policy_restricted_caller` | the premises, the eight refusals and their absent effects, the no-privilege caller's ACL error, the not-over-refusing controls, and the FORCE pair | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index cb3b19a84..67d1b5061 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -707,4 +707,6 @@ guard_tests 438 # by collection over the cluster files on this tree. # cluster_tests 588 -> 590 for #1341's caller-privilege tests in test_tier_partition.py and # test_partition_lifecycle.py, one each, by collection over the cluster files on this tree. -cluster_tests 590 +# cluster_tests 590 -> 591 for #1341's test_rls_direct_storage.py, the twin of +# rls_direct_storage.sh: one test, by collection over the cluster files on this tree. +cluster_tests 591 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 36270ee64..a1b5f034b 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -145,6 +145,7 @@ "projections", "recovery_minimal_wal", "rewrite_storage_oid", + "rls_direct_storage", "rollup", "scan_decode_cost", "serializable", diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py new file mode 100644 index 000000000..fd3397e20 --- /dev/null +++ b/test/pytest/test_rls_direct_storage.py @@ -0,0 +1,187 @@ +"""The direct-storage paths refuse a relation with RLS in force (#563): the pytest twin +of `test/rls_direct_storage.sh` (#1341). Every check carries the shell's name. + +Row-level security is applied by the REWRITER to a query's range table entry. These +functions open the relation's storage and read or write it, so no policy is ever +consulted. The fixture that matters is a role that HOLDS SELECT and INSERT and is +restricted by a policy: a role with no privilege is refused by the ACL check first, +and with the RLS guard deleted a no-privilege arm stays green. Only a granted but +restricted caller reaches the RLS check with every other refuser cleared. + +The bar is check_enable_rls(...) == RLS_ENABLED. The owner without FORCE (and a +superuser) is RLS_NONE_ENV and stays allowed; under FORCE a NON-superuser owner is +refused too, which is why an owner-only rule would not close the hole. + +REAL LOGINS, NOT `SET ROLE`, as in the shell twin. Every role holds USAGE on this +test's schema, as the shell's roles hold it on public, so a "permission denied" is +never the schema's. +""" +import os + +import pytest + +RLS, NONE, OWNER = "t_rls_py", "t_rlsnone_py", "t_owner_rls_py" +ROWS = 200 + + +def _login(cluster, role, schema, sql): + """-> (first value or None, error or None) for `sql` on a fresh login as `role`.""" + import psycopg + dsn = f"host=127.0.0.1 port={cluster.port} user={role} dbname=postgres" + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(f'SET search_path TO "{schema}", pgcolumnar, public') + cur = c.execute(sql) + row = cur.fetchone() if cur.description else None + return (row[0] if row else None), None + except psycopg.Error as e: + return None, e + + +def _verdict(cluster, schema, sql, role=RLS): + """-> rls | acl | allowed | other: , as the shell's verdict.""" + _, err = _login(cluster, role, schema, sql) + if err is None: + return "allowed" + msg = err.diag.message_primary or "" + if "row-level security" in msg: + return "rls" + if "permission denied" in msg: + return "acl" + return f"other: {err.sqlstate} {msg}" + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + row = cur.fetchone() + return row[0] if row else None + + +def test_the_direct_storage_paths_refuse_a_policy_restricted_caller(pgc_conn, pgc_cluster, server_dir, expect): + conn = pgc_conn + out = str(server_dir) + schema = _one(conn, "SELECT current_schema()") + with conn.cursor() as cur: + cur.execute("CREATE TABLE rls_t (id int, owner_name text, ssn text) USING pgcolumnar") + cur.execute(f"INSERT INTO rls_t SELECT g, CASE WHEN g = 1 THEN '{RLS}' ELSE 'someone' END," + f" 'ssn-'||g FROM generate_series(1,{ROWS}) g") + cur.execute("CREATE TABLE plain_t (id int, v text) USING pgcolumnar") + cur.execute(f"INSERT INTO plain_t SELECT g, 'v'||g FROM generate_series(1,{ROWS}) g") + for t in ("rls_t", "plain_t"): + cur.execute(f"SELECT pgcolumnar.add_projection('{t}','p1',ARRAY['id'],ARRAY['id'])") + cur.execute(f"SELECT pgcolumnar.rebuild_projections('{t}')") + for r in (RLS, NONE, OWNER): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} NOSUPERUSER LOGIN; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + cur.execute(f'GRANT USAGE ON SCHEMA "{schema}" TO {r}') + cur.execute(f"GRANT pg_read_server_files, pg_write_server_files TO {RLS}") + # SELECT and INSERT DELIBERATELY: this role must clear every bar but RLS. + cur.execute(f"GRANT SELECT, INSERT ON rls_t TO {RLS}") + cur.execute(f"GRANT SELECT, INSERT ON plain_t TO {RLS}") + # #567 revoked EXECUTE on the projection readers from PUBLIC; without these + # grants the projection arms stop at the grant layer and prove nothing here. + for r in (RLS, NONE, OWNER): + cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO {r}") + cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO {RLS}") + cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") + cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") + # Sources exported from rls_t BY THE OWNER, so their shape matches the import + # target: a differently shaped source fails on column counts, not on RLS. + cur.execute(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/src.parquet')") + cur.execute(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/src.arrow')") + cur.execute(f"COPY rls_t TO '{out}/src.txt'") + + def as_rls(sql): + return _login(pgc_cluster, RLS, schema, sql) + + # ---- premises, every one run BY the restricted role + expect.text(str(as_rls("SELECT 1")[0]), "1", "premise: the role can open a session") + expect.text(str(as_rls("SELECT has_table_privilege('rls_t','SELECT')")[0]), "True", + "premise: the role HOLDS select, so the ACL check cannot be what refuses") + got, _ = as_rls("SELECT count(*) FROM rls_t") + expect.num(-1 if got is None else got, 1, "premise: the policy is in force for ordinary SQL") + expect.text(str(as_rls("SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE')")[0]), + "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") + expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, + "premise: the owner sees every row, so the policy hides something real") + expect.text("yes" if all(os.path.getsize(f"{out}/{f}") > 0 for f in ("src.parquet", "src.arrow", "src.txt")) + else "no", "yes", "premise: the source files the import arms need exist") + + # ---- the refusals: on unfixed code each returns "allowed", which is the leak + def v(sql): + return _verdict(pgc_cluster, schema, sql) + + expect.text(v("SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), "rls", + "read_projection refuses a policy-restricted caller") + expect.text(v("SELECT count(*) FROM pgcolumnar.reconstruct_via_projection('rls_t','p1')"), "rls", + "reconstruct_via_projection refuses one too") + expect.text(v(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/leak1.parquet')"), "rls", + "export_parquet refuses one") + expect.text(v(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/leak2.arrow')"), "rls", + "export_arrow refuses one") + expect.text(v(f"SELECT pgcolumnar.parallel_export_parquet('rls_t','{out}/leak3',2)"), "rls", + "parallel_export_parquet refuses one") + expect.text(v(f"SELECT pgcolumnar.import_parquet('rls_t','{out}/src.parquet')"), "rls", + "import_parquet refuses a policy-restricted writer") + expect.text(v(f"SELECT pgcolumnar.import_arrow('rls_t','{out}/src.arrow')"), "rls", + "import_arrow refuses one") + expect.text(v(f"SELECT pgcolumnar.parallel_copy('rls_t','{out}/src.txt',2)"), "rls", + "parallel_copy refuses one") + + # The refusal must not have done the work anyway. A message is not a behaviour. + written = [f for f in ("leak1.parquet", "leak2.arrow", "leak3") if os.path.exists(f"{out}/{f}")] + expect.text(" ".join(written) or "none", "none", "no export file was written for the refused caller") + expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, "and no imported row landed in the restricted table") + + # ---- a caller WITHOUT privilege sees the ACL error, not the RLS one + _, err = _login(pgc_cluster, NONE, schema, "SELECT count(*) FROM rls_t") + expect.num(1 if err is not None and "permission denied" in (err.diag.message_primary or "") else 0, 1, + "premise: the no-privilege role is told permission denied by ordinary SQL") + _, err = _login(pgc_cluster, NONE, schema, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')") + msg = (err.diag.message_primary or "") if err is not None else "" + expect.num(1 if "permission denied" in msg else 0, 1, + "and read_projection tells it the same thing, not that RLS exists") + expect.num(1 if "row-level security" in msg else 0, 0, + "so no caller without privilege learns RLS state from these functions") + + # ---- not over-refusing: a fix that refused everyone would pass the arms above + expect.text(v("SELECT count(*) FROM pgcolumnar.read_projection('plain_t','p1')"), "allowed", + "a table with NO policy is still readable by the same role") + expect.text(v(f"SELECT pgcolumnar.export_parquet('plain_t','{out}/ok1.parquet')"), "allowed", + "and still exportable by it") + # Against the LIVE count: on unfixed code the import arms leak rows into rls_t. + owner_rows = _one(conn, "SELECT count(*) FROM rls_t") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), owner_rows, + "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") + import psycopg + try: + _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") + except psycopg.Error: + pass + exported = -1 # no file: a refused export is a red arm, not a crash + if os.path.exists(f"{out}/owner.parquet"): + exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," + " owner_name text, ssn text)") + expect.num(exported, owner_rows, "and the owner can still export it") + + # ---- FORCE ROW LEVEL SECURITY reaches a NON-superuser owner too + with conn.cursor() as cur: + cur.execute("CREATE TABLE force_t (id int, owner_name text) USING pgcolumnar") + cur.execute("INSERT INTO force_t SELECT g, 'nobody' FROM generate_series(1,50) g") + cur.execute("SELECT pgcolumnar.add_projection('force_t','p1',ARRAY['id'],ARRAY['id'])") + cur.execute("SELECT pgcolumnar.rebuild_projections('force_t')") + cur.execute(f"ALTER TABLE force_t OWNER TO {OWNER}") + cur.execute("ALTER TABLE force_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_none ON force_t FOR SELECT TO {OWNER} USING (owner_name = current_user)") + expect.text(_one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'force_t'::regclass"), + OWNER, "premise: t_owner owns force_t") + got, _ = _login(pgc_cluster, OWNER, schema, "SELECT count(*) FROM pgcolumnar.read_projection('force_t','p1')") + expect.num(-1 if got is None else got, 50, "without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED") + with conn.cursor() as cur: + cur.execute("ALTER TABLE force_t FORCE ROW LEVEL SECURITY") + _, err = _login(pgc_cluster, OWNER, schema, "SELECT count(*) FROM pgcolumnar.read_projection('force_t','p1')") + expect.num(1 if err is not None and "row-level security" in (err.diag.message_primary or "") else 0, 1, + "under FORCE the non-superuser owner IS refused, so owner-only is no substitute") From 4a02d4a89ef2f727037a9a4eddd4f68dbd7a95f0 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:17:49 +0000 Subject: [PATCH 09/41] test: rls_direct_storage makes its import sources before RLS is on, and every owner arm reports a number when refused, in both twins (#1341) A guard widened to refuse the owner stopped the fixture's own export, so a premise reddened for the product code and the pytest twin crashed in setup rather than failing an arm by name. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/pytest/test_rls_direct_storage.py | 16 ++++++++++------ test/rls_direct_storage.sh | 12 +++++++----- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py index fd3397e20..624396f55 100644 --- a/test/pytest/test_rls_direct_storage.py +++ b/test/pytest/test_rls_direct_storage.py @@ -85,14 +85,15 @@ def test_the_direct_storage_paths_refuse_a_policy_restricted_caller(pgc_conn, pg for r in (RLS, NONE, OWNER): cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO {r}") cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO {RLS}") - cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") - cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") - cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") # Sources exported from rls_t BY THE OWNER, so their shape matches the import # target: a differently shaped source fails on column counts, not on RLS. + # BEFORE RLS is enabled, so no version of the guard under test can stop them. cur.execute(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/src.parquet')") cur.execute(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/src.arrow')") cur.execute(f"COPY rls_t TO '{out}/src.txt'") + cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") + cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") def as_rls(sql): return _login(pgc_cluster, RLS, schema, sql) @@ -154,14 +155,17 @@ def v(sql): "and still exportable by it") # Against the LIVE count: on unfixed code the import arms leak rows into rls_t. owner_rows = _one(conn, "SELECT count(*) FROM rls_t") - expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), owner_rows, - "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") import psycopg + try: + owner_read = _one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')") + except psycopg.Error: + owner_read = -1 # refused: a red arm by name, not a crash + expect.num(owner_read, owner_rows, "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") try: _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") except psycopg.Error: pass - exported = -1 # no file: a refused export is a red arm, not a crash + exported = 0 # no file holds no rows: a refused export is a red arm, not a crash if os.path.exists(f"{out}/owner.parquet"): exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," " owner_name text, ssn text)") diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 8c301443f..f8254e659 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -58,16 +58,18 @@ psql_run "GRANT SELECT, INSERT ON plain_t TO t_rls;" # the RLS check, so they would report acl and prove nothing about this fix. psql_run "GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO t_rls;" psql_run "GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO t_rls;" -psql_run "ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY;" -psql_run "CREATE POLICY p_own ON rls_t FOR SELECT TO t_rls USING (owner_name = current_user);" -psql_run "CREATE POLICY p_ins ON rls_t FOR INSERT TO t_rls WITH CHECK (owner_name = current_user);" -# A source file the role may import, produced by the owner from the open table. +# A source file the role may import, produced by the owner from the open table, +# BEFORE row-level security is enabled, so no version of the guard under test can +# stop the fixture from being made. # Exported from rls_t BY THE OWNER, so the source shape matches the import # target. A source taken from a differently shaped table makes the import arms # fail on column counts, which looks like a refusal and is not one. psql_run "SELECT pgcolumnar.export_parquet('rls_t','$OUT/src.parquet');" psql_run "SELECT pgcolumnar.export_arrow('rls_t','$OUT/src.arrow');" psql_run "COPY rls_t TO '$OUT/src.txt';" +psql_run "ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY;" +psql_run "CREATE POLICY p_own ON rls_t FOR SELECT TO t_rls USING (owner_name = current_user);" +psql_run "CREATE POLICY p_ins ON rls_t FOR INSERT TO t_rls WITH CHECK (owner_name = current_user);" as_rls() { # as_rls -> the ERROR line, connfail, or noerror local out @@ -170,7 +172,7 @@ check "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)" \ # here echoed "ran", so a refused owner export passed it. q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" >/dev/null 2>&1 check_num "and the owner can still export it" \ - "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo absent)" "$owner_rows" + "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo 0)" "$owner_rows" # ---- FORCE ROW LEVEL SECURITY reaches the owner too ------------------------- # From 5f1fe16c31b38919836afd38a66a279057f10f9a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:22:00 +0000 Subject: [PATCH 10/41] test: rls_direct_storage seeds the ledger, sixteen arms dated by three mutations of the RLS guard (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 26 ++++++++++++++++++++++++++ test/check_ledger_budget.txt | 20 ++++++++++++++++++-- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 098b45e26..0f2683fd0 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2643,6 +2643,32 @@ rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partitio rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp ATTACH PARTITION lp_3 FOR VALUES FROM ('2026-03-01') TO ('2026-04-01') 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 CONCURRENTLY 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) +rls_direct_storage rls_direct_storage a table with NO policy is still readable by the same role 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage and no imported row landed in the restricted table 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage and read_projection tells it the same thing, not that RLS exists 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL +rls_direct_storage rls_direct_storage and still exportable by it 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage export_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage import_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage import_parquet refuses a policy-restricted writer 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage no export file was written for the refused caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage parallel_copy refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage parallel_export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage premise: t_owner owns force_t 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the no-privilege role is told permission denied by ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the owner sees every row, so the policy hides something real 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the policy is in force for ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the source files the import arms need exist 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage read_projection refuses a policy-restricted caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL +rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function rollup rollup DROP TABLE of a rollup's source is refused while the rollup exists (2BP01, #1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) rollup rollup DROP VIEW of a rollup takes its partials and its catalog rows with it (#1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 7f5b72915..97b3f6cc6 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 230 +suites_not_covered 229 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2559,4 +2559,20 @@ suites_not_covered 230 # never 2346 + 6 = 2352 # dated 772 + 5 = 777 # 2352 + 777 == 3129 -checks_never_observed_red 2352 +# +# #1341, rls_direct_storage seeded with its pytest twin (OffgridwithJD): 2352 -> 2362, +# and suites_not_covered 230 -> 229. The suite had no rows. It enters with all 26 +# checks from five-major runs (26/26 on each of 15-19). Sixteen are dated by three +# mutations of the RLS guard: deleted (the eight refusals, nothing written, nothing +# imported, and FORCE), widened to refuse RLS_NONE_ENV (the owner's read, the owner's +# export and the owner without FORCE), and moved ahead of read_projection's ACL check +# (the two disclosure arms). The owner's export was measured by a test both of whose +# branches echoed "ran"; it now reads the export back, and main's suite survives the +# widened guard on it. The other 10 arrive never: the premises and the no-policy +# controls. +# +# ledger rows 3129 + 26 = 3155 +# never 2352 + 10 = 2362 +# dated 777 + 16 = 793 +# 2362 + 793 == 3155 +checks_never_observed_red 2362 From 553d23882fc506d505bf6b49fa4ccc791f640873 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 18:17:31 +0000 Subject: [PATCH 11/41] test: rls_direct_storage's two scan-dependent premises become checks, in both twins (#1341, #1397 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 4 ++-- test/pytest/test_rls_direct_storage.py | 4 ++-- test/rls_direct_storage.sh | 9 +++++++-- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 0f2683fd0..f518976dd 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2657,8 +2657,6 @@ rls_direct_storage rls_direct_storage parallel_copy refuses one 15;16;17;18;19 2 rls_direct_storage rls_direct_storage parallel_export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage premise: t_owner owns force_t 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the no-privilege role is told permission denied by ordinary SQL 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage premise: the owner sees every row, so the policy hides something real 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage premise: the policy is in force for ordinary SQL 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - @@ -2667,6 +2665,8 @@ rls_direct_storage rls_direct_storage read_projection refuses a policy-restricte rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 never - rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py index 624396f55..4ac2bb936 100644 --- a/test/pytest/test_rls_direct_storage.py +++ b/test/pytest/test_rls_direct_storage.py @@ -103,11 +103,11 @@ def as_rls(sql): expect.text(str(as_rls("SELECT has_table_privilege('rls_t','SELECT')")[0]), "True", "premise: the role HOLDS select, so the ACL check cannot be what refuses") got, _ = as_rls("SELECT count(*) FROM rls_t") - expect.num(-1 if got is None else got, 1, "premise: the policy is in force for ordinary SQL") + expect.num(-1 if got is None else got, 1, "the policy is in force for ordinary SQL") expect.text(str(as_rls("SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE')")[0]), "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, - "premise: the owner sees every row, so the policy hides something real") + "the owner sees every row, so the policy hides something real") expect.text("yes" if all(os.path.getsize(f"{out}/{f}") > 0 for f in ("src.parquet", "src.arrow", "src.txt")) else "no", "yes", "premise: the source files the import arms need exist") diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index f8254e659..3040ac79a 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -92,18 +92,23 @@ verdict() { # verdict -> rls | acl | allowed | other } # ---- premises, every one run BY the restricted role ------------------------- +# +# Two of these read the table through the columnar scan, so a product change can +# redden them, and they are checks, not premises (#1236; #1397 review): the policy's +# row count, which the custom scan dropping its quals turned into every row, and the +# owner's count. check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At -c 'SELECT 1;' 2>&1 | head -1)" "1" check "premise: the role HOLDS select, so the ACL check cannot be what refuses" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c "SELECT has_table_privilege('rls_t','SELECT');" 2>&1 | head -1)" "t" -check_num "premise: the policy is in force for ordinary SQL" \ +check_num "the policy is in force for ordinary SQL" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c 'SELECT count(*) FROM rls_t;' 2>&1 | head -1)" "1" check "premise: the role holds EXECUTE, so the grant layer is not what refuses" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c "SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE');" 2>&1 | head -1)" "t" -check_num "premise: the owner sees every row, so the policy hides something real" \ +check_num "the owner sees every row, so the policy hides something real" \ "$(q 'SELECT count(*) FROM rls_t;')" "$ROWS" check "premise: the source files the import arms need exist" \ "$([ -s "$OUT/src.parquet" ] && [ -s "$OUT/src.arrow" ] && [ -s "$OUT/src.txt" ] && echo yes || echo no)" "yes" From 69002615a2b8ae24f39b063d0c671a555e950270 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 18:19:16 +0000 Subject: [PATCH 12/41] test: the RLS policy check is dated by the custom scan dropping its quals (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 2 +- test/check_ledger_budget.txt | 13 ++++++++++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index f518976dd..eb686434c 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2666,7 +2666,7 @@ rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 97b3f6cc6..f1049fa28 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2575,4 +2575,15 @@ suites_not_covered 229 # never 2352 + 10 = 2362 # dated 777 + 16 = 793 # 2362 + 793 == 3155 -checks_never_observed_red 2362 +# +# #1341, rls_direct_storage's two scan-dependent premises (OffgridwithJD, #1397 review): +# 2362 -> 2361. Both read the table through the columnar scan, so they are checks and +# lose the prefix, renamed with their rows. "the policy is in force for ordinary SQL" +# is dated by the custom scan dropping its quals (the restricted role then saw all 200 +# rows); the owner's count arrives never. Five-major runs: 26/26 on 15-19. +# +# ledger rows 3155 = 3155 +# never 2362 - 1 = 2361 +# dated 793 + 1 = 794 +# 2361 + 794 == 3155 +checks_never_observed_red 2361 From 24e7b44299a7652669039ade04d8310deb7a6d76 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 19:55:03 +0000 Subject: [PATCH 13/41] test: rls_direct_storage's source-files check loses the premise prefix, and the twin fails it by name rather than raising (#1383 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 2 +- test/pytest/test_rls_direct_storage.py | 18 ++++++++++++------ test/rls_direct_storage.sh | 5 +++-- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index eb686434c..a49363113 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2660,13 +2660,13 @@ rls_direct_storage rls_direct_storage premise: the no-privilege role is told per rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage premise: the source files the import arms need exist 15;16;17;18;19 never - rls_direct_storage rls_direct_storage read_projection refuses a policy-restricted caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) +rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 never - rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py index 4ac2bb936..692a0d5f9 100644 --- a/test/pytest/test_rls_direct_storage.py +++ b/test/pytest/test_rls_direct_storage.py @@ -108,8 +108,9 @@ def as_rls(sql): "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, "the owner sees every row, so the policy hides something real") - expect.text("yes" if all(os.path.getsize(f"{out}/{f}") > 0 for f in ("src.parquet", "src.arrow", "src.txt")) - else "no", "yes", "premise: the source files the import arms need exist") + expect.text("yes" if all(os.path.exists(f"{out}/{f}") and os.path.getsize(f"{out}/{f}") > 0 + for f in ("src.parquet", "src.arrow", "src.txt")) + else "no", "yes", "the source files the import arms need exist") # ---- the refusals: on unfixed code each returns "allowed", which is the leak def v(sql): @@ -165,10 +166,15 @@ def v(sql): _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") except psycopg.Error: pass - exported = 0 # no file holds no rows: a refused export is a red arm, not a crash - if os.path.exists(f"{out}/owner.parquet"): - exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," - " owner_name text, ssn text)") + # No file, an empty one, or one that does not read holds no rows: a refused or + # empty export is a red arm by name, not a crash (#1383 review). + exported = 0 + if os.path.exists(f"{out}/owner.parquet") and os.path.getsize(f"{out}/owner.parquet") > 0: + try: + exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," + " owner_name text, ssn text)") + except psycopg.Error: + exported = 0 expect.num(exported, owner_rows, "and the owner can still export it") # ---- FORCE ROW LEVEL SECURITY reaches a NON-superuser owner too diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 3040ac79a..e26f99424 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -96,7 +96,8 @@ verdict() { # verdict -> rls | acl | allowed | other # Two of these read the table through the columnar scan, so a product change can # redden them, and they are checks, not premises (#1236; #1397 review): the policy's # row count, which the custom scan dropping its quals turned into every row, and the -# owner's count. +# owner's count. The source files are a third: the owner's export writes them, so an +# export that writes nothing reddens that check (#1383 review). check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At -c 'SELECT 1;' 2>&1 | head -1)" "1" check "premise: the role HOLDS select, so the ACL check cannot be what refuses" \ @@ -110,7 +111,7 @@ check "premise: the role holds EXECUTE, so the grant layer is not what refuses" -c "SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE');" 2>&1 | head -1)" "t" check_num "the owner sees every row, so the policy hides something real" \ "$(q 'SELECT count(*) FROM rls_t;')" "$ROWS" -check "premise: the source files the import arms need exist" \ +check "the source files the import arms need exist" \ "$([ -s "$OUT/src.parquet" ] && [ -s "$OUT/src.arrow" ] && [ -s "$OUT/src.txt" ] && echo yes || echo no)" "yes" # ---- the refusals ----------------------------------------------------------- From 81a31405464a9e4993c17e6c7fcf988d1b918b01 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 19:56:59 +0000 Subject: [PATCH 14/41] test: the source-files check is dated by an export that writes nothing (#1383 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 4 ++-- test/check_ledger_budget.txt | 12 +++++++++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index a49363113..6ee5f9293 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2647,7 +2647,7 @@ rls_direct_storage rls_direct_storage a table with NO policy is still readable b rls_direct_storage rls_direct_storage and no imported row landed in the restricted table 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage and read_projection tells it the same thing, not that RLS exists 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage and still exportable by it 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 export_parquet writes no file (total = 0);the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage export_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage import_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) @@ -2666,7 +2666,7 @@ rls_direct_storage rls_direct_storage so no caller without privilege learns RLS rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) -rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 2026-10-14 export_parquet writes no file (total = 0) rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index f1049fa28..30fb0236f 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2586,4 +2586,14 @@ suites_not_covered 229 # never 2362 - 1 = 2361 # dated 793 + 1 = 794 # 2361 + 794 == 3155 -checks_never_observed_red 2361 +# +# #1383 review (OffgridwithJD): 2361 -> 2360. "the source files the import arms need exist" +# loses the premise prefix: the owner's export writes those files, and an export that +# writes nothing reddens it in both twins (the twin now fails it by name instead of +# raising), with the owner-export arm beside it. Five-major runs: 26/26 on 15-19. +# +# ledger rows 3155 = 3155 +# never 2361 - 1 = 2360 +# dated 794 + 1 = 795 +# 2360 + 795 == 3155 +checks_never_observed_red 2360 From 9ce70ba05d41221fc02d52cc347048fce5f82c54 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 03:41:45 +0000 Subject: [PATCH 15/41] test: objstore_crlf gets its pytest twin, every check by the shell's name (#1341) Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 11 ++++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_objstore_crlf.py | 99 +++++++++++++++++++++++++++++ 4 files changed, 114 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_objstore_crlf.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 648373c3e..47a28c02d 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -171,6 +171,7 @@ behaviour, the source of that number is named. - [123. test_maintenance_crash.py: a crash part-way through maintenance loses nothing](#123-test_maintenance_crashpy-a-crash-part-way-through-maintenance-loses-nothing) - [124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table](#124-test_serializablepy-serializable-refuses-a-write-skew-on-a-columnar-table) - [125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts](#125-test_rls_direct_storagepy-the-direct-storage-paths-refuse-a-caller-a-policy-restricts) +- [126. test_objstore_crlf.py: a CR or LF in an object-store URL is refused before it is sent](#126-test_objstore_crlfpy-a-cr-or-lf-in-an-object-store-url-is-refused-before-it-is-sent) ## 1. How to read a test in here @@ -7111,3 +7112,13 @@ The pytest twin of `test/rls_direct_storage.sh` (#563, ported for #1341), every | test | what it holds | | --- | --- | | `test_the_direct_storage_paths_refuse_a_policy_restricted_caller` | the premises, the eight refusals and their absent effects, the no-privilege caller's ACL error, the not-over-refusing controls, and the FORCE pair | + +## 126. test_objstore_crlf.py: a CR or LF in an object-store URL is refused before it is sent + +The pytest twin of `test/objstore_crlf.sh` (ported for #1341), every check by the shell's name. A URL path is written into the request line, so a CR/LF in it would split the request. The guard is in the single connect path, so read_parquet and the REST catalog client's request are both refused, naming CR or LF; and a listener of the file's own records that nothing carrying the injected line reached it. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_a_cr_or_lf_in_a_url_is_refused_before_it_is_sent` | read_parquet and a REST `catalog_uri` with a CR/LF in the path are refused naming CR or LF, the listener receives no injected line, and the backend is alive after each | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 67d1b5061..25b7e0a2d 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -709,4 +709,6 @@ guard_tests 438 # test_partition_lifecycle.py, one each, by collection over the cluster files on this tree. # cluster_tests 590 -> 591 for #1341's test_rls_direct_storage.py, the twin of # rls_direct_storage.sh: one test, by collection over the cluster files on this tree. -cluster_tests 591 +# cluster_tests 591 -> 592 for #1341's test_objstore_crlf.py, the twin of objstore_crlf.sh: +# one test, by collection over the cluster files on this tree. +cluster_tests 592 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index a1b5f034b..d468a93aa 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -129,6 +129,7 @@ "native_reclaim_frag", "native_vacuum_race", "objstore_credentials", + "objstore_crlf", "parallel_am_scan", "parallel_scan_cost", "partition_lifecycle", diff --git a/test/pytest/test_objstore_crlf.py b/test/pytest/test_objstore_crlf.py new file mode 100644 index 000000000..fb2f2eeda --- /dev/null +++ b/test/pytest/test_objstore_crlf.py @@ -0,0 +1,99 @@ +"""A CR or LF in an object-store URL is refused before any request is sent: the pytest +twin of `test/objstore_crlf.sh` (#1341). Every check carries the shell's name. + +A URL path is written into the request line, so a CR/LF in it would split the +request. The guard is in the single connect path, so both read_parquet and the REST +catalog client's http_request are covered. Each arm asserts the refusal names CR or LF, +and the read arm also asserts the listener received nothing carrying the injected line. + +The listener is this file's own socket in a thread, as the shell twin's is a script +under `test/`; the two harnesses do not call each other. +""" +import socket +import threading + +import pytest + + +class _Listener: + """Accept at most one connection on 127.0.0.1 and keep whatever bytes it sends.""" + + def __init__(self): + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.sock.bind(("127.0.0.1", 0)) + self.sock.listen(1) + self.sock.settimeout(5) + self.port = self.sock.getsockname()[1] + self.data = b"" + self.thread = threading.Thread(target=self._run, daemon=True) + self.thread.start() + + def _run(self): + try: + conn, _ = self.sock.accept() + except OSError: + return + conn.settimeout(2) + try: + while len(self.data) < 4096: + chunk = conn.recv(1024) + if not chunk: + break + self.data += chunk + except OSError: + pass + try: + conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nhi") + finally: + conn.close() + + def close(self): + self.thread.join(timeout=10) + self.sock.close() + + +def _err(conn, sql): + """-> the primary message of the error `sql` raises, or ''.""" + import psycopg + try: + with conn.cursor() as cur: + cur.execute(sql) + if cur.description: + cur.fetchall() + return "" + except psycopg.Error as e: + return e.diag.message_primary or "" + + +def _alive(conn): + with conn.cursor() as cur: + cur.execute("SELECT 1") + return cur.fetchone()[0] + + +def test_a_cr_or_lf_in_a_url_is_refused_before_it_is_sent(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("SET pgcolumnar.objstore_allowed_endpoints = '127.0.0.1'") + + lst = _Listener() + url = f"http://127.0.0.1:{lst.port}/x\r\nX-Injected: yes" + msg = _err(pgc_conn, f"SELECT * FROM pgcolumnar.read_parquet('{url}') AS t(v int)") + lst.close() + expect.num(1 if ("CR or LF" in msg) else 0, 1, + "a CR/LF in the URL path is rejected before the request is sent") + expect.num(1 if b"X-Injected" in lst.data else 0, 0, + "no injected line reached the listener (no request splitting)") + expect.num(_alive(pgc_conn), 1, "backend alive") + + # The REST catalog client goes through the same connect path. + cat = f"http://127.0.0.1:{lst.port}/x\r\nX-Injected: yes" + with pgc_conn.cursor() as cur: + cur.execute("DROP SERVER IF EXISTS crlf_cat_py CASCADE") + cur.execute(f"CREATE SERVER crlf_cat_py FOREIGN DATA WRAPPER pgcolumnar_iceberg_catalog" + f" OPTIONS (catalog_uri '{cat}')") + cur.execute("CREATE USER MAPPING FOR CURRENT_USER SERVER crlf_cat_py OPTIONS (token 'x')") + msg = _err(pgc_conn, "SELECT pgcolumnar.iceberg_rest_table_location('crlf_cat_py', 'db', 'events')") + expect.num(1 if ("CR or LF" in msg) else 0, 1, + "a CR/LF in a REST catalog_uri is rejected too (objstore_http_request path)") + expect.num(_alive(pgc_conn), 1, "backend alive after the REST attempt") From ab6ae272ba0db0cde251438e2ca3fe646acecc48 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 03:45:48 +0000 Subject: [PATCH 16/41] test: objstore_crlf seeds the ledger, with three arms dated (#1341) Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 5 +++++ test/check_ledger_budget.txt | 15 +++++++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 6ee5f9293..394bab980 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2364,6 +2364,11 @@ objstore_credentials objstore_credentials placement: secret on a SERVER is HV00D objstore_credentials objstore_credentials premise: bob's error was a real refusal, not silence 15;16;17;18;19 never - objstore_credentials objstore_credentials premise: sigv4 fixture server is up 15;16;17;18;19 never - objstore_credentials objstore_credentials premise: without mapping or environment, even a superuser gets 28000 15;16;17;18;19 never - +objstore_crlf objstore_crlf a CR/LF in a REST catalog_uri is rejected too (objstore_http_request path) 15;16;17;18;19 2026-10-14 the request-line guard on the URL path removed +objstore_crlf objstore_crlf a CR/LF in the URL path is rejected before the request is sent 15;16;17;18;19 2026-10-14 the request-line guard on the URL path removed +objstore_crlf objstore_crlf backend alive 15;16;17;18;19 never - +objstore_crlf objstore_crlf backend alive after the REST attempt 15;16;17;18;19 never - +objstore_crlf objstore_crlf no injected line reached the listener (no request splitting) 15;16;17;18;19 2026-10-14 the request-line guard on the URL path removed parallel_am_scan parallel_am_scan a parallel index build indexes every row of the table 15;16;17;18;19 never - parallel_am_scan parallel_am_scan a parallel table-AM scan returns the same row count as serial 15;16;17;18;19 never - parallel_am_scan parallel_am_scan premise: ANALYZE printed a rows= line per launched worker 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 30fb0236f..8d819d75e 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 229 +suites_not_covered 228 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2596,4 +2596,15 @@ suites_not_covered 229 # never 2361 - 1 = 2360 # dated 794 + 1 = 795 # 2360 + 795 == 3155 -checks_never_observed_red 2360 +# +# #1341, objstore_crlf seeded, with its pytest twin (OffgridwithJD): 2360 -> 2362, and +# suites_not_covered 229 -> 228. The suite had no rows. It enters with all 5 checks +# from five-major runs (5/5 on each of 15-19). Three are dated by the request-line +# guard on the URL path removed: neither refusal named CR or LF, and the listener +# received the injected line. The two backend-alive checks arrive never. +# +# ledger rows 3155 + 5 = 3160 +# never 2360 + 2 = 2362 +# dated 795 + 3 = 798 +# 2362 + 798 == 3160 +checks_never_observed_red 2362 From 92d957b62015dcdb0e45ba6bab091d5321e62e3f Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 03:09:11 +0000 Subject: [PATCH 17/41] test: objstore_allowlist gets its pytest twin, and the admin-override arm runs with the configured list empty, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 --- test/objstore_allowlist.sh | 3 + test/pytest/TESTS.md | 13 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_objstore_allowlist.py | 160 +++++++++++++++++++++++++ 5 files changed, 180 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_objstore_allowlist.py diff --git a/test/objstore_allowlist.sh b/test/objstore_allowlist.sh index be046261b..69b7f90a0 100755 --- a/test/objstore_allowlist.sh +++ b/test/objstore_allowlist.sh @@ -96,6 +96,9 @@ check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U allow_nobody -d "$PGC_DB" -qtA -c 'SELECT 1' 2>&1)" "1" check "a non-superuser cannot widen the list (42501)" \ "$(sqlstate_of "SET pgcolumnar.objstore_allowed_endpoints = '10.0.0.1'" allow_nobody)" "42501" +# WITH THE CONFIGURED LIST EMPTY (#1341). It was still "other.example, 127.0.0.1" here, +# so the read worked whatever the session SET did, and the arm could not fail. +set_list "" check_num "a superuser session SET works (admin override)" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -qtA \ -c "SET pgcolumnar.objstore_allowed_endpoints = '127.0.0.1'" -c "$Q" 2>/dev/null | tail -1)" "1000" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 47a28c02d..711ecae07 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -172,6 +172,7 @@ behaviour, the source of that number is named. - [124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table](#124-test_serializablepy-serializable-refuses-a-write-skew-on-a-columnar-table) - [125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts](#125-test_rls_direct_storagepy-the-direct-storage-paths-refuse-a-caller-a-policy-restricts) - [126. test_objstore_crlf.py: a CR or LF in an object-store URL is refused before it is sent](#126-test_objstore_crlfpy-a-cr-or-lf-in-an-object-store-url-is-refused-before-it-is-sent) +- [127. test_objstore_allowlist.py: the endpoint allow-list, empty by default](#127-test_objstore_allowlistpy-the-endpoint-allow-list-empty-by-default) ## 1. How to read a test in here @@ -7122,3 +7123,15 @@ The pytest twin of `test/objstore_crlf.sh` (ported for #1341), every check by th | test | what it holds | | --- | --- | | `test_a_cr_or_lf_in_a_url_is_refused_before_it_is_sent` | read_parquet and a REST `catalog_uri` with a CR/LF in the path are refused naming CR or LF, the listener receives no injected line, and the backend is alive after each | + +## 127. test_objstore_allowlist.py: the endpoint allow-list, empty by default + +The pytest twin of `test/objstore_allowlist.sh` (#393, ported for #1341), every check by the shell's name. An empty `pgcolumnar.objstore_allowed_endpoints` refuses all remote reads with 42501 and names the setting; a local read is untouched. A list admits exactly what it names, host or host:port, any entry of several. Only a superuser may change it (SUSET), and a superuser session SET is the admin override, asserted with the configured list empty so only the SET can admit the read. A link-local address is refused even when listed, before any connection, naming the range. + +The list is changed by ALTER SYSTEM and a reload, and reset in a `finally`. The fixture server is the file's own Range-capable `http.server`, serving one Parquet file from `server_dir`. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_endpoint_allow_list` | the default deny, the list's matching, SUSET and the admin override, and the link-local refusal | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 25b7e0a2d..c7650f3f4 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -711,4 +711,6 @@ guard_tests 438 # rls_direct_storage.sh: one test, by collection over the cluster files on this tree. # cluster_tests 591 -> 592 for #1341's test_objstore_crlf.py, the twin of objstore_crlf.sh: # one test, by collection over the cluster files on this tree. -cluster_tests 592 +# cluster_tests 592 -> 593 for #1341's test_objstore_allowlist.py, the twin of +# objstore_allowlist.sh: one test, by collection over the cluster files on this tree. +cluster_tests 593 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index d468a93aa..d78edc881 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -128,6 +128,7 @@ "native_reclaim_cycles", "native_reclaim_frag", "native_vacuum_race", + "objstore_allowlist", "objstore_credentials", "objstore_crlf", "parallel_am_scan", diff --git a/test/pytest/test_objstore_allowlist.py b/test/pytest/test_objstore_allowlist.py new file mode 100644 index 000000000..f3916d7b9 --- /dev/null +++ b/test/pytest/test_objstore_allowlist.py @@ -0,0 +1,160 @@ +"""The object-store endpoint allow-list, empty by default (#393): the pytest twin of +`test/objstore_allowlist.sh` (#1341). Every check carries the shell's name. + +`pgcolumnar.objstore_allowed_endpoints` gates every remote scheme: empty (the default) +refuses all remote access, so pg_read_server_files is not an SSRF primitive out of the +box, and link-local ranges (169.254.0.0/16, the cloud-metadata path) are refused even +when listed. The setting is SUSET, because a USERSET list would let any role widen its +own allow-list. + +The list is changed the operator's way, ALTER SYSTEM and a reload, and reset in a +`finally`, because the cluster is shared with every other test. A superuser session SET +is exercised once, as the admin override. + +THE FIXTURE SERVER IS THIS FILE'S OWN: a Range-capable http.server in a thread, serving +one Parquet file from `server_dir`. The shell twin's server is a script under `test/`, +and the two harnesses do not call each other. +""" +import os +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import pytest + + +class _Ranged(BaseHTTPRequestHandler): + """HEAD and GET with an optional `Range: bytes=lo-hi`, from one directory.""" + root = None + + def log_message(self, *args): + pass + + def _path(self): + p = os.path.join(self.root, os.path.basename(self.path.split("?")[0])) + return p if os.path.isfile(p) else None + + def do_HEAD(self): + p = self._path() + if p is None: + self.send_error(404) + return + self.send_response(200) + self.send_header("Content-Length", str(os.path.getsize(p))) + self.send_header("Accept-Ranges", "bytes") + self.end_headers() + + def do_GET(self): + p = self._path() + if p is None: + self.send_error(404) + return + data = open(p, "rb").read() + rng = self.headers.get("Range") + if rng and rng.startswith("bytes="): + lo, _, hi = rng[len("bytes="):].partition("-") + lo = int(lo) + hi = min(int(hi) if hi else len(data) - 1, len(data) - 1) + body = data[lo:hi + 1] + self.send_response(206) + self.send_header("Content-Range", f"bytes {lo}-{hi}/{len(data)}") + else: + body = data + self.send_response(200) + self.send_header("Content-Length", str(len(body))) + self.send_header("Accept-Ranges", "bytes") + self.end_headers() + self.wfile.write(body) + + +def _state(cluster, sql, role="postgres", pre=None): + """-> (sqlstate or '', primary message or '', first value or None) on a fresh session.""" + import psycopg + dsn = f"host=127.0.0.1 port={cluster.port} user={role} dbname=postgres" + try: + with psycopg.connect(dsn, autocommit=True) as c: + if pre: + c.execute(pre) + row = c.execute(sql).fetchone() + return "", "", (row[0] if row else None) + except psycopg.Error as e: + return e.sqlstate or "?", e.diag.message_primary or "", None + + +def test_the_endpoint_allow_list(pgc_cluster, pgc_conn, server_dir, expect): + pa = expect.require_module("pyarrow", name="pyarrow is required to write the fixture") + if pa is None: + return + pq = expect.require_module("pyarrow.parquet", name="pyarrow is required to write the fixture") + pq.write_table(pa.table({"a": pa.array(range(1000), pa.int64())}), str(server_dir / "fixture.parquet")) + + _Ranged.root = str(server_dir) + srv = ThreadingHTTPServer(("127.0.0.1", 0), _Ranged) + port = srv.server_address[1] + threading.Thread(target=srv.serve_forever, daemon=True).start() + + def set_list(value): + with pgc_conn.cursor() as cur: + cur.execute(f"ALTER SYSTEM SET pgcolumnar.objstore_allowed_endpoints = '{value}'") + cur.execute("SELECT pg_reload_conf()") + time.sleep(0.5) + + url = f"http://127.0.0.1:{port}/fixture.parquet" + q = f"SELECT count(*) FROM pgcolumnar.read_parquet('{url}') AS (a int8)" + try: + import urllib.request + with urllib.request.urlopen(urllib.request.Request(url, method="HEAD"), timeout=5) as r: + expect.num(r.status, 200, "premise: fixture server is up") + + # ---- the default: deny everything remote ---------------------------- + expect.text(_state(pgc_cluster, "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")[2], + "", "premise: the list defaults to empty") + state, msg, _ = _state(pgc_cluster, q) + expect.text(state, "42501", "default: an http read is refused 42501") + expect.text("yes" if "objstore_allowed_endpoints" in msg else "no", "yes", + "default: the refusal names the GUC") + local = f"SELECT count(*) FROM pgcolumnar.read_parquet('{server_dir / 'fixture.parquet'}') AS (a int8)" + expect.num(_state(pgc_cluster, local)[2], 1000, "default: a local read is untouched by the list") + + # ---- the list admits exactly what it names --------------------------- + set_list("127.0.0.1") + expect.num(_state(pgc_cluster, q)[2], 1000, "listed host: the read works") + set_list("127.0.0.2") + expect.text(_state(pgc_cluster, q)[0], "42501", "unlisted host: refused 42501") + set_list(f"127.0.0.1:{port}") + expect.num(_state(pgc_cluster, q)[2], 1000, "host:port entry, right port: works") + set_list("127.0.0.1:1") + expect.text(_state(pgc_cluster, q)[0], "42501", "host:port entry, wrong port: refused 42501") + set_list("other.example, 127.0.0.1") + expect.num(_state(pgc_cluster, q)[2], 1000, "a multi-entry list matches any entry") + + # ---- SUSET: only a superuser may change it --------------------------- + with pgc_conn.cursor() as cur: + cur.execute("DO $$ BEGIN CREATE ROLE allow_nobody_py LOGIN; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + expect.num(_state(pgc_cluster, "SELECT 1", role="allow_nobody_py")[2], 1, + "premise: the role can open a session") + expect.text(_state(pgc_cluster, "SET pgcolumnar.objstore_allowed_endpoints = '10.0.0.1'", + role="allow_nobody_py")[0], "42501", + "a non-superuser cannot widen the list (42501)") + # WITH THE CONFIGURED LIST EMPTY, so only the session's own SET can admit the + # read (#1341): with 127.0.0.1 still configured, the read passed whatever the + # SET did. + set_list("") + expect.num(_state(pgc_cluster, q, pre="SET pgcolumnar.objstore_allowed_endpoints = '127.0.0.1'")[2], 1000, + "a superuser session SET works (admin override)") + + # ---- link-local: refused even when listed ---------------------------- + # 42501 proves the refusal came BEFORE any connection: a real connect to + # 169.254/16 would surface 08006 or hang toward a timeout. + set_list("169.254.169.254, 127.0.0.1") + ll = "SELECT count(*) FROM pgcolumnar.read_parquet('http://169.254.169.254:1/x.parquet') AS (a int8)" + state, msg, _ = _state(pgc_cluster, ll) + expect.text(state, "42501", "link-local: refused 42501 despite being listed") + expect.num(1 if "link-local" in msg else 0, 1, "link-local: the refusal names the range, not the list") + expect.num(_state(pgc_cluster, q)[2], 1000, "link-local: the listed ordinary host still works beside it") + finally: + with pgc_conn.cursor() as cur: + cur.execute("ALTER SYSTEM RESET pgcolumnar.objstore_allowed_endpoints") + cur.execute("SELECT pg_reload_conf()") + srv.shutdown() From 5ab2afdb563f0a66bea15671dbbdde8bf4b1c4f9 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 03:36:36 +0000 Subject: [PATCH 18/41] test: the allow-list twin's default premise states 'empty' rather than comparing two empty strings (#1341) Co-Authored-By: Claude Opus 5.5 --- test/pytest/test_objstore_allowlist.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/pytest/test_objstore_allowlist.py b/test/pytest/test_objstore_allowlist.py index f3916d7b9..f809b4bfa 100644 --- a/test/pytest/test_objstore_allowlist.py +++ b/test/pytest/test_objstore_allowlist.py @@ -107,8 +107,9 @@ def set_list(value): expect.num(r.status, 200, "premise: fixture server is up") # ---- the default: deny everything remote ---------------------------- - expect.text(_state(pgc_cluster, "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")[2], - "", "premise: the list defaults to empty") + setting = _state(pgc_cluster, "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")[2] + expect.text("empty" if setting == "" else f"set: [{setting}]", "empty", + "premise: the list defaults to empty") state, msg, _ = _state(pgc_cluster, q) expect.text(state, "42501", "default: an http read is refused 42501") expect.text("yes" if "objstore_allowed_endpoints" in msg else "no", "yes", From bcb991a73ad9c35ce63d29de8d221412d340c71a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 03:40:03 +0000 Subject: [PATCH 19/41] test: objstore_allowlist seeds the ledger, with four arms dated (#1341) Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 16 ++++++++++++++++ test/check_ledger_budget.txt | 18 ++++++++++++++++-- 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 394bab980..9b2b89002 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2337,6 +2337,22 @@ native_reclaim native_reclaim repeated recluster reuses space (file plateaus) 15 native_reclaim native_reclaim reused-block data matches heap mirror 15;16;17;18;19 never - native_reclaim native_reclaim row count correct after cycle 15;16;17;18;19 never - native_reclaim native_reclaim ten row groups, every one half deleted 15;16;17;18;19 2026-10-14 the write state ignores stripe_row_limit +objstore_allowlist objstore_allowlist a multi-entry list matches any entry 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist a non-superuser cannot widen the list (42501) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it +objstore_allowlist objstore_allowlist a superuser session SET works (admin override) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it +objstore_allowlist objstore_allowlist default: a local read is untouched by the list 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist default: an http read is refused 42501 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint +objstore_allowlist objstore_allowlist default: the refusal names the GUC 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint +objstore_allowlist objstore_allowlist host:port entry, right port: works 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist host:port entry, wrong port: refused 42501 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist link-local: refused 42501 despite being listed 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist link-local: the listed ordinary host still works beside it 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist link-local: the refusal names the range, not the list 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist listed host: the read works 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist premise: fixture server is up 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist premise: the list defaults to empty 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist premise: the role can open a session 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist unlisted host: refused 42501 15;16;17;18;19 never - objstore_credentials objstore_credentials a mapping with an access key and no secret is refused before any request (28000) 15;16;17;18;19 2026-10-14 the user-mapping credential pairing check removed (an access key alone crashes the backend) objstore_credentials objstore_credentials a role with no mapping of its own reads through the PUBLIC mapping 15;16;17;18;19 2026-10-14 the user-mapping lookup has no PUBLIC fallback objstore_credentials objstore_credentials a role's own mapping is used before the PUBLIC one (bob's wrong secret is still refused 28000) 15;16;17;18;19 2026-10-14 the user-mapping lookup tries PUBLIC before the role's own diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 8d819d75e..198681ab8 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 228 +suites_not_covered 227 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2607,4 +2607,18 @@ suites_not_covered 228 # never 2360 + 2 = 2362 # dated 795 + 3 = 798 # 2362 + 798 == 3160 -checks_never_observed_red 2362 +# +# #1341, objstore_allowlist seeded, with its twin and a fixed admin-override arm +# (OffgridwithJD): 2362 -> 2374, and suites_not_covered 228 -> 227. The suite had no +# rows. It enters with all 16 checks from five-major runs (16/16 on each of 15-19). +# Four are dated: the two default-deny arms by an empty list that admits every +# endpoint; the non-superuser arm and the admin-override arm by the setting defined +# PGC_SIGHUP. Main's admin-override arm survived that one, because the configured +# list still named 127.0.0.1; it now runs with the list empty. The other 12 arrive +# never. +# +# ledger rows 3160 + 16 = 3176 +# never 2362 + 12 = 2374 +# dated 798 + 4 = 802 +# 2374 + 802 == 3176 +checks_never_observed_red 2374 From 111d4eb1a83e610dcf86a6f679ddf0cdbffd7ada Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 20:09:14 +0000 Subject: [PATCH 20/41] test: objstore_allowlist's default check loses the premise prefix, and the shell names 'empty' itself (#1379 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 2 +- test/objstore_allowlist.sh | 8 ++++++-- test/pytest/test_objstore_allowlist.py | 2 +- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 9b2b89002..9add04f81 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2350,8 +2350,8 @@ objstore_allowlist objstore_allowlist link-local: the listed ordinary host still objstore_allowlist objstore_allowlist link-local: the refusal names the range, not the list 15;16;17;18;19 never - objstore_allowlist objstore_allowlist listed host: the read works 15;16;17;18;19 never - objstore_allowlist objstore_allowlist premise: fixture server is up 15;16;17;18;19 never - -objstore_allowlist objstore_allowlist premise: the list defaults to empty 15;16;17;18;19 never - objstore_allowlist objstore_allowlist premise: the role can open a session 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist the list defaults to empty 15;16;17;18;19 never - objstore_allowlist objstore_allowlist unlisted host: refused 42501 15;16;17;18;19 never - objstore_credentials objstore_credentials a mapping with an access key and no secret is refused before any request (28000) 15;16;17;18;19 2026-10-14 the user-mapping credential pairing check removed (an access key alone crashes the backend) objstore_credentials objstore_credentials a role with no mapping of its own reads through the PUBLIC mapping 15;16;17;18;19 2026-10-14 the user-mapping lookup has no PUBLIC fallback diff --git a/test/objstore_allowlist.sh b/test/objstore_allowlist.sh index 69b7f90a0..f2e3d863a 100755 --- a/test/objstore_allowlist.sh +++ b/test/objstore_allowlist.sh @@ -69,8 +69,12 @@ URL="http://127.0.0.1:$HTTP_PORT/fixture.parquet" Q="SELECT count(*) FROM pgcolumnar.read_parquet('$URL') AS (a int8)" # ---- the default: deny everything remote ------------------------------------ -check "premise: the list defaults to empty" \ - "$(q "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")" "" +# A check, not a premise (#1236; #1379 review): the default is the product's. And +# "empty" is produced by the query itself, so a capture that is empty because the +# query never ran cannot pass. +check "the list defaults to empty" \ + "$(q "SELECT CASE WHEN current_setting('pgcolumnar.objstore_allowed_endpoints') = '' THEN 'empty' + ELSE 'set: [' || current_setting('pgcolumnar.objstore_allowed_endpoints') || ']' END")" "empty" check "default: an http read is refused 42501" "$(sqlstate_of "$Q")" "42501" check "default: the refusal names the GUC" \ "$([ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -qtA \ diff --git a/test/pytest/test_objstore_allowlist.py b/test/pytest/test_objstore_allowlist.py index f809b4bfa..71d41bdca 100644 --- a/test/pytest/test_objstore_allowlist.py +++ b/test/pytest/test_objstore_allowlist.py @@ -109,7 +109,7 @@ def set_list(value): # ---- the default: deny everything remote ---------------------------- setting = _state(pgc_cluster, "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")[2] expect.text("empty" if setting == "" else f"set: [{setting}]", "empty", - "premise: the list defaults to empty") + "the list defaults to empty") state, msg, _ = _state(pgc_cluster, q) expect.text(state, "42501", "default: an http read is refused 42501") expect.text("yes" if "objstore_allowed_endpoints" in msg else "no", "yes", From b0303e6d1cd6cd495fde1f6eb04f65b51c4be33a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 20:12:26 +0000 Subject: [PATCH 21/41] test: the allow-list default check, unlisted host and wrong port are dated (#1379 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 10 +++++----- test/check_ledger_budget.txt | 13 ++++++++++++- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 9add04f81..6c5ca89a4 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2341,18 +2341,18 @@ objstore_allowlist objstore_allowlist a multi-entry list matches any entry 15;16 objstore_allowlist objstore_allowlist a non-superuser cannot widen the list (42501) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it objstore_allowlist objstore_allowlist a superuser session SET works (admin override) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it objstore_allowlist objstore_allowlist default: a local read is untouched by the list 15;16;17;18;19 never - -objstore_allowlist objstore_allowlist default: an http read is refused 42501 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint -objstore_allowlist objstore_allowlist default: the refusal names the GUC 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint +objstore_allowlist objstore_allowlist default: an http read is refused 42501 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint;the allow-list check never refuses;the allow-list's default is 127.0.0.1, not empty +objstore_allowlist objstore_allowlist default: the refusal names the GUC 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint;the allow-list check never refuses;the allow-list's default is 127.0.0.1, not empty objstore_allowlist objstore_allowlist host:port entry, right port: works 15;16;17;18;19 never - -objstore_allowlist objstore_allowlist host:port entry, wrong port: refused 42501 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist host:port entry, wrong port: refused 42501 15;16;17;18;19 2026-10-14 the allow-list check never refuses objstore_allowlist objstore_allowlist link-local: refused 42501 despite being listed 15;16;17;18;19 never - objstore_allowlist objstore_allowlist link-local: the listed ordinary host still works beside it 15;16;17;18;19 never - objstore_allowlist objstore_allowlist link-local: the refusal names the range, not the list 15;16;17;18;19 never - objstore_allowlist objstore_allowlist listed host: the read works 15;16;17;18;19 never - objstore_allowlist objstore_allowlist premise: fixture server is up 15;16;17;18;19 never - objstore_allowlist objstore_allowlist premise: the role can open a session 15;16;17;18;19 never - -objstore_allowlist objstore_allowlist the list defaults to empty 15;16;17;18;19 never - -objstore_allowlist objstore_allowlist unlisted host: refused 42501 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist the list defaults to empty 15;16;17;18;19 2026-10-14 the allow-list's default is 127.0.0.1, not empty +objstore_allowlist objstore_allowlist unlisted host: refused 42501 15;16;17;18;19 2026-10-14 the allow-list check never refuses objstore_credentials objstore_credentials a mapping with an access key and no secret is refused before any request (28000) 15;16;17;18;19 2026-10-14 the user-mapping credential pairing check removed (an access key alone crashes the backend) objstore_credentials objstore_credentials a role with no mapping of its own reads through the PUBLIC mapping 15;16;17;18;19 2026-10-14 the user-mapping lookup has no PUBLIC fallback objstore_credentials objstore_credentials a role's own mapping is used before the PUBLIC one (bob's wrong secret is still refused 28000) 15;16;17;18;19 2026-10-14 the user-mapping lookup tries PUBLIC before the role's own diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 198681ab8..0cb4293c1 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2621,4 +2621,15 @@ suites_not_covered 227 # never 2362 + 12 = 2374 # dated 798 + 4 = 802 # 2374 + 802 == 3176 -checks_never_observed_red 2374 +# +# #1379 review (OffgridwithJD): 2374 -> 2371. "the list defaults to empty" loses the premise +# prefix, dated by the default changed to 127.0.0.1; the shell now has the query name +# "empty" itself rather than comparing with an empty string. The allow-list check that +# never refuses also dates "unlisted host" and "host:port entry, wrong port". +# Five-major runs: 16/16 on 15-19, the twin passing with pyarrow required. +# +# ledger rows 3176 = 3176 +# never 2374 - 3 = 2371 +# dated 802 + 3 = 805 +# 2371 + 805 == 3176 +checks_never_observed_red 2371 From 3baef6a2eb3ea1131760a2c4f9d6ad4752656b0a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:41:50 +0000 Subject: [PATCH 22/41] test: objstore_userinfo gets its pytest twin, every check by the shell's name (#1341) Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 15 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_objstore_userinfo.py | 127 ++++++++++++++++++++++++++ 4 files changed, 146 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_objstore_userinfo.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 711ecae07..d52a68b34 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -173,6 +173,7 @@ behaviour, the source of that number is named. - [125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts](#125-test_rls_direct_storagepy-the-direct-storage-paths-refuse-a-caller-a-policy-restricts) - [126. test_objstore_crlf.py: a CR or LF in an object-store URL is refused before it is sent](#126-test_objstore_crlfpy-a-cr-or-lf-in-an-object-store-url-is-refused-before-it-is-sent) - [127. test_objstore_allowlist.py: the endpoint allow-list, empty by default](#127-test_objstore_allowlistpy-the-endpoint-allow-list-empty-by-default) +- [128. test_objstore_userinfo.py: userinfo in a URL is refused on every entry point](#128-test_objstore_userinfopy-userinfo-in-a-url-is-refused-on-every-entry-point) ## 1. How to read a test in here @@ -7135,3 +7136,17 @@ The list is changed by ALTER SYSTEM and a reload, and reset in a `finally`. The | test | what it holds | | --- | --- | | `test_the_endpoint_allow_list` | the default deny, the list's matching, SUSET and the admin override, and the link-local refusal | + +## 128. test_objstore_userinfo.py: userinfo in a URL is refused on every entry point + +The pytest twin of `test/objstore_userinfo.sh` (#706, #995, #997, ported for #1341), every check by the shell's name. `user@host` is refused with the parse guard's 22023 on the http read and write paths, in an s3 or gs bucket authority, and in an endpoint a foreign server configures. 22023 comes only from the parse guards and 42501 only from the allow-list, so the SQLSTATE says which layer refused, and the message pins the guard. `test_objstore_endpoint_userinfo.py` covers the endpoint half independently and is not this file's pair. + +The controls are what make the arms mean anything. With no endpoint configured every s3 URL is refused 28000, the clean one included, and 28000 has three causes, so each control names the refusal it expects. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_userinfo_in_a_url_is_refused_by_the_parse_guard` | read_parquet and export_parquet over http, export_arrow, and the s3 and gs bucket authority, read and write: 22023, naming userinfo | +| `test_a_clean_url_reaches_the_refusal_it_should` | a clean s3 URL demands an endpoint (naming `AWS_ENDPOINT_URL`), gs demands a credential, a malformed URL and an `@` in a key are not userinfo, a clean non-allowed endpoint is still 42501, and the backend is alive | +| `test_userinfo_in_a_configured_endpoint_is_refused` | both endpoint shapes, `u:p@` and `user@`, are 22023 naming userinfo and the endpoint; a clean endpoint gets past the parse | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index c7650f3f4..e8dd7d78b 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -713,4 +713,6 @@ guard_tests 438 # one test, by collection over the cluster files on this tree. # cluster_tests 592 -> 593 for #1341's test_objstore_allowlist.py, the twin of # objstore_allowlist.sh: one test, by collection over the cluster files on this tree. -cluster_tests 593 +# cluster_tests 593 -> 596 for #1341's test_objstore_userinfo.py, the twin of +# objstore_userinfo.sh: three tests, by collection over the cluster files on this tree. +cluster_tests 596 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index d78edc881..e28cec9b5 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -131,6 +131,7 @@ "objstore_allowlist", "objstore_credentials", "objstore_crlf", + "objstore_userinfo", "parallel_am_scan", "parallel_scan_cost", "partition_lifecycle", diff --git a/test/pytest/test_objstore_userinfo.py b/test/pytest/test_objstore_userinfo.py new file mode 100644 index 000000000..39979dccc --- /dev/null +++ b/test/pytest/test_objstore_userinfo.py @@ -0,0 +1,127 @@ +"""Userinfo in a URL is refused on every entry point (#706, #995, #997): the pytest +twin of `test/objstore_userinfo.sh` (#1341). Every check carries the shell's name. + +The read path, the write path, the s3 and gs bucket authority, and an endpoint a +foreign server configures each refuse `user@host` with the parse guard's 22023. +22023 comes only from the parse guards here and 42501 only from the allow-list, so +the SQLSTATE says WHICH layer refused, and the message pins which 22023 guard. + +THE CONTROLS ARE WHAT MAKE THE ARMS MEAN ANYTHING. With no endpoint configured every +s3 URL is refused 28000, the clean one included, so a userinfo refusal must be told +apart from an unreachable object store: a clean URL has to reach the endpoint demand, +and say so. 28000 has three causes in os_resolve_s3 (no endpoint, no credential, the +allow-list's authorization), so each control names the one it expects. + +No object-store server is needed: every refusal here fires before any network I/O. +The allow-list is set per session, as test_objstore_endpoint_userinfo.py does; it is +superuser-settable. Nothing here reads anything under `test/*.sh`. +""" +import pytest + +ALLOWED = "127.0.0.1" +URL = "http://u:p@127.0.0.1:1/x.parquet" +S3UI = "s3://u:p@mybucket/x.parquet" + + +def _err(conn, sql): + """-> (sqlstate, primary message) of the error `sql` raises, or ('', '').""" + import psycopg + try: + with conn.cursor() as cur: + cur.execute(sql) + if cur.description: + cur.fetchall() + return "", "" + except psycopg.Error as e: + return e.sqlstate or "?", e.diag.message_primary or "" + + +def _has(text, word): + return 1 if word in text else 0 + + +@pytest.fixture +def ui(pgc_conn): + with pgc_conn.cursor() as cur: + cur.execute(f"SET pgcolumnar.objstore_allowed_endpoints = '{ALLOWED}'") + cur.execute("CREATE TABLE ex (id int, v text) USING pgcolumnar") + cur.execute("INSERT INTO ex SELECT g, 'v'||g FROM generate_series(1,100) g") + return pgc_conn + + +def test_userinfo_in_a_url_is_refused_by_the_parse_guard(ui, expect): + """The http read and write paths, and the s3 and gs bucket authority.""" + rp = f"SELECT * FROM pgcolumnar.read_parquet('{URL}') AS t(v int)" + state, msg = _err(ui, rp) + expect.text(state, "22023", "read_parquet refuses userinfo (22023, the parse guard)") + expect.num(_has(msg, "userinfo"), 1, "and the read_parquet message names userinfo") + + ep = f"SELECT pgcolumnar.export_parquet('ex', '{URL}')" + state, msg = _err(ui, ep) + expect.text(state, "22023", "export_parquet refuses userinfo (22023, not an allow-list 42501)") + expect.num(_has(msg, "userinfo"), 1, "and the export_parquet message names userinfo") + expect.text(_err(ui, "SELECT pgcolumnar.export_arrow('ex', 'http://u@127.0.0.1:1/x.arrow')")[0], "22023", + "export_arrow refuses userinfo through the same handle (22023)") + + s3r = f"SELECT * FROM pgcolumnar.read_parquet('{S3UI}') AS t(v int)" + state, msg = _err(ui, s3r) + expect.text(state, "22023", "read_parquet refuses userinfo in an s3:// bucket authority (22023)") + expect.num(_has(msg, "userinfo"), 1, "and the s3 read message names userinfo") + expect.text(_err(ui, "SELECT * FROM pgcolumnar.read_parquet('gs://u@mybucket/x.parquet') AS t(v int)")[0], + "22023", "a gs:// bucket authority is refused by the same guard") + s3w = f"SELECT pgcolumnar.export_parquet('ex', '{S3UI}')" + state, msg = _err(ui, s3w) + expect.text(state, "22023", + "export_parquet refuses it on the WRITE path too, where acceptance writes to a bucket nobody named") + expect.num(_has(msg, "userinfo"), 1, "and the s3 write message names userinfo") + + +def test_a_clean_url_reaches_the_refusal_it_should(ui, expect): + """The controls: a clean s3 URL demands an endpoint, gs demands a credential, a + malformed URL is the bucket/key refusal, '@' in a KEY is not userinfo, and a clean + URL to a non-allowed endpoint is still the allow-list's 42501.""" + clean = "SELECT * FROM pgcolumnar.read_parquet('s3://mybucket/x.parquet') AS t(v int)" + state, msg = _err(ui, clean) + expect.text(state, "28000", "control: a clean s3:// URL is not refused as userinfo, it demands an endpoint") + expect.num(_has(msg, "AWS_ENDPOINT_URL"), 1, + "control: and that refusal NAMES AWS_ENDPOINT_URL, pinning which 28000 fired") + expect.num(_has(msg, "userinfo"), 0, + "control: and it does not name userinfo, so the new guard did not fire on it") + _, msg = _err(ui, "SELECT * FROM pgcolumnar.read_parquet('gs://mybucket/x.parquet') AS t(v int)") + expect.num(_has(msg, "AWS_ACCESS_KEY_ID"), 1, + "control: the gs:// refusal names a CREDENTIAL, not the endpoint -- same 28000, other cause") + _, msg = _err(ui, "SELECT * FROM pgcolumnar.read_parquet('s3://nokey') AS t(v int)") + expect.num(_has(msg, "userinfo"), 0, "control: a malformed s3:// URL is still the bucket/key refusal, naming no userinfo") + state, msg = _err(ui, "SELECT * FROM pgcolumnar.read_parquet('s3://mybucket/my@file.parquet') AS t(v int)") + expect.num(_has(msg, "userinfo"), 0, "control: an '@' in the KEY is not userinfo, and is not refused as it") + expect.text(state, "28000", "control: and such a URL still reaches the endpoint demand, so it was not refused earlier") + expect.text(_err(ui, "SELECT pgcolumnar.export_parquet('ex', 'http://127.0.0.2:1/x.parquet')")[0], "42501", + "a clean URL to a non-allowed endpoint is still the allow-list's 42501") + with ui.cursor() as cur: + cur.execute("SELECT 1") + expect.num(cur.fetchone()[0], 1, "backend alive") + + +def test_userinfo_in_a_configured_endpoint_is_refused(ui, expect): + """The endpoint a foreign server configures, which the bucket guard cannot see. + Two shapes: with a colon in the userinfo, and without one, which used to keep a + valid port and reach only the allow-list.""" + with ui.cursor() as cur: + for srv, endpoint in (("s3ui_py", "http://u:p@127.0.0.1:1"), ("s3ui2_py", "http://user@127.0.0.1:1"), + ("s3ok_py", "http://127.0.0.1:1")): + cur.execute(f"CREATE SERVER IF NOT EXISTS {srv} FOREIGN DATA WRAPPER pgcolumnar_parquet" + f" OPTIONS (endpoint '{endpoint}')") + cur.execute("CREATE FOREIGN TABLE ft_ui (id int) SERVER s3ui_py OPTIONS (path 's3://mybucket/x.parquet')") + cur.execute("CREATE FOREIGN TABLE ft_ui2 (id int) SERVER s3ui2_py OPTIONS (path 's3://mybucket/x.parquet')") + cur.execute("CREATE FOREIGN TABLE ft_ok (id int) SERVER s3ok_py OPTIONS (path 's3://mybucket/x.parquet')") + state, msg = _err(ui, "SELECT * FROM ft_ui") + expect.text(state, "22023", "a userinfo endpoint is refused by the parse guard (22023)") + expect.num(_has(msg, "userinfo"), 1, "and the message names userinfo") + expect.num(_has(msg, "endpoint"), 1, "and it names the ENDPOINT, not the s3:// URL, which carries none") + state, msg = _err(ui, "SELECT * FROM ft_ui2") + expect.text(state, "22023", "the user@host shape is refused by the same guard (22023, not an allow-list 42501)") + expect.num(_has(msg, "userinfo"), 1, "and its message names userinfo too") + state, msg = _err(ui, "SELECT * FROM ft_ok") + expect.num(_has(msg, "userinfo"), 0, "control: a clean endpoint is not refused as userinfo") + expect.text("yes" if state != "22023" else "no", "yes", + "control: and it gets past the parse to a connection failure, not 22023") From df720734275ecf5d4fed9d9d7ef4d28d798a1ee7 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:28:16 +0000 Subject: [PATCH 23/41] test: a projection sorted on a type with no ordering is refused (0A000) and leaves nothing declared, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 --- test/projections.sh | 14 ++++++++++++++ test/pytest/TESTS.md | 1 + test/pytest/expected_tests.txt | 4 +++- test/pytest/test_projections.py | 21 +++++++++++++++++++++ 4 files changed, 39 insertions(+), 1 deletion(-) diff --git a/test/projections.sh b/test/projections.sh index bf126fd5a..e4e5470a1 100755 --- a/test/projections.sh +++ b/test/projections.sh @@ -448,4 +448,18 @@ psql_run "INSERT INTO dt2 SELECT g, g FROM generate_series(200, 300) g;" check "control: a drop in its own transaction never left one" \ "$(proj_orphans dt2)" "0" +# ---- a sort key with no ordering is refused (#1341) -------------------------- +# +# add_projection does not check the sort column's type. The writer does, when it +# is built: json has no btree comparison, so the back-fill over existing rows +# must refuse with 0A000 rather than sort by a comparison function that does not +# exist. Last in the suite, because without the check that sort is not a clean +# error. +psql_run "CREATE TABLE pjs (id int, j json) USING pgcolumnar; + INSERT INTO pjs SELECT g, json_build_object('k', g) FROM generate_series(1, 500) g;" +check "a projection sorted on a type with no ordering (json) is refused (0A000)" \ + "$(proj_sqlstate "SELECT pgcolumnar.add_projection('pjs', 'pjp', ARRAY['id','j'], ARRAY['j'])")" "0A000" +check "and nothing of it is declared" \ + "$(q "SELECT count(*) FROM pgcolumnar.projection_declaration WHERE name = 'pjp';")" "0" + pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index d52a68b34..f19ca19cc 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -4434,6 +4434,7 @@ failing for a reason that has nothing to do with projections. | `test_the_first_add_records_the_base_and_the_new_projection` | both rows appear at once, with the base's columns, empty sort key, name and shared storage id | | `test_a_second_projection_may_have_no_sort_key` | a projection without a sort key, and three distinct storage ids | | `test_a_bad_projection_is_refused_by_its_own_code` | seven refusals, each by its measured SQLSTATE rather than by "it errored" | +| `test_a_sort_key_with_no_ordering_is_refused` | #1341: a projection sorted on `json`, which has no btree ordering, is refused `0A000` when the back-fill builds its writer, and nothing of it is declared | | `test_a_projection_on_a_heap_table_is_refused` | the refusal that is about the ACCESS METHOD, not the arguments | | `test_drop_removes_one_projection_and_leaves_the_rest` | drop is surgical, and the table is still readable after the DDL | | `test_a_projection_added_late_is_back_filled_from_the_existing_rows` | a projection added after the rows exist is populated from them, not left empty | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index e8dd7d78b..3c595766c 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -715,4 +715,6 @@ guard_tests 438 # objstore_allowlist.sh: one test, by collection over the cluster files on this tree. # cluster_tests 593 -> 596 for #1341's test_objstore_userinfo.py, the twin of # objstore_userinfo.sh: three tests, by collection over the cluster files on this tree. -cluster_tests 596 +# cluster_tests 596 -> 597 for #1341's test_a_sort_key_with_no_ordering_is_refused in +# test_projections.py, by collection over the cluster files on this tree. +cluster_tests 597 diff --git a/test/pytest/test_projections.py b/test/pytest/test_projections.py index b284143bf..1aad7783b 100644 --- a/test/pytest/test_projections.py +++ b/test/pytest/test_projections.py @@ -253,6 +253,27 @@ def test_a_bad_projection_is_refused_by_its_own_code(cat, expect, name, code, sq expect.sqlstate(exc.value, code, name) +def test_a_sort_key_with_no_ordering_is_refused(pgc_conn, expect): + """add_projection does not check the sort column's type; the writer does, when + the back-fill builds it (#1341). json has no btree comparison, so the refusal is + 0A000 with the type named, rather than a sort by a comparison that does not + exist, and nothing of the projection is left declared.""" + import psycopg + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE pjs (id int, j json) USING pgcolumnar") + cur.execute("INSERT INTO pjs SELECT g, json_build_object('k', g) FROM generate_series(1, 500) g") + try: + with pgc_conn.cursor() as cur: + cur.execute("SELECT pgcolumnar.add_projection('pjs', 'pjp', ARRAY['id','j'], ARRAY['j'])") + got = "ok" + except psycopg.Error as e: + got = e.sqlstate or "?" + expect.text(got, "0A000", "a projection sorted on a type with no ordering (json) is refused (0A000)") + with pgc_conn.cursor() as cur: + expect.num(_one(cur, "SELECT count(*) FROM pgcolumnar.projection_declaration WHERE name = 'pjp'"), 0, + "and nothing of it is declared") + + def test_a_projection_on_a_heap_table_is_refused(cat, expect): """SEPARATE, because it needs a heap table the other refusals do not, and because the refusal is about the ACCESS METHOD rather than about the arguments.""" From 9dcdf628b93355c1e370331af615bd1d9385b78b Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:31:51 +0000 Subject: [PATCH 24/41] test: the sort-type arm's catalog check waits for a server that accepts connections (#1341) Co-Authored-By: Claude Opus 5.5 --- test/projections.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/test/projections.sh b/test/projections.sh index e4e5470a1..d6d8d68bd 100755 --- a/test/projections.sh +++ b/test/projections.sh @@ -453,12 +453,15 @@ check "control: a drop in its own transaction never left one" \ # add_projection does not check the sort column's type. The writer does, when it # is built: json has no btree comparison, so the back-fill over existing rows # must refuse with 0A000 rather than sort by a comparison function that does not -# exist. Last in the suite, because without the check that sort is not a clean -# error. +# exist. Last in the suite, because without the check that sort crashes the +# backend (measured). psql_run "CREATE TABLE pjs (id int, j json) USING pgcolumnar; INSERT INTO pjs SELECT g, json_build_object('k', g) FROM generate_series(1, 500) g;" check "a projection sorted on a type with no ordering (json) is refused (0A000)" \ "$(proj_sqlstate "SELECT pgcolumnar.add_projection('pjs', 'pjp', ARRAY['id','j'], ARRAY['j'])")" "0A000" +# On a server that accepts connections again: without the check the sort crashes +# its backend (measured), and this arm must answer for the catalog, not for that. +for _i in $(seq 1 60); do [ "$(q 'SELECT 1' 2>/dev/null)" = 1 ] && break; sleep 0.5; done check "and nothing of it is declared" \ "$(q "SELECT count(*) FROM pgcolumnar.projection_declaration WHERE name = 'pjp';")" "0" From 326a2c18bfab42a1c8c7bcb48eaa0e33ff237946 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:34:57 +0000 Subject: [PATCH 25/41] test: projections seeds the ledger, with the sort-type arm dated (#1341) Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 76 ++++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 15 ++++++- 2 files changed, 89 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 6c5ca89a4..d0bb3388d 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2604,6 +2604,82 @@ projection_update projection_update projection matches the heap before any updat projection_update projection_update read_projection matches the live base after that update 15;16;17;18;19 2026-10-14 projection rows storing the first projected column's value in every projected column projection_update projection_update read_projection still matches the live base 15;16;17;18;19 2026-10-14 projection rows storing the first projected column's value in every projected column projection_update projection_update reconstruct via the projection still rebuilds every live row 15;16;17;18;19 2026-10-14 projection rows storing the first projected column's value in every projected column +projections projections DROP TABLE removes its declaration (#304) 15;16;17;18;19 never - +projections projections GUC off: no projection scan 15;16;17;18;19 never - +projections projections a mid-transaction drop leaves no orphan projection storage 15;16;17;18;19 never - +projections projections a projection added after a write in the same transaction gets the later rows 15;16;17;18;19 never - +projections projections a projection dropped mid-transaction is gone, not still being written 15;16;17;18;19 never - +projections projections a projection sorted on a type with no ordering (json) is refused (0A000) 15;16;17;18;19 2026-10-14 the projection writer without its sort-type ordering check (the back-fill sort then crashes the backend) +projections projections add on heap table rejected 15;16;17;18;19 never - +projections projections aggregate over projection scan matches oracle 15;16;17;18;19 never - +projections projections an orphan left by an older build is present 15;16;17;18;19 never - +projections projections an unrestricted base scan matches oracle after delete 15;16;17;18;19 never - +projections projections an unrestricted query reads the base, not the projection 15;16;17;18;19 never - +projections projections and it removed the orphan 15;16;17;18;19 never - +projections projections and leaves the other table's declaration alone 15;16;17;18;19 never - +projections projections and nothing of it is declared 15;16;17;18;19 never - +projections projections and the base table still took every row 15;16;17;18;19 never - +projections projections and they are the right rows, not merely the right number 15;16;17;18;19 never - +projections projections back-fill: p2 matches base (b column) 15;16;17;18;19 never - +projections projections back-fill: p2 populated from existing rows 15;16;17;18;19 never - +projections projections base + p2 remain 15;16;17;18;19 never - +projections projections base id 0 columns are all attrs 15;16;17;18;19 never - +projections projections base id 0 name 15;16;17;18;19 never - +projections projections base id 0 sort_key empty 15;16;17;18;19 never - +projections projections base proj_storage_id == base 15;16;17;18;19 never - +projections projections control: a drop in its own transaction never left one 15;16;17;18;19 never - +projections projections control: with no write before add_projection the projection was always right 15;16;17;18;19 never - +projections projections distinct storage ids 15;16;17;18;19 never - +projections projections drop base rejected 15;16;17;18;19 never - +projections projections drop unknown rejected 15;16;17;18;19 never - +projections projections duplicate column rejected 15;16;17;18;19 never - +projections projections duplicate name rejected 15;16;17;18;19 never - +projections projections empty columns rejected 15;16;17;18;19 never - +projections projections fp chunks carry min/max skip metadata 15;16;17;18;19 never - +projections projections fp count after delete matches base 15;16;17;18;19 never - +projections projections fp fan-out matches base (a,c) 15;16;17;18;19 never - +projections projections fp reflects deletes (a,c) 15;16;17;18;19 never - +projections projections fp row count matches base 15;16;17;18;19 never - +projections projections fp2 multi-stripe fan-out matches base 15;16;17;18;19 never - +projections projections fp2 spans multiple projection row groups 15;16;17;18;19 never - +projections projections fq fan-out matches base (b) 15;16;17;18;19 never - +projections projections new snapshot projection scan sees both batches 15;16;17;18;19 never - +projections projections no projection rows before first add 15;16;17;18;19 never - +projections projections non-covering query (references b) uses the base 15;16;17;18;19 never - +projections projections old snapshot projection scan does not see post-snapshot rows 15;16;17;18;19 never - +projections projections p1 columns 15;16;17;18;19 never - +projections projections p1 gone after drop 15;16;17;18;19 never - +projections projections p1 has its own storage id 15;16;17;18;19 never - +projections projections p1 id is 1 15;16;17;18;19 never - +projections projections p1 sort_key 15;16;17;18;19 never - +projections projections p2 columns 15;16;17;18;19 never - +projections projections p2 id is 2 15;16;17;18;19 never - +projections projections p2 sort_key empty 15;16;17;18;19 never - +projections projections planner still uses projection after vacuum 15;16;17;18;19 never - +projections projections premise: the base table holds every committed row 15;16;17;18;19 never - +projections projections premise: the drop really removed the projection 15;16;17;18;19 never - +projections projections projection chosen for covering + sort-key query 15;16;17;18;19 never - +projections projections projection matches base after second vacuum 15;16;17;18;19 never - +projections projections projection scan matches oracle after delete 15;16;17;18;19 never - +projections projections projection survives vacuum 15;16;17;18;19 never - +projections projections projection-scan matches oracle after vacuum 15;16;17;18;19 never - +projections projections projection-scan results match heap oracle 15;16;17;18;19 never - +projections projections read_projection base rejected 15;16;17;18;19 never - +projections projections read_projection matches base after vacuum 15;16;17;18;19 never - +projections projections reconstruct (a,b,c) matches base after vacuum 15;16;17;18;19 never - +projections projections reconstruct full row matches base 15;16;17;18;19 never - +projections projections reconstruct matches base after delete + NULLs 15;16;17;18;19 never - +projections projections reconstruct row count matches base 15;16;17;18;19 never - +projections projections session A baseline via projection sees batch 1 15;16;17;18;19 never - +projections projections so a rebuild still works for the rest of the database 15;16;17;18;19 never - +projections projections sort key not in columns 15;16;17;18;19 never - +projections projections storage id resolves 15;16;17;18;19 never - +projections projections table populated 15;16;17;18;19 never - +projections projections table still readable after DDL 15;16;17;18;19 never - +projections projections the rebuild does not abort on it 15;16;17;18;19 never - +projections projections two declared projections to start 15;16;17;18;19 never - +projections projections two rows after first add (base + p1) 15;16;17;18;19 never - +projections projections unknown column rejected 15;16;17;18;19 never - range_pruning range_pruning a clustered overlap near the end skips the four vectors only their upper bounds rule out 15;16;17;18;19 2026-10-13 native_range_excludes never using max_upper (haveUpper forced false) range_pruning range_pruning a clustered overlap skips vectors instead of decoding them 15;16;17;18;19 never - range_pruning range_pruning a scattered column skips nothing, which is what the documentation says 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 0cb4293c1..0e9fac106 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 227 +suites_not_covered 226 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2632,4 +2632,15 @@ suites_not_covered 227 # never 2374 - 3 = 2371 # dated 802 + 3 = 805 # 2371 + 805 == 3176 -checks_never_observed_red 2371 +# +# #1341, projections seeded, with the sort-type arms (OffgridwithJD): 2371 -> 2446, and +# suites_not_covered 227 -> 226. The suite had no rows. It enters with all 76 checks +# from five-major runs (76/76 on each of 15-19). One is dated by the projection +# writer without its sort-type ordering check: a json sort key's back-fill then +# crashed the backend instead of refusing with 0A000. The other 75 arrive never. +# +# ledger rows 3176 + 76 = 3252 +# never 2371 + 75 = 2446 +# dated 805 + 1 = 806 +# 2446 + 806 == 3252 +checks_never_observed_red 2446 From 875429d8d2ce3db28cbef043ed2f736e39164f3e Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 22:00:44 +0000 Subject: [PATCH 26/41] test: the projections drop row takes the check's name, without the premise: prefix (#1341) Review of #1375 by jdatcmd: CI's suites (PG 17) ledger gate found the row seeded as 'premise: the drop really removed the projection' while projections.sh:410 and test_projections.py:795 name the check without the prefix, so the gate saw one check not in the ledger and one orphan row. The row is renamed and keeps its other fields. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index d0bb3388d..3f9562f27 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2657,7 +2657,6 @@ projections projections p2 id is 2 15;16;17;18;19 never - projections projections p2 sort_key empty 15;16;17;18;19 never - projections projections planner still uses projection after vacuum 15;16;17;18;19 never - projections projections premise: the base table holds every committed row 15;16;17;18;19 never - -projections projections premise: the drop really removed the projection 15;16;17;18;19 never - projections projections projection chosen for covering + sort-key query 15;16;17;18;19 never - projections projections projection matches base after second vacuum 15;16;17;18;19 never - projections projections projection scan matches oracle after delete 15;16;17;18;19 never - @@ -2676,6 +2675,7 @@ projections projections sort key not in columns 15;16;17;18;19 never - projections projections storage id resolves 15;16;17;18;19 never - projections projections table populated 15;16;17;18;19 never - projections projections table still readable after DDL 15;16;17;18;19 never - +projections projections the drop really removed the projection 15;16;17;18;19 never - projections projections the rebuild does not abort on it 15;16;17;18;19 never - projections projections two declared projections to start 15;16;17;18;19 never - projections projections two rows after first add (base + p1) 15;16;17;18;19 never - From 9cd207fb31e7f643ebd9bf3ff5442157ed66b1e9 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:15:00 +0000 Subject: [PATCH 27/41] test: analyze() refuses a named column the table does not have, before writing any statistics, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 --- test/analyze_function.sh | 18 ++++++++++++++++++ test/pytest/TESTS.md | 1 + test/pytest/expected_tests.txt | 4 +++- test/pytest/test_analyze_function.py | 26 ++++++++++++++++++++++++++ 4 files changed, 48 insertions(+), 1 deletion(-) diff --git a/test/analyze_function.sh b/test/analyze_function.sh index 5f3c3f883..d1e3d4c3a 100755 --- a/test/analyze_function.sh +++ b/test/analyze_function.sh @@ -979,5 +979,23 @@ check_text "the two routes agree which storage a WITH DATA matview owns" \ check_text "pgcolumnar.analyze() reaches a matview created WITH DATA" \ "$mv_state" "succeeded" +# ---- a named column that does not exist is refused (#1341) -------------------- +# +# Silently collecting nothing for it is the failure that looks like success, and +# the refusal comes before any work: a real column named beside it gets no +# statistics either. A fresh table, so pg_stats starts empty for it. +q "DROP TABLE IF EXISTS af_unk; + CREATE TABLE af_unk (a int, b text) USING pgcolumnar; + INSERT INTO af_unk SELECT g, 'v'||g FROM generate_series(1, 5000) g;" >/dev/null +unk_out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -X -At \ + -v ON_ERROR_STOP=1 -v VERBOSITY=verbose -c "SELECT pgcolumnar.analyze('af_unk'::regclass, ARRAY['a', 'nosuch'])" 2>&1)" +check_text "analyze() refuses a named column the table does not have, and names it" \ + "$(sed -n 's/^ERROR: *\([0-9A-Z]\{5\}\): \(.*\)$/\1 \2/p' <<<"$unk_out" | head -1)" \ + "P0001 pgcolumnar.analyze(): column \"nosuch\" does not exist in af_unk" +check_num "and writes no statistics, not even for the real column named beside it" \ + "$(q "SELECT count(*) FROM pg_stats WHERE tablename = 'af_unk'")" "0" +q "SELECT pgcolumnar.analyze('af_unk'::regclass, ARRAY['a'])" >/dev/null +check_num "control: the same call naming only the real column writes its statistics" \ + "$(q "SELECT count(*) FROM pg_stats WHERE tablename = 'af_unk' AND attname = 'a'")" "1" pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index f19ca19cc..3643b6594 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -5849,6 +5849,7 @@ THE FIX IS NOT ONE LINE, AND A GREP IS WHY IT LOOKED LIKE ONE. `pgcolumnar."anal | test | what it holds | | --- | --- | | `test_analyze_function` | the 500,000-row fixture: null_frac exact on a fraction core's sample cannot express, n_distinct exact from one column, the statistics it does not compute left in place, the histogram's true maximum and its smallest non-most-common bound, no most-common value inside the histogram, every bound a value the column holds, the three MCV frequencies over total rows rather than non-null rows, the per-column statistics target honoured, and a column at `SET STATISTICS 0` left alone | +| `test_a_named_column_that_does_not_exist_is_refused` | #1341: `analyze(rel, ARRAY['a','nosuch'])` is refused, naming `nosuch`, and writes no statistics, not even for `a`; naming `a` alone does | | `test_histogram_bounds_are_a_positional_stride` | eleven distinct rows at target 3, where core's `values[floor(i*(nv-1)/(nhist-1))]` and `percentile_disc`'s `ceil(p*nv)-1` land on different values; the expectation comes from an independent oracle AND a hand-worked figure that must agree first | | `test_null_frac_counts_live_rows_not_ones_a_delete_left` | #485: null_frac came from the zone maps, which count what was WRITTEN, so one pg_stats row carried two statistics normalised against different populations; both must imply the same table | | `test_the_documented_statistics_are_the_ones_written` | the list in `docs/sql-reference.md` parsed and compared against what the function populates, plus the two negatives the doc states in prose | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 3c595766c..74101bb4b 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -717,4 +717,6 @@ guard_tests 438 # objstore_userinfo.sh: three tests, by collection over the cluster files on this tree. # cluster_tests 596 -> 597 for #1341's test_a_sort_key_with_no_ordering_is_refused in # test_projections.py, by collection over the cluster files on this tree. -cluster_tests 597 +# cluster_tests 597 -> 598 for #1341's test_a_named_column_that_does_not_exist_is_refused in +# test_analyze_function.py, by collection over the cluster files on this tree. +cluster_tests 598 diff --git a/test/pytest/test_analyze_function.py b/test/pytest/test_analyze_function.py index 31b1c6258..68849cde2 100644 --- a/test/pytest/test_analyze_function.py +++ b/test/pytest/test_analyze_function.py @@ -497,6 +497,32 @@ def _truth(count): "a column at SET STATISTICS 0 is left alone, because that is what zero means") +def test_a_named_column_that_does_not_exist_is_refused(pgc_conn, expect): + """Silently collecting nothing for a column that does not exist is the failure + that looks like success (#1341). The refusal comes before any work: a real + column named beside it gets no statistics either. The control names only the + real column, so "no statistics" is not a table that never gets any.""" + import psycopg + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE af_unk (a int, b text) USING pgcolumnar") + cur.execute("INSERT INTO af_unk SELECT g, 'v'||g FROM generate_series(1, 5000) g") + try: + with pgc_conn.cursor() as cur: + cur.execute("SELECT pgcolumnar.analyze('af_unk'::regclass, ARRAY['a', 'nosuch'])") + got = "ok" + except psycopg.Error as e: + got = f"{e.sqlstate} {e.diag.message_primary}" + expect.text(got, 'P0001 pgcolumnar.analyze(): column "nosuch" does not exist in af_unk', + "analyze() refuses a named column the table does not have, and names it") + stats = "SELECT count(*) FROM pg_stats WHERE tablename = 'af_unk'" + with pgc_conn.cursor() as cur: + cur.execute(stats) + expect.num(cur.fetchone()[0], 0, "and writes no statistics, not even for the real column named beside it") + cur.execute("SELECT pgcolumnar.analyze('af_unk'::regclass, ARRAY['a'])") + cur.execute(stats + " AND attname = 'a'") + expect.num(cur.fetchone()[0], 1, "control: the same call naming only the real column writes its statistics") + + def test_histogram_bounds_are_a_positional_stride(pgc_conn, expect): """Core's bounds are POSITIONS, not quantiles, and the difference is observable. From ce3f5a8b288f7f54e2ca5e5b8032a33c9245d408 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:26:03 +0000 Subject: [PATCH 28/41] test: the unknown-column test declines below PostgreSQL 18, as its siblings do (#1341) Co-Authored-By: Claude Opus 5.5 --- test/pytest/test_analyze_function.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/pytest/test_analyze_function.py b/test/pytest/test_analyze_function.py index 68849cde2..8b73f334f 100644 --- a/test/pytest/test_analyze_function.py +++ b/test/pytest/test_analyze_function.py @@ -503,6 +503,10 @@ def test_a_named_column_that_does_not_exist_is_refused(pgc_conn, expect): column named beside it gets no statistics either. The control names only the real column, so "no statistics" is not a table that never gets any.""" import psycopg + readable, _raw, major = _major(pgc_conn) + if not readable or major < 18: + _decline_unnamed(expect, major) + return with pgc_conn.cursor() as cur: cur.execute("CREATE TABLE af_unk (a int, b text) USING pgcolumnar") cur.execute("INSERT INTO af_unk SELECT g, 'v'||g FROM generate_series(1, 5000) g") From be7d28ef5b2ac687dd3767c68811b10e90da97c6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:30:42 +0000 Subject: [PATCH 29/41] test: analyze_function seeds the ledger, with the unknown-column arms dated (#1341) Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 72 ++++++++++++++++++++++++++++++++++++ test/check_ledger_budget.txt | 17 ++++++++- 2 files changed, 87 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 3f9562f27..80b62cf66 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -72,6 +72,78 @@ am_contracts am_contracts premise: the racing sessions contended for keys 15;16; am_contracts am_contracts premise: the truncate table holds 5000 rows 15;16;17;18;19 never - am_contracts am_contracts rows the parent buffered before a savepoint survive a TRUNCATE rolled back to it 15;16;17;18;19 2026-10-14 ExecutorEnd flushes no pending writes;TRUNCATE empties the old storage in place (RelationTruncate before the swap) am_contracts am_contracts sessions racing DO NOTHING on the same keys leave each key once, with no error 15;16;17;18;19 2026-10-14 a speculative row that lost the race is never withdrawn;speculative insert refused as unsupported, as before #1341 +analyze_function analyze_function a column at SET STATISTICS 0 is left alone, because that is what zero means 18;19 never - +analyze_function analyze_function analyze() refuses a named column the table does not have, and names it 18;19 2026-10-14 analyze() without its unknown-column check +analyze_function analyze_function and 42's 18;19 never - +analyze_function analyze_function and 7's frequency exactly, over total rows rather than non-null rows 18;19 never - +analyze_function analyze_function and 99's 18;19 never - +analyze_function analyze_function and core's own number for it is not the truth, as the arithmetic requires 18;19 never - +analyze_function analyze_function and correlation is absent, as the doc says 18;19 never - +analyze_function analyze_function and exactly for a fraction core's sample cannot express 18;19 never - +analyze_function analyze_function and it is an ordered ladder rather than two extremes 18;19 never - +analyze_function analyze_function and it is sorted ascending, which a histogram must be to be usable 18;19 never - +analyze_function analyze_function and no most-common value is inside skew's histogram either 18;19 never - +analyze_function analyze_function and so does the most-common list, which is capped by the same target 18;19 never - +analyze_function analyze_function and the smallest non-most-common value at the bottom 18;19 never - +analyze_function analyze_function and without a WARNING, which is how a mistyped argument is dropped 18;19 never - +analyze_function analyze_function and writes no statistics, not even for the real column named beside it 18;19 2026-10-14 analyze() without its unknown-column check +analyze_function analyze_function control: the same call naming only the real column writes its statistics 18;19 never - +analyze_function analyze_function every bound is a value the column actually holds 18;19 never - +analyze_function analyze_function every statistic the doc lists is one the function writes 18;19 never - +analyze_function analyze_function histogram_bounds are core's positional stride, not evenly spaced quantiles 18;19 never - +analyze_function analyze_function no most-common value appears in histogram_bounds, which would double-count it 18;19 never - +analyze_function analyze_function null_frac and the most-common frequencies agree on how many rows there are 18;19 never - +analyze_function analyze_function null_frac counts live rows, not rows a DELETE left behind 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() 15;16;17 never - +analyze_function analyze_function pgcolumnar.analyze() covers every column whichever storage row the heap returns first 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() leaves the statistics it does not compute in place 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() puts the true maximum at the top of histogram_bounds 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() ran for the zero-target column without raising 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() ran without raising for the MCV column 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() ran without raising, so the statistics below are its own 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() reaches a matview created WITH DATA 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() reports n_distinct exactly, from one column 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() reports null_frac exactly, from reading the column 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() still refuses a relation that has never been written 18;19 never - +analyze_function analyze_function pgcolumnar.analyze() writes the three repeated values as most_common_vals 18;19 never - +analyze_function analyze_function premise: 7 survived the delete and is still a most-common value 18;19 never - +analyze_function analyze_function premise: a covering projection gave the table a second storage row 18;19 never - +analyze_function analyze_function premise: a histogram exists for cat, so the exclusion below is not vacuous 18;19 never - +analyze_function analyze_function premise: a histogram was written, so the comparison below is not vacuous 18;19 never - +analyze_function analyze_function premise: a never-written columnar table has no storage row to find 18;19 never - +analyze_function analyze_function premise: and it really is one row in 500000 18;19 never - +analyze_function analyze_function premise: and no whole number of sampled rows gives that fraction, so core cannot report it 18;19 never - +analyze_function analyze_function premise: and nothing else in the column repeats, so the list is exactly three 18;19 never - +analyze_function analyze_function premise: and the global default differs from it, so the two are distinguishable 18;19 never - +analyze_function analyze_function premise: core ANALYZE produced a null_frac at all, so the numbers below exist 18;19 never - +analyze_function analyze_function premise: core samples fewer rows than the table holds, so its number is an estimate 18;19 never - +analyze_function analyze_function premise: every column has statistics when the base row is resolved 18;19 never - +analyze_function analyze_function premise: skew really is at a non-default statistics target 18;19 never - +analyze_function analyze_function premise: the MCV fixture has the row count the frequencies divide by 18;19 never - +analyze_function analyze_function premise: the MCV list really is gone before we write, so what follows is ours 18;19 never - +analyze_function analyze_function premise: the base storage covers every column index 18;19 never - +analyze_function analyze_function premise: the base storage row was first 18;19 never - +analyze_function analyze_function premise: the capture helper reports a success when there is one 18;19 never - +analyze_function analyze_function premise: the delete left fewer live rows than the zone maps describe 18;19 never - +analyze_function analyze_function premise: the doc lists the statistics it claims to write 18;19 never - +analyze_function analyze_function premise: the fixture has the cardinality this check compares against 18;19 never - +analyze_function analyze_function premise: the fixture is exactly one-in-ten NULL 18;19 never - +analyze_function analyze_function premise: the fixture stays on the absolute-count side of core's 10% rule 18;19 never - +analyze_function analyze_function premise: the fixture's column count has not changed 18;19 never - +analyze_function analyze_function premise: the independent oracle agrees with the hand-worked bounds 18;19 never - +analyze_function analyze_function premise: the k7 fixture is exactly one row in seven NULL 18;19 never - +analyze_function analyze_function premise: the matview's relation_oid finds a storage row 18;19 never - +analyze_function analyze_function premise: the outlier really is in the table 18;19 never - +analyze_function analyze_function premise: the projection's storage covers fewer of them 18;19 never - +analyze_function analyze_function premise: the stride fixture has no repeated value, so no MCV is excluded 18;19 never - +analyze_function analyze_function premise: the update put the projection's row first 18;19 never - +analyze_function analyze_function premise: value 42 appears exactly 54,000 times 18;19 never - +analyze_function analyze_function premise: value 7 appears exactly 90,000 times 18;19 never - +analyze_function analyze_function premise: value 99 appears exactly 27,000 times 18;19 never - +analyze_function analyze_function reltuples is untouched, as the doc says 18;19 never - +analyze_function analyze_function so the bottom bound is the smallest non-most-common value, not the column minimum 18;19 never - +analyze_function analyze_function the histogram honours the column's statistics target, not the global default 18;19 never - +analyze_function analyze_function the two routes agree which storage a WITH DATA matview owns 18;19 never - analyze_stats analyze_stats ANALYZE now collects statistics at all 15;16;17;18;19 never - analyze_stats analyze_stats ANALYZE on a wide table completes well inside its derived cap 15;16;17;18;19 never - analyze_stats analyze_stats a fast machine still gets the floor, not a bound that shrinks with it 15;16;17;18;19 2026-09-24 the cap loses its absolute floor, so the bound shrinks with the machine it runs on;the floor moved by one, so the boundary where floor and derived bound meet is off by one diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 0e9fac106..1e7a3c2a6 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 226 +suites_not_covered 225 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2643,4 +2643,17 @@ suites_not_covered 226 # never 2371 + 75 = 2446 # dated 805 + 1 = 806 # 2446 + 806 == 3252 -checks_never_observed_red 2446 +# +# #1341, analyze_function seeded, with the unknown-column arms (OffgridwithJD): 2446 -> +# 2516, and suites_not_covered 226 -> 225. The suite had no rows. It enters with all 72 +# checks from five-major runs (71/71 on 18-19, and the one named version SKIP on +# 15-17, where pgcolumnar.analyze() needs PostgreSQL 18). Two are dated, on PG18, by +# analyze() without its unknown-column check: the call then succeeded and wrote +# statistics for the real column named beside the missing one. The other 70 arrive +# never. +# +# ledger rows 3252 + 72 = 3324 +# never 2446 + 70 = 2516 +# dated 806 + 2 = 808 +# 2516 + 808 == 3324 +checks_never_observed_red 2516 From e56de60d9fa187d094cb494bbf2c895bee8b1bfd Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 19:53:04 +0000 Subject: [PATCH 30/41] test: the unknown-column twin declines on PG15-17, so the registry says so (#1341) test_a_named_column_that_does_not_exist_is_refused calls pgcolumnar.analyze(), which needs PostgreSQL 18, so it declines on 15, 16 and 17 like the other analyze_function twins. It was missing from expected_unrunnable.txt, so CI's PG17 cluster job collected and passed 578 and still exited 67 on the unexpected decline. Measured on PG17: test_analyze_function.py exits 67 without these rows and 0 with them. Co-Authored-By: Claude Opus 5.5 --- test/pytest/expected_unrunnable.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/test/pytest/expected_unrunnable.txt b/test/pytest/expected_unrunnable.txt index b967980e0..731240aae 100644 --- a/test/pytest/expected_unrunnable.txt +++ b/test/pytest/expected_unrunnable.txt @@ -24,18 +24,21 @@ # pg_restore_attribute_stats, which core added in PostgreSQL 18. 15 test_analyze_differential.py::test_analyze_differential 15 test_analyze_function.py::test_analyze_function +15 test_analyze_function.py::test_a_named_column_that_does_not_exist_is_refused 15 test_analyze_function.py::test_histogram_bounds_are_a_positional_stride 15 test_analyze_function.py::test_null_frac_counts_live_rows_not_ones_a_delete_left 15 test_analyze_function.py::test_the_documented_statistics_are_the_ones_written 15 test_analyze_function.py::test_a_covering_projection_does_not_decide_which_columns_get_statistics 16 test_analyze_differential.py::test_analyze_differential 16 test_analyze_function.py::test_analyze_function +16 test_analyze_function.py::test_a_named_column_that_does_not_exist_is_refused 16 test_analyze_function.py::test_histogram_bounds_are_a_positional_stride 16 test_analyze_function.py::test_null_frac_counts_live_rows_not_ones_a_delete_left 16 test_analyze_function.py::test_the_documented_statistics_are_the_ones_written 16 test_analyze_function.py::test_a_covering_projection_does_not_decide_which_columns_get_statistics 17 test_analyze_differential.py::test_analyze_differential 17 test_analyze_function.py::test_analyze_function +17 test_analyze_function.py::test_a_named_column_that_does_not_exist_is_refused 17 test_analyze_function.py::test_histogram_bounds_are_a_positional_stride 17 test_analyze_function.py::test_null_frac_counts_live_rows_not_ones_a_delete_left 17 test_analyze_function.py::test_the_documented_statistics_are_the_ones_written From 193ef09e102313292996a42eb02fbf108c2ebbce Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:14:01 +0000 Subject: [PATCH 31/41] test: set_options refuses an unknown compression name and keeps the stored one (#1341) Co-Authored-By: Claude Opus 5.5 --- test/encode_effort.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/test/encode_effort.sh b/test/encode_effort.sh index 7299f5c4a..f86cbac12 100755 --- a/test/encode_effort.sh +++ b/test/encode_effort.sh @@ -75,6 +75,18 @@ check "a valid encode_effort is stored" \ "$(q "SELECT encode_effort FROM pgcolumnar.options WHERE regclass = 'ef_a'::regclass;" | tail -1)" \ "fast" +# The compression option's own guard (#1341), its sibling. Without it an unknown +# name is stored, and the C reader maps a name it does not know to -1 and falls +# back to the instance default: the table silently compresses as something else. +psql_run "SELECT pgcolumnar.set_options('ef_a', compression => 'zstd');" >/dev/null +badc="$(psql_run "SELECT pgcolumnar.set_options('ef_a', compression => 'brotli');" 2>&1 || true)" +check "an unknown compression name is refused" \ + "$(case "$badc" in *'unknown columnar compression "brotli"'*) echo yes ;; *) echo "no ($badc)" ;; esac)" \ + "yes" +check "and the compression set before it is still the one stored" \ + "$(q "SELECT compression FROM pgcolumnar.options WHERE regclass = 'ef_a'::regclass;" | tail -1)" \ + "zstd" + # --- 2. it changes what the writer does --------------------------------------- psql_run "DROP TABLE IF EXISTS ef_full; DROP TABLE IF EXISTS ef_fast; From f41f38bc5b54d0590761af197eda0e03b14b3a53 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:26:37 +0000 Subject: [PATCH 32/41] test: encode_effort seeds the ledger, with the compression-name arms dated (#1341) Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 12 ++++++++++++ test/check_ledger_budget.txt | 15 +++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 80b62cf66..5411d8719 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -711,6 +711,18 @@ differential differential wide point 15;16;17;18;19 never - differential differential wide range 15;16;17;18;19 never - differential differential wide row proj 15;16;17;18;19 never - differential differential wide row scan 15;16;17;18;19 never - +encode_effort encode_effort a table with no option set matches full 15;16;17;18;19 never - +encode_effort encode_effort a valid encode_effort is stored 15;16;17;18;19 never - +encode_effort encode_effort an unknown compression name is refused 15;16;17;18;19 2026-10-14 set_options without its compression-name check +encode_effort encode_effort an unknown encode_effort is refused 15;16;17;18;19 never - +encode_effort encode_effort and a filtered read agrees 15;16;17;18;19 never - +encode_effort encode_effort and an index fetch finds a fast-written row 15;16;17;18;19 never - +encode_effort encode_effort and is identical where FSST does not win 15;16;17;18;19 never - +encode_effort encode_effort and the compression set before it is still the one stored 15;16;17;18;19 2026-10-14 set_options without its compression-name check +encode_effort encode_effort and the refused value was not stored 15;16;17;18;19 never - +encode_effort encode_effort fast stores more than full on a shape where FSST wins 15;16;17;18;19 never - +encode_effort encode_effort reset_options clears it 15;16;17;18;19 never - +encode_effort encode_effort the rows are identical either way 15;16;17;18;19 never - encode_post_codec encode_post_codec a chunk is not stored larger than it would be with no encoding at all 15;16;17;18;19 never - encode_post_codec encode_post_codec and that column stays far below the no-encoding size, so the win is real 15;16;17;18;19 never - encode_post_codec encode_post_codec premise: both fixtures hold every row 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 1e7a3c2a6..b4449e49a 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 225 +suites_not_covered 224 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2656,4 +2656,15 @@ suites_not_covered 225 # never 2446 + 70 = 2516 # dated 806 + 2 = 808 # 2516 + 808 == 3324 -checks_never_observed_red 2516 +# +# #1341, encode_effort seeded, with the compression-name arms (OffgridwithJD): 2516 -> +# 2526, and suites_not_covered 225 -> 224. The suite had no rows. It enters with all +# 12 checks from five-major runs (12/12 on each of 15-19). Two are dated by +# set_options without its compression-name check: "brotli" was stored over the +# zstd set before it. The other 10 arrive never. +# +# ledger rows 3324 + 12 = 3336 +# never 2516 + 10 = 2526 +# dated 808 + 2 = 810 +# 2526 + 810 == 3336 +checks_never_observed_red 2526 From e688e0fd659bd3a84bbfcc339ee5c1abe556ef13 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 21:18:02 +0000 Subject: [PATCH 33/41] test: every relation-taking entry point names the suite that asserts its bar, and four that had none get arms here, in both twins (#1386) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/entry_point_privilege.sh | 71 ++++++++++++++++++++++- test/pytest/test_entry_point_privilege.py | 54 ++++++++++++++++- 2 files changed, 119 insertions(+), 6 deletions(-) diff --git a/test/entry_point_privilege.sh b/test/entry_point_privilege.sh index b7a28ea29..b5857ab3d 100755 --- a/test/entry_point_privilege.sh +++ b/test/entry_point_privilege.sh @@ -136,7 +136,10 @@ CATALOG_GOVERNED="$(echo $CATALOG_GOVERNED)" # Relation-taking entry points that are supposed to REFUSE an unprivileged # caller, asserted behaviourally rather than believed. -GUARDED_BEHAVIOURAL="read_projection reconstruct_via_projection vm_is_visible vm_selftest get_storage_id" +# #1386 added the four after get_storage_id: each had no arm anywhere that calls it as +# a caller its bar should stop (cluster_hilbert and recluster_hilbert share cluster's +# owner check in code, which is a reason to expect a refusal, not an assertion of one). +GUARDED_BEHAVIOURAL="read_projection reconstruct_via_projection vm_is_visible vm_selftest get_storage_id cluster_hilbert recluster_hilbert truncate require_caller_select" GUARDED_BEHAVIOURAL="$(echo $GUARDED_BEHAVIOURAL)" # ---- the enumeration, from the catalog ------------------------------------- @@ -277,6 +280,66 @@ check_num "the catalog-governed class is the six from #560" "$catalog_n" "6" # engineer who closes one must come here and remove the name. check_num "no relation-taking C entry point is still ungated" "$pinned_n" "0" +# ---- every claimed-guarded entry point names where its bar is asserted (#1386) ---- +# +# "Claimed-guarded" was the else branch: every relation-taking name that is neither +# catalog-governed nor pinned. A new relation-taking function with no check at all +# landed there and nothing reddened (measured: zz_mutant_leak(regclass) left this +# suite 31/31). So the claim is now written down, name by name, with the suite whose +# arms call it as a caller the bar should stop. A name with no entry is red, and so +# is an entry whose suite does not exist or never calls the function, or that names +# a function this population no longer has. +BAR_ASSERTED_IN=" +add_projection:vacuum_lock_privilege +drop_projection:vacuum_lock_privilege +vacuum:vacuum_lock_privilege +vacuum_sorted:vacuum_lock_privilege +cluster:vacuum_lock_privilege +recluster:vacuum_lock_privilege +compact:vacuum_lock_privilege +compact_rewrite:vacuum_lock_privilege +export_arrow:import_export_privilege +export_parquet:import_export_privilege +import_arrow:import_export_privilege +import_parquet:import_export_privilege +parallel_copy:import_export_privilege +parallel_export_parquet:import_export_privilege +expire:ttl_expire +column_extent:column_extent +maintenance_due:maintenance_due +refresh_rollup:rollup +rebase_rollup:rollup +rollup_source_groups:rollup +rollup_source_retired:rollup +tier_partition:tier_partition +partition_create_ahead:partition_lifecycle +" +# Relation-taking, with no bar to assert, each for its reason: +# partition_drop_candidates, partition_drop_plan return what pg_inherits and the +# partition bounds already tell any role, and the plan is text: the statements +# it lists run as whoever runs them, under core's own privilege checks. +NO_BAR_NEEDED="partition_drop_candidates partition_drop_plan" +bar_home() { awk -v f="$1" -F: '$1 == f { print $2 }' <<<"$BAR_ASSERTED_IN"; } +unhomed="" badhome="" stalehome="" +for fn in $relfns; do + case " $CATALOG_GOVERNED $KNOWN_UNGUARDED $NO_BAR_NEEDED " in *" $fn "*) continue ;; esac + # The behavioural arms below are this suite's own home for the names they call. + case " $GUARDED_BEHAVIOURAL " in *" $fn "*) continue ;; esac + home="$(bar_home "$fn")" + if [ -z "$home" ]; then unhomed="$unhomed $fn"; continue; fi + f="$(dirname "${BASH_SOURCE[0]}")/$home.sh" + [ -r "$f" ] && grep -q "pgcolumnar\.$fn(" "$f" || badhome="$badhome $fn:$home" +done +for fn in $(awk -F: 'NF == 2 { print $1 }' <<<"$BAR_ASSERTED_IN") $NO_BAR_NEEDED; do + case " $relfns " in *" $fn "*) ;; *) stalehome="$stalehome $fn" ;; esac +done +check "every claimed-guarded entry point names the suite that asserts its bar (#1386)" \ + "$([ -z "${unhomed// /}" ] && echo none || echo "$unhomed")" "none" +check "and each named suite exists and calls that entry point" \ + "$([ -z "${badhome// /}" ] && echo none || echo "$badhome")" "none" +check "and no entry names a function that is not a relation-taking entry point" \ + "$([ -z "${stalehome// /}" ] && echo none || echo "$stalehome")" "none" + # THE COUNT ABOVE IS NOT ENOUGH, AND THIS SUITE PROVED IT ON ITSELF. # # When #558 and #562 merged, four of the nine names here became genuinely @@ -353,7 +416,8 @@ call_for() { # call_for -> a valid argument vector for that function case "$1" in read_projection|reconstruct_via_projection) echo "SELECT pgcolumnar.$1('ep_target','p1');" ;; vm_is_visible|vm_selftest) echo "SELECT pgcolumnar.$1('ep_target',0);" ;; - get_storage_id) echo "SELECT pgcolumnar.$1('ep_target');" ;; + get_storage_id|truncate|require_caller_select) echo "SELECT pgcolumnar.$1('ep_target');" ;; + cluster_hilbert|recluster_hilbert) echo "SELECT pgcolumnar.$1('ep_target', 'id');" ;; export_arrow|export_parquet|import_arrow|import_parquet) echo "SELECT pgcolumnar.$1('ep_target','/nonexistent/dir/x');" ;; *) echo "" ;; @@ -386,7 +450,8 @@ done check "premise: t_ep now holds EXECUTE, so a refusal below can only be the C check" \ "$(q "SELECT bool_and(has_function_privilege('t_ep', p.oid, 'EXECUTE')) FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace - WHERE n.nspname='pgcolumnar' AND p.proname IN ('read_projection','reconstruct_via_projection','vm_is_visible','vm_selftest');")" \ + WHERE n.nspname='pgcolumnar' AND p.proname IN ('read_projection','reconstruct_via_projection','vm_is_visible','vm_selftest', + 'cluster_hilbert','recluster_hilbert','truncate','require_caller_select');")" \ "t" for fn in $GUARDED_BEHAVIOURAL; do diff --git a/test/pytest/test_entry_point_privilege.py b/test/pytest/test_entry_point_privilege.py index 19c1210bb..52f251b8d 100644 --- a/test/pytest/test_entry_point_privilege.py +++ b/test/pytest/test_entry_point_privilege.py @@ -49,7 +49,34 @@ "vm_is_visible": "SELECT pgcolumnar.vm_is_visible('ep_target',0)", "vm_selftest": "SELECT pgcolumnar.vm_selftest('ep_target',0)", "get_storage_id": "SELECT pgcolumnar.get_storage_id('ep_target')", + # #1386: no arm anywhere called these as a caller their bar should stop. + "cluster_hilbert": "SELECT pgcolumnar.cluster_hilbert('ep_target', 'id')", + "recluster_hilbert": "SELECT pgcolumnar.recluster_hilbert('ep_target', 'id')", + "truncate": "SELECT pgcolumnar.truncate('ep_target')", + "require_caller_select": "SELECT pgcolumnar.require_caller_select('ep_target')", } +# Where each other claimed-guarded entry point's bar is asserted (#1386): the pytest +# file whose tests call it as a caller its bar should stop. A suite with no pytest twin +# yet is named as such, so the name is still accounted for and a new one is not. +NO_TWIN_YET = "no pytest twin yet: " +BAR_ASSERTED_IN = { + **{f: "test_vacuum_lock_privilege.py" for f in ("add_projection", "drop_projection", "vacuum", + "vacuum_sorted", "cluster", "recluster", + "compact", "compact_rewrite")}, + **{f: "test_import_export_privilege.py" for f in ("export_arrow", "export_parquet", "import_arrow", + "import_parquet", "parallel_copy", + "parallel_export_parquet")}, + "expire": "test_ttl_expire.py", + "column_extent": "test_column_extent.py", + "maintenance_due": NO_TWIN_YET + "maintenance_due.sh", + **{f: "test_rollup.py" for f in ("refresh_rollup", "rebase_rollup", "rollup_source_groups", + "rollup_source_retired")}, + "tier_partition": "test_tier_partition.py", + "partition_create_ahead": "test_partition_lifecycle.py", +} +# Relation-taking, with no bar to assert: they return what pg_inherits and the +# partition bounds already tell any role, and the plan's statements run as their caller. +NO_BAR_NEEDED = {"partition_drop_candidates", "partition_drop_plan"} FILE_SIBLINGS = ("export_arrow", "export_parquet", "import_arrow", "import_parquet") @@ -142,6 +169,24 @@ def test_every_relation_taking_entry_point_has_a_decided_bar(pgc_conn, pgc_clust expect.num(catalog_n, 6, "the catalog-governed class is the six from #560") expect.num(pinned_n, 0, "no relation-taking C entry point is still ungated") + # ---- every claimed-guarded entry point names where its bar is asserted (#1386) + claimed = sorted(relfns - CATALOG_GOVERNED - KNOWN_UNGUARDED - NO_BAR_NEEDED - set(GUARDED_BEHAVIOURAL)) + unhomed = [f for f in claimed if f not in BAR_ASSERTED_IN] + badhome = [] + for f in claimed: + home = BAR_ASSERTED_IN.get(f) + if home is None or home.startswith(NO_TWIN_YET): + continue + path = pathlib.Path(__file__).resolve().parent / home + if not (path.is_file() and f"pgcolumnar.{f}(" in path.read_text()): + badhome.append(f"{f}:{home}") + stale = sorted((set(BAR_ASSERTED_IN) | NO_BAR_NEEDED) - relfns) + expect.text(" ".join(unhomed) or "none", "none", + "every claimed-guarded entry point names the suite that asserts its bar (#1386)") + expect.text(" ".join(badhome) or "none", "none", "and each named suite exists and calls that entry point") + expect.text(" ".join(stale) or "none", "none", + "and no entry names a function that is not a relation-taking entry point") + # ---- the behavioural arms with conn.cursor() as cur: cur.execute("CREATE TABLE ep_target (id int, v text) USING pgcolumnar") @@ -166,12 +211,15 @@ def test_every_relation_taking_entry_point_has_a_decided_bar(pgc_conn, pgc_clust cur.execute(f"GRANT EXECUTE ON FUNCTION {sig} TO {ROLE}") expect.text(str(_one(conn, f"SELECT bool_and(has_function_privilege('{ROLE}', p.oid, 'EXECUTE')) {in_ext}" " AND p.proname IN ('read_projection','reconstruct_via_projection'," - "'vm_is_visible','vm_selftest')")), "True", + "'vm_is_visible','vm_selftest','cluster_hilbert','recluster_hilbert'," + "'truncate','require_caller_select')")), "True", "premise: t_ep now holds EXECUTE, so a refusal below can only be the C check") import psycopg - expect.num(len(GUARDED_BEHAVIOURAL), 5, "premise: the behavioural loop covers the five guarded names") - for fn, sql in sorted(GUARDED_BEHAVIOURAL.items()): + expect.num(len(GUARDED_BEHAVIOURAL), 9, "premise: the behavioural loop covers the nine guarded names") + # In declaration order, as the shell's loop runs: the projection readers before the + # Hilbert rewrites. + for fn, sql in GUARDED_BEHAVIOURAL.items(): try: _one(conn, sql) owner = "ok" From ab57d12766ac0b28990ed848c30de4c73897deda Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 21:21:16 +0000 Subject: [PATCH 34/41] test: the bar map and the four new arms enter the ledger, dated by their mutations (#1386) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 11 +++++++++++ test/check_ledger_budget.txt | 16 +++++++++++++++- test/pytest/TESTS.md | 2 +- 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 5411d8719..e3fb491be 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -734,10 +734,14 @@ encode_post_codec encode_post_codec the tail column holds no row the heap does n encode_post_codec encode_post_codec the tail column preserves its checksum 15;16;17;18;19 2026-10-13 the reader returning row 5 of every row group as NULL encode_post_codec encode_post_codec the tail column reads back exactly what the heap holds 15;16;17;18;19 2026-10-13 the reader returning row 5 of every row group as NULL encode_post_codec encode_post_codec while a column where encoding genuinely wins still encodes 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege and each named suite exists and calls that entry point 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege and no entry names a function that is not a relation-taking entry point 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege cluster_hilbert refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 cluster_verb_validate without its owner check entry_point_privilege entry_point_privilege coverage: inputs == sum(buckets) 15;16;17;18;19 never - entry_point_privilege entry_point_privilege coverage: no entry point is unbucketed 15;16;17;18;19 2026-10-14 the install script gains a pgcolumnar function no bucket names entry_point_privilege entry_point_privilege coverage: relation-taking == pinned + claimed-guarded + catalog-governed 15;16;17;18;19 never - entry_point_privilege entry_point_privilege every catalog-governed name is still a relation-taking entry point 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege every claimed-guarded entry point names the suite that asserts its bar (#1386) 15;16;17;18;19 2026-10-14 the install script gains a relation-taking function with no privilege check entry_point_privilege entry_point_privilege export_arrow refuses a caller with no privilege before touching the file 15;16;17;18;19 never - entry_point_privilege entry_point_privilege export_parquet refuses a caller with no privilege before touching the file 15;16;17;18;19 2026-10-14 export_parquet without its SELECT check entry_point_privilege entry_point_privilege get_storage_id refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 get_storage_id without its SELECT check @@ -754,15 +758,22 @@ entry_point_privilege entry_point_privilege premise: t_ep now holds EXECUTE, so entry_point_privilege entry_point_privilege premise: the extension is installed and C entry points were found 15;16;17;18;19 never - entry_point_privilege entry_point_privilege premise: the install script declares MODULE_PATHNAME symbols at all 15;16;17;18;19 never - entry_point_privilege entry_point_privilege read_projection refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 read_projection without its SELECT check +entry_point_privilege entry_point_privilege recluster_hilbert refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 cluster_verb_validate without its owner check entry_point_privilege entry_point_privilege reconstruct_via_projection refuses an unprivileged caller with 42501 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege require_caller_select refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 require_caller_select checks nothing entry_point_privilege entry_point_privilege the catalog and the install script agree on the symbol set 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the catalog-governed class is the six from #560 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege the owner can call cluster_hilbert, so a refusal below is a refusal 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the owner can call get_storage_id, so a refusal below is a refusal 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the owner can call read_projection, so a refusal below is a refusal 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege the owner can call recluster_hilbert, so a refusal below is a refusal 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the owner can call reconstruct_via_projection, so a refusal below is a refusal 15;16;17;18;19 2026-10-14 reconstruct_via_projection refuses even a superuser owner +entry_point_privilege entry_point_privilege the owner can call require_caller_select, so a refusal below is a refusal 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege the owner can call truncate, so a refusal below is a refusal 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the owner can call vm_is_visible, so a refusal below is a refusal 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the owner can call vm_selftest, so a refusal below is a refusal 15;16;17;18;19 never - entry_point_privilege entry_point_privilege the relation-by-other-means class is the three that are known 15;16;17;18;19 never - +entry_point_privilege entry_point_privilege truncate refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 truncate without its owner check entry_point_privilege entry_point_privilege vm_is_visible refuses an unprivileged caller with 42501 15;16;17;18;19 2026-10-14 vm_is_visible without its owner check entry_point_privilege entry_point_privilege vm_selftest refuses an unprivileged caller with 42501 15;16;17;18;19 never - fdw_plan_privilege fdw_plan_privilege a privileged planner estimates the Iceberg table from its manifests 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index b4449e49a..1054948d7 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2667,4 +2667,18 @@ suites_not_covered 224 # never 2516 + 10 = 2526 # dated 808 + 2 = 810 # 2526 + 810 == 3336 -checks_never_observed_red 2526 +# +# #1386, every relation-taking entry point names the suite that asserts its bar +# (OffgridwithJD): 2526 -> 2532. 11 new entry_point_privilege checks: the three map arms, and +# an owner call and a 42501 arm for each of cluster_hilbert, recluster_hilbert, truncate +# and require_caller_select, which no arm called as a refused caller. Dated: the map arm +# by a relation-taking function with no check (it left the suite 31/31 before), and +# each new 42501 arm by its check removed (truncate's owner check, require_caller_select +# checking nothing, and cluster_verb_validate's owner check for both Hilbert verbs). +# Five-major runs: 42/42 on 15-19. +# +# ledger rows 3336 + 11 = 3347 +# never 2526 + 6 = 2532 +# dated 810 + 5 = 815 +# 2532 + 815 == 3347 +checks_never_observed_red 2532 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 3643b6594..beeab22f1 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6940,7 +6940,7 @@ The pytest twin of `test/entry_point_privilege.sh` (#569, ported for #1341), eve | test | what it holds | | --- | --- | -| `test_every_relation_taking_entry_point_has_a_decided_bar` | the enumeration premises, the symbol-set agreement, the bucket and class counts, the five 42501 refusals with their owner premises, and the four file siblings refused at the table | +| `test_every_relation_taking_entry_point_has_a_decided_bar` | the enumeration premises, the symbol-set agreement, the bucket and class counts, every claimed-guarded entry point named with the pytest file that asserts its bar (#1386), the nine 42501 refusals with their owner calls, and the four file siblings refused at the table | ## 110. test_am_contracts.py: contracts the access method owes core DML and transactions From eec3f3c24949e2d4b0530d22761f872c3427e518 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 20:26:12 +0000 Subject: [PATCH 35/41] test: six rollup checks that read the result of vacuum, compact_rewrite, expire or a refresh lose the premise prefix, in both twins (#1339) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 12 ++++++------ test/pytest/test_rollup.py | 12 ++++++------ test/rollup.sh | 12 ++++++------ 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index e3fb491be..e47738929 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2932,17 +2932,16 @@ rollup rollup another role cannot add a group row to the owner's rollup (42501) rollup rollup another role cannot read the source's groups through rollup_source_groups (42501) 15;16;17;18;19 2026-10-14 rollup_source_groups without its require_caller_select guard rollup rollup another role sees none of its rollup catalog rows 15;16;17;18;19 2026-10-13 the rollup catalogs without row-level security;the rollup catalogs without their grants rollup rollup both rollups over the source refreshed, so each has a view, partials and catalog rows 15;16;17;18;19 never - +rollup rollup compact_rewrite rewrote a row group 15;16;17;18;19 never - rollup rollup each row group's TID range holds exactly that group's rows 15;16;17;18;19 2026-10-13 row_tid mapping 290 rows a block instead of the AM's +rollup rollup expire retired the group in bucket D alone and kept the one that shares it 15;16;17;18;19 never - +rollup rollup expire retired the group the delete thinned 15;16;17;18;19 never - +rollup rollup expire retired the old row groups 15;16;17;18;19 never - rollup rollup nor its retired groups through rollup_source_retired (42501) 15;16;17;18;19 2026-10-14 rollup_source_retired without its require_caller_select guard +rollup rollup pgcolumnar.vacuum gave the source's row groups new identities 15;16;17;18;19 never - rollup rollup premise: and the fixture has NULL-key and NULL-bucket groups 15;16;17;18;19 never - -rollup rollup premise: compact_rewrite rewrote a row group 15;16;17;18;19 never - -rollup rollup premise: expire retired the group in bucket D alone and kept the one that shares it 15;16;17;18;19 never - -rollup rollup premise: expire retired the group the delete thinned 15;16;17;18;19 never - -rollup rollup premise: expire retired the old row groups 15;16;17;18;19 never - -rollup rollup premise: pgcolumnar.vacuum gave the source's row groups new identities 15;16;17;18;19 never - rollup rollup premise: the owner's rollup has group rows another role could reach 15;16;17;18;19 never - rollup rollup premise: the rollup functions exist 15;16;17;18;19 never - -rollup rollup premise: the rollup holds partials the delete below cannot reach 15;16;17;18;19 never - rollup rollup premise: the source holds every row 15;16;17;18;19 never - rollup rollup rebase_rollup of a table that is not a rollup is refused (42704) 15;16;17;18;19 2026-10-14 rebase_rollup without its not-a-rollup check rollup rollup rebase_rollup records the source's current row groups 15;16;17;18;19 2026-10-13 rebase_rollup returning 0 @@ -2960,6 +2959,7 @@ rollup rollup the paused refresh folded what was there before it began 15;16;17; rollup rollup the refresh after a small append read the source at all 15;16;17;18;19 never - rollup rollup the restored rollup reads as a recompute of the restored source 15;16;17;18;19 never - rollup rollup the restored source holds the dumped rows under the storage id recorded for the old one 15;16;17;18;19 never - +rollup rollup the rollup holds partials the delete below cannot reach 15;16;17;18;19 never - rollup rollup the rollup holds rows before the rewrite 15;16;17;18;19 never - rollup rollup the rollup table exists with the documented columns 15;16;17;18;19 never - rollup rollup the rollup to be dumped folds its source 15;16;17;18;19 never - diff --git a/test/pytest/test_rollup.py b/test/pytest/test_rollup.py index d97783416..aedc36fb6 100644 --- a/test/pytest/test_rollup.py +++ b/test/pytest/test_rollup.py @@ -194,7 +194,7 @@ def identity(): with conn.cursor() as cur: cur.execute("SELECT pgcolumnar.vacuum('feed')") expect.text("renumbered" if identity() != before else "same", "renumbered", - "premise: pgcolumnar.vacuum gave the source's row groups new identities") + "pgcolumnar.vacuum gave the source's row groups new identities") vb = _matches(conn) expect.at_least(_one(conn, "SELECT count(*) FROM gauge_15m"), 1, "the rollup holds rows before the rewrite") expect.text(f"{_refresh(conn)}|{_matches(conn)}", f"55000|{vb}", @@ -219,7 +219,7 @@ def identity(): f" FROM gauge_15m__partials WHERE {far}") far_before = _one(conn, at_far) expect.at_least(_one(conn, f"SELECT count(*) FROM gauge_15m__partials WHERE {far}"), 1, - "premise: the rollup holds partials the delete below cannot reach") + "the rollup holds partials the delete below cannot reach") with conn.cursor() as cur: cur.execute("DELETE FROM feed WHERE sensor = 4 AND at < '2026-02-01 01:00:00+00'") expect.text(_folds(conn), "folded", @@ -253,7 +253,7 @@ def identity(): rewrote = _one(conn, "SELECT pgcolumnar.compact_rewrite('feed', 0.02)") except psycopg.Error: rewrote = -1 - expect.at_least(rewrote, 1, "premise: compact_rewrite rewrote a row group") + expect.at_least(rewrote, 1, "compact_rewrite rewrote a row group") _refresh(conn) expect.text(_matches(conn), _oracle(conn), "after compact_rewrite the rollup equals a full recompute") expect.text(_refresh(conn), "0", "and the refresh after the rewrite recomputes nothing") @@ -316,7 +316,7 @@ def identity(): expect.at_least(old_buckets, 1, "the history rollup holds old buckets") with conn.cursor() as cur: cur.execute("SELECT pgcolumnar.set_options('aged', ttl_column => 'at', ttl_interval => '14 days')") - expect.at_least(_one(conn, "SELECT pgcolumnar.expire('aged')"), 1, "premise: expire retired the old row groups") + expect.at_least(_one(conn, "SELECT pgcolumnar.expire('aged')"), 1, "expire retired the old row groups") expect.text(f"{_refresh(conn, 'aged_hourly')}|{_one(conn, hist)}", f"0|{before_hist}", "a refresh after retention folds nothing, refuses nothing, and keeps every bucket") expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.rollup_group rg JOIN pgcolumnar.retired_group t" @@ -361,7 +361,7 @@ def layer(table): expect.text(_one(conn, f"SELECT count(*) FILTER (WHERE at < {h} + interval '40 min') || '|'" f" || count(*) FILTER (WHERE at >= {h} + interval '40 min' AND at < {h} + interval '1 hour')" " FROM layered"), "0|1", - "premise: expire retired the group in bucket D alone and kept the one that shares it") + "expire retired the group in bucket D alone and kept the one that shares it") lq = "SELECT md5(string_agg(bucket || ':' || count || ':' || max_reading, ',' ORDER BY bucket)) FROM layered_hourly" lr = (f"SELECT md5(string_agg(b || ':' || n || ':' || m, ',' ORDER BY b)) FROM (SELECT date_bin('1 hour', at," f" '{ORIGIN}') b, count(*) n, max(reading) m FROM layered GROUP BY 1) x") @@ -403,7 +403,7 @@ def layer(table): cur.execute("DELETE FROM thinned WHERE at < now() - interval '14 days' AND reading % 5 = 0") cur.execute("SELECT pgcolumnar.set_options('thinned', ttl_column => 'at', ttl_interval => '14 days')") expect.num(_one(conn, "SELECT pgcolumnar.expire('thinned')"), 1, - "premise: expire retired the group the delete thinned") + "expire retired the group the delete thinned") tq = "SELECT string_agg(bucket || ':' || count || ':' || max_reading, ',' ORDER BY bucket) FROM thinned_hourly" before_thin = _one(conn, tq) expect.text(f"{_refresh(conn, 'thinned_hourly')}|{_one(conn, tq)}", f"0A000|{before_thin}", diff --git a/test/rollup.sh b/test/rollup.sh index da9db2510..2404a6108 100755 --- a/test/rollup.sh +++ b/test/rollup.sh @@ -167,7 +167,7 @@ groups_before="$(q "SELECT pgcolumnar.get_storage_id('src') || ':' || string_agg q "SELECT pgcolumnar.vacuum('src');" >/dev/null groups_after="$(q "SELECT pgcolumnar.get_storage_id('src') || ':' || string_agg(group_number::text, ',' ORDER BY group_number) FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('src');")" -check "premise: pgcolumnar.vacuum gave the source's row groups new identities" \ +check "pgcolumnar.vacuum gave the source's row groups new identities" \ "$([ -n "$groups_after" ] && [ "$groups_after" != "$groups_before" ] && echo renumbered || echo "same: $groups_after")" "renumbered" vb="$(exact "SELECT * FROM r_hourly")" check_num "the rollup holds rows before the rewrite" "$(at_least "$(q "SELECT count(*) FROM r_hourly;")" 1)" "1" @@ -191,7 +191,7 @@ check "and after the rebase the rollup equals a full recompute" "$(matches)" "$( partials_at() { q "SELECT count(*) || '|' || md5(string_agg(ctid::text, ',' ORDER BY ctid)) FROM r_hourly__partials WHERE $1;"; } FAR="bucket >= '2026-03-05 00:00:00+00'" # beyond the time range of every group the delete below reaches far_before="$(partials_at "$FAR")" -check_num "premise: the rollup holds partials the delete below cannot reach" \ +check_num "the rollup holds partials the delete below cannot reach" \ "$(at_least "$(q "SELECT count(*) FROM r_hourly__partials WHERE $FAR;")" 1)" "1" q "DELETE FROM src WHERE dev = 3 AND ts < '2026-03-01 06:00:00+00';" >/dev/null check "a refresh after a delete is not refused" \ @@ -218,7 +218,7 @@ check "after a delete and an append that straddles its bucket, a refresh counts # compact_rewrite drops a group's dead rows by rewriting the group. q "DELETE FROM src WHERE ts >= '2026-03-06 00:00:00+00' AND ts < '2026-03-06 12:00:00+00';" >/dev/null -check_num "premise: compact_rewrite rewrote a row group" \ +check_num "compact_rewrite rewrote a row group" \ "$(at_least "$(q "SELECT pgcolumnar.compact_rewrite('src', 0.01);")" 1)" "1" q "SELECT pgcolumnar.refresh_rollup('r_hourly');" >/dev/null check "after compact_rewrite the rollup equals a full recompute" "$(matches)" "$(oracle)" @@ -278,7 +278,7 @@ ob="$(q "SELECT count(*) FROM r_hist WHERE bucket < now() - interval '100 days'; check_num "the history rollup holds old buckets" "$(at_least "${ob:-0}" 1)" "1" q "SELECT pgcolumnar.set_options('hist', ttl_column => 'ts', ttl_interval => '30 days');" >/dev/null retired="$(q "SELECT pgcolumnar.expire('hist');")" -check_num "premise: expire retired the old row groups" "$(at_least "${retired:-0}" 1)" "1" +check_num "expire retired the old row groups" "$(at_least "${retired:-0}" 1)" "1" check "a refresh after retention folds nothing, refuses nothing, and keeps every bucket" \ "$(sqlstate_or "SELECT pgcolumnar.refresh_rollup('r_hist');")|$(q "SELECT md5(string_agg(bucket || ':' || count || ':' || sum_v, ',' ORDER BY bucket)) FROM r_hist;")" \ "0|$hb" @@ -310,7 +310,7 @@ layered() { # layered TABLE: the three groups, a daily rollup r_TABLE refreshed q "SELECT pgcolumnar.expire('$1');" >/dev/null } layered hd -check "premise: expire retired the group in bucket D alone and kept the one that shares it" \ +check "expire retired the group in bucket D alone and kept the one that shares it" \ "$(q "SELECT count(*) FILTER (WHERE ts < $D + interval '12 hours') || '|' || count(*) FILTER (WHERE ts >= $D + interval '12 hours' AND ts < $D + interval '1 day') FROM hd;")" "0|47" HDQ="SELECT md5(string_agg(bucket || ':' || count || ':' || sum_v, ',' ORDER BY bucket)) FROM r_hd" HDR="SELECT md5(string_agg(b || ':' || n || ':' || s, ',' ORDER BY b)) FROM (SELECT date_bin('1 day', ts, '$ORIGIN') b, count(*) n, sum(v)::numeric s FROM hd GROUP BY 1) x" @@ -343,7 +343,7 @@ q "CREATE TABLE dx (ts timestamptz, v int) USING pgcolumnar; SELECT pgcolumnar.refresh_rollup('r_dx'); DELETE FROM dx WHERE ts < now() - interval '30 days' AND v <= 400; SELECT pgcolumnar.set_options('dx', ttl_column => 'ts', ttl_interval => '30 days');" >/dev/null -check_num "premise: expire retired the group the delete thinned" "$(q "SELECT pgcolumnar.expire('dx');")" "1" +check_num "expire retired the group the delete thinned" "$(q "SELECT pgcolumnar.expire('dx');")" "1" dx_before="$(q "SELECT string_agg(bucket || ':' || count, ',' ORDER BY bucket) FROM r_dx;")" check "a delete in a group that expire then retires is refused, and the rollup is unchanged" \ "$(sqlstate "SELECT pgcolumnar.refresh_rollup('r_dx');")|$(q "SELECT string_agg(bucket || ':' || count, ',' ORDER BY bucket) FROM r_dx;")" \ From 1526d8ebaeda0044f2b0d9aea1b919e5e1e9de4a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 20:33:03 +0000 Subject: [PATCH 36/41] test: five of the six rollup checks are dated by vacuum, compact_rewrite and expire mutations (#1339) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 18 +++++++++--------- test/check_ledger_budget.txt | 16 +++++++++++++++- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index e47738929..11961a204 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2864,16 +2864,16 @@ rls_direct_storage rls_direct_storage without FORCE the owner is allowed, becaus rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function rollup rollup DROP TABLE of a rollup's source is refused while the rollup exists (2BP01, #1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) rollup rollup DROP VIEW of a rollup takes its partials and its catalog rows with it (#1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) -rollup rollup a delete in a bucket that holds retired history is refused, and the rollup is unchanged 15;16;17;18;19 2026-10-13 refresh_rollup not storing the retired groups' ranges as history;refresh_rollup recording new groups without their time range;refresh_rollup without the history refusal -rollup rollup a delete in a bucket whose history the same refresh retires is refused, and the rollup is unchanged 15;16;17;18;19 2026-10-13 refresh_rollup blocking on stored history only, not on groups the same refresh retires;refresh_rollup recording new groups without their time range;refresh_rollup without the history refusal -rollup rollup a delete in a group that expire then retires is refused, and the rollup is unchanged 15;16;17;18;19 2026-10-13 expire recording a retired group's deleted count as 0;refresh_rollup not refusing a retiring group whose deleted count moved +rollup rollup a delete in a bucket that holds retired history is refused, and the rollup is unchanged 15;16;17;18;19 2026-10-14 expire retires nothing;refresh_rollup not storing the retired groups' ranges as history;refresh_rollup recording new groups without their time range;refresh_rollup without the history refusal +rollup rollup a delete in a bucket whose history the same refresh retires is refused, and the rollup is unchanged 15;16;17;18;19 2026-10-14 expire retires nothing;refresh_rollup blocking on stored history only, not on groups the same refresh retires;refresh_rollup recording new groups without their time range;refresh_rollup without the history refusal +rollup rollup a delete in a group that expire then retires is refused, and the rollup is unchanged 15;16;17;18;19 2026-10-14 expire recording a retired group's deleted count as 0;expire retires nothing;refresh_rollup not refusing a retiring group whose deleted count moved rollup rollup a delete in buckets with no history, next to retired ones, folds: those buckets equal a recompute and D keeps its history 15;16;17;18;19 2026-10-13 refresh_rollup recomputing every bucket once any group changed;refresh_rollup recording new groups without their time range;refresh_rollup refusing any change but appends (phase 1, main before this branch) rollup rollup a group_by column the source does not have is refused 15;16;17;18;19 2026-10-14 create_rollup without its group_by column check rollup rollup a refresh after a delete is not refused 15;16;17;18;19 2026-10-13 refresh_rollup refusing any change but appends (phase 1, main before this branch) rollup rollup a refresh after a small append reads under a fifth of the source 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup's per-group subquery without OFFSET 0, so it becomes a join and reads every group rollup rollup a refresh after appends folds exactly the appended rows, late ones included 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's rollup rollup a refresh after retention folds nothing, refuses nothing, and keeps every bucket 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup not dropping groups retention retired -rollup rollup a refresh after the source was rewritten is refused, naming rebase_rollup 15;16;17;18;19 2026-10-13 refresh_rollup without the source identity check +rollup rollup a refresh after the source was rewritten is refused, naming rebase_rollup 15;16;17;18;19 2026-10-14 pgcolumnar.vacuum rewrites nothing;refresh_rollup without the source identity check rollup rollup a refresh at REPEATABLE READ is refused 15;16;17;18;19 2026-10-13 refresh_rollup at any isolation level rollup rollup a refresh folds a bulk append 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's rollup rollup a refresh in a zone with a DST change recomputes a whole day bucket 15;16;17;18;19 2026-10-13 refresh_rollup adding the width in the session's zone;refresh_rollup recomputing every bucket once any group changed;refresh_rollup recording new groups without their time range;refresh_rollup refusing any change but appends (phase 1, main before this branch) @@ -2932,13 +2932,13 @@ rollup rollup another role cannot add a group row to the owner's rollup (42501) rollup rollup another role cannot read the source's groups through rollup_source_groups (42501) 15;16;17;18;19 2026-10-14 rollup_source_groups without its require_caller_select guard rollup rollup another role sees none of its rollup catalog rows 15;16;17;18;19 2026-10-13 the rollup catalogs without row-level security;the rollup catalogs without their grants rollup rollup both rollups over the source refreshed, so each has a view, partials and catalog rows 15;16;17;18;19 never - -rollup rollup compact_rewrite rewrote a row group 15;16;17;18;19 never - +rollup rollup compact_rewrite rewrote a row group 15;16;17;18;19 2026-10-14 compact_rewrite reports 0 whatever it rewrote rollup rollup each row group's TID range holds exactly that group's rows 15;16;17;18;19 2026-10-13 row_tid mapping 290 rows a block instead of the AM's -rollup rollup expire retired the group in bucket D alone and kept the one that shares it 15;16;17;18;19 never - -rollup rollup expire retired the group the delete thinned 15;16;17;18;19 never - -rollup rollup expire retired the old row groups 15;16;17;18;19 never - +rollup rollup expire retired the group in bucket D alone and kept the one that shares it 15;16;17;18;19 2026-10-14 expire retires nothing +rollup rollup expire retired the group the delete thinned 15;16;17;18;19 2026-10-14 expire reports 0 whatever it retired;expire retires nothing +rollup rollup expire retired the old row groups 15;16;17;18;19 2026-10-14 expire reports 0 whatever it retired;expire retires nothing rollup rollup nor its retired groups through rollup_source_retired (42501) 15;16;17;18;19 2026-10-14 rollup_source_retired without its require_caller_select guard -rollup rollup pgcolumnar.vacuum gave the source's row groups new identities 15;16;17;18;19 never - +rollup rollup pgcolumnar.vacuum gave the source's row groups new identities 15;16;17;18;19 2026-10-14 pgcolumnar.vacuum rewrites nothing rollup rollup premise: and the fixture has NULL-key and NULL-bucket groups 15;16;17;18;19 never - rollup rollup premise: the owner's rollup has group rows another role could reach 15;16;17;18;19 never - rollup rollup premise: the rollup functions exist 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 1054948d7..d5de2922e 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2681,4 +2681,18 @@ suites_not_covered 224 # never 2526 + 6 = 2532 # dated 810 + 5 = 815 # 2532 + 815 == 3347 -checks_never_observed_red 2532 +# +# #1339, six rollup premises that read a product operation's result (OffgridwithJD, from +# jdatcmd's #1372 review): 2532 -> 2527. They lose the prefix, renamed with their rows. +# Five are dated: vacuum rewriting nothing (the renumbering check), compact_rewrite and +# expire reporting 0 (their checks), and expire retiring nothing (the bucket-D check, +# with the two expire counts and three refusal arms beside it). "the rollup holds +# partials the delete below cannot reach" arrives never: only a refresh that writes no +# partials reddens it, and that reddens most of the suite. Five-major runs: 82/82, and +# 105/105 on PG18 after the rebase onto d5c0a5e, which brought in 23 checks. +# +# ledger rows 3347 = 3347 +# never 2532 - 5 = 2527 +# dated 815 + 5 = 820 +# 2527 + 820 == 3347 +checks_never_observed_red 2527 From eaf10204bc82eb538235a5b5fad037c954d5763c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:00:55 +0000 Subject: [PATCH 37/41] test: a rollup's min and max keep each source column's collation across a refresh, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 1 + test/pytest/expected_tests.txt | 4 +++- test/pytest/test_rollup.py | 34 ++++++++++++++++++++++++++++++++++ test/rollup.sh | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 70 insertions(+), 1 deletion(-) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index beeab22f1..52480759f 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6735,6 +6735,7 @@ Phase 1 removal proofs, each red on its arm: no `OFFSET 0` (the cost arm), no is | `test_rollup_phase_one` | the rollup equals a full recompute after the first refresh, appends with late rows, a bulk append, and a rebase; a refresh with nothing new folds nothing; rows committed during a refresh are folded by the next, once; two concurrent refreshes fold each row once; zones agree; a refresh after a small append reads under a fifth of the source; a rewritten source is refused until `rebase_rollup`; after a delete, an update, a straddling append, `compact_rewrite` or a deleted NULL-ts row the rollup equals a full recompute, rewriting no partial outside the changed groups' range; a recompute that would reach retained history is refused with `0A000` and changes nothing; a day bucket is recomputed whole across a DST switch; a heap rollup behaves the same; REPEATABLE READ is refused; an owner who is not a superuser creates and refreshes a rollup, and another role sees none of it; bad definitions are refused with `22023`; each group's TID range holds exactly its rows | | `test_a_rollup_and_its_source_are_dropped_together` | the view depends on its source, as a view over it would (#1376): `DROP TABLE` of the source is refused (`2BP01`), `CASCADE` takes the view, its partials table and its catalog rows, and `DROP VIEW` takes the partials and the rows. A role that cannot read a table cannot tie a rollup row to it (`42501`), so it cannot block that table's drop. Nor can it name another role's table as its partials (`42501`, and that table survives its view's drop), or a non-view as its rollup (`42809`); and a written row's view, partials and source cannot be changed (`0A000`) | | `test_rollup_dump_and_restore` | a dump carries a rollup's definition; after a restore into another database, and into the same one after DROP EXTENSION, the restored source has the recorded storage id, and the owner's refresh is refused with `55000` naming `rebase_rollup`, the rollup untouched; after the rebase a refresh folds nothing, then exactly the rows appended since, and the rollup equals a full recompute (#1377) | +| `test_min_and_max_keep_each_columns_collation` | #1341: min and max over a `"C"` column and an en_US column equal a recompute after a refresh that folds new rows into an existing bucket, where the two collations disagree; the partials columns carry the default collation, so the fold's own comparison is what is tested. No en_US collation: `cannot_run` | | `test_row_tid_refuses_a_row_number_below_one` | `row_tid(0)` and `row_tid(-1)` are refused with `22003` and `row_tid(1)` is accepted. A test of its own, last in the file: without the floor `row_tid(0)` crashed the backend | ## 98. test_timeseries_recipes.py: the gap-fill, LOCF and interpolation recipes answer what they claim diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 74101bb4b..786ac5aa1 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -719,4 +719,6 @@ guard_tests 438 # test_projections.py, by collection over the cluster files on this tree. # cluster_tests 597 -> 598 for #1341's test_a_named_column_that_does_not_exist_is_refused in # test_analyze_function.py, by collection over the cluster files on this tree. -cluster_tests 598 +# cluster_tests 598 -> 599 for #1341's test_min_and_max_keep_each_columns_collation in +# test_rollup.py, by collection over the cluster files on this tree. +cluster_tests 599 diff --git a/test/pytest/test_rollup.py b/test/pytest/test_rollup.py index aedc36fb6..791972564 100644 --- a/test/pytest/test_rollup.py +++ b/test/pytest/test_rollup.py @@ -781,6 +781,40 @@ def restore(db): admin("postgres", f"DROP DATABASE IF EXISTS {db} WITH (FORCE)") +def test_min_and_max_keep_each_columns_collation(pgc_own_db, expect): + """The partials columns carry the database's default collation, and a refresh + that updates a bucket folds with least()/greatest(), which must compare in the + SOURCE column's collation (#1341). One "C" column and one en_US column, so + whichever is the default, the other differs from it. Lower-case rows first, + then upper-case into the same bucket: under C 'B' sorts before 'a', under en_US + after it, and only the second refresh's fold can get that wrong.""" + conn = pgc_own_db + coll = _one(conn, "SELECT collname FROM pg_collation WHERE collname IN ('en_US.utf8', 'en_US.UTF-8', 'en_US')" + " ORDER BY collname LIMIT 1") + if coll is None: + expect.cannot_run("MISSING_DEPENDENCY", "no en_US collation", + name="the server has no en_US collation") + return + with conn.cursor() as cur: + cur.execute(f'CREATE TABLE ct (ts timestamptz, sc text COLLATE "C", su text COLLATE "{coll}") USING pgcolumnar') + cur.execute("SELECT pgcolumnar.set_options('ct', stripe_row_limit => 1000)") + cur.execute(f"INSERT INTO ct SELECT '{ORIGIN}'::timestamptz + (g % 24) * interval '1 hour', 'a'||g, 'a'||g" + " FROM generate_series(1, 2000) g") + cur.execute(f"SELECT pgcolumnar.create_rollup('r_ct', 'ct', 'ts', '1 day', '{ORIGIN}', '{{}}'," + " '{min,max,min,max}', '{sc,sc,su,su}')") + cur.execute("SELECT pgcolumnar.refresh_rollup('r_ct')") + cur.execute(f"INSERT INTO ct SELECT '{ORIGIN}'::timestamptz + (g % 24) * interval '1 hour', 'B'||g, 'B'||g" + " FROM generate_series(1, 2000) g") + folded = _one(conn, "SELECT pgcolumnar.refresh_rollup('r_ct')") + expect.text(f"{folded}|{_one(conn, 'SELECT count(*) FROM r_ct')}", "2000|1", + "premise: the second refresh folds new rows into the existing bucket") + expect.text(_one(conn, "SELECT min_sc || '|' || max_sc || '|' || min_su || '|' || max_su FROM r_ct"), + _one(conn, "SELECT min(sc) || '|' || max(sc) || '|' || min(su) || '|' || max(su) FROM ct"), + "min and max over a C column and an en_US column keep each column's own order across a refresh") + expect.text(str(_one(conn, "SELECT min(sc) <> min(su) FROM ct")), "True", + "premise: the two collations disagree on these rows, so the arm can tell them apart") + + def test_row_tid_refuses_a_row_number_below_one(pgc_conn, expect): """Row numbers start at 1 (#1341). LAST IN THE FILE, AND A TEST OF ITS OWN: without the floor, row_tid(0) crashed the backend (measured on PG17), which would take the diff --git a/test/rollup.sh b/test/rollup.sh index 2404a6108..ec7f70acf 100755 --- a/test/rollup.sh +++ b/test/rollup.sh @@ -622,6 +622,38 @@ psql_run "CREATE TABLE lr_p5 (bucket timestamptz) USING pgcolumnar; CREATE VIEW check "a rollup row's view, partials and source cannot be changed once written (0A000, #1376)" \ "$(sqlstate "UPDATE pgcolumnar.rollup SET partials = 'lr_p6' WHERE rollup = 'lr_v5'::regclass")" "0A000" +# ---- min and max keep the source column's collation (#1341) ------------------ +# +# The partials columns are created with the type alone, so they carry the +# database's default collation. A refresh that updates a bucket folds the new +# partial into the stored one with least()/greatest(), and that comparison must be +# the source column's. Two text columns, "C" and en_US, so whichever is the +# database default, the other differs from it. Lower-case rows first, then +# upper-case rows into the SAME buckets: under C 'B' sorts before 'a', under en_US +# after it, and only the second refresh's fold can get that wrong. +CT_COLL="$(q "SELECT collname FROM pg_collation WHERE collname IN ('en_US.utf8', 'en_US.UTF-8', 'en_US') ORDER BY collname LIMIT 1;")" +if [ -n "$CT_COLL" ]; then + q "CREATE TABLE ct (ts timestamptz, sc text COLLATE \"C\", su text COLLATE \"$CT_COLL\") USING pgcolumnar; + SELECT pgcolumnar.set_options('ct', stripe_row_limit => 1000); + INSERT INTO ct SELECT '$ORIGIN'::timestamptz + (g % 24) * interval '1 hour', 'a'||g, 'a'||g FROM generate_series(1, 2000) g;" >/dev/null + q "SELECT pgcolumnar.create_rollup('r_ct', 'ct', 'ts', '1 day', '$ORIGIN', '{}', '{min,max,min,max}', '{sc,sc,su,su}');" >/dev/null + q "SELECT pgcolumnar.refresh_rollup('r_ct');" >/dev/null + q "INSERT INTO ct SELECT '$ORIGIN'::timestamptz + (g % 24) * interval '1 hour', 'B'||g, 'B'||g FROM generate_series(1, 2000) g;" >/dev/null + check "premise: the second refresh folds new rows into the existing bucket" \ + "$(q "SELECT pgcolumnar.refresh_rollup('r_ct');")|$(q "SELECT count(*) FROM r_ct;")" "2000|1" + check "min and max over a C column and an en_US column keep each column's own order across a refresh" \ + "$(q "SELECT min_sc || '|' || max_sc || '|' || min_su || '|' || max_su FROM r_ct;")" \ + "$(q "SELECT min(sc) || '|' || max(sc) || '|' || min(su) || '|' || max(su) FROM ct;")" + check "premise: the two collations disagree on these rows, so the arm can tell them apart" \ + "$(q "SELECT (min(sc) <> min(su))::text FROM ct;")" "true" +else + for _n in "premise: the second refresh folds new rows into the existing bucket" \ + "min and max over a C column and an en_US column keep each column's own order across a refresh" \ + "premise: the two collations disagree on these rows, so the arm can tell them apart"; do + check_skip "$_n" "SKIP $_n (no en_US collation)" "the server has no en_US collation" + done +fi + # Row numbers start at 1 (#1341). LAST IN THE SUITE ON PURPOSE: without the floor, # row_tid(0) crashed the backend (measured on PG17), and a crash mid-suite takes the # checks after it down with it, premises included. From 0aa42e7075736ecf0f47c67e411a8975d1675ca6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:02:24 +0000 Subject: [PATCH 38/41] test: the collation premise compares both minimums in C, since the columns' collations differ (#1341) Co-Authored-By: Claude Opus 5.5 --- test/pytest/test_rollup.py | 4 +++- test/rollup.sh | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/test/pytest/test_rollup.py b/test/pytest/test_rollup.py index 791972564..fcd8828a7 100644 --- a/test/pytest/test_rollup.py +++ b/test/pytest/test_rollup.py @@ -811,7 +811,9 @@ def test_min_and_max_keep_each_columns_collation(pgc_own_db, expect): expect.text(_one(conn, "SELECT min_sc || '|' || max_sc || '|' || min_su || '|' || max_su FROM r_ct"), _one(conn, "SELECT min(sc) || '|' || max(sc) || '|' || min(su) || '|' || max(su) FROM ct"), "min and max over a C column and an en_US column keep each column's own order across a refresh") - expect.text(str(_one(conn, "SELECT min(sc) <> min(su) FROM ct")), "True", + # Both sides in "C": the two columns' collations differ, so a bare comparison + # has no collation to use. Equality does not depend on which one is chosen. + expect.text(str(_one(conn, 'SELECT min(sc) COLLATE "C" <> min(su) COLLATE "C" FROM ct')), "True", "premise: the two collations disagree on these rows, so the arm can tell them apart") diff --git a/test/rollup.sh b/test/rollup.sh index ec7f70acf..e220a7cf2 100755 --- a/test/rollup.sh +++ b/test/rollup.sh @@ -645,7 +645,7 @@ if [ -n "$CT_COLL" ]; then "$(q "SELECT min_sc || '|' || max_sc || '|' || min_su || '|' || max_su FROM r_ct;")" \ "$(q "SELECT min(sc) || '|' || max(sc) || '|' || min(su) || '|' || max(su) FROM ct;")" check "premise: the two collations disagree on these rows, so the arm can tell them apart" \ - "$(q "SELECT (min(sc) <> min(su))::text FROM ct;")" "true" + "$(q "SELECT (min(sc) COLLATE \"C\" <> min(su) COLLATE \"C\")::text FROM ct;")" "true" else for _n in "premise: the second refresh folds new rows into the existing bucket" \ "min and max over a C column and an en_US column keep each column's own order across a refresh" \ From 66e96a1e2915e0af852482e94d02451ea0878c51 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 02:29:24 +0000 Subject: [PATCH 39/41] test: ledger rows for the rollup collation arms (#1341) Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 3 +++ test/check_ledger_budget.txt | 13 ++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 11961a204..f141d707d 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2937,12 +2937,15 @@ rollup rollup each row group's TID range holds exactly that group's rows 15;16;1 rollup rollup expire retired the group in bucket D alone and kept the one that shares it 15;16;17;18;19 2026-10-14 expire retires nothing rollup rollup expire retired the group the delete thinned 15;16;17;18;19 2026-10-14 expire reports 0 whatever it retired;expire retires nothing rollup rollup expire retired the old row groups 15;16;17;18;19 2026-10-14 expire reports 0 whatever it retired;expire retires nothing +rollup rollup min and max over a C column and an en_US column keep each column's own order across a refresh 15;16;17;18;19 2026-10-14 the rollup fold of min and max compares in the partials' default collation rollup rollup nor its retired groups through rollup_source_retired (42501) 15;16;17;18;19 2026-10-14 rollup_source_retired without its require_caller_select guard rollup rollup pgcolumnar.vacuum gave the source's row groups new identities 15;16;17;18;19 2026-10-14 pgcolumnar.vacuum rewrites nothing rollup rollup premise: and the fixture has NULL-key and NULL-bucket groups 15;16;17;18;19 never - rollup rollup premise: the owner's rollup has group rows another role could reach 15;16;17;18;19 never - rollup rollup premise: the rollup functions exist 15;16;17;18;19 never - +rollup rollup premise: the second refresh folds new rows into the existing bucket 15;16;17;18;19 never - rollup rollup premise: the source holds every row 15;16;17;18;19 never - +rollup rollup premise: the two collations disagree on these rows, so the arm can tell them apart 15;16;17;18;19 never - rollup rollup rebase_rollup of a table that is not a rollup is refused (42704) 15;16;17;18;19 2026-10-14 rebase_rollup without its not-a-rollup check rollup rollup rebase_rollup records the source's current row groups 15;16;17;18;19 2026-10-13 rebase_rollup returning 0 rollup rollup refreshes from sessions in different time zones build the same rollup 15;16;17;18;19 2026-10-13 refresh_rollup bucketing with date_trunc, which follows the session's zone diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index d5de2922e..6ee71cac9 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2695,4 +2695,15 @@ suites_not_covered 224 # never 2532 - 5 = 2527 # dated 815 + 5 = 820 # 2527 + 820 == 3347 -checks_never_observed_red 2527 +# +# #1341, a rollup's min and max keep each source column's collation (OffgridwithJD): +# 2527 -> 2529. Three new rollup checks. The arm is dated by the fold comparing in +# the partials' default collation (on an en_US database the C column's min became +# a1 where the recompute gives B1). Its two premises arrive never. Five-major runs +# on the files merged with #1359: rollup 85/85 on 15-19. +# +# ledger rows 3347 + 3 = 3350 +# never 2527 + 2 = 2529 +# dated 820 + 1 = 821 +# 2529 + 821 == 3350 +checks_never_observed_red 2529 From a22277caa220e6c1270eda3082512bea879fc6ff Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 19:58:27 +0000 Subject: [PATCH 40/41] test: the rollup refresh-count check loses the premise prefix, in both twins and the skip list (#1372 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 2 +- test/pytest/test_rollup.py | 2 +- test/rollup.sh | 6 ++++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index f141d707d..381d12c9a 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2943,7 +2943,6 @@ rollup rollup pgcolumnar.vacuum gave the source's row groups new identities 15;1 rollup rollup premise: and the fixture has NULL-key and NULL-bucket groups 15;16;17;18;19 never - rollup rollup premise: the owner's rollup has group rows another role could reach 15;16;17;18;19 never - rollup rollup premise: the rollup functions exist 15;16;17;18;19 never - -rollup rollup premise: the second refresh folds new rows into the existing bucket 15;16;17;18;19 never - rollup rollup premise: the source holds every row 15;16;17;18;19 never - rollup rollup premise: the two collations disagree on these rows, so the arm can tell them apart 15;16;17;18;19 never - rollup rollup rebase_rollup of a table that is not a rollup is refused (42704) 15;16;17;18;19 2026-10-14 rebase_rollup without its not-a-rollup check @@ -2966,6 +2965,7 @@ rollup rollup the rollup holds partials the delete below cannot reach 15;16;17;1 rollup rollup the rollup holds rows before the rewrite 15;16;17;18;19 never - rollup rollup the rollup table exists with the documented columns 15;16;17;18;19 never - rollup rollup the rollup to be dumped folds its source 15;16;17;18;19 never - +rollup rollup the second refresh folds new rows into the existing bucket 15;16;17;18;19 never - rollup rollup the second refresh of the NULL-key rollup folded the append 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's rollup rollup the zone rollups hold a row per key of the recompute 15;16;17;18;19 2026-10-13 refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's rollup rollup two refreshes started together fold the new rows once between them 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup without the self-conflicting lock;row_tid mapping 290 rows a block instead of the AM's diff --git a/test/pytest/test_rollup.py b/test/pytest/test_rollup.py index fcd8828a7..7ad7858ec 100644 --- a/test/pytest/test_rollup.py +++ b/test/pytest/test_rollup.py @@ -807,7 +807,7 @@ def test_min_and_max_keep_each_columns_collation(pgc_own_db, expect): " FROM generate_series(1, 2000) g") folded = _one(conn, "SELECT pgcolumnar.refresh_rollup('r_ct')") expect.text(f"{folded}|{_one(conn, 'SELECT count(*) FROM r_ct')}", "2000|1", - "premise: the second refresh folds new rows into the existing bucket") + "the second refresh folds new rows into the existing bucket") expect.text(_one(conn, "SELECT min_sc || '|' || max_sc || '|' || min_su || '|' || max_su FROM r_ct"), _one(conn, "SELECT min(sc) || '|' || max(sc) || '|' || min(su) || '|' || max(su) FROM ct"), "min and max over a C column and an en_US column keep each column's own order across a refresh") diff --git a/test/rollup.sh b/test/rollup.sh index e220a7cf2..02fb163e7 100755 --- a/test/rollup.sh +++ b/test/rollup.sh @@ -639,7 +639,9 @@ if [ -n "$CT_COLL" ]; then q "SELECT pgcolumnar.create_rollup('r_ct', 'ct', 'ts', '1 day', '$ORIGIN', '{}', '{min,max,min,max}', '{sc,sc,su,su}');" >/dev/null q "SELECT pgcolumnar.refresh_rollup('r_ct');" >/dev/null q "INSERT INTO ct SELECT '$ORIGIN'::timestamptz + (g % 24) * interval '1 hour', 'B'||g, 'B'||g FROM generate_series(1, 2000) g;" >/dev/null - check "premise: the second refresh folds new rows into the existing bucket" \ + # A check, not a premise (#1236; #1372 review): refresh_rollup's return value and + # the bucket count are both product output. + check "the second refresh folds new rows into the existing bucket" \ "$(q "SELECT pgcolumnar.refresh_rollup('r_ct');")|$(q "SELECT count(*) FROM r_ct;")" "2000|1" check "min and max over a C column and an en_US column keep each column's own order across a refresh" \ "$(q "SELECT min_sc || '|' || max_sc || '|' || min_su || '|' || max_su FROM r_ct;")" \ @@ -647,7 +649,7 @@ if [ -n "$CT_COLL" ]; then check "premise: the two collations disagree on these rows, so the arm can tell them apart" \ "$(q "SELECT (min(sc) COLLATE \"C\" <> min(su) COLLATE \"C\")::text FROM ct;")" "true" else - for _n in "premise: the second refresh folds new rows into the existing bucket" \ + for _n in "the second refresh folds new rows into the existing bucket" \ "min and max over a C column and an en_US column keep each column's own order across a refresh" \ "premise: the two collations disagree on these rows, so the arm can tell them apart"; do check_skip "$_n" "SKIP $_n (no en_US collation)" "the server has no en_US collation" From f07760fa667a5329fecfb32fc769500dfaef1f79 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 20:03:04 +0000 Subject: [PATCH 41/41] test: the refresh-count check is dated by refresh_rollup returning one more than it folded (#1372 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 32 ++++++++++++++++---------------- test/check_ledger_budget.txt | 14 +++++++++++++- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 381d12c9a..a543b0391 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2871,13 +2871,13 @@ rollup rollup a delete in buckets with no history, next to retired ones, folds: rollup rollup a group_by column the source does not have is refused 15;16;17;18;19 2026-10-14 create_rollup without its group_by column check rollup rollup a refresh after a delete is not refused 15;16;17;18;19 2026-10-13 refresh_rollup refusing any change but appends (phase 1, main before this branch) rollup rollup a refresh after a small append reads under a fifth of the source 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup's per-group subquery without OFFSET 0, so it becomes a join and reads every group -rollup rollup a refresh after appends folds exactly the appended rows, late ones included 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's -rollup rollup a refresh after retention folds nothing, refuses nothing, and keeps every bucket 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup not dropping groups retention retired +rollup rollup a refresh after appends folds exactly the appended rows, late ones included 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's +rollup rollup a refresh after retention folds nothing, refuses nothing, and keeps every bucket 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup not dropping groups retention retired;refresh_rollup returns one more than it folded rollup rollup a refresh after the source was rewritten is refused, naming rebase_rollup 15;16;17;18;19 2026-10-14 pgcolumnar.vacuum rewrites nothing;refresh_rollup without the source identity check rollup rollup a refresh at REPEATABLE READ is refused 15;16;17;18;19 2026-10-13 refresh_rollup at any isolation level -rollup rollup a refresh folds a bulk append 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's +rollup rollup a refresh folds a bulk append 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's rollup rollup a refresh in a zone with a DST change recomputes a whole day bucket 15;16;17;18;19 2026-10-13 refresh_rollup adding the width in the session's zone;refresh_rollup recomputing every bucket once any group changed;refresh_rollup recording new groups without their time range;refresh_rollup refusing any change but appends (phase 1, main before this branch) -rollup rollup a refresh with nothing new folds nothing 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group +rollup rollup a refresh with nothing new folds nothing 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded rollup rollup a restore keeps the rollup's definition 15;16;17;18;19 2026-10-14 the tree before #1377: pgcolumnar.rollup not dumped, no source_oid rollup rollup a role cannot name another role's table as its rollup's partials (42501), and that table survives its view's drop (#1376) 15;16;17;18;19 2026-10-14 the rollup trigger with no ownership, kind or no-change checks (8a7fab2, #1406 review) rollup rollup a role cannot tie a rollup row to a table it cannot read (42501, #1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) @@ -2892,7 +2892,7 @@ rollup rollup a ts_column that is not timestamptz is refused 15;16;17;18;19 2026 rollup rollup a width with months is refused 15;16;17;18;19 2026-10-13 create_rollup without the months refusal rollup rollup after a delete and an append that straddles its bucket, a refresh counts the new rows once 15;16;17;18;19 2026-10-13 rebase_rollup recording no time ranges;refresh_rollup folding new groups' rows into dirty buckets too;refresh_rollup recomputing every bucket once any group changed;refresh_rollup refusing any change but appends (phase 1, main before this branch) rollup rollup after a delete of rows whose ts is NULL the NULL-bucket rollup equals a full recompute 15;16;17;18;19 2026-10-13 refresh_rollup never dirtying the NULL bucket;refresh_rollup recomputing every bucket once any group changed;refresh_rollup refusing any change but appends (phase 1, main before this branch) -rollup rollup after a rebase the next refresh folds exactly the appended rows 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;row_tid mapping 290 rows a block instead of the AM's +rollup rollup after a rebase the next refresh folds exactly the appended rows 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's rollup rollup after a restore into another database its owner's refresh is refused (55000), the rollup untouched 15;16;17;18;19 2026-10-14 refresh_rollup without the source_oid check: a restored source under the recorded storage id is folded twice;the tree before #1377: pgcolumnar.rollup not dumped, no source_oid rollup rollup after a restore into the same database its refresh is refused (55000), the rollup untouched 15;16;17;18;19 2026-10-14 refresh_rollup without the source_oid check: a restored source under the recorded storage id is folded twice;the tree before #1377: pgcolumnar.rollup not dumped, no source_oid rollup rollup after a restore the rollup's source cannot be dropped while the rollup exists (2BP01) 15;16;17;18;19 2026-10-14 the tree without #1406's dependency trigger: a restored rollup does not depend on its source @@ -2904,7 +2904,7 @@ rollup rollup an aggregate column the source does not have is refused 15;16;17;1 rollup rollup an aggregate that does not combine is refused 15;16;17;18;19 2026-10-13 create_rollup without the five-function allow-list rollup rollup and DROP TABLE ... CASCADE takes the rollup's view, partials and catalog rows with it (#1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) rollup rollup and after a delete the rollup equals a full recompute 15;16;17;18;19 2026-10-13 rebase_rollup recording no time ranges;refresh_rollup recomputing every bucket once any group changed;refresh_rollup refusing any change but appends (phase 1, main before this branch) -rollup rollup and after retention an append still folds 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup not dropping groups retention retired;row_tid mapping 290 rows a block instead of the AM's +rollup rollup and after retention an append still folds 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup not dropping groups retention retired;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's rollup rollup and after the bulk append the rollup equals a full recompute 15;16;17;18;19 2026-10-13 refresh_rollup combining min with greatest;refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup recording the folded groups from a fresh read after the fold;row_tid mapping 290 rows a block instead of the AM's rollup rollup and after the first refresh the rollup equals a full recompute 15;16;17;18;19 2026-10-13 refresh_rollup folding only the first new group rollup rollup and after the rebase the rollup equals a full recompute 15;16;17;18;19 2026-10-13 rebase_rollup clearing the rollup's rows;refresh_rollup combining min with greatest;refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup recording the folded groups from a fresh read after the fold;row_tid mapping 290 rows a block instead of the AM's @@ -2917,11 +2917,11 @@ rollup rollup and leaves the rollup as it was 15;16;17;18;19 2026-10-13 refresh_ rollup rollup and leaves the rollup's rows alone 15;16;17;18;19 2026-10-13 rebase_rollup clearing the rollup's rows rollup rollup and no refused rollup left a table behind 15;16;17;18;19 2026-10-14 create_rollup allowing * with any function;create_rollup without its timestamptz check on ts_column;create_rollup without the equal-length check;create_rollup without the months refusal rollup rollup and reads what a recompute gives 15;16;17;18;19 2026-10-13 refresh_rollup bucketing with date_trunc, which follows the session's zone;rollup_source_groups without SECURITY DEFINER;the rollup catalogs without their grants -rollup rollup and refreshes it 15;16;17;18;19 2026-10-13 rollup_source_groups without SECURITY DEFINER;the rollup catalogs without their grants +rollup rollup and refreshes it 15;16;17;18;19 2026-10-14 refresh_rollup returns one more than it folded;rollup_source_groups without SECURITY DEFINER;the rollup catalogs without their grants rollup rollup and that is not zero 15;16;17;18;19 never - -rollup rollup and the next refresh, with nothing changed since, recomputes nothing 15;16;17;18;19 2026-10-13 refresh_rollup not recounting a changed group that is still there;refresh_rollup refusing any change but appends (phase 1, main before this branch) +rollup rollup and the next refresh, with nothing changed since, recomputes nothing 15;16;17;18;19 2026-10-14 refresh_rollup not recounting a changed group that is still there;refresh_rollup refusing any change but appends (phase 1, main before this branch);refresh_rollup returns one more than it folded rollup rollup and the recent buckets equal a recompute, while the expired ones keep their history 15;16;17;18;19 2026-10-13 refresh_rollup bucketing with date_trunc, which follows the session's zone;refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup not dropping groups retention retired;row_tid mapping 290 rows a block instead of the AM's -rollup rollup and the refresh after the rewrite recomputes nothing 15;16;17;18;19 2026-10-13 refresh_rollup not forgetting a changed group that is gone;refresh_rollup refusing any change but appends (phase 1, main before this branch) +rollup rollup and the refresh after the rewrite recomputes nothing 15;16;17;18;19 2026-10-14 refresh_rollup not forgetting a changed group that is gone;refresh_rollup refusing any change but appends (phase 1, main before this branch);refresh_rollup returns one more than it folded rollup rollup and the refresh rewrote no partial outside the changed groups' time range 15;16;17;18;19 2026-10-13 refresh_rollup recomputing every bucket once any group changed rollup rollup and the refusal names rebase_rollup 15;16;17;18;19 2026-10-14 refresh_rollup without the source_oid check: a restored source under the recorded storage id is folded twice;the tree before #1377: pgcolumnar.rollup not dumped, no source_oid rollup rollup and the restored rollup equals a full recompute 15;16;17;18;19 2026-10-14 refresh_rollup without the source_oid check: a restored source under the recorded storage id is folded twice;the tree before #1377: pgcolumnar.rollup not dumped, no source_oid @@ -2952,12 +2952,12 @@ rollup rollup row_tid refuses a row number below 1 (22003), and row 1 is accepte rollup rollup the database holding a rollup dumps 15;16;17;18;19 never - rollup rollup the dump restores into another database 15;16;17;18;19 never - rollup rollup the dump restores into its own database after DROP EXTENSION, again under the recorded storage id 15;16;17;18;19 never - -rollup rollup the first refresh folds every row 15;16;17;18;19 2026-10-13 refresh_rollup folding only the first new group -rollup rollup the history rollup folded every row 15;16;17;18;19 2026-10-13 refresh_rollup folding only the first new group +rollup rollup the first refresh folds every row 15;16;17;18;19 2026-10-14 refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded +rollup rollup the history rollup folded every row 15;16;17;18;19 2026-10-14 refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded rollup rollup the history rollup holds old buckets 15;16;17;18;19 never - -rollup rollup the next refresh folds the rows committed during the last one, once 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup recording the folded groups from a fresh read after the fold;row_tid mapping 290 rows a block instead of the AM's +rollup rollup the next refresh folds the rows committed during the last one, once 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup recording the folded groups from a fresh read after the fold;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's rollup rollup the owner reads its source's groups through the definer helpers 15;16;17;18;19 never - -rollup rollup the paused refresh folded what was there before it began 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's +rollup rollup the paused refresh folded what was there before it began 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's rollup rollup the refresh after a small append read the source at all 15;16;17;18;19 never - rollup rollup the restored rollup reads as a recompute of the restored source 15;16;17;18;19 never - rollup rollup the restored source holds the dumped rows under the storage id recorded for the old one 15;16;17;18;19 never - @@ -2965,10 +2965,10 @@ rollup rollup the rollup holds partials the delete below cannot reach 15;16;17;1 rollup rollup the rollup holds rows before the rewrite 15;16;17;18;19 never - rollup rollup the rollup table exists with the documented columns 15;16;17;18;19 never - rollup rollup the rollup to be dumped folds its source 15;16;17;18;19 never - -rollup rollup the second refresh folds new rows into the existing bucket 15;16;17;18;19 never - -rollup rollup the second refresh of the NULL-key rollup folded the append 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's +rollup rollup the second refresh folds new rows into the existing bucket 15;16;17;18;19 2026-10-14 refresh_rollup returns one more than it folded +rollup rollup the second refresh of the NULL-key rollup folded the append 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded;row_tid mapping 290 rows a block instead of the AM's rollup rollup the zone rollups hold a row per key of the recompute 15;16;17;18;19 2026-10-13 refresh_rollup folding only the first new group;row_tid mapping 290 rows a block instead of the AM's -rollup rollup two refreshes started together fold the new rows once between them 15;16;17;18;19 2026-10-13 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup without the self-conflicting lock;row_tid mapping 290 rows a block instead of the AM's +rollup rollup two refreshes started together fold the new rows once between them 15;16;17;18;19 2026-10-14 refresh_rollup folding every group, ignoring the recorded snapshot;refresh_rollup folding only the first new group;refresh_rollup returns one more than it folded;refresh_rollup without the self-conflicting lock;row_tid mapping 290 rows a block instead of the AM's serializable serializable a serializable read of a columnar table holds SIRead locks on columnar's row_group and delete_vector catalog 15;16;17;18;19 2026-10-15 PgColumnarCatalogSnapshot returns a SNAPSHOT_SELF copy, so a serializable read takes no SIRead lock on columnar's catalog serializable serializable at REPEATABLE READ both doctors' transactions commit and no one stays on call, in one row group and in two, as on heap 15;16;17;18;19 never - serializable serializable at REPEATABLE READ both ward transactions commit and two are on call, by a scan and by an index, as on heap 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 6ee71cac9..f83a54eed 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2706,4 +2706,16 @@ suites_not_covered 224 # never 2527 + 2 = 2529 # dated 820 + 1 = 821 # 2529 + 821 == 3350 -checks_never_observed_red 2529 +# +# #1372 review (OffgridwithJD): 2529 -> 2528. "the second refresh folds new rows into the +# existing bucket" loses the premise prefix: refresh_rollup's return value is product +# output. refresh_rollup returning one more than it folded reddens it, dating it, and +# fifteen other rollup arms that count what a refresh folded, all already dated. +# Five-major runs: rollup 85/85 on 15-19, and 108/108 on PG18 after the rebase onto +# d5c0a5e, which brought in 23 checks (#1391's row_tid arm still runs last). +# +# ledger rows 3350 = 3350 +# never 2529 - 1 = 2528 +# dated 821 + 1 = 822 +# 2528 + 822 == 3350 +checks_never_observed_red 2528