From d0ca319fb50b474ac5f143f4edaef2a69eb18c08 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 19:16:33 +0000 Subject: [PATCH 01/14] test: the ledger refuses a mutation split on ';', and the 12 split rows are reworded The mutation column is a ';'-separated set, written back sorted. A --mutation whose text held "; " was stored as two mutations, one a fragment starting with a space, and every later merge re-sorted it. #1427's recovery_minimal_wal text did this to 8 rows, and #1423, #1424 and #1426 each carried them as a change of their own. Four older harness_selftest rows were split the same way. - merge refuses a --mutation holding ';' (rc 2, ledger untouched). - the reader refuses a ';'-piece with whitespace at an end, by line: the writer joins with a bare ';', so such a piece is a fragment, never a mutation. - the 12 rows are reworded without ';', from their original text in history. Only field 6 changes; the census stays 3007 / 2282 / 725. Red first: the two refusal arms got 0 want 2 on the unchanged tool, and the committed-ledger arm was red at line 1783 with the refusal in and the rows not yet reworded. guard_tests 435 -> 438 by collection. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 24 +++++++-------- test/pgc_ledger.py | 16 ++++++++++ test/pytest/TESTS.md | 20 ++++++++++++ test/pytest/expected_tests.txt | 5 ++- test/pytest/test_mutation_ledger.py | 48 +++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+), 13 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 734b4a672..694fb5d6e 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1805,8 +1805,8 @@ harness_selftest 540-an-arm-must-carry-its-measurement a new arm that cannot say harness_selftest 540-an-arm-must-carry-its-measurement a repaired arm must have its row removed, so the list only shrinks 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: a determinate or carrying arm is not swept up 15;16;17;18;19 2026-10-12 part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: a determinate or carrying awk arm is not swept up 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's awk_determinate treats a comparison against > 0 as indeterminate (return 1 made return 0) -harness_selftest 540-an-arm-must-carry-its-measurement control: the carve-outs hold and the innocent arm above the helper is not named 15;16;17;18;19 2026-09-24 else echo 0; fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent); then echo 1;the shell if/then/else spelling removed from the matcher, so `$(if [ C ] -harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a shell if/then/else that discards its operand 15;16;17;18;19 2026-09-24 else echo 0; fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent); then echo 1;the shell if/then/else spelling removed from the matcher, so `$(if [ C ] +harness_selftest 540-an-arm-must-carry-its-measurement control: the carve-outs hold and the innocent arm above the helper is not named 15;16;17;18;19 2026-09-24 the shell if/then/else spelling removed from the matcher, so a one-line `$(if [ C ]... then echo 1... else echo 0... fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent) +harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a shell if/then/else that discards its operand 15;16;17;18;19 2026-09-24 the shell if/then/else spelling removed from the matcher, so a one-line `$(if [ C ]... then echo 1... else echo 0... fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent) harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a threshold arm and a floor arm 15;16;17;18;19 2026-10-12 part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees an awk verdict that discards its operands 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's awk_determinate treats a comparison against > 0 as indeterminate (return 1 made return 0) harness_selftest 540-an-arm-must-carry-its-measurement premise: the comment fixture quotes a whole recorder call, not a fragment 15;16;17;18;19 never - @@ -1852,10 +1852,10 @@ harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary harness_selftest 580-a-hand-rebuild-must-record a PostgreSQL symbol is NOT exempt, so the check still has a subject 15;16;17;18;19 2026-10-12 pgc_symbol_is_toolchain deleted from the real lib.sh;the symbol exemption matches every name, so the check has no subject left and a genuinely mislinked major would resolve clean harness_selftest 580-a-hand-rebuild-must-record a log that is not there also produces a diagnosis 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason -harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it, driven alone with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp harness_selftest 580-a-hand-rebuild-must-record an empty log still produces a diagnosis, because silence is the bug 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all -harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it, driven alone with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record and it distinguishes missing from empty, so the two causes do not merge 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record and it names the log as empty rather than printing a bare banner 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record and it never reaches the pg_config check it would otherwise fail on 15;16;17;18;19 2026-09-25 rebuild.sh's fail-closed assertion is removed, so a missing exemption function leaves the undefined list empty and the check reports success @@ -2609,14 +2609,14 @@ range_pruning range_pruning premise: the range type collates differently from it range_pruning range_pruning so does a containment probe 15;16;17;18;19 never - range_pruning range_pruning the overlap qual is pushed down, where it used to be dropped 15;16;17;18;19 never - range_pruning range_pruning the same for a containment probe far to its right 15;16;17;18;19 never - -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) recovery_minimal_wal recovery_minimal_wal premise: the cluster runs at wal_level=minimal 15;16;17;18;19 never - rewrite_storage_oid rewrite_storage_oid a matview created WITH DATA leaves its storage row pointing at itself 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite repair gate no longer covers CreateTableAsStmt, so a matview created WITH DATA keeps the transient's OID rewrite_storage_oid rewrite_storage_oid a type rewrite leaves the storage row pointing at the live table 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite hook does not retarget storage.relation_oid diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 1d880f395..83c6666dc 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -312,6 +312,16 @@ def read_ledger(path): f"{path}:{n}: major {m!r} is neither a number nor " f"{MAJOR_UNKNOWN!r}, so this row names no version it applies to") muts = set() if f[5] == NONE else {m for m in f[5].split(";") if m} + # A PIECE WITH WHITESPACE AT AN END IS A FRAGMENT, not a mutation. The writer + # joins the set with a bare ";", so it never produces one; a text that held + # "; " does, when it is split on the ";" (12 rows on the shipped ledger, 8 of + # them from one recovery_minimal_wal mutation, re-sorted by every later merge). + for m in sorted(muts): + if m != m.strip(): + raise LedgerError( + f"{path}:{n}: mutation {m!r} has whitespace at an end, so it is a " + f"fragment of one mutation's text split on ';'. Reword that mutation " + f"without ';' and keep the row's other mutations as they are.") key = (f[0], f[1], f[2]) # A REPEATED KEY, REFUSED RATHER THAN OVERWRITTEN. This was # `rows[key] = [...]`, so a duplicated row collapsed silently and the LAST line @@ -530,6 +540,12 @@ def cmd_merge(args): "--target names what a mutation killed, so it needs --mutation to say " "which mutation. On its own it attributes nothing.") + if args.mutation and ";" in args.mutation: + raise LedgerError( + f"--mutation {args.mutation!r} holds ';', which separates mutations in the " + f"ledger's mutation column, so it would be stored as several. Reword it " + f"without ';' (', and' reads the same).") + if args.mutation: failed = sorted({key for _p, seen in runs for key, vs in seen.items() if any(v == "FAIL" for v, _m in vs)}) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 4425ba7e6..ed83fde0e 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2445,6 +2445,26 @@ Last-write-wins records the most recent attack rather than the catalogue the col exists to become. One `--mutation` copied across several logs attributes a deliberate change to failures it had nothing to do with, and is refused. +### `test_a_mutation_named_with_a_semicolon_is_refused_rather_than_split` + +The mutation column is a `;`-separated set, written back sorted, so a `--mutation` +whose text held `; ` was stored as two mutations, one a fragment starting with a space. +Every later merge re-sorted it, and three branches in a row carried the same eight +recovery_minimal_wal rows as a change of their own. `merge` now refuses such a text with +rc 2 and leaves the ledger as it was; the same text with `, and` is recorded whole. + +### `test_a_ledger_row_whose_mutation_holds_a_fragment_is_refused` + +The writer joins the set with a bare `;`, so a piece with whitespace at an end is a +fragment of a split sentence, never a mutation it wrote. The reader refuses the ledger +by line. Two whole mutations separated by a bare `;` still read. + +### `test_the_committed_ledger_names_no_mutation_fragment` + +The committed ledger, read by the same reader. Red on the tree before this change at +line 1783, the first of the 12 rows it rewords: 8 from #1427's recovery_minimal_wal +mutation and 4 older harness_selftest rows. + ### `test_a_mutation_with_two_targets_can_be_recorded_when_both_are_named` A mutation with two GENUINE targets is ordinary: reverting the diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 783fc01c6..8e812d734 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -285,7 +285,10 @@ # tests, the twin of selftest part 640, by collection over the NO_CLUSTER files. # guard_tests 433 -> 435 for #1420's two arms in test_writes_wrote_rows.py (a setting # on a watched connection or cursor is refused), by collection over NO_CLUSTER. -guard_tests 435 +# guard_tests 435 -> 438 for three arms in test_mutation_ledger.py: a mutation text holding +# ';' is refused at merge, a ledger piece with whitespace at an end is refused, and the +# committed ledger holds none, by collection over NO_CLUSTER. +guard_tests 438 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index a06e83064..07650ec7f 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -125,6 +125,54 @@ def test_the_mutation_column_accumulates_rather_than_overwriting(tmp_path, expec "one mutation cannot be attributed across several runs at once") +def test_a_mutation_named_with_a_semicolon_is_refused_rather_than_split(tmp_path, expect): + """The mutation column is a ";"-separated SET, written back sorted. A `--mutation` + whose text held "; " was stored as two "mutations", one a sentence fragment that + starts with a space, and the next merge sorted the fragment first. Measured on the + shipped ledger: 12 rows, 8 of them from one recovery_minimal_wal mutation (#1427), + re-sorted by three later branches until each looked like a change of its own. + """ + ledger = _w(tmp_path, "l.tsv", "") + red = _w(tmp_path, "r.log", RED) + out, rc = _run("merge", "--ledger", ledger, "--date", "2026-10-15", + "--mutation", "the sync not registered; this build then asserts", red) + expect.num(rc, 2, "a mutation whose text holds ';' is refused, not split in two") + expect.num(out.count("separates mutations"), 1, "and the refusal says why") + expect.num(len(_rows(ledger)), 0, "and the ledger is left as it was") + + out, rc = _run("merge", "--ledger", ledger, "--date", "2026-10-15", + "--mutation", "the sync not registered, and this build then asserts", red) + expect.num(rc, 0, f"control: the same text without ';' is recorded: {out.strip()}") + expect.text({r[2]: r[5] for r in _rows(ledger)}["first check"], + "the sync not registered, and this build then asserts", + "and it is recorded whole, as one mutation") + + +def test_a_ledger_row_whose_mutation_holds_a_fragment_is_refused(tmp_path, expect): + """A ";"-piece with whitespace at either end cannot be a mutation the writer + produced: it joins the set with a bare ";". It is the fragment of a split sentence, + so the ledger is refused by line rather than read as two mutations.""" + green = _w(tmp_path, "g.log", GREEN) + split = _w(tmp_path, "split.tsv", + "demo\tpart1\tfirst check\t18\t2026-10-15\tthe sync not registered; this build asserts\n") + out, rc = _run("merge", "--ledger", split, green) + expect.num(rc, 2, "a mutation piece that starts with a space is refused") + expect.num(out.count("split.tsv:1:"), 1, "and the refusal names the line") + + two = _w(tmp_path, "two.tsv", + "demo\tpart1\tfirst check\t18\t2026-10-15\tbloom neutered;SAOP 128 -> 0\n") + out, rc = _run("merge", "--ledger", two, green) + expect.num(rc, 0, f"control: two whole mutations separated by a bare ';' still read: {out.strip()}") + + +def test_the_committed_ledger_names_no_mutation_fragment(tmp_path, expect): + """The committed ledger read by the same reader, so a split row on main is an + integrity failure of the file every gate reads, not a cosmetic one.""" + copy = _w(tmp_path, "ledger.tsv", (REPO / "test" / "check_ledger.tsv").read_text()) + out, rc = _run("merge", "--ledger", copy, _w(tmp_path, "g.log", GREEN)) + expect.num(rc, 0, f"the committed ledger reads with no fragment in it: {out.strip()[:400]}") + + def test_a_mutation_with_two_targets_can_be_recorded_when_both_are_named(tmp_path, expect): """#1014. A mutation with TWO GENUINE TARGETS is ordinary, not exotic. From 05fe29e4d579126e7b067831bd97865557132bfa Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 00:48:54 +0000 Subject: [PATCH 02/14] test: recluster recomputes the column extent, in both twins (#1341 item 75) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/column_extent.sh | 15 +++++++++++++++ test/pytest/test_column_extent.py | 16 ++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/test/column_extent.sh b/test/column_extent.sh index bba291034..066a15d78 100755 --- a/test/column_extent.sh +++ b/test/column_extent.sh @@ -213,6 +213,21 @@ q "CREATE TABLE vr (ts timestamptz, v int) USING pgcolumnar; q "VACUUM vr;" >/dev/null check "after VACUUM retires a fully deleted group the extent is the live rows'" "$(ext vr ts)" "$(truth vr ts)" +# recluster rewrites every group too, and the changelog names it with the three +# above (#1341 item 75): the rows a DELETE removed are not written back, so its +# extent must be the live rows', not wider by rows no longer stored. The deleted +# rows hold the minimum, so a stale extent keeps it. +q "CREATE TABLE rc (ts timestamptz, v int) USING pgcolumnar; + SELECT pgcolumnar.set_options('rc', stripe_row_limit => 1000, chunk_group_row_limit => 1000); + INSERT INTO rc SELECT '2026-01-01'::timestamptz + g * interval '1 min', (g * 7919) % 3001 FROM generate_series(1, 3000) g; + DELETE FROM rc WHERE ts < '2026-01-01'::timestamptz + interval '400 min';" >/dev/null +rc_before="$(ext rc ts)" +reclustered="$(q "SELECT pgcolumnar.recluster('rc', 'ts', 'v');")" +check "recluster rewrites the table's row groups" "$([ "${reclustered:-0}" -ge 1 ] && echo yes || echo "no ($reclustered)")" "yes" +check "after recluster the extent is the live rows'" "$(ext rc ts)" "$(truth rc ts)" +check "and it did move: the deleted rows held the old minimum" \ + "$([ -n "$rc_before" ] && [ "$rc_before" != "$(ext rc ts)" ] && echo yes || echo "no ($rc_before)")" "yes" + # ---- no complete extent where the zone maps cannot give one ---------------------- q "CREATE TABLE ac (ts timestamptz) USING pgcolumnar; SELECT pgcolumnar.set_options('ac', stripe_row_limit => 1000, chunk_group_row_limit => 1000); diff --git a/test/pytest/test_column_extent.py b/test/pytest/test_column_extent.py index add40cc65..f0074520a 100644 --- a/test/pytest/test_column_extent.py +++ b/test/pytest/test_column_extent.py @@ -247,6 +247,22 @@ def test_a_shrink_is_recomputed_never_guessed(pgc_own_db, expect): expect.text(_ext(conn, "vr", "ts"), _truth(conn, "vr", "ts"), "after VACUUM retires a fully deleted group the extent is the live rows'") + # recluster, named by the changelog with the three above (#1341 item 75): the + # deleted rows are not written back, and they held the minimum. + _run(conn, "CREATE TABLE rc (ts timestamptz, v int) USING pgcolumnar") + _small_groups(conn, "rc") + _run(conn, "INSERT INTO rc SELECT '2025-05-05'::timestamptz + g * interval '3 min', (g * 104729) % 2711" + " FROM generate_series(1, 3600) g", + "DELETE FROM rc WHERE ts < '2025-05-05'::timestamptz + interval '1500 min'") + before = _ext(conn, "rc", "ts") + reclustered = _one(conn, "SELECT pgcolumnar.recluster('rc', 'ts', 'v')") + expect.text("yes" if (reclustered or 0) >= 1 else f"no ({reclustered})", "yes", + "recluster rewrites the table's row groups") + after = _ext(conn, "rc", "ts") + expect.text(after, _truth(conn, "rc", "ts"), "after recluster the extent is the live rows'") + expect.text("yes" if before and before != after else f"no ({before})", "yes", + "and it did move: the deleted rows held the old minimum") + def test_no_complete_extent_where_the_zone_maps_cannot_give_one(pgc_own_db, expect): conn = pgc_own_db From 639fc193b96d885c0603bf36d71d2c11e6dbd96f Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 00:51:14 +0000 Subject: [PATCH 03/14] test: the recluster extent arms enter the ledger, dated by the retirement-mark mutation (#1341 item 75) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 11 +++++++---- test/check_ledger_budget.txt | 14 +++++++++++++- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 694fb5d6e..54b8eaed2 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -393,11 +393,13 @@ column_extent column_extent a transaction's own flushed groups are inside the ex column_extent column_extent a type with no ordering gets no extent 15;16;17;18;19 2026-10-13 a recompute that skips values without an ordering;pgcolumnar.column_extent returns a row with no complete extent column_extent column_extent a writer that finds no complete extent recomputes one over every group 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent a writer that rolls back leaves the extent as the data has it 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit;no extent rows written at PRE_PREPARE;the writer's fold keeps the largest minimum -column_extent column_extent after VACUUM retires a fully deleted group the extent is the live rows' 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) -column_extent column_extent after compact_rewrite the extent is the live rows' 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;compact_rewrite returning 0 before rewriting anything;no extent row is ever written (PgColumnarExtentPreCommit returns at once) -column_extent column_extent after expire the extent is recomputed to the data that is left 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after VACUUM retires a fully deleted group the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after compact_rewrite the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;compact_rewrite returning 0 before rewriting anything;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after expire the extent is recomputed to the data that is left 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after recluster the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute column_extent column_extent and a plan made in that transaction estimates as the zone maps do 15;16;17;18;19 2026-10-13 the extent a transaction reads leaves out its own pending fold (the tree before the fix) -column_extent column_extent and it did move: the retired groups held the old minimum 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent and it did move: the deleted rows held the old minimum 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute +column_extent column_extent and it did move: the retired groups held the old minimum 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent and the estimate is the one the zone maps give 15;16;17;18;19 2026-10-13 the planner takes the extent's minimum as its upper bound too;the planner's zone-map fallback removed: with no complete extent it reads nothing and bounds nothing column_extent column_extent and the extent covers what they loaded 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit;no extent rows written at PRE_PREPARE;the writer's fold keeps the largest minimum column_extent column_extent and the extent still covers all fifty 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit @@ -414,6 +416,7 @@ column_extent column_extent planning a bucketed GROUP BY scans zone_map not at a column_extent column_extent premise: the extent function exists 15;16;17;18;19 never - column_extent column_extent premise: the transaction's plans gave two estimates 15;16;17;18;19 never - column_extent column_extent premise: the typed fixture spans several row groups 15;16;17;18;19 never - +column_extent column_extent recluster rewrites the table's row groups 15;16;17;18;19 never - column_extent column_extent the extent catalog is not readable by an ordinary role (it holds column values) 15;16;17;18;19 2026-10-13 the column_extent catalog granted to PUBLIC column_extent column_extent the extent of ty.d is the data's min and max, in the type's order 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent the extent of ty.i is the data's min and max, in the type's order 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index cea363447..e8678ba9d 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2533,4 +2533,16 @@ suites_not_covered 230 # never 2344 + 1 = 2345 # dated 767 + 3 = 770 # 2345 + 770 == 3115 -checks_never_observed_red 2345 +# +# #1341 item 75, the changelog names recluster with expire, compact_rewrite and +# VACUUM as recomputing the column extent, and only the other three had arms: +# 2345 -> 2346. column_extent gains 3 checks from five-major runs (42/42 on each +# of 15-19). Two are dated on PG17 by the existing mutation "PgColumnarRetireGroup +# without marking the storage for a recompute", which also re-dates the four +# arms it already held. "recluster rewrites the table's row groups" arrives never. +# +# ledger rows 3115 + 3 = 3118 +# never 2345 + 1 = 2346 +# dated 770 + 2 = 772 +# 2346 + 772 == 3118 +checks_never_observed_red 2346 From 14f6aeb94c6e29ee04673f74e5cb809b3fa81975 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 16:05:52 +0000 Subject: [PATCH 04/14] test: tier_partition and partition_create_ahead run on the caller's own privileges, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/partition_lifecycle.sh | 31 ++++++++++++++ test/pytest/TESTS.md | 2 + test/pytest/expected_tests.txt | 4 +- test/pytest/test_partition_lifecycle.py | 38 +++++++++++++++++ test/pytest/test_tier_partition.py | 54 +++++++++++++++++++++++++ test/tier_partition.sh | 44 ++++++++++++++++++++ 6 files changed, 172 insertions(+), 1 deletion(-) diff --git a/test/partition_lifecycle.sh b/test/partition_lifecycle.sh index 9c649e886..5eb52be5f 100755 --- a/test/partition_lifecycle.sh +++ b/test/partition_lifecycle.sh @@ -241,6 +241,37 @@ check "a parent with a DEFAULT partition is refused" \ check "and nothing was created for it" \ "$(kids td)|$(q "SELECT count(*) FROM pg_proc WHERE proname = 'partition_create_ahead';")" "1|1" +# ---- the caller's own privileges (#1341) ------------------------------------- +# +# partition_create_ahead is not SECURITY DEFINER, so its CREATE TABLE ... PARTITION OF +# runs as the caller, and core refuses a role that does not own the parent. Made +# SECURITY DEFINER, it would run as the extension's owner and let any caller add +# partitions, and the access-exclusive lock that comes with them, to a table it does +# not own. Both roles hold USAGE and CREATE on the schema, so ownership is all that +# separates them. +for r in pl_own pl_peer; do + psql_run "DROP ROLE IF EXISTS $r; CREATE ROLE $r LOGIN NOSUPERUSER; + GRANT USAGE, CREATE ON SCHEMA public TO $r; GRANT USAGE ON SCHEMA pgcolumnar TO $r;" +done +as_role() { # as_role ROLE SQL -> the single value SQL returns as ROLE, or the SQLSTATE it fails with + local out + out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" -d "$PGC_DB" \ + -X -At -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$2" 2>&1) + case "$out" in + *ERROR:*|*FATAL:*) sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' <<<"$out" | head -1 ;; + *) printf '%s\n' "$out" | tail -1 ;; + esac +} +q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own;" >/dev/null +check "premise: pl_own owns the parent, and neither role is a superuser" \ + "$(q "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass;")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ + "pl_own|0" +check "premise: pl_peer can open a session" "$(as_role pl_peer "SELECT 1")" "1" +check "a role that does not own the parent is refused by its own privileges (42501), and creates nothing" \ + "$(as_role pl_peer "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "42501|0" +check "a non-superuser owner creates partitions ahead on its own table" \ + "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "3|3" + # ==== the drop half: partition_drop_plan (#1297) ================================ # # partition_drop_plan(parent regclass, cutoff timestamptz) -> SETOF text diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index ed83fde0e..effcdd57c 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6643,6 +6643,7 @@ Removal proofs, each red in both twins: the prototype's zone handling; `CREATE T | --- | --- | | `test_partition_create_ahead` | from nothing it reaches now plus the lead, columnar, contiguous and one width wide; a second call creates nothing, and asking further ahead creates only the difference; rows land in columnar partitions; the parent is never held ACCESS EXCLUSIVE; west of UTC, for both key types, partitions continue a hand-made one contiguously and a row at every boundary routes; east of UTC, a timestamp parent covers the local now and starts on the width's boundary; inputs that broke the first version (a generated column, a 63-character name, more than 1000 partitions, a MAXVALUE bound) are handled or refused before any DDL; a DEFAULT partition is refused and nothing is created | | `test_partition_drop_plan` | the drop half: `partition_drop_plan` lists `DETACH ... CONCURRENTLY` and `DROP` for the partitions wholly before the cutoff, in range order, from partitions created newest first. A cutoff on an upper bound includes the partition, and one before them all plans nothing. Running the plan, each statement top-level, leaves the rest with their rows and drops the storage rows. A `timestamp` key with a `timestamp` cutoff plans the same in Tokyo and Denver, and a cutoff of the other type is refused (`42804`). MINVALUE is listed once past, and MAXVALUE never. Quoted schema and partition names run as returned. A sub-partitioned partition is planned whole and dropped with its children. A tiered partition is planned with `DROP FOREIGN TABLE`, and its Parquet file stays. A partition whose name holds the drop block's dollar-quote tag drops as planned. A detach cancelled in its second phase is planned as `FINALIZE` and dropped. The DEFAULT and non-timestamp parents are refused (`55000`) | +| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates its partitions ahead | ## 94. test_approx_aggregates.py: hll and tdigest over columnar tables, the compatibility statement @@ -6791,6 +6792,7 @@ On a private database, because foreign servers are per database. A local path st | `test_tier_partition_moves_a_partition_exactly_once` | the move returns the rows moved; the parent's answers are the same; the cold range is a foreign partition with the same name and bound, reading the exported file; no pgcolumnar catalog rows are left; pruning keeps the ranges apart; an insert into the tiered range is refused; five refusals move nothing; the parent is held ACCESS EXCLUSIVE until commit, and a rollback leaves the rows local; a write in flight is exported; a fault after each of the five steps leaves the rows exactly once, still local | | `test_tier_partition_refuses_what_it_cannot_move_and_survives_concurrency` | at each isolation level a row committed between the snapshot and the lock survives, and only READ COMMITTED moves; a UNIQUE parent and four kinds of dependent are refused with no file left, and the partition moves once they are gone; grants and comment carry over; a racing insert leaves every row once; lock_wait times a held parent out with nothing attached; the caller's lock_timeout survives; a retry over a failed attempt's file moves the partition | | `test_tier_partition_moves_a_virtual_generated_column` | PostgreSQL 18+: a partition whose parent has a virtual generated column moves and reads back the same; below 18, `UNSUPPORTED_MAJOR`, listed in `expected_unrunnable.txt` | +| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501`, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers its own partition | ## 101. test_prepared_writes.py: a write prepared and finished in one session leaves the backend alive diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 8e812d734..cb3b19a84 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -705,4 +705,6 @@ guard_tests 438 # cluster_tests 586 -> 587 for #1341's test_serializable.py, one test, by collection # cluster_tests 587 -> 588 for #1341's daemon TimeZone test in test_autovacuum_expire.py, # by collection over the cluster files on this tree. -cluster_tests 588 +# cluster_tests 588 -> 590 for #1341's caller-privilege tests in test_tier_partition.py and +# test_partition_lifecycle.py, one each, by collection over the cluster files on this tree. +cluster_tests 590 diff --git a/test/pytest/test_partition_lifecycle.py b/test/pytest/test_partition_lifecycle.py index c7bcad322..814a912c2 100644 --- a/test/pytest/test_partition_lifecycle.py +++ b/test/pytest/test_partition_lifecycle.py @@ -595,3 +595,41 @@ def name(t): for t in ("xnl", "xls", "xr2")), "|".join(f"55000 {name(t)} is not RANGE-partitioned on one column" for t in ("xnl", "xls", "xr2")), "a drop plan for a parent not RANGE-partitioned on one column is refused, LIST and two-column RANGE included") + + +def test_partition_create_ahead_runs_on_the_callers_own_privileges(pgc_conn, pgc_cluster, expect): + """partition_create_ahead is not SECURITY DEFINER, so its CREATE TABLE ... PARTITION + OF runs as the caller and core refuses a role that does not own the parent (#1341). + Made SECURITY DEFINER, it would run as the extension's owner and let any caller add + partitions to a table it does not own. Both roles hold USAGE and CREATE on the + schema, so ownership is all that separates them.""" + conn = pgc_conn + schema = _one(conn, "SELECT current_schema()") + with conn.cursor() as cur: + for r in ("pl_own_py", "pl_peer_py"): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} LOGIN NOSUPERUSER; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f'GRANT USAGE, CREATE ON SCHEMA "{schema}" TO {r}') + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + cur.execute("CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") + cur.execute("ALTER TABLE pv OWNER TO pl_own_py") + + def as_role(role, sql): + try: + with psycopg.connect(f"host=127.0.0.1 port={pgc_cluster.port} user={role} dbname=postgres", + autocommit=True) as c: + c.execute(f'SET search_path TO "{schema}", pgcolumnar, public') + return str(c.execute(sql).fetchone()[0]) + except psycopg.Error as e: + return e.sqlstate or "?" + + ahead_pv = "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)" + owner = _one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass") + supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own_py', 'pl_peer_py') AND rolsuper") + expect.text(f"{owner}|{supers}", "pl_own_py|0", + "premise: pl_own owns the parent, and neither role is a superuser") + expect.text(as_role("pl_peer_py", "SELECT 1"), "1", "premise: pl_peer can open a session") + expect.text(f"{as_role('pl_peer_py', ahead_pv)}|{_kids(conn, 'pv')}", + "42501|0", "a role that does not own the parent is refused by its own privileges (42501), and creates nothing") + expect.text(f"{as_role('pl_own_py', ahead_pv)}|{_kids(conn, 'pv')}", + "3|3", "a non-superuser owner creates partitions ahead on its own table") diff --git a/test/pytest/test_tier_partition.py b/test/pytest/test_tier_partition.py index ad306382a..b994d3480 100644 --- a/test/pytest/test_tier_partition.py +++ b/test/pytest/test_tier_partition.py @@ -530,3 +530,57 @@ def test_tier_partition_moves_a_virtual_generated_column(pgc_own_db, server_dir, kind = _one(conn, "SELECT relkind::text FROM pg_class WHERE oid = 'cv_mar'::regclass") expect.text(f"{moved}|{kind}|{'same' if before and before == _one(conn, digest) else 'differs'}", "71|f|same", "a virtual generated column moves and reads back the same") + + +def test_tier_partition_runs_on_the_callers_own_privileges(pgc_own_db, pgc_cluster, server_dir, expect): + """tier_partition is not SECURITY DEFINER, so a role that does not own the table is + refused by its own privileges (#1341). Made SECURITY DEFINER, it would run as the + extension's owner and let any caller replace a partition it does not own. Both roles + hold every other privilege the call needs, so ownership is all that separates them.""" + conn = pgc_own_db + db = _one(conn, "SELECT current_database()") + with conn.cursor() as cur: + cur.execute("SET timezone = 'UTC'") + cur.execute("CREATE SERVER tq FOREIGN DATA WRAPPER pgcolumnar_parquet") + for r in ("tier_own_py", "tier_peer_py"): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} LOGIN NOSUPERUSER; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f"GRANT pg_write_server_files TO {r}") + cur.execute(f"GRANT USAGE ON FOREIGN SERVER tq TO {r}") + cur.execute(f"GRANT USAGE, CREATE ON SCHEMA public TO {r}") + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + _build(conn, "po") + with conn.cursor() as cur: + for t in ("po", "po_mar", "po_apr"): + cur.execute(f"ALTER TABLE {t} OWNER TO tier_own_py") + cur.execute("GRANT SELECT ON po, po_mar TO tier_peer_py") + before = _digest(conn, "po") + + def as_role(role, sql): + try: + with psycopg.connect(f"host=127.0.0.1 port={pgc_cluster.port} user={role} dbname={db}", + autocommit=True) as c: + c.execute("SET timezone = 'UTC'") + c.execute(sql) + return "ok" + except psycopg.Error as e: + return e.sqlstate or "?" + + owners = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" + " WHERE relname IN ('po', 'po_mar', 'po_apr')") + supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own_py', 'tier_peer_py') AND rolsuper") + expect.text(f"{owners}|{supers}", "tier_own_py|0", + "premise: tier_own owns the parent and its partitions, and neither role is a superuser") + expect.text(as_role("tier_peer_py", "SELECT count(*) FROM po_mar"), "ok", + "premise: tier_peer can open a session and read the partition") + peer_path = server_dir / "po_peer.parquet" + expect.text(as_role("tier_peer_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{peer_path}')"), + "42501", "a role that does not own the table is refused by its own privileges (42501)") + expect.text(f"{_digest(conn, 'po')}|{_mar_shape(conn, 'po')}", f"{before}|r:pgcolumnar:{BOUND}", + "and the refused call leaves the partition local and attached, every row once") + expect.num(_one(conn, "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid" + " WHERE c.relname LIKE 'po\\_%'"), 0, "with no foreign table behind it") + print(f"-- the refused call's export file: {'written' if peer_path.exists() else 'absent'}") + expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{server_dir / 'po_own.parquet'}')") + + f"|{_mar_shape(conn, 'po')}", f"ok|f:-:{BOUND}", + "a non-superuser owner tiers its own partition") diff --git a/test/tier_partition.sh b/test/tier_partition.sh index 03b3540d8..e543a5e3a 100755 --- a/test/tier_partition.sh +++ b/test/tier_partition.sh @@ -399,6 +399,50 @@ else "PostgreSQL $PGC_MAJOR has no virtual generated columns" fi +# ---- the caller's own privileges (#1341) ------------------------------------- +# +# tier_partition is not SECURITY DEFINER: every step runs as the caller, so the +# owner's ALTER TABLE ... DETACH is what stops a role that does not own the table. +# Made SECURITY DEFINER, it would run as the extension's owner and let any caller +# detach, drop and replace a partition it does not own. Two roles, each holding +# every other privilege the call needs (the server-file role for the export, USAGE +# on the server and CREATE on the schema for the foreign table, and SELECT on the +# partition), so ownership is the only thing between them. +for r in tier_own tier_peer; do + psql_run "DROP ROLE IF EXISTS $r; CREATE ROLE $r LOGIN NOSUPERUSER; + GRANT pg_write_server_files TO $r; GRANT USAGE ON FOREIGN SERVER tier_pq TO $r; + GRANT USAGE, CREATE ON SCHEMA public TO $r; GRANT USAGE ON SCHEMA pgcolumnar TO $r;" +done +sqlstate_as() { # sqlstate_as ROLE SQL -> the SQLSTATE SQL fails with as ROLE, or ok + local out + out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" -d "$PGC_DB" \ + -X -At -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$2" 2>&1 >/dev/null) + [ -z "$out" ] && { echo ok; return; } + echo "$out" | sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' | head -1 +} +build po +psql_run "ALTER TABLE po OWNER TO tier_own; ALTER TABLE po_jan OWNER TO tier_own; + ALTER TABLE po_feb OWNER TO tier_own; GRANT SELECT ON po, po_jan TO tier_peer;" +pb="$(digest po)" +check "premise: tier_own owns the parent and its partitions, and neither role is a superuser" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ + "tier_own|0" +check "premise: tier_peer can open a session and read the partition" \ + "$(sqlstate_as tier_peer "SELECT count(*) FROM po_jan")" "ok" +check "a role that does not own the table is refused by its own privileges (42501)" \ + "$(sqlstate_as tier_peer "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_peer.parquet')")" "42501" +check "and the refused call leaves the partition local and attached, every row once" \ + "$(digest po)|$(jan_shape po)" "$pb|r:pgcolumnar:$JAN_BOUND" +check_num "with no foreign table behind it" \ + "$(q "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid WHERE c.relname LIKE 'po\\_%';")" "0" +# Measured, not asserted: the export runs before the DETACH that refuses, so the +# file is written first. A role holding the server-file role and SELECT can write +# that file with export_parquet directly, so this grants nothing new. +echo "-- the refused call's export file: $([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" +check "a non-superuser owner tiers its own partition" \ + "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_own.parquet')")|$(jan_shape po)" \ + "ok|f:-:$JAN_BOUND" + # ---- a failure after each step ---------------------------------------------- for step in export detach drop create attach; do build f From a4b0ac6be4f89106e11baeaaaf32d6c49ef0871e Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 16:15:14 +0000 Subject: [PATCH 05/14] test: the caller-privilege owner controls get tables of their own, and the refused tier writes no file, in both twins (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/partition_lifecycle.sh | 9 ++++++--- test/pytest/TESTS.md | 4 ++-- test/pytest/test_partition_lifecycle.py | 13 +++++++++---- test/pytest/test_tier_partition.py | 20 ++++++++++++------- test/tier_partition.sh | 26 +++++++++++++++---------- 5 files changed, 46 insertions(+), 26 deletions(-) diff --git a/test/partition_lifecycle.sh b/test/partition_lifecycle.sh index 5eb52be5f..c8c89a1d9 100755 --- a/test/partition_lifecycle.sh +++ b/test/partition_lifecycle.sh @@ -262,15 +262,18 @@ as_role() { # as_role ROLE SQL -> the single value SQL returns as ROLE, or the S *) printf '%s\n' "$out" | tail -1 ;; esac } -q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own;" >/dev/null +# The owner's control has a parent of its own (pw2), so a refusal that let the peer +# through cannot make the control fail for the peer's reason. +q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own; + CREATE TABLE pw2 (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pw2 OWNER TO pl_own;" >/dev/null check "premise: pl_own owns the parent, and neither role is a superuser" \ - "$(q "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass;")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE oid IN ('pv'::regclass, 'pw2'::regclass);")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ "pl_own|0" check "premise: pl_peer can open a session" "$(as_role pl_peer "SELECT 1")" "1" check "a role that does not own the parent is refused by its own privileges (42501), and creates nothing" \ "$(as_role pl_peer "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "42501|0" check "a non-superuser owner creates partitions ahead on its own table" \ - "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "3|3" + "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pw2', '1 day', 2)")|$(kids pw2)" "3|3" # ==== the drop half: partition_drop_plan (#1297) ================================ # diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index effcdd57c..bb1d13ebb 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6643,7 +6643,7 @@ Removal proofs, each red in both twins: the prototype's zone handling; `CREATE T | --- | --- | | `test_partition_create_ahead` | from nothing it reaches now plus the lead, columnar, contiguous and one width wide; a second call creates nothing, and asking further ahead creates only the difference; rows land in columnar partitions; the parent is never held ACCESS EXCLUSIVE; west of UTC, for both key types, partitions continue a hand-made one contiguously and a row at every boundary routes; east of UTC, a timestamp parent covers the local now and starts on the width's boundary; inputs that broke the first version (a generated column, a 63-character name, more than 1000 partitions, a MAXVALUE bound) are handled or refused before any DDL; a DEFAULT partition is refused and nothing is created | | `test_partition_drop_plan` | the drop half: `partition_drop_plan` lists `DETACH ... CONCURRENTLY` and `DROP` for the partitions wholly before the cutoff, in range order, from partitions created newest first. A cutoff on an upper bound includes the partition, and one before them all plans nothing. Running the plan, each statement top-level, leaves the rest with their rows and drops the storage rows. A `timestamp` key with a `timestamp` cutoff plans the same in Tokyo and Denver, and a cutoff of the other type is refused (`42804`). MINVALUE is listed once past, and MAXVALUE never. Quoted schema and partition names run as returned. A sub-partitioned partition is planned whole and dropped with its children. A tiered partition is planned with `DROP FOREIGN TABLE`, and its Parquet file stays. A partition whose name holds the drop block's dollar-quote tag drops as planned. A detach cancelled in its second phase is planned as `FINALIZE` and dropped. The DEFAULT and non-timestamp parents are refused (`55000`) | -| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates its partitions ahead | +| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates partitions ahead on a parent of its own | ## 94. test_approx_aggregates.py: hll and tdigest over columnar tables, the compatibility statement @@ -6792,7 +6792,7 @@ On a private database, because foreign servers are per database. A local path st | `test_tier_partition_moves_a_partition_exactly_once` | the move returns the rows moved; the parent's answers are the same; the cold range is a foreign partition with the same name and bound, reading the exported file; no pgcolumnar catalog rows are left; pruning keeps the ranges apart; an insert into the tiered range is refused; five refusals move nothing; the parent is held ACCESS EXCLUSIVE until commit, and a rollback leaves the rows local; a write in flight is exported; a fault after each of the five steps leaves the rows exactly once, still local | | `test_tier_partition_refuses_what_it_cannot_move_and_survives_concurrency` | at each isolation level a row committed between the snapshot and the lock survives, and only READ COMMITTED moves; a UNIQUE parent and four kinds of dependent are refused with no file left, and the partition moves once they are gone; grants and comment carry over; a racing insert leaves every row once; lock_wait times a held parent out with nothing attached; the caller's lock_timeout survives; a retry over a failed attempt's file moves the partition | | `test_tier_partition_moves_a_virtual_generated_column` | PostgreSQL 18+: a partition whose parent has a virtual generated column moves and reads back the same; below 18, `UNSUPPORTED_MAJOR`, listed in `expected_unrunnable.txt` | -| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501`, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers its own partition | +| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501` before any file is written, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers a partition of its own table | ## 101. test_prepared_writes.py: a write prepared and finished in one session leaves the backend alive diff --git a/test/pytest/test_partition_lifecycle.py b/test/pytest/test_partition_lifecycle.py index 814a912c2..67791e7c0 100644 --- a/test/pytest/test_partition_lifecycle.py +++ b/test/pytest/test_partition_lifecycle.py @@ -611,8 +611,11 @@ def test_partition_create_ahead_runs_on_the_callers_own_privileges(pgc_conn, pgc "EXCEPTION WHEN duplicate_object THEN NULL; END $$") cur.execute(f'GRANT USAGE, CREATE ON SCHEMA "{schema}" TO {r}') cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") - cur.execute("CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") - cur.execute("ALTER TABLE pv OWNER TO pl_own_py") + # The owner's control has a parent of its own (pw2), so a refusal that let the + # peer through cannot make the control fail for the peer's reason. + for t in ("pv", "pw2"): + cur.execute(f"CREATE TABLE {t} (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") + cur.execute(f"ALTER TABLE {t} OWNER TO pl_own_py") def as_role(role, sql): try: @@ -624,12 +627,14 @@ def as_role(role, sql): return e.sqlstate or "?" ahead_pv = "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)" - owner = _one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'pv'::regclass") + owner = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" + " WHERE oid IN ('pv'::regclass, 'pw2'::regclass)") supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own_py', 'pl_peer_py') AND rolsuper") expect.text(f"{owner}|{supers}", "pl_own_py|0", "premise: pl_own owns the parent, and neither role is a superuser") expect.text(as_role("pl_peer_py", "SELECT 1"), "1", "premise: pl_peer can open a session") expect.text(f"{as_role('pl_peer_py', ahead_pv)}|{_kids(conn, 'pv')}", "42501|0", "a role that does not own the parent is refused by its own privileges (42501), and creates nothing") - expect.text(f"{as_role('pl_own_py', ahead_pv)}|{_kids(conn, 'pv')}", + ahead_pw2 = "SELECT pgcolumnar.partition_create_ahead('pw2', '1 day', 2)" + expect.text(f"{as_role('pl_own_py', ahead_pw2)}|{_kids(conn, 'pw2')}", "3|3", "a non-superuser owner creates partitions ahead on its own table") diff --git a/test/pytest/test_tier_partition.py b/test/pytest/test_tier_partition.py index b994d3480..929b99a17 100644 --- a/test/pytest/test_tier_partition.py +++ b/test/pytest/test_tier_partition.py @@ -549,10 +549,14 @@ def test_tier_partition_runs_on_the_callers_own_privileges(pgc_own_db, pgc_clust cur.execute(f"GRANT USAGE ON FOREIGN SERVER tq TO {r}") cur.execute(f"GRANT USAGE, CREATE ON SCHEMA public TO {r}") cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") - _build(conn, "po") + # The owner's control has a table of its own (pw), so a refusal that let the peer + # through cannot make the control fail for the peer's reason. + for t in ("po", "pw"): + _build(conn, t) + with conn.cursor() as cur: + for r in (t, f"{t}_mar", f"{t}_apr"): + cur.execute(f"ALTER TABLE {r} OWNER TO tier_own_py") with conn.cursor() as cur: - for t in ("po", "po_mar", "po_apr"): - cur.execute(f"ALTER TABLE {t} OWNER TO tier_own_py") cur.execute("GRANT SELECT ON po, po_mar TO tier_peer_py") before = _digest(conn, "po") @@ -567,7 +571,7 @@ def as_role(role, sql): return e.sqlstate or "?" owners = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" - " WHERE relname IN ('po', 'po_mar', 'po_apr')") + " WHERE relname IN ('po', 'po_mar', 'po_apr', 'pw', 'pw_mar', 'pw_apr')") supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own_py', 'tier_peer_py') AND rolsuper") expect.text(f"{owners}|{supers}", "tier_own_py|0", "premise: tier_own owns the parent and its partitions, and neither role is a superuser") @@ -580,7 +584,9 @@ def as_role(role, sql): "and the refused call leaves the partition local and attached, every row once") expect.num(_one(conn, "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid" " WHERE c.relname LIKE 'po\\_%'"), 0, "with no foreign table behind it") - print(f"-- the refused call's export file: {'written' if peer_path.exists() else 'absent'}") - expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{server_dir / 'po_own.parquet'}')") - + f"|{_mar_shape(conn, 'po')}", f"ok|f:-:{BOUND}", + # The LOCK ... IN SHARE MODE ahead of the export is the first statement to need + # more than SELECT, so the refusal comes before any file is written. + expect.text("written" if peer_path.exists() else "absent", "absent", "and the refused call wrote no export file") + expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('pw_mar', 'tq', '{server_dir / 'pw_own.parquet'}')") + + f"|{_mar_shape(conn, 'pw')}", f"ok|f:-:{BOUND}", "a non-superuser owner tiers its own partition") diff --git a/test/tier_partition.sh b/test/tier_partition.sh index e543a5e3a..186370e20 100755 --- a/test/tier_partition.sh +++ b/test/tier_partition.sh @@ -402,7 +402,8 @@ fi # ---- the caller's own privileges (#1341) ------------------------------------- # # tier_partition is not SECURITY DEFINER: every step runs as the caller, so the -# owner's ALTER TABLE ... DETACH is what stops a role that does not own the table. +# owner-only steps (its LOCK ... IN SHARE MODE first) stop a role that does not own +# the table. # Made SECURITY DEFINER, it would run as the extension's owner and let any caller # detach, drop and replace a partition it does not own. Two roles, each holding # every other privilege the call needs (the server-file role for the export, USAGE @@ -420,12 +421,17 @@ sqlstate_as() { # sqlstate_as ROLE SQL -> the SQLSTATE SQL fails with as ROLE, o [ -z "$out" ] && { echo ok; return; } echo "$out" | sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' | head -1 } -build po -psql_run "ALTER TABLE po OWNER TO tier_own; ALTER TABLE po_jan OWNER TO tier_own; - ALTER TABLE po_feb OWNER TO tier_own; GRANT SELECT ON po, po_jan TO tier_peer;" +# The owner's control has a table of its own (pw), so a refusal that let the peer +# through cannot make the control fail for the peer's reason. +for t in po pw; do + build $t + psql_run "ALTER TABLE $t OWNER TO tier_own; ALTER TABLE ${t}_jan OWNER TO tier_own; + ALTER TABLE ${t}_feb OWNER TO tier_own;" +done +psql_run "GRANT SELECT ON po, po_jan TO tier_peer;" pb="$(digest po)" check "premise: tier_own owns the parent and its partitions, and neither role is a superuser" \ - "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb', 'pw', 'pw_jan', 'pw_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ "tier_own|0" check "premise: tier_peer can open a session and read the partition" \ "$(sqlstate_as tier_peer "SELECT count(*) FROM po_jan")" "ok" @@ -435,12 +441,12 @@ check "and the refused call leaves the partition local and attached, every row o "$(digest po)|$(jan_shape po)" "$pb|r:pgcolumnar:$JAN_BOUND" check_num "with no foreign table behind it" \ "$(q "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid WHERE c.relname LIKE 'po\\_%';")" "0" -# Measured, not asserted: the export runs before the DETACH that refuses, so the -# file is written first. A role holding the server-file role and SELECT can write -# that file with export_parquet directly, so this grants nothing new. -echo "-- the refused call's export file: $([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" +# The first statement to need more than SELECT is the LOCK ... IN SHARE MODE ahead +# of the export, so the refusal comes before any file is written (measured on 15-19). +check "and the refused call wrote no export file" \ + "$([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" "absent" check "a non-superuser owner tiers its own partition" \ - "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_own.parquet')")|$(jan_shape po)" \ + "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('pw_jan', 'tier_pq', '$W/pw_own.parquet')")|$(jan_shape pw)" \ "ok|f:-:$JAN_BOUND" # ---- a failure after each step ---------------------------------------------- From bfa284c960061776b15ec263cae2937e03835cb6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 16:22:33 +0000 Subject: [PATCH 06/14] test: the caller-privilege arms enter the ledger, five dated by SECURITY DEFINER (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 11 +++++++++++ test/check_ledger_budget.txt | 16 +++++++++++++++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 54b8eaed2..098b45e26 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2399,6 +2399,7 @@ partition_lifecycle partition_lifecycle a cutoff on a partition's upper bound in partition_lifecycle partition_lifecycle a drop plan for a parent not RANGE-partitioned on one column is refused, LIST and two-column RANGE included 15;16;17;18;19 2026-10-14 partition_drop_candidates refusing only a parent with no partition key, not LIST or a multi-column RANGE;partition_drop_candidates without its RANGE-on-one-column check partition_lifecycle partition_lifecycle a drop plan for a parent not partitioned on a timestamp is refused 15;16;17;18;19 2026-10-14 a parent partitioned on a non-timestamp key planned instead of refused;partition_drop_candidates without its timestamp-key check partition_lifecycle partition_lifecycle a drop plan for a parent with a DEFAULT partition is refused 15;16;17;18;19 2026-10-13 a parent with a DEFAULT partition planned instead of refused +partition_lifecycle partition_lifecycle a non-superuser owner creates partitions ahead on its own table 15;16;17;18;19 never - partition_lifecycle partition_lifecycle a parent partitioned on a key that is not a timestamp is refused before any DDL 15;16;17;18;19 2026-10-13 partition_create_ahead without its timestamp key check partition_lifecycle partition_lifecycle a parent that is not RANGE-partitioned on one column is refused before any DDL 15;16;17;18;19 2026-10-13 partition_create_ahead without its RANGE-on-one-column check partition_lifecycle partition_lifecycle a parent with a 63-byte name gets partitions whose names keep their dates 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it;partition_create_ahead stopping one width short of now plus the lead;partition_create_ahead without shortening a long parent name;partition_create_ahead's name bound one byte too long @@ -2410,6 +2411,7 @@ partition_lifecycle partition_lifecycle a partition named with the drop block's partition_lifecycle partition_lifecycle a partition that is itself partitioned is planned whole 15;16;17;18;19 2026-10-13 a partition that is itself partitioned left out of the plan;a partition whose upper bound equals the cutoff left out;a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY partition_lifecycle partition_lifecycle a pending detach is finalised, not started again 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;a pending detach planned as a new concurrent detach, not FINALIZE;a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing partition_lifecycle partition_lifecycle a plan for names that need quoting runs as returned 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;names in the plan with their quotes removed;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing +partition_lifecycle partition_lifecycle a role that does not own the parent is refused by its own privileges (42501), and creates nothing 15;16;17;18;19 2026-10-14 partition_create_ahead made SECURITY DEFINER partition_lifecycle partition_lifecycle a second call creates nothing 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it;partition_create_ahead starting from now, not the highest upper bound partition_lifecycle partition_lifecycle a second plan for the same cutoff is empty, and the partitions it kept are still attached 15;16;17;18;19 2026-10-13 the cutoff filter removed: every partition planned;the lower bound read where the upper is meant partition_lifecycle partition_lifecycle a tiered partition is planned with DROP FOREIGN TABLE 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;a plain DROP, not refusing while the partition is still attached;a tiered partition planned with DROP TABLE;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY @@ -2441,6 +2443,8 @@ partition_lifecycle partition_lifecycle premise: a partition is named with the d partition_lifecycle partition_lifecycle premise: a timestamp parent continued from a hand-made partition, in a session west of UTC 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: a timestamp parent was given partitions from a session east of UTC 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: a timestamptz parent continued from a hand-made partition, in a session west of UTC 15;16;17;18;19 never - +partition_lifecycle partition_lifecycle premise: pl_own owns the parent, and neither role is a superuser 15;16;17;18;19 never - +partition_lifecycle partition_lifecycle premise: pl_peer can open a session 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the January partition was tiered to Parquet 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the call inside the transaction created partitions 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the drop fixture has several partitions, each holding rows 15;16;17;18;19 never - @@ -2884,6 +2888,7 @@ tier_partition tier_partition a failure after drop leaves the rows exactly once, tier_partition tier_partition a failure after export leaves the rows exactly once, still local 15;16;17;18;19 never - tier_partition tier_partition a fault step that is not one of tier_partition's is refused 15;16;17;18;19 2026-10-13 the tier_fault_after check hook accepting any value tier_partition tier_partition a grant to a role whose name needs quoting is carried to the foreign partition 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition not carrying the grants;tier_partition returning 0 rows moved +tier_partition tier_partition a non-superuser owner tiers its own partition 15;16;17;18;19 never - tier_partition tier_partition a parent held by a long reader times the move out, as lock_wait says 15;16;17;18;19 2026-10-13 tier_partition without the lock_wait bound tier_partition tier_partition a parent with a UNIQUE index is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without the UNIQUE-parent refusal tier_partition tier_partition a partition a view depends on is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without the dependents check @@ -2896,6 +2901,7 @@ tier_partition tier_partition a partition with a trigger of its own is refused b tier_partition tier_partition a query on the other range scans only the local partition, beside a foreign one 15;16;17;18;19 never - tier_partition tier_partition a query on the tiered range scans only the foreign partition 15;16;17;18;19 never - tier_partition tier_partition a retry over the failed attempt's file moves the partition, exactly once 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition refusing every existing path (the first version's rule);tier_partition returning 0 rows moved +tier_partition tier_partition a role that does not own the table is refused by its own privileges (42501) 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition a rollback after the whole call leaves the rows local, exactly once 15;16;17;18;19 never - tier_partition tier_partition a server of another wrapper is refused 15;16;17;18;19 2026-10-14 tier_partition without its wrong-wrapper branch, so the iceberg wrapper refuses only after the export;tier_partition without the wrapper check tier_partition tier_partition a server that does not exist is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without its missing-server branch, so core refuses only after the export @@ -2913,6 +2919,8 @@ tier_partition tier_partition and no foreign table behind it after detach 15;16; tier_partition tier_partition and no foreign table behind it after drop 15;16;17;18;19 never - tier_partition tier_partition and no foreign table behind it after export 15;16;17;18;19 never - tier_partition tier_partition and the refusals moved nothing 15;16;17;18;19 2026-10-13 tier_partition without the path refusal +tier_partition tier_partition and the refused call leaves the partition local and attached, every row once 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER +tier_partition tier_partition and the refused call wrote no export file 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition and the timed-out move leaves the rows local, with nothing attached 15;16;17;18;19 2026-10-13 tier_partition without the lock_wait bound tier_partition tier_partition at read committed a row committed between the caller's snapshot and the lock survives 15;16;17;18;19 never - tier_partition tier_partition at read committed the move goes ahead and exports that row 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition returning 0 rows moved @@ -2937,6 +2945,8 @@ tier_partition tier_partition premise: the first attempt failed after export and tier_partition tier_partition premise: the move started while the insert was in flight, and waited for it 15;16;17;18;19 never - tier_partition tier_partition premise: the parent holds every row 15;16;17;18;19 never - tier_partition tier_partition premise: the storage id was read before the move 15;16;17;18;19 never - +tier_partition tier_partition premise: tier_own owns the parent and its partitions, and neither role is a superuser 15;16;17;18;19 never - +tier_partition tier_partition premise: tier_peer can open a session and read the partition 15;16;17;18;19 never - tier_partition tier_partition the answers over the parent are the same after the move 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix tier_partition tier_partition the call holds ACCESS EXCLUSIVE on the parent until commit, as the docs say 15;16;17;18;19 never - tier_partition tier_partition the cold partition is now a foreign partition with its own name and bound 15;16;17;18;19 2026-10-13 tier_partition attaching with the upper bound shifted a minute;tier_partition naming the foreign table with a suffix @@ -2947,6 +2957,7 @@ tier_partition tier_partition the local partition leaves no pgcolumnar storage r tier_partition tier_partition the move ran inside the caller's transaction 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved tier_partition tier_partition the other partition is still local after the refusals 15;16;17;18;19 2026-10-13 tier_partition without the path refusal tier_partition tier_partition tier_partition moves the cold partition and returns the rows it moved 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved +tier_partition tier_partition with no foreign table behind it 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition with the dependents gone, the same partition moves 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved;tier_partition without the dependents check timeseries_recipes timeseries_recipes premise: the readings table is columnar 15;16;17;18;19 never - timeseries_recipes timeseries_recipes premise: the reference has 24 hours, 6 of them empty and one all-NULL 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index e8678ba9d..7f5b72915 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2545,4 +2545,18 @@ suites_not_covered 230 # never 2345 + 1 = 2346 # dated 770 + 2 = 772 # 2346 + 772 == 3118 -checks_never_observed_red 2346 +# +# #1341, tier_partition and partition_create_ahead on the caller's own privileges +# (OffgridwithJD): 2346 -> 2352. Eleven new checks, all in already-covered suites. +# Five are dated, by each function made SECURITY DEFINER: tier_partition's refusal, +# the partition left local, no foreign table and no export file (four), and +# partition_create_ahead's refusal (one). The two owner controls run on tables of +# their own, so neither mutation reddens them; they and four premises arrive never. +# Five-major runs: tier_partition 79/79 (80 on 18-19, which run the virtual column) +# and partition_lifecycle 64/64 on 15-19. +# +# ledger rows 3118 + 11 = 3129 +# never 2346 + 6 = 2352 +# dated 772 + 5 = 777 +# 2352 + 777 == 3129 +checks_never_observed_red 2352 From 278615052b5fe87a4c49689835b87230ffaff984 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:13:58 +0000 Subject: [PATCH 07/14] test: rls_direct_storage's owner-export arm measures the export, read back; both its branches echoed "ran" (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/rls_direct_storage.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 355bda4b9..8c301443f 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -166,8 +166,11 @@ check "and still exportable by it" \ owner_rows="$(q 'SELECT count(*) FROM rls_t;')" check "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)" \ "$(q "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1');" 2>&1 | head -1)" "$owner_rows" -check "and the owner can still export it" \ - "$([ -n "$(q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" 2>&1)" ] && echo ran || echo ran)" "ran" +# Measured as the rows the export holds, read back: both branches of the old test +# here echoed "ran", so a refused owner export passed it. +q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" >/dev/null 2>&1 +check_num "and the owner can still export it" \ + "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo absent)" "$owner_rows" # ---- FORCE ROW LEVEL SECURITY reaches the owner too ------------------------- # From cd0f18dcc933d00cd6ebd350956ebdc825eac7c2 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:13:58 +0000 Subject: [PATCH 08/14] test: rls_direct_storage gets its pytest twin, every check by the shell's name (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/pytest/TESTS.md | 11 ++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_rls_direct_storage.py | 187 +++++++++++++++++++++++++ 4 files changed, 202 insertions(+), 1 deletion(-) create mode 100644 test/pytest/test_rls_direct_storage.py diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index bb1d13ebb..648373c3e 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -170,6 +170,7 @@ behaviour, the source of that number is named. - [122. test_prepared_across_sessions.py: a prepared transaction, seen from other sessions and across a crash](#122-test_prepared_across_sessionspy-a-prepared-transaction-seen-from-other-sessions-and-across-a-crash) - [123. test_maintenance_crash.py: a crash part-way through maintenance loses nothing](#123-test_maintenance_crashpy-a-crash-part-way-through-maintenance-loses-nothing) - [124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table](#124-test_serializablepy-serializable-refuses-a-write-skew-on-a-columnar-table) +- [125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts](#125-test_rls_direct_storagepy-the-direct-storage-paths-refuse-a-caller-a-policy-restricts) ## 1. How to read a test in here @@ -7100,3 +7101,13 @@ The pytest twin of `test/serializable.sh` (#1341), every check by the shell's na | test | what it holds | | --- | --- | | `test_serializable_refuses_a_write_skew` | the catalog SIRead locks, the four write skews refused by SSI with 40001 against heap, and the REPEATABLE READ controls | + +## 125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts + +The pytest twin of `test/rls_direct_storage.sh` (#563, ported for #1341), every check by the shell's name. Row-level security is applied by the rewriter, and these functions read and write storage with no query to rewrite, so each refuses a relation whose policies apply to the caller. The caller HOLDS SELECT, INSERT, EXECUTE and the server-file roles, so the RLS check is the only refuser left. Eight entry points are refused and nothing was written or imported; a role with no privilege is told "permission denied" and not that RLS exists; a table with no policy, the owner without FORCE, and the owner's export (read back, row for row) stay allowed; and under FORCE a non-superuser owner is refused too. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_direct_storage_paths_refuse_a_policy_restricted_caller` | the premises, the eight refusals and their absent effects, the no-privilege caller's ACL error, the not-over-refusing controls, and the FORCE pair | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index cb3b19a84..67d1b5061 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -707,4 +707,6 @@ guard_tests 438 # by collection over the cluster files on this tree. # cluster_tests 588 -> 590 for #1341's caller-privilege tests in test_tier_partition.py and # test_partition_lifecycle.py, one each, by collection over the cluster files on this tree. -cluster_tests 590 +# cluster_tests 590 -> 591 for #1341's test_rls_direct_storage.py, the twin of +# rls_direct_storage.sh: one test, by collection over the cluster files on this tree. +cluster_tests 591 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 36270ee64..a1b5f034b 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -145,6 +145,7 @@ "projections", "recovery_minimal_wal", "rewrite_storage_oid", + "rls_direct_storage", "rollup", "scan_decode_cost", "serializable", diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py new file mode 100644 index 000000000..fd3397e20 --- /dev/null +++ b/test/pytest/test_rls_direct_storage.py @@ -0,0 +1,187 @@ +"""The direct-storage paths refuse a relation with RLS in force (#563): the pytest twin +of `test/rls_direct_storage.sh` (#1341). Every check carries the shell's name. + +Row-level security is applied by the REWRITER to a query's range table entry. These +functions open the relation's storage and read or write it, so no policy is ever +consulted. The fixture that matters is a role that HOLDS SELECT and INSERT and is +restricted by a policy: a role with no privilege is refused by the ACL check first, +and with the RLS guard deleted a no-privilege arm stays green. Only a granted but +restricted caller reaches the RLS check with every other refuser cleared. + +The bar is check_enable_rls(...) == RLS_ENABLED. The owner without FORCE (and a +superuser) is RLS_NONE_ENV and stays allowed; under FORCE a NON-superuser owner is +refused too, which is why an owner-only rule would not close the hole. + +REAL LOGINS, NOT `SET ROLE`, as in the shell twin. Every role holds USAGE on this +test's schema, as the shell's roles hold it on public, so a "permission denied" is +never the schema's. +""" +import os + +import pytest + +RLS, NONE, OWNER = "t_rls_py", "t_rlsnone_py", "t_owner_rls_py" +ROWS = 200 + + +def _login(cluster, role, schema, sql): + """-> (first value or None, error or None) for `sql` on a fresh login as `role`.""" + import psycopg + dsn = f"host=127.0.0.1 port={cluster.port} user={role} dbname=postgres" + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(f'SET search_path TO "{schema}", pgcolumnar, public') + cur = c.execute(sql) + row = cur.fetchone() if cur.description else None + return (row[0] if row else None), None + except psycopg.Error as e: + return None, e + + +def _verdict(cluster, schema, sql, role=RLS): + """-> rls | acl | allowed | other: , as the shell's verdict.""" + _, err = _login(cluster, role, schema, sql) + if err is None: + return "allowed" + msg = err.diag.message_primary or "" + if "row-level security" in msg: + return "rls" + if "permission denied" in msg: + return "acl" + return f"other: {err.sqlstate} {msg}" + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + row = cur.fetchone() + return row[0] if row else None + + +def test_the_direct_storage_paths_refuse_a_policy_restricted_caller(pgc_conn, pgc_cluster, server_dir, expect): + conn = pgc_conn + out = str(server_dir) + schema = _one(conn, "SELECT current_schema()") + with conn.cursor() as cur: + cur.execute("CREATE TABLE rls_t (id int, owner_name text, ssn text) USING pgcolumnar") + cur.execute(f"INSERT INTO rls_t SELECT g, CASE WHEN g = 1 THEN '{RLS}' ELSE 'someone' END," + f" 'ssn-'||g FROM generate_series(1,{ROWS}) g") + cur.execute("CREATE TABLE plain_t (id int, v text) USING pgcolumnar") + cur.execute(f"INSERT INTO plain_t SELECT g, 'v'||g FROM generate_series(1,{ROWS}) g") + for t in ("rls_t", "plain_t"): + cur.execute(f"SELECT pgcolumnar.add_projection('{t}','p1',ARRAY['id'],ARRAY['id'])") + cur.execute(f"SELECT pgcolumnar.rebuild_projections('{t}')") + for r in (RLS, NONE, OWNER): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} NOSUPERUSER LOGIN; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + cur.execute(f'GRANT USAGE ON SCHEMA "{schema}" TO {r}') + cur.execute(f"GRANT pg_read_server_files, pg_write_server_files TO {RLS}") + # SELECT and INSERT DELIBERATELY: this role must clear every bar but RLS. + cur.execute(f"GRANT SELECT, INSERT ON rls_t TO {RLS}") + cur.execute(f"GRANT SELECT, INSERT ON plain_t TO {RLS}") + # #567 revoked EXECUTE on the projection readers from PUBLIC; without these + # grants the projection arms stop at the grant layer and prove nothing here. + for r in (RLS, NONE, OWNER): + cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO {r}") + cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO {RLS}") + cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") + cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") + # Sources exported from rls_t BY THE OWNER, so their shape matches the import + # target: a differently shaped source fails on column counts, not on RLS. + cur.execute(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/src.parquet')") + cur.execute(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/src.arrow')") + cur.execute(f"COPY rls_t TO '{out}/src.txt'") + + def as_rls(sql): + return _login(pgc_cluster, RLS, schema, sql) + + # ---- premises, every one run BY the restricted role + expect.text(str(as_rls("SELECT 1")[0]), "1", "premise: the role can open a session") + expect.text(str(as_rls("SELECT has_table_privilege('rls_t','SELECT')")[0]), "True", + "premise: the role HOLDS select, so the ACL check cannot be what refuses") + got, _ = as_rls("SELECT count(*) FROM rls_t") + expect.num(-1 if got is None else got, 1, "premise: the policy is in force for ordinary SQL") + expect.text(str(as_rls("SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE')")[0]), + "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") + expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, + "premise: the owner sees every row, so the policy hides something real") + expect.text("yes" if all(os.path.getsize(f"{out}/{f}") > 0 for f in ("src.parquet", "src.arrow", "src.txt")) + else "no", "yes", "premise: the source files the import arms need exist") + + # ---- the refusals: on unfixed code each returns "allowed", which is the leak + def v(sql): + return _verdict(pgc_cluster, schema, sql) + + expect.text(v("SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), "rls", + "read_projection refuses a policy-restricted caller") + expect.text(v("SELECT count(*) FROM pgcolumnar.reconstruct_via_projection('rls_t','p1')"), "rls", + "reconstruct_via_projection refuses one too") + expect.text(v(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/leak1.parquet')"), "rls", + "export_parquet refuses one") + expect.text(v(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/leak2.arrow')"), "rls", + "export_arrow refuses one") + expect.text(v(f"SELECT pgcolumnar.parallel_export_parquet('rls_t','{out}/leak3',2)"), "rls", + "parallel_export_parquet refuses one") + expect.text(v(f"SELECT pgcolumnar.import_parquet('rls_t','{out}/src.parquet')"), "rls", + "import_parquet refuses a policy-restricted writer") + expect.text(v(f"SELECT pgcolumnar.import_arrow('rls_t','{out}/src.arrow')"), "rls", + "import_arrow refuses one") + expect.text(v(f"SELECT pgcolumnar.parallel_copy('rls_t','{out}/src.txt',2)"), "rls", + "parallel_copy refuses one") + + # The refusal must not have done the work anyway. A message is not a behaviour. + written = [f for f in ("leak1.parquet", "leak2.arrow", "leak3") if os.path.exists(f"{out}/{f}")] + expect.text(" ".join(written) or "none", "none", "no export file was written for the refused caller") + expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, "and no imported row landed in the restricted table") + + # ---- a caller WITHOUT privilege sees the ACL error, not the RLS one + _, err = _login(pgc_cluster, NONE, schema, "SELECT count(*) FROM rls_t") + expect.num(1 if err is not None and "permission denied" in (err.diag.message_primary or "") else 0, 1, + "premise: the no-privilege role is told permission denied by ordinary SQL") + _, err = _login(pgc_cluster, NONE, schema, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')") + msg = (err.diag.message_primary or "") if err is not None else "" + expect.num(1 if "permission denied" in msg else 0, 1, + "and read_projection tells it the same thing, not that RLS exists") + expect.num(1 if "row-level security" in msg else 0, 0, + "so no caller without privilege learns RLS state from these functions") + + # ---- not over-refusing: a fix that refused everyone would pass the arms above + expect.text(v("SELECT count(*) FROM pgcolumnar.read_projection('plain_t','p1')"), "allowed", + "a table with NO policy is still readable by the same role") + expect.text(v(f"SELECT pgcolumnar.export_parquet('plain_t','{out}/ok1.parquet')"), "allowed", + "and still exportable by it") + # Against the LIVE count: on unfixed code the import arms leak rows into rls_t. + owner_rows = _one(conn, "SELECT count(*) FROM rls_t") + expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), owner_rows, + "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") + import psycopg + try: + _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") + except psycopg.Error: + pass + exported = -1 # no file: a refused export is a red arm, not a crash + if os.path.exists(f"{out}/owner.parquet"): + exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," + " owner_name text, ssn text)") + expect.num(exported, owner_rows, "and the owner can still export it") + + # ---- FORCE ROW LEVEL SECURITY reaches a NON-superuser owner too + with conn.cursor() as cur: + cur.execute("CREATE TABLE force_t (id int, owner_name text) USING pgcolumnar") + cur.execute("INSERT INTO force_t SELECT g, 'nobody' FROM generate_series(1,50) g") + cur.execute("SELECT pgcolumnar.add_projection('force_t','p1',ARRAY['id'],ARRAY['id'])") + cur.execute("SELECT pgcolumnar.rebuild_projections('force_t')") + cur.execute(f"ALTER TABLE force_t OWNER TO {OWNER}") + cur.execute("ALTER TABLE force_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_none ON force_t FOR SELECT TO {OWNER} USING (owner_name = current_user)") + expect.text(_one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'force_t'::regclass"), + OWNER, "premise: t_owner owns force_t") + got, _ = _login(pgc_cluster, OWNER, schema, "SELECT count(*) FROM pgcolumnar.read_projection('force_t','p1')") + expect.num(-1 if got is None else got, 50, "without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED") + with conn.cursor() as cur: + cur.execute("ALTER TABLE force_t FORCE ROW LEVEL SECURITY") + _, err = _login(pgc_cluster, OWNER, schema, "SELECT count(*) FROM pgcolumnar.read_projection('force_t','p1')") + expect.num(1 if err is not None and "row-level security" in (err.diag.message_primary or "") else 0, 1, + "under FORCE the non-superuser owner IS refused, so owner-only is no substitute") From 4a02d4a89ef2f727037a9a4eddd4f68dbd7a95f0 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:17:49 +0000 Subject: [PATCH 09/14] test: rls_direct_storage makes its import sources before RLS is on, and every owner arm reports a number when refused, in both twins (#1341) A guard widened to refuse the owner stopped the fixture's own export, so a premise reddened for the product code and the pytest twin crashed in setup rather than failing an arm by name. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/pytest/test_rls_direct_storage.py | 16 ++++++++++------ test/rls_direct_storage.sh | 12 +++++++----- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py index fd3397e20..624396f55 100644 --- a/test/pytest/test_rls_direct_storage.py +++ b/test/pytest/test_rls_direct_storage.py @@ -85,14 +85,15 @@ def test_the_direct_storage_paths_refuse_a_policy_restricted_caller(pgc_conn, pg for r in (RLS, NONE, OWNER): cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO {r}") cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO {RLS}") - cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") - cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") - cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") # Sources exported from rls_t BY THE OWNER, so their shape matches the import # target: a differently shaped source fails on column counts, not on RLS. + # BEFORE RLS is enabled, so no version of the guard under test can stop them. cur.execute(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/src.parquet')") cur.execute(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/src.arrow')") cur.execute(f"COPY rls_t TO '{out}/src.txt'") + cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") + cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") def as_rls(sql): return _login(pgc_cluster, RLS, schema, sql) @@ -154,14 +155,17 @@ def v(sql): "and still exportable by it") # Against the LIVE count: on unfixed code the import arms leak rows into rls_t. owner_rows = _one(conn, "SELECT count(*) FROM rls_t") - expect.num(_one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), owner_rows, - "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") import psycopg + try: + owner_read = _one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')") + except psycopg.Error: + owner_read = -1 # refused: a red arm by name, not a crash + expect.num(owner_read, owner_rows, "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") try: _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") except psycopg.Error: pass - exported = -1 # no file: a refused export is a red arm, not a crash + exported = 0 # no file holds no rows: a refused export is a red arm, not a crash if os.path.exists(f"{out}/owner.parquet"): exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," " owner_name text, ssn text)") diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 8c301443f..f8254e659 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -58,16 +58,18 @@ psql_run "GRANT SELECT, INSERT ON plain_t TO t_rls;" # the RLS check, so they would report acl and prove nothing about this fix. psql_run "GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO t_rls;" psql_run "GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO t_rls;" -psql_run "ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY;" -psql_run "CREATE POLICY p_own ON rls_t FOR SELECT TO t_rls USING (owner_name = current_user);" -psql_run "CREATE POLICY p_ins ON rls_t FOR INSERT TO t_rls WITH CHECK (owner_name = current_user);" -# A source file the role may import, produced by the owner from the open table. +# A source file the role may import, produced by the owner from the open table, +# BEFORE row-level security is enabled, so no version of the guard under test can +# stop the fixture from being made. # Exported from rls_t BY THE OWNER, so the source shape matches the import # target. A source taken from a differently shaped table makes the import arms # fail on column counts, which looks like a refusal and is not one. psql_run "SELECT pgcolumnar.export_parquet('rls_t','$OUT/src.parquet');" psql_run "SELECT pgcolumnar.export_arrow('rls_t','$OUT/src.arrow');" psql_run "COPY rls_t TO '$OUT/src.txt';" +psql_run "ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY;" +psql_run "CREATE POLICY p_own ON rls_t FOR SELECT TO t_rls USING (owner_name = current_user);" +psql_run "CREATE POLICY p_ins ON rls_t FOR INSERT TO t_rls WITH CHECK (owner_name = current_user);" as_rls() { # as_rls -> the ERROR line, connfail, or noerror local out @@ -170,7 +172,7 @@ check "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)" \ # here echoed "ran", so a refused owner export passed it. q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" >/dev/null 2>&1 check_num "and the owner can still export it" \ - "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo absent)" "$owner_rows" + "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo 0)" "$owner_rows" # ---- FORCE ROW LEVEL SECURITY reaches the owner too ------------------------- # From 5f1fe16c31b38919836afd38a66a279057f10f9a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 14:22:00 +0000 Subject: [PATCH 10/14] test: rls_direct_storage seeds the ledger, sixteen arms dated by three mutations of the RLS guard (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 26 ++++++++++++++++++++++++++ test/check_ledger_budget.txt | 20 ++++++++++++++++++-- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 098b45e26..0f2683fd0 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2643,6 +2643,32 @@ rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partitio rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp ATTACH PARTITION lp_3 FOR VALUES FROM ('2026-03-01') TO ('2026-04-01') 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 CONCURRENTLY 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) +rls_direct_storage rls_direct_storage a table with NO policy is still readable by the same role 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage and no imported row landed in the restricted table 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage and read_projection tells it the same thing, not that RLS exists 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL +rls_direct_storage rls_direct_storage and still exportable by it 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage export_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage import_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage import_parquet refuses a policy-restricted writer 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage no export file was written for the refused caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage parallel_copy refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage parallel_export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage premise: t_owner owns force_t 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the no-privilege role is told permission denied by ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the owner sees every row, so the policy hides something real 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the policy is in force for ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the source files the import arms need exist 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage read_projection refuses a policy-restricted caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL +rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function rollup rollup DROP TABLE of a rollup's source is refused while the rollup exists (2BP01, #1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) rollup rollup DROP VIEW of a rollup takes its partials and its catalog rows with it (#1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 7f5b72915..97b3f6cc6 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 230 +suites_not_covered 229 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2559,4 +2559,20 @@ suites_not_covered 230 # never 2346 + 6 = 2352 # dated 772 + 5 = 777 # 2352 + 777 == 3129 -checks_never_observed_red 2352 +# +# #1341, rls_direct_storage seeded with its pytest twin (OffgridwithJD): 2352 -> 2362, +# and suites_not_covered 230 -> 229. The suite had no rows. It enters with all 26 +# checks from five-major runs (26/26 on each of 15-19). Sixteen are dated by three +# mutations of the RLS guard: deleted (the eight refusals, nothing written, nothing +# imported, and FORCE), widened to refuse RLS_NONE_ENV (the owner's read, the owner's +# export and the owner without FORCE), and moved ahead of read_projection's ACL check +# (the two disclosure arms). The owner's export was measured by a test both of whose +# branches echoed "ran"; it now reads the export back, and main's suite survives the +# widened guard on it. The other 10 arrive never: the premises and the no-policy +# controls. +# +# ledger rows 3129 + 26 = 3155 +# never 2352 + 10 = 2362 +# dated 777 + 16 = 793 +# 2362 + 793 == 3155 +checks_never_observed_red 2362 From 553d23882fc506d505bf6b49fa4ccc791f640873 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 18:17:31 +0000 Subject: [PATCH 11/14] test: rls_direct_storage's two scan-dependent premises become checks, in both twins (#1341, #1397 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 4 ++-- test/pytest/test_rls_direct_storage.py | 4 ++-- test/rls_direct_storage.sh | 9 +++++++-- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 0f2683fd0..f518976dd 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2657,8 +2657,6 @@ rls_direct_storage rls_direct_storage parallel_copy refuses one 15;16;17;18;19 2 rls_direct_storage rls_direct_storage parallel_export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage premise: t_owner owns force_t 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the no-privilege role is told permission denied by ordinary SQL 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage premise: the owner sees every row, so the policy hides something real 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage premise: the policy is in force for ordinary SQL 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - @@ -2667,6 +2665,8 @@ rls_direct_storage rls_direct_storage read_projection refuses a policy-restricte rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 never - rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py index 624396f55..4ac2bb936 100644 --- a/test/pytest/test_rls_direct_storage.py +++ b/test/pytest/test_rls_direct_storage.py @@ -103,11 +103,11 @@ def as_rls(sql): expect.text(str(as_rls("SELECT has_table_privilege('rls_t','SELECT')")[0]), "True", "premise: the role HOLDS select, so the ACL check cannot be what refuses") got, _ = as_rls("SELECT count(*) FROM rls_t") - expect.num(-1 if got is None else got, 1, "premise: the policy is in force for ordinary SQL") + expect.num(-1 if got is None else got, 1, "the policy is in force for ordinary SQL") expect.text(str(as_rls("SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE')")[0]), "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, - "premise: the owner sees every row, so the policy hides something real") + "the owner sees every row, so the policy hides something real") expect.text("yes" if all(os.path.getsize(f"{out}/{f}") > 0 for f in ("src.parquet", "src.arrow", "src.txt")) else "no", "yes", "premise: the source files the import arms need exist") diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index f8254e659..3040ac79a 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -92,18 +92,23 @@ verdict() { # verdict -> rls | acl | allowed | other } # ---- premises, every one run BY the restricted role ------------------------- +# +# Two of these read the table through the columnar scan, so a product change can +# redden them, and they are checks, not premises (#1236; #1397 review): the policy's +# row count, which the custom scan dropping its quals turned into every row, and the +# owner's count. check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At -c 'SELECT 1;' 2>&1 | head -1)" "1" check "premise: the role HOLDS select, so the ACL check cannot be what refuses" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c "SELECT has_table_privilege('rls_t','SELECT');" 2>&1 | head -1)" "t" -check_num "premise: the policy is in force for ordinary SQL" \ +check_num "the policy is in force for ordinary SQL" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c 'SELECT count(*) FROM rls_t;' 2>&1 | head -1)" "1" check "premise: the role holds EXECUTE, so the grant layer is not what refuses" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c "SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE');" 2>&1 | head -1)" "t" -check_num "premise: the owner sees every row, so the policy hides something real" \ +check_num "the owner sees every row, so the policy hides something real" \ "$(q 'SELECT count(*) FROM rls_t;')" "$ROWS" check "premise: the source files the import arms need exist" \ "$([ -s "$OUT/src.parquet" ] && [ -s "$OUT/src.arrow" ] && [ -s "$OUT/src.txt" ] && echo yes || echo no)" "yes" From 69002615a2b8ae24f39b063d0c671a555e950270 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 18:19:16 +0000 Subject: [PATCH 12/14] test: the RLS policy check is dated by the custom scan dropping its quals (#1341) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 2 +- test/check_ledger_budget.txt | 13 ++++++++++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index f518976dd..eb686434c 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2666,7 +2666,7 @@ rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 97b3f6cc6..f1049fa28 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2575,4 +2575,15 @@ suites_not_covered 229 # never 2352 + 10 = 2362 # dated 777 + 16 = 793 # 2362 + 793 == 3155 -checks_never_observed_red 2362 +# +# #1341, rls_direct_storage's two scan-dependent premises (OffgridwithJD, #1397 review): +# 2362 -> 2361. Both read the table through the columnar scan, so they are checks and +# lose the prefix, renamed with their rows. "the policy is in force for ordinary SQL" +# is dated by the custom scan dropping its quals (the restricted role then saw all 200 +# rows); the owner's count arrives never. Five-major runs: 26/26 on 15-19. +# +# ledger rows 3155 = 3155 +# never 2362 - 1 = 2361 +# dated 793 + 1 = 794 +# 2361 + 794 == 3155 +checks_never_observed_red 2361 From 24e7b44299a7652669039ade04d8310deb7a6d76 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 19:55:03 +0000 Subject: [PATCH 13/14] test: rls_direct_storage's source-files check loses the premise prefix, and the twin fails it by name rather than raising (#1383 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 2 +- test/pytest/test_rls_direct_storage.py | 18 ++++++++++++------ test/rls_direct_storage.sh | 5 +++-- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index eb686434c..a49363113 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2660,13 +2660,13 @@ rls_direct_storage rls_direct_storage premise: the no-privilege role is told per rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage premise: the source files the import arms need exist 15;16;17;18;19 never - rls_direct_storage rls_direct_storage read_projection refuses a policy-restricted caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) +rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 never - rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py index 4ac2bb936..692a0d5f9 100644 --- a/test/pytest/test_rls_direct_storage.py +++ b/test/pytest/test_rls_direct_storage.py @@ -108,8 +108,9 @@ def as_rls(sql): "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, "the owner sees every row, so the policy hides something real") - expect.text("yes" if all(os.path.getsize(f"{out}/{f}") > 0 for f in ("src.parquet", "src.arrow", "src.txt")) - else "no", "yes", "premise: the source files the import arms need exist") + expect.text("yes" if all(os.path.exists(f"{out}/{f}") and os.path.getsize(f"{out}/{f}") > 0 + for f in ("src.parquet", "src.arrow", "src.txt")) + else "no", "yes", "the source files the import arms need exist") # ---- the refusals: on unfixed code each returns "allowed", which is the leak def v(sql): @@ -165,10 +166,15 @@ def v(sql): _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") except psycopg.Error: pass - exported = 0 # no file holds no rows: a refused export is a red arm, not a crash - if os.path.exists(f"{out}/owner.parquet"): - exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," - " owner_name text, ssn text)") + # No file, an empty one, or one that does not read holds no rows: a refused or + # empty export is a red arm by name, not a crash (#1383 review). + exported = 0 + if os.path.exists(f"{out}/owner.parquet") and os.path.getsize(f"{out}/owner.parquet") > 0: + try: + exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," + " owner_name text, ssn text)") + except psycopg.Error: + exported = 0 expect.num(exported, owner_rows, "and the owner can still export it") # ---- FORCE ROW LEVEL SECURITY reaches a NON-superuser owner too diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 3040ac79a..e26f99424 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -96,7 +96,8 @@ verdict() { # verdict -> rls | acl | allowed | other # Two of these read the table through the columnar scan, so a product change can # redden them, and they are checks, not premises (#1236; #1397 review): the policy's # row count, which the custom scan dropping its quals turned into every row, and the -# owner's count. +# owner's count. The source files are a third: the owner's export writes them, so an +# export that writes nothing reddens that check (#1383 review). check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At -c 'SELECT 1;' 2>&1 | head -1)" "1" check "premise: the role HOLDS select, so the ACL check cannot be what refuses" \ @@ -110,7 +111,7 @@ check "premise: the role holds EXECUTE, so the grant layer is not what refuses" -c "SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE');" 2>&1 | head -1)" "t" check_num "the owner sees every row, so the policy hides something real" \ "$(q 'SELECT count(*) FROM rls_t;')" "$ROWS" -check "premise: the source files the import arms need exist" \ +check "the source files the import arms need exist" \ "$([ -s "$OUT/src.parquet" ] && [ -s "$OUT/src.arrow" ] && [ -s "$OUT/src.txt" ] && echo yes || echo no)" "yes" # ---- the refusals ----------------------------------------------------------- From 81a31405464a9e4993c17e6c7fcf988d1b918b01 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 19:56:59 +0000 Subject: [PATCH 14/14] test: the source-files check is dated by an export that writes nothing (#1383 review) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho --- test/check_ledger.tsv | 4 ++-- test/check_ledger_budget.txt | 12 +++++++++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index a49363113..6ee5f9293 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2647,7 +2647,7 @@ rls_direct_storage rls_direct_storage a table with NO policy is still readable b rls_direct_storage rls_direct_storage and no imported row landed in the restricted table 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage and read_projection tells it the same thing, not that RLS exists 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL rls_direct_storage rls_direct_storage and still exportable by it 15;16;17;18;19 never - -rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 export_parquet writes no file (total = 0);the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage export_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage import_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) @@ -2666,7 +2666,7 @@ rls_direct_storage rls_direct_storage so no caller without privilege learns RLS rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) -rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 2026-10-14 export_parquet writes no file (total = 0) rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index f1049fa28..30fb0236f 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2586,4 +2586,14 @@ suites_not_covered 229 # never 2362 - 1 = 2361 # dated 793 + 1 = 794 # 2361 + 794 == 3155 -checks_never_observed_red 2361 +# +# #1383 review (OffgridwithJD): 2361 -> 2360. "the source files the import arms need exist" +# loses the premise prefix: the owner's export writes those files, and an export that +# writes nothing reddens it in both twins (the twin now fails it by name instead of +# raising), with the owner-export arm beside it. Five-major runs: 26/26 on 15-19. +# +# ledger rows 3155 = 3155 +# never 2361 - 1 = 2360 +# dated 794 + 1 = 795 +# 2360 + 795 == 3155 +checks_never_observed_red 2360