diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 734b4a672..3f9562f27 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -393,11 +393,13 @@ column_extent column_extent a transaction's own flushed groups are inside the ex column_extent column_extent a type with no ordering gets no extent 15;16;17;18;19 2026-10-13 a recompute that skips values without an ordering;pgcolumnar.column_extent returns a row with no complete extent column_extent column_extent a writer that finds no complete extent recomputes one over every group 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent a writer that rolls back leaves the extent as the data has it 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit;no extent rows written at PRE_PREPARE;the writer's fold keeps the largest minimum -column_extent column_extent after VACUUM retires a fully deleted group the extent is the live rows' 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) -column_extent column_extent after compact_rewrite the extent is the live rows' 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;compact_rewrite returning 0 before rewriting anything;no extent row is ever written (PgColumnarExtentPreCommit returns at once) -column_extent column_extent after expire the extent is recomputed to the data that is left 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after VACUUM retires a fully deleted group the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after compact_rewrite the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;compact_rewrite returning 0 before rewriting anything;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after expire the extent is recomputed to the data that is left 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent after recluster the extent is the live rows' 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute column_extent column_extent and a plan made in that transaction estimates as the zone maps do 15;16;17;18;19 2026-10-13 the extent a transaction reads leaves out its own pending fold (the tree before the fix) -column_extent column_extent and it did move: the retired groups held the old minimum 15;16;17;18;19 2026-10-14 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) +column_extent column_extent and it did move: the deleted rows held the old minimum 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute +column_extent column_extent and it did move: the retired groups held the old minimum 15;16;17;18;19 2026-10-15 PgColumnarRetireGroup without marking the storage for a recompute;a recompute that leaves the rows it folded;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent and the estimate is the one the zone maps give 15;16;17;18;19 2026-10-13 the planner takes the extent's minimum as its upper bound too;the planner's zone-map fallback removed: with no complete extent it reads nothing and bounds nothing column_extent column_extent and the extent covers what they loaded 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit;no extent rows written at PRE_PREPARE;the writer's fold keeps the largest minimum column_extent column_extent and the extent still covers all fifty 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once);no extent row written at pre-commit @@ -414,6 +416,7 @@ column_extent column_extent planning a bucketed GROUP BY scans zone_map not at a column_extent column_extent premise: the extent function exists 15;16;17;18;19 never - column_extent column_extent premise: the transaction's plans gave two estimates 15;16;17;18;19 never - column_extent column_extent premise: the typed fixture spans several row groups 15;16;17;18;19 never - +column_extent column_extent recluster rewrites the table's row groups 15;16;17;18;19 never - column_extent column_extent the extent catalog is not readable by an ordinary role (it holds column values) 15;16;17;18;19 2026-10-13 the column_extent catalog granted to PUBLIC column_extent column_extent the extent of ty.d is the data's min and max, in the type's order 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) column_extent column_extent the extent of ty.i is the data's min and max, in the type's order 15;16;17;18;19 2026-10-14 a writer that finds no complete extent does not recompute one;no extent row is ever written (PgColumnarExtentPreCommit returns at once) @@ -1805,8 +1808,8 @@ harness_selftest 540-an-arm-must-carry-its-measurement a new arm that cannot say harness_selftest 540-an-arm-must-carry-its-measurement a repaired arm must have its row removed, so the list only shrinks 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: a determinate or carrying arm is not swept up 15;16;17;18;19 2026-10-12 part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: a determinate or carrying awk arm is not swept up 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's awk_determinate treats a comparison against > 0 as indeterminate (return 1 made return 0) -harness_selftest 540-an-arm-must-carry-its-measurement control: the carve-outs hold and the innocent arm above the helper is not named 15;16;17;18;19 2026-09-24 else echo 0; fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent); then echo 1;the shell if/then/else spelling removed from the matcher, so `$(if [ C ] -harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a shell if/then/else that discards its operand 15;16;17;18;19 2026-09-24 else echo 0; fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent); then echo 1;the shell if/then/else spelling removed from the matcher, so `$(if [ C ] +harness_selftest 540-an-arm-must-carry-its-measurement control: the carve-outs hold and the innocent arm above the helper is not named 15;16;17;18;19 2026-09-24 the shell if/then/else spelling removed from the matcher, so a one-line `$(if [ C ]... then echo 1... else echo 0... fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent) +harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a shell if/then/else that discards its operand 15;16;17;18;19 2026-09-24 the shell if/then/else spelling removed from the matcher, so a one-line `$(if [ C ]... then echo 1... else echo 0... fi)` is invisible again (mutation B, the same-line recorder conjunct removed from the `[` rule, reddens the SAME two arms -- they are not independent) harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees a threshold arm and a floor arm 15;16;17;18;19 2026-10-12 part 540's sweep stops skipping arms whose text ends in $ (the awk 'if (arms ~ /\$/) next' made 'if (0) next') harness_selftest 540-an-arm-must-carry-its-measurement control: the sweep sees an awk verdict that discards its operands 15;16;17;18;19 2026-10-12 part 540's awk sweep never reports an indeterminate verdict (the print guarded by 0);part 540's awk_determinate treats a comparison against > 0 as indeterminate (return 1 made return 0) harness_selftest 540-an-arm-must-carry-its-measurement premise: the comment fixture quotes a whole recorder call, not a fragment 15;16;17;18;19 never - @@ -1852,10 +1855,10 @@ harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary harness_selftest 580-a-hand-rebuild-must-record a PostgreSQL symbol is NOT exempt, so the check still has a subject 15;16;17;18;19 2026-10-12 pgc_symbol_is_toolchain deleted from the real lib.sh;the symbol exemption matches every name, so the check has no subject left and a genuinely mislinked major would resolve clean harness_selftest 580-a-hand-rebuild-must-record a log that is not there also produces a diagnosis 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason -harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it, driven alone with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp harness_selftest 580-a-hand-rebuild-must-record an empty log still produces a diagnosis, because silence is the bug 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all -harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it, driven alone with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record and it distinguishes missing from empty, so the two causes do not merge 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record and it names the log as empty rather than printing a bare banner 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all harness_selftest 580-a-hand-rebuild-must-record and it never reaches the pg_config check it would otherwise fail on 15;16;17;18;19 2026-09-25 rebuild.sh's fail-closed assertion is removed, so a missing exemption function leaves the undefined list empty and the check reports success @@ -2334,6 +2337,22 @@ native_reclaim native_reclaim repeated recluster reuses space (file plateaus) 15 native_reclaim native_reclaim reused-block data matches heap mirror 15;16;17;18;19 never - native_reclaim native_reclaim row count correct after cycle 15;16;17;18;19 never - native_reclaim native_reclaim ten row groups, every one half deleted 15;16;17;18;19 2026-10-14 the write state ignores stripe_row_limit +objstore_allowlist objstore_allowlist a multi-entry list matches any entry 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist a non-superuser cannot widen the list (42501) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it +objstore_allowlist objstore_allowlist a superuser session SET works (admin override) 15;16;17;18;19 2026-10-14 objstore_allowed_endpoints defined PGC_SIGHUP, so no session may SET it +objstore_allowlist objstore_allowlist default: a local read is untouched by the list 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist default: an http read is refused 42501 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint;the allow-list check never refuses;the allow-list's default is 127.0.0.1, not empty +objstore_allowlist objstore_allowlist default: the refusal names the GUC 15;16;17;18;19 2026-10-14 an empty endpoint allow-list admits every endpoint;the allow-list check never refuses;the allow-list's default is 127.0.0.1, not empty +objstore_allowlist objstore_allowlist host:port entry, right port: works 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist host:port entry, wrong port: refused 42501 15;16;17;18;19 2026-10-14 the allow-list check never refuses +objstore_allowlist objstore_allowlist link-local: refused 42501 despite being listed 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist link-local: the listed ordinary host still works beside it 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist link-local: the refusal names the range, not the list 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist listed host: the read works 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist premise: fixture server is up 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist premise: the role can open a session 15;16;17;18;19 never - +objstore_allowlist objstore_allowlist the list defaults to empty 15;16;17;18;19 2026-10-14 the allow-list's default is 127.0.0.1, not empty +objstore_allowlist objstore_allowlist unlisted host: refused 42501 15;16;17;18;19 2026-10-14 the allow-list check never refuses objstore_credentials objstore_credentials a mapping with an access key and no secret is refused before any request (28000) 15;16;17;18;19 2026-10-14 the user-mapping credential pairing check removed (an access key alone crashes the backend) objstore_credentials objstore_credentials a role with no mapping of its own reads through the PUBLIC mapping 15;16;17;18;19 2026-10-14 the user-mapping lookup has no PUBLIC fallback objstore_credentials objstore_credentials a role's own mapping is used before the PUBLIC one (bob's wrong secret is still refused 28000) 15;16;17;18;19 2026-10-14 the user-mapping lookup tries PUBLIC before the role's own @@ -2361,6 +2380,11 @@ objstore_credentials objstore_credentials placement: secret on a SERVER is HV00D objstore_credentials objstore_credentials premise: bob's error was a real refusal, not silence 15;16;17;18;19 never - objstore_credentials objstore_credentials premise: sigv4 fixture server is up 15;16;17;18;19 never - objstore_credentials objstore_credentials premise: without mapping or environment, even a superuser gets 28000 15;16;17;18;19 never - +objstore_crlf objstore_crlf a CR/LF in a REST catalog_uri is rejected too (objstore_http_request path) 15;16;17;18;19 2026-10-14 the request-line guard on the URL path removed +objstore_crlf objstore_crlf a CR/LF in the URL path is rejected before the request is sent 15;16;17;18;19 2026-10-14 the request-line guard on the URL path removed +objstore_crlf objstore_crlf backend alive 15;16;17;18;19 never - +objstore_crlf objstore_crlf backend alive after the REST attempt 15;16;17;18;19 never - +objstore_crlf objstore_crlf no injected line reached the listener (no request splitting) 15;16;17;18;19 2026-10-14 the request-line guard on the URL path removed parallel_am_scan parallel_am_scan a parallel index build indexes every row of the table 15;16;17;18;19 never - parallel_am_scan parallel_am_scan a parallel table-AM scan returns the same row count as serial 15;16;17;18;19 never - parallel_am_scan parallel_am_scan premise: ANALYZE printed a rows= line per launched worker 15;16;17;18;19 never - @@ -2396,6 +2420,7 @@ partition_lifecycle partition_lifecycle a cutoff on a partition's upper bound in partition_lifecycle partition_lifecycle a drop plan for a parent not RANGE-partitioned on one column is refused, LIST and two-column RANGE included 15;16;17;18;19 2026-10-14 partition_drop_candidates refusing only a parent with no partition key, not LIST or a multi-column RANGE;partition_drop_candidates without its RANGE-on-one-column check partition_lifecycle partition_lifecycle a drop plan for a parent not partitioned on a timestamp is refused 15;16;17;18;19 2026-10-14 a parent partitioned on a non-timestamp key planned instead of refused;partition_drop_candidates without its timestamp-key check partition_lifecycle partition_lifecycle a drop plan for a parent with a DEFAULT partition is refused 15;16;17;18;19 2026-10-13 a parent with a DEFAULT partition planned instead of refused +partition_lifecycle partition_lifecycle a non-superuser owner creates partitions ahead on its own table 15;16;17;18;19 never - partition_lifecycle partition_lifecycle a parent partitioned on a key that is not a timestamp is refused before any DDL 15;16;17;18;19 2026-10-13 partition_create_ahead without its timestamp key check partition_lifecycle partition_lifecycle a parent that is not RANGE-partitioned on one column is refused before any DDL 15;16;17;18;19 2026-10-13 partition_create_ahead without its RANGE-on-one-column check partition_lifecycle partition_lifecycle a parent with a 63-byte name gets partitions whose names keep their dates 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it;partition_create_ahead stopping one width short of now plus the lead;partition_create_ahead without shortening a long parent name;partition_create_ahead's name bound one byte too long @@ -2407,6 +2432,7 @@ partition_lifecycle partition_lifecycle a partition named with the drop block's partition_lifecycle partition_lifecycle a partition that is itself partitioned is planned whole 15;16;17;18;19 2026-10-13 a partition that is itself partitioned left out of the plan;a partition whose upper bound equals the cutoff left out;a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY partition_lifecycle partition_lifecycle a pending detach is finalised, not started again 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;a pending detach planned as a new concurrent detach, not FINALIZE;a plain DROP, not refusing while the partition is still attached;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing partition_lifecycle partition_lifecycle a plan for names that need quoting runs as returned 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;names in the plan with their quotes removed;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing +partition_lifecycle partition_lifecycle a role that does not own the parent is refused by its own privileges (42501), and creates nothing 15;16;17;18;19 2026-10-14 partition_create_ahead made SECURITY DEFINER partition_lifecycle partition_lifecycle a second call creates nothing 15;16;17;18;19 2026-10-14 partition_create_ahead creating its last partition without attaching it;partition_create_ahead starting from now, not the highest upper bound partition_lifecycle partition_lifecycle a second plan for the same cutoff is empty, and the partitions it kept are still attached 15;16;17;18;19 2026-10-13 the cutoff filter removed: every partition planned;the lower bound read where the upper is meant partition_lifecycle partition_lifecycle a tiered partition is planned with DROP FOREIGN TABLE 15;16;17;18;19 2026-10-13 a partition whose upper bound equals the cutoff left out;a plain DROP, not refusing while the partition is still attached;a tiered partition planned with DROP TABLE;the cutoff filter removed: every partition planned;the lower bound read where the upper is meant;the plan detaches but drops nothing;the plan detaches without CONCURRENTLY @@ -2438,6 +2464,8 @@ partition_lifecycle partition_lifecycle premise: a partition is named with the d partition_lifecycle partition_lifecycle premise: a timestamp parent continued from a hand-made partition, in a session west of UTC 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: a timestamp parent was given partitions from a session east of UTC 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: a timestamptz parent continued from a hand-made partition, in a session west of UTC 15;16;17;18;19 never - +partition_lifecycle partition_lifecycle premise: pl_own owns the parent, and neither role is a superuser 15;16;17;18;19 never - +partition_lifecycle partition_lifecycle premise: pl_peer can open a session 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the January partition was tiered to Parquet 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the call inside the transaction created partitions 15;16;17;18;19 never - partition_lifecycle partition_lifecycle premise: the drop fixture has several partitions, each holding rows 15;16;17;18;19 never - @@ -2576,6 +2604,82 @@ projection_update projection_update projection matches the heap before any updat projection_update projection_update read_projection matches the live base after that update 15;16;17;18;19 2026-10-14 projection rows storing the first projected column's value in every projected column projection_update projection_update read_projection still matches the live base 15;16;17;18;19 2026-10-14 projection rows storing the first projected column's value in every projected column projection_update projection_update reconstruct via the projection still rebuilds every live row 15;16;17;18;19 2026-10-14 projection rows storing the first projected column's value in every projected column +projections projections DROP TABLE removes its declaration (#304) 15;16;17;18;19 never - +projections projections GUC off: no projection scan 15;16;17;18;19 never - +projections projections a mid-transaction drop leaves no orphan projection storage 15;16;17;18;19 never - +projections projections a projection added after a write in the same transaction gets the later rows 15;16;17;18;19 never - +projections projections a projection dropped mid-transaction is gone, not still being written 15;16;17;18;19 never - +projections projections a projection sorted on a type with no ordering (json) is refused (0A000) 15;16;17;18;19 2026-10-14 the projection writer without its sort-type ordering check (the back-fill sort then crashes the backend) +projections projections add on heap table rejected 15;16;17;18;19 never - +projections projections aggregate over projection scan matches oracle 15;16;17;18;19 never - +projections projections an orphan left by an older build is present 15;16;17;18;19 never - +projections projections an unrestricted base scan matches oracle after delete 15;16;17;18;19 never - +projections projections an unrestricted query reads the base, not the projection 15;16;17;18;19 never - +projections projections and it removed the orphan 15;16;17;18;19 never - +projections projections and leaves the other table's declaration alone 15;16;17;18;19 never - +projections projections and nothing of it is declared 15;16;17;18;19 never - +projections projections and the base table still took every row 15;16;17;18;19 never - +projections projections and they are the right rows, not merely the right number 15;16;17;18;19 never - +projections projections back-fill: p2 matches base (b column) 15;16;17;18;19 never - +projections projections back-fill: p2 populated from existing rows 15;16;17;18;19 never - +projections projections base + p2 remain 15;16;17;18;19 never - +projections projections base id 0 columns are all attrs 15;16;17;18;19 never - +projections projections base id 0 name 15;16;17;18;19 never - +projections projections base id 0 sort_key empty 15;16;17;18;19 never - +projections projections base proj_storage_id == base 15;16;17;18;19 never - +projections projections control: a drop in its own transaction never left one 15;16;17;18;19 never - +projections projections control: with no write before add_projection the projection was always right 15;16;17;18;19 never - +projections projections distinct storage ids 15;16;17;18;19 never - +projections projections drop base rejected 15;16;17;18;19 never - +projections projections drop unknown rejected 15;16;17;18;19 never - +projections projections duplicate column rejected 15;16;17;18;19 never - +projections projections duplicate name rejected 15;16;17;18;19 never - +projections projections empty columns rejected 15;16;17;18;19 never - +projections projections fp chunks carry min/max skip metadata 15;16;17;18;19 never - +projections projections fp count after delete matches base 15;16;17;18;19 never - +projections projections fp fan-out matches base (a,c) 15;16;17;18;19 never - +projections projections fp reflects deletes (a,c) 15;16;17;18;19 never - +projections projections fp row count matches base 15;16;17;18;19 never - +projections projections fp2 multi-stripe fan-out matches base 15;16;17;18;19 never - +projections projections fp2 spans multiple projection row groups 15;16;17;18;19 never - +projections projections fq fan-out matches base (b) 15;16;17;18;19 never - +projections projections new snapshot projection scan sees both batches 15;16;17;18;19 never - +projections projections no projection rows before first add 15;16;17;18;19 never - +projections projections non-covering query (references b) uses the base 15;16;17;18;19 never - +projections projections old snapshot projection scan does not see post-snapshot rows 15;16;17;18;19 never - +projections projections p1 columns 15;16;17;18;19 never - +projections projections p1 gone after drop 15;16;17;18;19 never - +projections projections p1 has its own storage id 15;16;17;18;19 never - +projections projections p1 id is 1 15;16;17;18;19 never - +projections projections p1 sort_key 15;16;17;18;19 never - +projections projections p2 columns 15;16;17;18;19 never - +projections projections p2 id is 2 15;16;17;18;19 never - +projections projections p2 sort_key empty 15;16;17;18;19 never - +projections projections planner still uses projection after vacuum 15;16;17;18;19 never - +projections projections premise: the base table holds every committed row 15;16;17;18;19 never - +projections projections projection chosen for covering + sort-key query 15;16;17;18;19 never - +projections projections projection matches base after second vacuum 15;16;17;18;19 never - +projections projections projection scan matches oracle after delete 15;16;17;18;19 never - +projections projections projection survives vacuum 15;16;17;18;19 never - +projections projections projection-scan matches oracle after vacuum 15;16;17;18;19 never - +projections projections projection-scan results match heap oracle 15;16;17;18;19 never - +projections projections read_projection base rejected 15;16;17;18;19 never - +projections projections read_projection matches base after vacuum 15;16;17;18;19 never - +projections projections reconstruct (a,b,c) matches base after vacuum 15;16;17;18;19 never - +projections projections reconstruct full row matches base 15;16;17;18;19 never - +projections projections reconstruct matches base after delete + NULLs 15;16;17;18;19 never - +projections projections reconstruct row count matches base 15;16;17;18;19 never - +projections projections session A baseline via projection sees batch 1 15;16;17;18;19 never - +projections projections so a rebuild still works for the rest of the database 15;16;17;18;19 never - +projections projections sort key not in columns 15;16;17;18;19 never - +projections projections storage id resolves 15;16;17;18;19 never - +projections projections table populated 15;16;17;18;19 never - +projections projections table still readable after DDL 15;16;17;18;19 never - +projections projections the drop really removed the projection 15;16;17;18;19 never - +projections projections the rebuild does not abort on it 15;16;17;18;19 never - +projections projections two declared projections to start 15;16;17;18;19 never - +projections projections two rows after first add (base + p1) 15;16;17;18;19 never - +projections projections unknown column rejected 15;16;17;18;19 never - range_pruning range_pruning a clustered overlap near the end skips the four vectors only their upper bounds rule out 15;16;17;18;19 2026-10-13 native_range_excludes never using max_upper (haveUpper forced false) range_pruning range_pruning a clustered overlap skips vectors instead of decoding them 15;16;17;18;19 never - range_pruning range_pruning a scattered column skips nothing, which is what the documentation says 15;16;17;18;19 never - @@ -2609,14 +2713,14 @@ range_pruning range_pruning premise: the range type collates differently from it range_pruning range_pruning so does a containment probe 15;16;17;18;19 never - range_pruning range_pruning the overlap qual is pushed down, where it used to be dropped 15;16;17;18;19 never - range_pruning range_pruning the same for a containment probe far to its right 15;16;17;18;19 never - -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered -recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c);the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table created and loaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 0 committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal a table truncated and reloaded in one transaction at wal_skip_threshold 64MB committed its rows, and the server came back through recovery 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and after the truncate the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 0 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) +recovery_minimal_wal recovery_minimal_wal and the columnar table holds what its heap mirror holds, at wal_skip_threshold 64MB 15;16;17;18;19 2026-10-15 the relfilenode created with smgrcreate instead of RelationCreateStorage, so no commit-time sync is registered, and this assert-enabled build then fails relcache.c's RelFileLocatorSkippingWAL assertion during the load (src/columnar_tableam.c) recovery_minimal_wal recovery_minimal_wal premise: the cluster runs at wal_level=minimal 15;16;17;18;19 never - rewrite_storage_oid rewrite_storage_oid a matview created WITH DATA leaves its storage row pointing at itself 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite repair gate no longer covers CreateTableAsStmt, so a matview created WITH DATA keeps the transient's OID rewrite_storage_oid rewrite_storage_oid a type rewrite leaves the storage row pointing at the live table 15;16;17;18;19 2026-10-12 the repair gate skips every relation, as if no inheritor were locked by the statement;the rewrite hook does not retarget storage.relation_oid @@ -2636,6 +2740,32 @@ rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partitio rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp ATTACH PARTITION lp_3 FOR VALUES FROM ('2026-03-01') TO ('2026-04-01') 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 CONCURRENTLY 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) +rls_direct_storage rls_direct_storage a table with NO policy is still readable by the same role 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage and no imported row landed in the restricted table 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage and read_projection tells it the same thing, not that RLS exists 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL +rls_direct_storage rls_direct_storage and still exportable by it 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage and the owner can still export it 15;16;17;18;19 2026-10-14 export_parquet writes no file (total = 0);the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage export_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage import_arrow refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage import_parquet refuses a policy-restricted writer 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage no export file was written for the refused caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage parallel_copy refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage parallel_export_parquet refuses one 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage premise: t_owner owns force_t 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the no-privilege role is told permission denied by ordinary SQL 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role HOLDS select, so the ACL check cannot be what refuses 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role can open a session 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage premise: the role holds EXECUTE, so the grant layer is not what refuses 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage read_projection refuses a policy-restricted caller 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage reconstruct_via_projection refuses one too 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage so no caller without privilege learns RLS state from these functions 15;16;17;18;19 2026-10-14 read_projection checks RLS before the table ACL +rls_direct_storage rls_direct_storage the OWNER can still read the RLS table's storage (RLS_NONE_ENV) 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too +rls_direct_storage rls_direct_storage the owner sees every row, so the policy hides something real 15;16;17;18;19 never - +rls_direct_storage rls_direct_storage the policy is in force for ordinary SQL 15;16;17;18;19 2026-10-14 the custom scan drops its quals (plan.qual = NIL) +rls_direct_storage rls_direct_storage the source files the import arms need exist 15;16;17;18;19 2026-10-14 export_parquet writes no file (total = 0) +rls_direct_storage rls_direct_storage under FORCE the non-superuser owner IS refused, so owner-only is no substitute 15;16;17;18;19 2026-10-14 PgColumnarRequireNoRowSecurity never refuses (the RLS guard deleted) +rls_direct_storage rls_direct_storage without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED 15;16;17;18;19 2026-10-14 the RLS guard refuses anything but RLS_NONE so the owner and superuser (RLS_NONE_ENV) are refused too rollup rollup * with anything but count is refused 15;16;17;18;19 2026-10-13 create_rollup allowing * with any function rollup rollup DROP TABLE of a rollup's source is refused while the rollup exists (2BP01, #1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) rollup rollup DROP VIEW of a rollup takes its partials and its catalog rows with it (#1376) 15;16;17;18;19 2026-10-14 a rollup's view, partials and source with no dependency between them, and no catalog cleanup (main before #1376's fix) @@ -2881,6 +3011,7 @@ tier_partition tier_partition a failure after drop leaves the rows exactly once, tier_partition tier_partition a failure after export leaves the rows exactly once, still local 15;16;17;18;19 never - tier_partition tier_partition a fault step that is not one of tier_partition's is refused 15;16;17;18;19 2026-10-13 the tier_fault_after check hook accepting any value tier_partition tier_partition a grant to a role whose name needs quoting is carried to the foreign partition 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition not carrying the grants;tier_partition returning 0 rows moved +tier_partition tier_partition a non-superuser owner tiers its own partition 15;16;17;18;19 never - tier_partition tier_partition a parent held by a long reader times the move out, as lock_wait says 15;16;17;18;19 2026-10-13 tier_partition without the lock_wait bound tier_partition tier_partition a parent with a UNIQUE index is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without the UNIQUE-parent refusal tier_partition tier_partition a partition a view depends on is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without the dependents check @@ -2893,6 +3024,7 @@ tier_partition tier_partition a partition with a trigger of its own is refused b tier_partition tier_partition a query on the other range scans only the local partition, beside a foreign one 15;16;17;18;19 never - tier_partition tier_partition a query on the tiered range scans only the foreign partition 15;16;17;18;19 never - tier_partition tier_partition a retry over the failed attempt's file moves the partition, exactly once 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition refusing every existing path (the first version's rule);tier_partition returning 0 rows moved +tier_partition tier_partition a role that does not own the table is refused by its own privileges (42501) 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition a rollback after the whole call leaves the rows local, exactly once 15;16;17;18;19 never - tier_partition tier_partition a server of another wrapper is refused 15;16;17;18;19 2026-10-14 tier_partition without its wrong-wrapper branch, so the iceberg wrapper refuses only after the export;tier_partition without the wrapper check tier_partition tier_partition a server that does not exist is refused before anything moves 15;16;17;18;19 2026-10-13 tier_partition without its missing-server branch, so core refuses only after the export @@ -2910,6 +3042,8 @@ tier_partition tier_partition and no foreign table behind it after detach 15;16; tier_partition tier_partition and no foreign table behind it after drop 15;16;17;18;19 never - tier_partition tier_partition and no foreign table behind it after export 15;16;17;18;19 never - tier_partition tier_partition and the refusals moved nothing 15;16;17;18;19 2026-10-13 tier_partition without the path refusal +tier_partition tier_partition and the refused call leaves the partition local and attached, every row once 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER +tier_partition tier_partition and the refused call wrote no export file 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition and the timed-out move leaves the rows local, with nothing attached 15;16;17;18;19 2026-10-13 tier_partition without the lock_wait bound tier_partition tier_partition at read committed a row committed between the caller's snapshot and the lock survives 15;16;17;18;19 never - tier_partition tier_partition at read committed the move goes ahead and exports that row 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix;tier_partition returning 0 rows moved @@ -2934,6 +3068,8 @@ tier_partition tier_partition premise: the first attempt failed after export and tier_partition tier_partition premise: the move started while the insert was in flight, and waited for it 15;16;17;18;19 never - tier_partition tier_partition premise: the parent holds every row 15;16;17;18;19 never - tier_partition tier_partition premise: the storage id was read before the move 15;16;17;18;19 never - +tier_partition tier_partition premise: tier_own owns the parent and its partitions, and neither role is a superuser 15;16;17;18;19 never - +tier_partition tier_partition premise: tier_peer can open a session and read the partition 15;16;17;18;19 never - tier_partition tier_partition the answers over the parent are the same after the move 15;16;17;18;19 2026-10-13 tier_partition naming the foreign table with a suffix tier_partition tier_partition the call holds ACCESS EXCLUSIVE on the parent until commit, as the docs say 15;16;17;18;19 never - tier_partition tier_partition the cold partition is now a foreign partition with its own name and bound 15;16;17;18;19 2026-10-13 tier_partition attaching with the upper bound shifted a minute;tier_partition naming the foreign table with a suffix @@ -2944,6 +3080,7 @@ tier_partition tier_partition the local partition leaves no pgcolumnar storage r tier_partition tier_partition the move ran inside the caller's transaction 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved tier_partition tier_partition the other partition is still local after the refusals 15;16;17;18;19 2026-10-13 tier_partition without the path refusal tier_partition tier_partition tier_partition moves the cold partition and returns the rows it moved 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved +tier_partition tier_partition with no foreign table behind it 15;16;17;18;19 2026-10-14 tier_partition made SECURITY DEFINER tier_partition tier_partition with the dependents gone, the same partition moves 15;16;17;18;19 2026-10-13 tier_partition returning 0 rows moved;tier_partition without the dependents check timeseries_recipes timeseries_recipes premise: the readings table is columnar 15;16;17;18;19 never - timeseries_recipes timeseries_recipes premise: the reference has 24 hours, 6 of them empty and one all-NULL 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index cea363447..0e9fac106 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 230 +suites_not_covered 226 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -2533,4 +2533,114 @@ suites_not_covered 230 # never 2344 + 1 = 2345 # dated 767 + 3 = 770 # 2345 + 770 == 3115 -checks_never_observed_red 2345 +# +# #1341 item 75, the changelog names recluster with expire, compact_rewrite and +# VACUUM as recomputing the column extent, and only the other three had arms: +# 2345 -> 2346. column_extent gains 3 checks from five-major runs (42/42 on each +# of 15-19). Two are dated on PG17 by the existing mutation "PgColumnarRetireGroup +# without marking the storage for a recompute", which also re-dates the four +# arms it already held. "recluster rewrites the table's row groups" arrives never. +# +# ledger rows 3115 + 3 = 3118 +# never 2345 + 1 = 2346 +# dated 770 + 2 = 772 +# 2346 + 772 == 3118 +# +# #1341, tier_partition and partition_create_ahead on the caller's own privileges +# (OffgridwithJD): 2346 -> 2352. Eleven new checks, all in already-covered suites. +# Five are dated, by each function made SECURITY DEFINER: tier_partition's refusal, +# the partition left local, no foreign table and no export file (four), and +# partition_create_ahead's refusal (one). The two owner controls run on tables of +# their own, so neither mutation reddens them; they and four premises arrive never. +# Five-major runs: tier_partition 79/79 (80 on 18-19, which run the virtual column) +# and partition_lifecycle 64/64 on 15-19. +# +# ledger rows 3118 + 11 = 3129 +# never 2346 + 6 = 2352 +# dated 772 + 5 = 777 +# 2352 + 777 == 3129 +# +# #1341, rls_direct_storage seeded with its pytest twin (OffgridwithJD): 2352 -> 2362, +# and suites_not_covered 230 -> 229. The suite had no rows. It enters with all 26 +# checks from five-major runs (26/26 on each of 15-19). Sixteen are dated by three +# mutations of the RLS guard: deleted (the eight refusals, nothing written, nothing +# imported, and FORCE), widened to refuse RLS_NONE_ENV (the owner's read, the owner's +# export and the owner without FORCE), and moved ahead of read_projection's ACL check +# (the two disclosure arms). The owner's export was measured by a test both of whose +# branches echoed "ran"; it now reads the export back, and main's suite survives the +# widened guard on it. The other 10 arrive never: the premises and the no-policy +# controls. +# +# ledger rows 3129 + 26 = 3155 +# never 2352 + 10 = 2362 +# dated 777 + 16 = 793 +# 2362 + 793 == 3155 +# +# #1341, rls_direct_storage's two scan-dependent premises (OffgridwithJD, #1397 review): +# 2362 -> 2361. Both read the table through the columnar scan, so they are checks and +# lose the prefix, renamed with their rows. "the policy is in force for ordinary SQL" +# is dated by the custom scan dropping its quals (the restricted role then saw all 200 +# rows); the owner's count arrives never. Five-major runs: 26/26 on 15-19. +# +# ledger rows 3155 = 3155 +# never 2362 - 1 = 2361 +# dated 793 + 1 = 794 +# 2361 + 794 == 3155 +# +# #1383 review (OffgridwithJD): 2361 -> 2360. "the source files the import arms need exist" +# loses the premise prefix: the owner's export writes those files, and an export that +# writes nothing reddens it in both twins (the twin now fails it by name instead of +# raising), with the owner-export arm beside it. Five-major runs: 26/26 on 15-19. +# +# ledger rows 3155 = 3155 +# never 2361 - 1 = 2360 +# dated 794 + 1 = 795 +# 2360 + 795 == 3155 +# +# #1341, objstore_crlf seeded, with its pytest twin (OffgridwithJD): 2360 -> 2362, and +# suites_not_covered 229 -> 228. The suite had no rows. It enters with all 5 checks +# from five-major runs (5/5 on each of 15-19). Three are dated by the request-line +# guard on the URL path removed: neither refusal named CR or LF, and the listener +# received the injected line. The two backend-alive checks arrive never. +# +# ledger rows 3155 + 5 = 3160 +# never 2360 + 2 = 2362 +# dated 795 + 3 = 798 +# 2362 + 798 == 3160 +# +# #1341, objstore_allowlist seeded, with its twin and a fixed admin-override arm +# (OffgridwithJD): 2362 -> 2374, and suites_not_covered 228 -> 227. The suite had no +# rows. It enters with all 16 checks from five-major runs (16/16 on each of 15-19). +# Four are dated: the two default-deny arms by an empty list that admits every +# endpoint; the non-superuser arm and the admin-override arm by the setting defined +# PGC_SIGHUP. Main's admin-override arm survived that one, because the configured +# list still named 127.0.0.1; it now runs with the list empty. The other 12 arrive +# never. +# +# ledger rows 3160 + 16 = 3176 +# never 2362 + 12 = 2374 +# dated 798 + 4 = 802 +# 2374 + 802 == 3176 +# +# #1379 review (OffgridwithJD): 2374 -> 2371. "the list defaults to empty" loses the premise +# prefix, dated by the default changed to 127.0.0.1; the shell now has the query name +# "empty" itself rather than comparing with an empty string. The allow-list check that +# never refuses also dates "unlisted host" and "host:port entry, wrong port". +# Five-major runs: 16/16 on 15-19, the twin passing with pyarrow required. +# +# ledger rows 3176 = 3176 +# never 2374 - 3 = 2371 +# dated 802 + 3 = 805 +# 2371 + 805 == 3176 +# +# #1341, projections seeded, with the sort-type arms (OffgridwithJD): 2371 -> 2446, and +# suites_not_covered 227 -> 226. The suite had no rows. It enters with all 76 checks +# from five-major runs (76/76 on each of 15-19). One is dated by the projection +# writer without its sort-type ordering check: a json sort key's back-fill then +# crashed the backend instead of refusing with 0A000. The other 75 arrive never. +# +# ledger rows 3176 + 76 = 3252 +# never 2371 + 75 = 2446 +# dated 805 + 1 = 806 +# 2446 + 806 == 3252 +checks_never_observed_red 2446 diff --git a/test/column_extent.sh b/test/column_extent.sh index bba291034..066a15d78 100755 --- a/test/column_extent.sh +++ b/test/column_extent.sh @@ -213,6 +213,21 @@ q "CREATE TABLE vr (ts timestamptz, v int) USING pgcolumnar; q "VACUUM vr;" >/dev/null check "after VACUUM retires a fully deleted group the extent is the live rows'" "$(ext vr ts)" "$(truth vr ts)" +# recluster rewrites every group too, and the changelog names it with the three +# above (#1341 item 75): the rows a DELETE removed are not written back, so its +# extent must be the live rows', not wider by rows no longer stored. The deleted +# rows hold the minimum, so a stale extent keeps it. +q "CREATE TABLE rc (ts timestamptz, v int) USING pgcolumnar; + SELECT pgcolumnar.set_options('rc', stripe_row_limit => 1000, chunk_group_row_limit => 1000); + INSERT INTO rc SELECT '2026-01-01'::timestamptz + g * interval '1 min', (g * 7919) % 3001 FROM generate_series(1, 3000) g; + DELETE FROM rc WHERE ts < '2026-01-01'::timestamptz + interval '400 min';" >/dev/null +rc_before="$(ext rc ts)" +reclustered="$(q "SELECT pgcolumnar.recluster('rc', 'ts', 'v');")" +check "recluster rewrites the table's row groups" "$([ "${reclustered:-0}" -ge 1 ] && echo yes || echo "no ($reclustered)")" "yes" +check "after recluster the extent is the live rows'" "$(ext rc ts)" "$(truth rc ts)" +check "and it did move: the deleted rows held the old minimum" \ + "$([ -n "$rc_before" ] && [ "$rc_before" != "$(ext rc ts)" ] && echo yes || echo "no ($rc_before)")" "yes" + # ---- no complete extent where the zone maps cannot give one ---------------------- q "CREATE TABLE ac (ts timestamptz) USING pgcolumnar; SELECT pgcolumnar.set_options('ac', stripe_row_limit => 1000, chunk_group_row_limit => 1000); diff --git a/test/objstore_allowlist.sh b/test/objstore_allowlist.sh index be046261b..f2e3d863a 100755 --- a/test/objstore_allowlist.sh +++ b/test/objstore_allowlist.sh @@ -69,8 +69,12 @@ URL="http://127.0.0.1:$HTTP_PORT/fixture.parquet" Q="SELECT count(*) FROM pgcolumnar.read_parquet('$URL') AS (a int8)" # ---- the default: deny everything remote ------------------------------------ -check "premise: the list defaults to empty" \ - "$(q "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")" "" +# A check, not a premise (#1236; #1379 review): the default is the product's. And +# "empty" is produced by the query itself, so a capture that is empty because the +# query never ran cannot pass. +check "the list defaults to empty" \ + "$(q "SELECT CASE WHEN current_setting('pgcolumnar.objstore_allowed_endpoints') = '' THEN 'empty' + ELSE 'set: [' || current_setting('pgcolumnar.objstore_allowed_endpoints') || ']' END")" "empty" check "default: an http read is refused 42501" "$(sqlstate_of "$Q")" "42501" check "default: the refusal names the GUC" \ "$([ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -qtA \ @@ -96,6 +100,9 @@ check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U allow_nobody -d "$PGC_DB" -qtA -c 'SELECT 1' 2>&1)" "1" check "a non-superuser cannot widen the list (42501)" \ "$(sqlstate_of "SET pgcolumnar.objstore_allowed_endpoints = '10.0.0.1'" allow_nobody)" "42501" +# WITH THE CONFIGURED LIST EMPTY (#1341). It was still "other.example, 127.0.0.1" here, +# so the read worked whatever the session SET did, and the arm could not fail. +set_list "" check_num "a superuser session SET works (admin override)" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -qtA \ -c "SET pgcolumnar.objstore_allowed_endpoints = '127.0.0.1'" -c "$Q" 2>/dev/null | tail -1)" "1000" diff --git a/test/partition_lifecycle.sh b/test/partition_lifecycle.sh index 9c649e886..c8c89a1d9 100755 --- a/test/partition_lifecycle.sh +++ b/test/partition_lifecycle.sh @@ -241,6 +241,40 @@ check "a parent with a DEFAULT partition is refused" \ check "and nothing was created for it" \ "$(kids td)|$(q "SELECT count(*) FROM pg_proc WHERE proname = 'partition_create_ahead';")" "1|1" +# ---- the caller's own privileges (#1341) ------------------------------------- +# +# partition_create_ahead is not SECURITY DEFINER, so its CREATE TABLE ... PARTITION OF +# runs as the caller, and core refuses a role that does not own the parent. Made +# SECURITY DEFINER, it would run as the extension's owner and let any caller add +# partitions, and the access-exclusive lock that comes with them, to a table it does +# not own. Both roles hold USAGE and CREATE on the schema, so ownership is all that +# separates them. +for r in pl_own pl_peer; do + psql_run "DROP ROLE IF EXISTS $r; CREATE ROLE $r LOGIN NOSUPERUSER; + GRANT USAGE, CREATE ON SCHEMA public TO $r; GRANT USAGE ON SCHEMA pgcolumnar TO $r;" +done +as_role() { # as_role ROLE SQL -> the single value SQL returns as ROLE, or the SQLSTATE it fails with + local out + out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" -d "$PGC_DB" \ + -X -At -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$2" 2>&1) + case "$out" in + *ERROR:*|*FATAL:*) sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' <<<"$out" | head -1 ;; + *) printf '%s\n' "$out" | tail -1 ;; + esac +} +# The owner's control has a parent of its own (pw2), so a refusal that let the peer +# through cannot make the control fail for the peer's reason. +q "CREATE TABLE pv (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pv OWNER TO pl_own; + CREATE TABLE pw2 (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts); ALTER TABLE pw2 OWNER TO pl_own;" >/dev/null +check "premise: pl_own owns the parent, and neither role is a superuser" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE oid IN ('pv'::regclass, 'pw2'::regclass);")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own', 'pl_peer') AND rolsuper;")" \ + "pl_own|0" +check "premise: pl_peer can open a session" "$(as_role pl_peer "SELECT 1")" "1" +check "a role that does not own the parent is refused by its own privileges (42501), and creates nothing" \ + "$(as_role pl_peer "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)")|$(kids pv)" "42501|0" +check "a non-superuser owner creates partitions ahead on its own table" \ + "$(as_role pl_own "SELECT pgcolumnar.partition_create_ahead('pw2', '1 day', 2)")|$(kids pw2)" "3|3" + # ==== the drop half: partition_drop_plan (#1297) ================================ # # partition_drop_plan(parent regclass, cutoff timestamptz) -> SETOF text diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 1d880f395..83c6666dc 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -312,6 +312,16 @@ def read_ledger(path): f"{path}:{n}: major {m!r} is neither a number nor " f"{MAJOR_UNKNOWN!r}, so this row names no version it applies to") muts = set() if f[5] == NONE else {m for m in f[5].split(";") if m} + # A PIECE WITH WHITESPACE AT AN END IS A FRAGMENT, not a mutation. The writer + # joins the set with a bare ";", so it never produces one; a text that held + # "; " does, when it is split on the ";" (12 rows on the shipped ledger, 8 of + # them from one recovery_minimal_wal mutation, re-sorted by every later merge). + for m in sorted(muts): + if m != m.strip(): + raise LedgerError( + f"{path}:{n}: mutation {m!r} has whitespace at an end, so it is a " + f"fragment of one mutation's text split on ';'. Reword that mutation " + f"without ';' and keep the row's other mutations as they are.") key = (f[0], f[1], f[2]) # A REPEATED KEY, REFUSED RATHER THAN OVERWRITTEN. This was # `rows[key] = [...]`, so a duplicated row collapsed silently and the LAST line @@ -530,6 +540,12 @@ def cmd_merge(args): "--target names what a mutation killed, so it needs --mutation to say " "which mutation. On its own it attributes nothing.") + if args.mutation and ";" in args.mutation: + raise LedgerError( + f"--mutation {args.mutation!r} holds ';', which separates mutations in the " + f"ledger's mutation column, so it would be stored as several. Reword it " + f"without ';' (', and' reads the same).") + if args.mutation: failed = sorted({key for _p, seen in runs for key, vs in seen.items() if any(v == "FAIL" for v, _m in vs)}) diff --git a/test/projections.sh b/test/projections.sh index bf126fd5a..d6d8d68bd 100755 --- a/test/projections.sh +++ b/test/projections.sh @@ -448,4 +448,21 @@ psql_run "INSERT INTO dt2 SELECT g, g FROM generate_series(200, 300) g;" check "control: a drop in its own transaction never left one" \ "$(proj_orphans dt2)" "0" +# ---- a sort key with no ordering is refused (#1341) -------------------------- +# +# add_projection does not check the sort column's type. The writer does, when it +# is built: json has no btree comparison, so the back-fill over existing rows +# must refuse with 0A000 rather than sort by a comparison function that does not +# exist. Last in the suite, because without the check that sort crashes the +# backend (measured). +psql_run "CREATE TABLE pjs (id int, j json) USING pgcolumnar; + INSERT INTO pjs SELECT g, json_build_object('k', g) FROM generate_series(1, 500) g;" +check "a projection sorted on a type with no ordering (json) is refused (0A000)" \ + "$(proj_sqlstate "SELECT pgcolumnar.add_projection('pjs', 'pjp', ARRAY['id','j'], ARRAY['j'])")" "0A000" +# On a server that accepts connections again: without the check the sort crashes +# its backend (measured), and this arm must answer for the catalog, not for that. +for _i in $(seq 1 60); do [ "$(q 'SELECT 1' 2>/dev/null)" = 1 ] && break; sleep 0.5; done +check "and nothing of it is declared" \ + "$(q "SELECT count(*) FROM pgcolumnar.projection_declaration WHERE name = 'pjp';")" "0" + pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 4425ba7e6..f19ca19cc 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -170,6 +170,10 @@ behaviour, the source of that number is named. - [122. test_prepared_across_sessions.py: a prepared transaction, seen from other sessions and across a crash](#122-test_prepared_across_sessionspy-a-prepared-transaction-seen-from-other-sessions-and-across-a-crash) - [123. test_maintenance_crash.py: a crash part-way through maintenance loses nothing](#123-test_maintenance_crashpy-a-crash-part-way-through-maintenance-loses-nothing) - [124. test_serializable.py: SERIALIZABLE refuses a write skew on a columnar table](#124-test_serializablepy-serializable-refuses-a-write-skew-on-a-columnar-table) +- [125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts](#125-test_rls_direct_storagepy-the-direct-storage-paths-refuse-a-caller-a-policy-restricts) +- [126. test_objstore_crlf.py: a CR or LF in an object-store URL is refused before it is sent](#126-test_objstore_crlfpy-a-cr-or-lf-in-an-object-store-url-is-refused-before-it-is-sent) +- [127. test_objstore_allowlist.py: the endpoint allow-list, empty by default](#127-test_objstore_allowlistpy-the-endpoint-allow-list-empty-by-default) +- [128. test_objstore_userinfo.py: userinfo in a URL is refused on every entry point](#128-test_objstore_userinfopy-userinfo-in-a-url-is-refused-on-every-entry-point) ## 1. How to read a test in here @@ -2445,6 +2449,26 @@ Last-write-wins records the most recent attack rather than the catalogue the col exists to become. One `--mutation` copied across several logs attributes a deliberate change to failures it had nothing to do with, and is refused. +### `test_a_mutation_named_with_a_semicolon_is_refused_rather_than_split` + +The mutation column is a `;`-separated set, written back sorted, so a `--mutation` +whose text held `; ` was stored as two mutations, one a fragment starting with a space. +Every later merge re-sorted it, and three branches in a row carried the same eight +recovery_minimal_wal rows as a change of their own. `merge` now refuses such a text with +rc 2 and leaves the ledger as it was; the same text with `, and` is recorded whole. + +### `test_a_ledger_row_whose_mutation_holds_a_fragment_is_refused` + +The writer joins the set with a bare `;`, so a piece with whitespace at an end is a +fragment of a split sentence, never a mutation it wrote. The reader refuses the ledger +by line. Two whole mutations separated by a bare `;` still read. + +### `test_the_committed_ledger_names_no_mutation_fragment` + +The committed ledger, read by the same reader. Red on the tree before this change at +line 1783, the first of the 12 rows it rewords: 8 from #1427's recovery_minimal_wal +mutation and 4 older harness_selftest rows. + ### `test_a_mutation_with_two_targets_can_be_recorded_when_both_are_named` A mutation with two GENUINE targets is ordinary: reverting the @@ -4410,6 +4434,7 @@ failing for a reason that has nothing to do with projections. | `test_the_first_add_records_the_base_and_the_new_projection` | both rows appear at once, with the base's columns, empty sort key, name and shared storage id | | `test_a_second_projection_may_have_no_sort_key` | a projection without a sort key, and three distinct storage ids | | `test_a_bad_projection_is_refused_by_its_own_code` | seven refusals, each by its measured SQLSTATE rather than by "it errored" | +| `test_a_sort_key_with_no_ordering_is_refused` | #1341: a projection sorted on `json`, which has no btree ordering, is refused `0A000` when the back-fill builds its writer, and nothing of it is declared | | `test_a_projection_on_a_heap_table_is_refused` | the refusal that is about the ACCESS METHOD, not the arguments | | `test_drop_removes_one_projection_and_leaves_the_rest` | drop is surgical, and the table is still readable after the DDL | | `test_a_projection_added_late_is_back_filled_from_the_existing_rows` | a projection added after the rows exist is populated from them, not left empty | @@ -6623,6 +6648,7 @@ Removal proofs, each red in both twins: the prototype's zone handling; `CREATE T | --- | --- | | `test_partition_create_ahead` | from nothing it reaches now plus the lead, columnar, contiguous and one width wide; a second call creates nothing, and asking further ahead creates only the difference; rows land in columnar partitions; the parent is never held ACCESS EXCLUSIVE; west of UTC, for both key types, partitions continue a hand-made one contiguously and a row at every boundary routes; east of UTC, a timestamp parent covers the local now and starts on the width's boundary; inputs that broke the first version (a generated column, a 63-character name, more than 1000 partitions, a MAXVALUE bound) are handled or refused before any DDL; a DEFAULT partition is refused and nothing is created | | `test_partition_drop_plan` | the drop half: `partition_drop_plan` lists `DETACH ... CONCURRENTLY` and `DROP` for the partitions wholly before the cutoff, in range order, from partitions created newest first. A cutoff on an upper bound includes the partition, and one before them all plans nothing. Running the plan, each statement top-level, leaves the rest with their rows and drops the storage rows. A `timestamp` key with a `timestamp` cutoff plans the same in Tokyo and Denver, and a cutoff of the other type is refused (`42804`). MINVALUE is listed once past, and MAXVALUE never. Quoted schema and partition names run as returned. A sub-partitioned partition is planned whole and dropped with its children. A tiered partition is planned with `DROP FOREIGN TABLE`, and its Parquet file stays. A partition whose name holds the drop block's dollar-quote tag drops as planned. A detach cancelled in its second phase is planned as `FINALIZE` and dropped. The DEFAULT and non-timestamp parents are refused (`55000`) | +| `test_partition_create_ahead_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the parent, holding USAGE and CREATE on the schema, is refused `42501` and creates nothing; a non-superuser owner creates partitions ahead on a parent of its own | ## 94. test_approx_aggregates.py: hll and tdigest over columnar tables, the compatibility statement @@ -6771,6 +6797,7 @@ On a private database, because foreign servers are per database. A local path st | `test_tier_partition_moves_a_partition_exactly_once` | the move returns the rows moved; the parent's answers are the same; the cold range is a foreign partition with the same name and bound, reading the exported file; no pgcolumnar catalog rows are left; pruning keeps the ranges apart; an insert into the tiered range is refused; five refusals move nothing; the parent is held ACCESS EXCLUSIVE until commit, and a rollback leaves the rows local; a write in flight is exported; a fault after each of the five steps leaves the rows exactly once, still local | | `test_tier_partition_refuses_what_it_cannot_move_and_survives_concurrency` | at each isolation level a row committed between the snapshot and the lock survives, and only READ COMMITTED moves; a UNIQUE parent and four kinds of dependent are refused with no file left, and the partition moves once they are gone; grants and comment carry over; a racing insert leaves every row once; lock_wait times a held parent out with nothing attached; the caller's lock_timeout survives; a retry over a failed attempt's file moves the partition | | `test_tier_partition_moves_a_virtual_generated_column` | PostgreSQL 18+: a partition whose parent has a virtual generated column moves and reads back the same; below 18, `UNSUPPORTED_MAJOR`, listed in `expected_unrunnable.txt` | +| `test_tier_partition_runs_on_the_callers_own_privileges` | not SECURITY DEFINER: a role that does not own the table, holding the server-file role, USAGE on the server, CREATE on the schema and SELECT on the partition, is refused `42501` before any file is written, and the partition stays local and attached with no foreign table behind it; a non-superuser owner tiers a partition of its own table | ## 101. test_prepared_writes.py: a write prepared and finished in one session leaves the backend alive @@ -7078,3 +7105,49 @@ The pytest twin of `test/serializable.sh` (#1341), every check by the shell's na | test | what it holds | | --- | --- | | `test_serializable_refuses_a_write_skew` | the catalog SIRead locks, the four write skews refused by SSI with 40001 against heap, and the REPEATABLE READ controls | + +## 125. test_rls_direct_storage.py: the direct-storage paths refuse a caller a policy restricts + +The pytest twin of `test/rls_direct_storage.sh` (#563, ported for #1341), every check by the shell's name. Row-level security is applied by the rewriter, and these functions read and write storage with no query to rewrite, so each refuses a relation whose policies apply to the caller. The caller HOLDS SELECT, INSERT, EXECUTE and the server-file roles, so the RLS check is the only refuser left. Eight entry points are refused and nothing was written or imported; a role with no privilege is told "permission denied" and not that RLS exists; a table with no policy, the owner without FORCE, and the owner's export (read back, row for row) stay allowed; and under FORCE a non-superuser owner is refused too. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_direct_storage_paths_refuse_a_policy_restricted_caller` | the premises, the eight refusals and their absent effects, the no-privilege caller's ACL error, the not-over-refusing controls, and the FORCE pair | + +## 126. test_objstore_crlf.py: a CR or LF in an object-store URL is refused before it is sent + +The pytest twin of `test/objstore_crlf.sh` (ported for #1341), every check by the shell's name. A URL path is written into the request line, so a CR/LF in it would split the request. The guard is in the single connect path, so read_parquet and the REST catalog client's request are both refused, naming CR or LF; and a listener of the file's own records that nothing carrying the injected line reached it. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_a_cr_or_lf_in_a_url_is_refused_before_it_is_sent` | read_parquet and a REST `catalog_uri` with a CR/LF in the path are refused naming CR or LF, the listener receives no injected line, and the backend is alive after each | + +## 127. test_objstore_allowlist.py: the endpoint allow-list, empty by default + +The pytest twin of `test/objstore_allowlist.sh` (#393, ported for #1341), every check by the shell's name. An empty `pgcolumnar.objstore_allowed_endpoints` refuses all remote reads with 42501 and names the setting; a local read is untouched. A list admits exactly what it names, host or host:port, any entry of several. Only a superuser may change it (SUSET), and a superuser session SET is the admin override, asserted with the configured list empty so only the SET can admit the read. A link-local address is refused even when listed, before any connection, naming the range. + +The list is changed by ALTER SYSTEM and a reload, and reset in a `finally`. The fixture server is the file's own Range-capable `http.server`, serving one Parquet file from `server_dir`. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_endpoint_allow_list` | the default deny, the list's matching, SUSET and the admin override, and the link-local refusal | + +## 128. test_objstore_userinfo.py: userinfo in a URL is refused on every entry point + +The pytest twin of `test/objstore_userinfo.sh` (#706, #995, #997, ported for #1341), every check by the shell's name. `user@host` is refused with the parse guard's 22023 on the http read and write paths, in an s3 or gs bucket authority, and in an endpoint a foreign server configures. 22023 comes only from the parse guards and 42501 only from the allow-list, so the SQLSTATE says which layer refused, and the message pins the guard. `test_objstore_endpoint_userinfo.py` covers the endpoint half independently and is not this file's pair. + +The controls are what make the arms mean anything. With no endpoint configured every s3 URL is refused 28000, the clean one included, and 28000 has three causes, so each control names the refusal it expects. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_userinfo_in_a_url_is_refused_by_the_parse_guard` | read_parquet and export_parquet over http, export_arrow, and the s3 and gs bucket authority, read and write: 22023, naming userinfo | +| `test_a_clean_url_reaches_the_refusal_it_should` | a clean s3 URL demands an endpoint (naming `AWS_ENDPOINT_URL`), gs demands a credential, a malformed URL and an `@` in a key are not userinfo, a clean non-allowed endpoint is still 42501, and the backend is alive | +| `test_userinfo_in_a_configured_endpoint_is_refused` | both endpoint shapes, `u:p@` and `user@`, are 22023 naming userinfo and the endpoint; a clean endpoint gets past the parse | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 783fc01c6..3c595766c 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -285,7 +285,10 @@ # tests, the twin of selftest part 640, by collection over the NO_CLUSTER files. # guard_tests 433 -> 435 for #1420's two arms in test_writes_wrote_rows.py (a setting # on a watched connection or cursor is refused), by collection over NO_CLUSTER. -guard_tests 435 +# guard_tests 435 -> 438 for three arms in test_mutation_ledger.py: a mutation text holding +# ';' is refused at merge, a ledger piece with whitespace at an end is refused, and the +# committed ledger holds none, by collection over NO_CLUSTER. +guard_tests 438 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and @@ -702,4 +705,16 @@ guard_tests 435 # cluster_tests 586 -> 587 for #1341's test_serializable.py, one test, by collection # cluster_tests 587 -> 588 for #1341's daemon TimeZone test in test_autovacuum_expire.py, # by collection over the cluster files on this tree. -cluster_tests 588 +# cluster_tests 588 -> 590 for #1341's caller-privilege tests in test_tier_partition.py and +# test_partition_lifecycle.py, one each, by collection over the cluster files on this tree. +# cluster_tests 590 -> 591 for #1341's test_rls_direct_storage.py, the twin of +# rls_direct_storage.sh: one test, by collection over the cluster files on this tree. +# cluster_tests 591 -> 592 for #1341's test_objstore_crlf.py, the twin of objstore_crlf.sh: +# one test, by collection over the cluster files on this tree. +# cluster_tests 592 -> 593 for #1341's test_objstore_allowlist.py, the twin of +# objstore_allowlist.sh: one test, by collection over the cluster files on this tree. +# cluster_tests 593 -> 596 for #1341's test_objstore_userinfo.py, the twin of +# objstore_userinfo.sh: three tests, by collection over the cluster files on this tree. +# cluster_tests 596 -> 597 for #1341's test_a_sort_key_with_no_ordering_is_refused in +# test_projections.py, by collection over the cluster files on this tree. +cluster_tests 597 diff --git a/test/pytest/test_column_extent.py b/test/pytest/test_column_extent.py index add40cc65..f0074520a 100644 --- a/test/pytest/test_column_extent.py +++ b/test/pytest/test_column_extent.py @@ -247,6 +247,22 @@ def test_a_shrink_is_recomputed_never_guessed(pgc_own_db, expect): expect.text(_ext(conn, "vr", "ts"), _truth(conn, "vr", "ts"), "after VACUUM retires a fully deleted group the extent is the live rows'") + # recluster, named by the changelog with the three above (#1341 item 75): the + # deleted rows are not written back, and they held the minimum. + _run(conn, "CREATE TABLE rc (ts timestamptz, v int) USING pgcolumnar") + _small_groups(conn, "rc") + _run(conn, "INSERT INTO rc SELECT '2025-05-05'::timestamptz + g * interval '3 min', (g * 104729) % 2711" + " FROM generate_series(1, 3600) g", + "DELETE FROM rc WHERE ts < '2025-05-05'::timestamptz + interval '1500 min'") + before = _ext(conn, "rc", "ts") + reclustered = _one(conn, "SELECT pgcolumnar.recluster('rc', 'ts', 'v')") + expect.text("yes" if (reclustered or 0) >= 1 else f"no ({reclustered})", "yes", + "recluster rewrites the table's row groups") + after = _ext(conn, "rc", "ts") + expect.text(after, _truth(conn, "rc", "ts"), "after recluster the extent is the live rows'") + expect.text("yes" if before and before != after else f"no ({before})", "yes", + "and it did move: the deleted rows held the old minimum") + def test_no_complete_extent_where_the_zone_maps_cannot_give_one(pgc_own_db, expect): conn = pgc_own_db diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 36270ee64..e28cec9b5 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -128,7 +128,10 @@ "native_reclaim_cycles", "native_reclaim_frag", "native_vacuum_race", + "objstore_allowlist", "objstore_credentials", + "objstore_crlf", + "objstore_userinfo", "parallel_am_scan", "parallel_scan_cost", "partition_lifecycle", @@ -145,6 +148,7 @@ "projections", "recovery_minimal_wal", "rewrite_storage_oid", + "rls_direct_storage", "rollup", "scan_decode_cost", "serializable", diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index a06e83064..07650ec7f 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -125,6 +125,54 @@ def test_the_mutation_column_accumulates_rather_than_overwriting(tmp_path, expec "one mutation cannot be attributed across several runs at once") +def test_a_mutation_named_with_a_semicolon_is_refused_rather_than_split(tmp_path, expect): + """The mutation column is a ";"-separated SET, written back sorted. A `--mutation` + whose text held "; " was stored as two "mutations", one a sentence fragment that + starts with a space, and the next merge sorted the fragment first. Measured on the + shipped ledger: 12 rows, 8 of them from one recovery_minimal_wal mutation (#1427), + re-sorted by three later branches until each looked like a change of its own. + """ + ledger = _w(tmp_path, "l.tsv", "") + red = _w(tmp_path, "r.log", RED) + out, rc = _run("merge", "--ledger", ledger, "--date", "2026-10-15", + "--mutation", "the sync not registered; this build then asserts", red) + expect.num(rc, 2, "a mutation whose text holds ';' is refused, not split in two") + expect.num(out.count("separates mutations"), 1, "and the refusal says why") + expect.num(len(_rows(ledger)), 0, "and the ledger is left as it was") + + out, rc = _run("merge", "--ledger", ledger, "--date", "2026-10-15", + "--mutation", "the sync not registered, and this build then asserts", red) + expect.num(rc, 0, f"control: the same text without ';' is recorded: {out.strip()}") + expect.text({r[2]: r[5] for r in _rows(ledger)}["first check"], + "the sync not registered, and this build then asserts", + "and it is recorded whole, as one mutation") + + +def test_a_ledger_row_whose_mutation_holds_a_fragment_is_refused(tmp_path, expect): + """A ";"-piece with whitespace at either end cannot be a mutation the writer + produced: it joins the set with a bare ";". It is the fragment of a split sentence, + so the ledger is refused by line rather than read as two mutations.""" + green = _w(tmp_path, "g.log", GREEN) + split = _w(tmp_path, "split.tsv", + "demo\tpart1\tfirst check\t18\t2026-10-15\tthe sync not registered; this build asserts\n") + out, rc = _run("merge", "--ledger", split, green) + expect.num(rc, 2, "a mutation piece that starts with a space is refused") + expect.num(out.count("split.tsv:1:"), 1, "and the refusal names the line") + + two = _w(tmp_path, "two.tsv", + "demo\tpart1\tfirst check\t18\t2026-10-15\tbloom neutered;SAOP 128 -> 0\n") + out, rc = _run("merge", "--ledger", two, green) + expect.num(rc, 0, f"control: two whole mutations separated by a bare ';' still read: {out.strip()}") + + +def test_the_committed_ledger_names_no_mutation_fragment(tmp_path, expect): + """The committed ledger read by the same reader, so a split row on main is an + integrity failure of the file every gate reads, not a cosmetic one.""" + copy = _w(tmp_path, "ledger.tsv", (REPO / "test" / "check_ledger.tsv").read_text()) + out, rc = _run("merge", "--ledger", copy, _w(tmp_path, "g.log", GREEN)) + expect.num(rc, 0, f"the committed ledger reads with no fragment in it: {out.strip()[:400]}") + + def test_a_mutation_with_two_targets_can_be_recorded_when_both_are_named(tmp_path, expect): """#1014. A mutation with TWO GENUINE TARGETS is ordinary, not exotic. diff --git a/test/pytest/test_objstore_allowlist.py b/test/pytest/test_objstore_allowlist.py new file mode 100644 index 000000000..71d41bdca --- /dev/null +++ b/test/pytest/test_objstore_allowlist.py @@ -0,0 +1,161 @@ +"""The object-store endpoint allow-list, empty by default (#393): the pytest twin of +`test/objstore_allowlist.sh` (#1341). Every check carries the shell's name. + +`pgcolumnar.objstore_allowed_endpoints` gates every remote scheme: empty (the default) +refuses all remote access, so pg_read_server_files is not an SSRF primitive out of the +box, and link-local ranges (169.254.0.0/16, the cloud-metadata path) are refused even +when listed. The setting is SUSET, because a USERSET list would let any role widen its +own allow-list. + +The list is changed the operator's way, ALTER SYSTEM and a reload, and reset in a +`finally`, because the cluster is shared with every other test. A superuser session SET +is exercised once, as the admin override. + +THE FIXTURE SERVER IS THIS FILE'S OWN: a Range-capable http.server in a thread, serving +one Parquet file from `server_dir`. The shell twin's server is a script under `test/`, +and the two harnesses do not call each other. +""" +import os +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import pytest + + +class _Ranged(BaseHTTPRequestHandler): + """HEAD and GET with an optional `Range: bytes=lo-hi`, from one directory.""" + root = None + + def log_message(self, *args): + pass + + def _path(self): + p = os.path.join(self.root, os.path.basename(self.path.split("?")[0])) + return p if os.path.isfile(p) else None + + def do_HEAD(self): + p = self._path() + if p is None: + self.send_error(404) + return + self.send_response(200) + self.send_header("Content-Length", str(os.path.getsize(p))) + self.send_header("Accept-Ranges", "bytes") + self.end_headers() + + def do_GET(self): + p = self._path() + if p is None: + self.send_error(404) + return + data = open(p, "rb").read() + rng = self.headers.get("Range") + if rng and rng.startswith("bytes="): + lo, _, hi = rng[len("bytes="):].partition("-") + lo = int(lo) + hi = min(int(hi) if hi else len(data) - 1, len(data) - 1) + body = data[lo:hi + 1] + self.send_response(206) + self.send_header("Content-Range", f"bytes {lo}-{hi}/{len(data)}") + else: + body = data + self.send_response(200) + self.send_header("Content-Length", str(len(body))) + self.send_header("Accept-Ranges", "bytes") + self.end_headers() + self.wfile.write(body) + + +def _state(cluster, sql, role="postgres", pre=None): + """-> (sqlstate or '', primary message or '', first value or None) on a fresh session.""" + import psycopg + dsn = f"host=127.0.0.1 port={cluster.port} user={role} dbname=postgres" + try: + with psycopg.connect(dsn, autocommit=True) as c: + if pre: + c.execute(pre) + row = c.execute(sql).fetchone() + return "", "", (row[0] if row else None) + except psycopg.Error as e: + return e.sqlstate or "?", e.diag.message_primary or "", None + + +def test_the_endpoint_allow_list(pgc_cluster, pgc_conn, server_dir, expect): + pa = expect.require_module("pyarrow", name="pyarrow is required to write the fixture") + if pa is None: + return + pq = expect.require_module("pyarrow.parquet", name="pyarrow is required to write the fixture") + pq.write_table(pa.table({"a": pa.array(range(1000), pa.int64())}), str(server_dir / "fixture.parquet")) + + _Ranged.root = str(server_dir) + srv = ThreadingHTTPServer(("127.0.0.1", 0), _Ranged) + port = srv.server_address[1] + threading.Thread(target=srv.serve_forever, daemon=True).start() + + def set_list(value): + with pgc_conn.cursor() as cur: + cur.execute(f"ALTER SYSTEM SET pgcolumnar.objstore_allowed_endpoints = '{value}'") + cur.execute("SELECT pg_reload_conf()") + time.sleep(0.5) + + url = f"http://127.0.0.1:{port}/fixture.parquet" + q = f"SELECT count(*) FROM pgcolumnar.read_parquet('{url}') AS (a int8)" + try: + import urllib.request + with urllib.request.urlopen(urllib.request.Request(url, method="HEAD"), timeout=5) as r: + expect.num(r.status, 200, "premise: fixture server is up") + + # ---- the default: deny everything remote ---------------------------- + setting = _state(pgc_cluster, "SELECT current_setting('pgcolumnar.objstore_allowed_endpoints')")[2] + expect.text("empty" if setting == "" else f"set: [{setting}]", "empty", + "the list defaults to empty") + state, msg, _ = _state(pgc_cluster, q) + expect.text(state, "42501", "default: an http read is refused 42501") + expect.text("yes" if "objstore_allowed_endpoints" in msg else "no", "yes", + "default: the refusal names the GUC") + local = f"SELECT count(*) FROM pgcolumnar.read_parquet('{server_dir / 'fixture.parquet'}') AS (a int8)" + expect.num(_state(pgc_cluster, local)[2], 1000, "default: a local read is untouched by the list") + + # ---- the list admits exactly what it names --------------------------- + set_list("127.0.0.1") + expect.num(_state(pgc_cluster, q)[2], 1000, "listed host: the read works") + set_list("127.0.0.2") + expect.text(_state(pgc_cluster, q)[0], "42501", "unlisted host: refused 42501") + set_list(f"127.0.0.1:{port}") + expect.num(_state(pgc_cluster, q)[2], 1000, "host:port entry, right port: works") + set_list("127.0.0.1:1") + expect.text(_state(pgc_cluster, q)[0], "42501", "host:port entry, wrong port: refused 42501") + set_list("other.example, 127.0.0.1") + expect.num(_state(pgc_cluster, q)[2], 1000, "a multi-entry list matches any entry") + + # ---- SUSET: only a superuser may change it --------------------------- + with pgc_conn.cursor() as cur: + cur.execute("DO $$ BEGIN CREATE ROLE allow_nobody_py LOGIN; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + expect.num(_state(pgc_cluster, "SELECT 1", role="allow_nobody_py")[2], 1, + "premise: the role can open a session") + expect.text(_state(pgc_cluster, "SET pgcolumnar.objstore_allowed_endpoints = '10.0.0.1'", + role="allow_nobody_py")[0], "42501", + "a non-superuser cannot widen the list (42501)") + # WITH THE CONFIGURED LIST EMPTY, so only the session's own SET can admit the + # read (#1341): with 127.0.0.1 still configured, the read passed whatever the + # SET did. + set_list("") + expect.num(_state(pgc_cluster, q, pre="SET pgcolumnar.objstore_allowed_endpoints = '127.0.0.1'")[2], 1000, + "a superuser session SET works (admin override)") + + # ---- link-local: refused even when listed ---------------------------- + # 42501 proves the refusal came BEFORE any connection: a real connect to + # 169.254/16 would surface 08006 or hang toward a timeout. + set_list("169.254.169.254, 127.0.0.1") + ll = "SELECT count(*) FROM pgcolumnar.read_parquet('http://169.254.169.254:1/x.parquet') AS (a int8)" + state, msg, _ = _state(pgc_cluster, ll) + expect.text(state, "42501", "link-local: refused 42501 despite being listed") + expect.num(1 if "link-local" in msg else 0, 1, "link-local: the refusal names the range, not the list") + expect.num(_state(pgc_cluster, q)[2], 1000, "link-local: the listed ordinary host still works beside it") + finally: + with pgc_conn.cursor() as cur: + cur.execute("ALTER SYSTEM RESET pgcolumnar.objstore_allowed_endpoints") + cur.execute("SELECT pg_reload_conf()") + srv.shutdown() diff --git a/test/pytest/test_objstore_crlf.py b/test/pytest/test_objstore_crlf.py new file mode 100644 index 000000000..fb2f2eeda --- /dev/null +++ b/test/pytest/test_objstore_crlf.py @@ -0,0 +1,99 @@ +"""A CR or LF in an object-store URL is refused before any request is sent: the pytest +twin of `test/objstore_crlf.sh` (#1341). Every check carries the shell's name. + +A URL path is written into the request line, so a CR/LF in it would split the +request. The guard is in the single connect path, so both read_parquet and the REST +catalog client's http_request are covered. Each arm asserts the refusal names CR or LF, +and the read arm also asserts the listener received nothing carrying the injected line. + +The listener is this file's own socket in a thread, as the shell twin's is a script +under `test/`; the two harnesses do not call each other. +""" +import socket +import threading + +import pytest + + +class _Listener: + """Accept at most one connection on 127.0.0.1 and keep whatever bytes it sends.""" + + def __init__(self): + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.sock.bind(("127.0.0.1", 0)) + self.sock.listen(1) + self.sock.settimeout(5) + self.port = self.sock.getsockname()[1] + self.data = b"" + self.thread = threading.Thread(target=self._run, daemon=True) + self.thread.start() + + def _run(self): + try: + conn, _ = self.sock.accept() + except OSError: + return + conn.settimeout(2) + try: + while len(self.data) < 4096: + chunk = conn.recv(1024) + if not chunk: + break + self.data += chunk + except OSError: + pass + try: + conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nhi") + finally: + conn.close() + + def close(self): + self.thread.join(timeout=10) + self.sock.close() + + +def _err(conn, sql): + """-> the primary message of the error `sql` raises, or ''.""" + import psycopg + try: + with conn.cursor() as cur: + cur.execute(sql) + if cur.description: + cur.fetchall() + return "" + except psycopg.Error as e: + return e.diag.message_primary or "" + + +def _alive(conn): + with conn.cursor() as cur: + cur.execute("SELECT 1") + return cur.fetchone()[0] + + +def test_a_cr_or_lf_in_a_url_is_refused_before_it_is_sent(pgc_conn, expect): + with pgc_conn.cursor() as cur: + cur.execute("SET pgcolumnar.objstore_allowed_endpoints = '127.0.0.1'") + + lst = _Listener() + url = f"http://127.0.0.1:{lst.port}/x\r\nX-Injected: yes" + msg = _err(pgc_conn, f"SELECT * FROM pgcolumnar.read_parquet('{url}') AS t(v int)") + lst.close() + expect.num(1 if ("CR or LF" in msg) else 0, 1, + "a CR/LF in the URL path is rejected before the request is sent") + expect.num(1 if b"X-Injected" in lst.data else 0, 0, + "no injected line reached the listener (no request splitting)") + expect.num(_alive(pgc_conn), 1, "backend alive") + + # The REST catalog client goes through the same connect path. + cat = f"http://127.0.0.1:{lst.port}/x\r\nX-Injected: yes" + with pgc_conn.cursor() as cur: + cur.execute("DROP SERVER IF EXISTS crlf_cat_py CASCADE") + cur.execute(f"CREATE SERVER crlf_cat_py FOREIGN DATA WRAPPER pgcolumnar_iceberg_catalog" + f" OPTIONS (catalog_uri '{cat}')") + cur.execute("CREATE USER MAPPING FOR CURRENT_USER SERVER crlf_cat_py OPTIONS (token 'x')") + msg = _err(pgc_conn, "SELECT pgcolumnar.iceberg_rest_table_location('crlf_cat_py', 'db', 'events')") + expect.num(1 if ("CR or LF" in msg) else 0, 1, + "a CR/LF in a REST catalog_uri is rejected too (objstore_http_request path)") + expect.num(_alive(pgc_conn), 1, "backend alive after the REST attempt") diff --git a/test/pytest/test_objstore_userinfo.py b/test/pytest/test_objstore_userinfo.py new file mode 100644 index 000000000..39979dccc --- /dev/null +++ b/test/pytest/test_objstore_userinfo.py @@ -0,0 +1,127 @@ +"""Userinfo in a URL is refused on every entry point (#706, #995, #997): the pytest +twin of `test/objstore_userinfo.sh` (#1341). Every check carries the shell's name. + +The read path, the write path, the s3 and gs bucket authority, and an endpoint a +foreign server configures each refuse `user@host` with the parse guard's 22023. +22023 comes only from the parse guards here and 42501 only from the allow-list, so +the SQLSTATE says WHICH layer refused, and the message pins which 22023 guard. + +THE CONTROLS ARE WHAT MAKE THE ARMS MEAN ANYTHING. With no endpoint configured every +s3 URL is refused 28000, the clean one included, so a userinfo refusal must be told +apart from an unreachable object store: a clean URL has to reach the endpoint demand, +and say so. 28000 has three causes in os_resolve_s3 (no endpoint, no credential, the +allow-list's authorization), so each control names the one it expects. + +No object-store server is needed: every refusal here fires before any network I/O. +The allow-list is set per session, as test_objstore_endpoint_userinfo.py does; it is +superuser-settable. Nothing here reads anything under `test/*.sh`. +""" +import pytest + +ALLOWED = "127.0.0.1" +URL = "http://u:p@127.0.0.1:1/x.parquet" +S3UI = "s3://u:p@mybucket/x.parquet" + + +def _err(conn, sql): + """-> (sqlstate, primary message) of the error `sql` raises, or ('', '').""" + import psycopg + try: + with conn.cursor() as cur: + cur.execute(sql) + if cur.description: + cur.fetchall() + return "", "" + except psycopg.Error as e: + return e.sqlstate or "?", e.diag.message_primary or "" + + +def _has(text, word): + return 1 if word in text else 0 + + +@pytest.fixture +def ui(pgc_conn): + with pgc_conn.cursor() as cur: + cur.execute(f"SET pgcolumnar.objstore_allowed_endpoints = '{ALLOWED}'") + cur.execute("CREATE TABLE ex (id int, v text) USING pgcolumnar") + cur.execute("INSERT INTO ex SELECT g, 'v'||g FROM generate_series(1,100) g") + return pgc_conn + + +def test_userinfo_in_a_url_is_refused_by_the_parse_guard(ui, expect): + """The http read and write paths, and the s3 and gs bucket authority.""" + rp = f"SELECT * FROM pgcolumnar.read_parquet('{URL}') AS t(v int)" + state, msg = _err(ui, rp) + expect.text(state, "22023", "read_parquet refuses userinfo (22023, the parse guard)") + expect.num(_has(msg, "userinfo"), 1, "and the read_parquet message names userinfo") + + ep = f"SELECT pgcolumnar.export_parquet('ex', '{URL}')" + state, msg = _err(ui, ep) + expect.text(state, "22023", "export_parquet refuses userinfo (22023, not an allow-list 42501)") + expect.num(_has(msg, "userinfo"), 1, "and the export_parquet message names userinfo") + expect.text(_err(ui, "SELECT pgcolumnar.export_arrow('ex', 'http://u@127.0.0.1:1/x.arrow')")[0], "22023", + "export_arrow refuses userinfo through the same handle (22023)") + + s3r = f"SELECT * FROM pgcolumnar.read_parquet('{S3UI}') AS t(v int)" + state, msg = _err(ui, s3r) + expect.text(state, "22023", "read_parquet refuses userinfo in an s3:// bucket authority (22023)") + expect.num(_has(msg, "userinfo"), 1, "and the s3 read message names userinfo") + expect.text(_err(ui, "SELECT * FROM pgcolumnar.read_parquet('gs://u@mybucket/x.parquet') AS t(v int)")[0], + "22023", "a gs:// bucket authority is refused by the same guard") + s3w = f"SELECT pgcolumnar.export_parquet('ex', '{S3UI}')" + state, msg = _err(ui, s3w) + expect.text(state, "22023", + "export_parquet refuses it on the WRITE path too, where acceptance writes to a bucket nobody named") + expect.num(_has(msg, "userinfo"), 1, "and the s3 write message names userinfo") + + +def test_a_clean_url_reaches_the_refusal_it_should(ui, expect): + """The controls: a clean s3 URL demands an endpoint, gs demands a credential, a + malformed URL is the bucket/key refusal, '@' in a KEY is not userinfo, and a clean + URL to a non-allowed endpoint is still the allow-list's 42501.""" + clean = "SELECT * FROM pgcolumnar.read_parquet('s3://mybucket/x.parquet') AS t(v int)" + state, msg = _err(ui, clean) + expect.text(state, "28000", "control: a clean s3:// URL is not refused as userinfo, it demands an endpoint") + expect.num(_has(msg, "AWS_ENDPOINT_URL"), 1, + "control: and that refusal NAMES AWS_ENDPOINT_URL, pinning which 28000 fired") + expect.num(_has(msg, "userinfo"), 0, + "control: and it does not name userinfo, so the new guard did not fire on it") + _, msg = _err(ui, "SELECT * FROM pgcolumnar.read_parquet('gs://mybucket/x.parquet') AS t(v int)") + expect.num(_has(msg, "AWS_ACCESS_KEY_ID"), 1, + "control: the gs:// refusal names a CREDENTIAL, not the endpoint -- same 28000, other cause") + _, msg = _err(ui, "SELECT * FROM pgcolumnar.read_parquet('s3://nokey') AS t(v int)") + expect.num(_has(msg, "userinfo"), 0, "control: a malformed s3:// URL is still the bucket/key refusal, naming no userinfo") + state, msg = _err(ui, "SELECT * FROM pgcolumnar.read_parquet('s3://mybucket/my@file.parquet') AS t(v int)") + expect.num(_has(msg, "userinfo"), 0, "control: an '@' in the KEY is not userinfo, and is not refused as it") + expect.text(state, "28000", "control: and such a URL still reaches the endpoint demand, so it was not refused earlier") + expect.text(_err(ui, "SELECT pgcolumnar.export_parquet('ex', 'http://127.0.0.2:1/x.parquet')")[0], "42501", + "a clean URL to a non-allowed endpoint is still the allow-list's 42501") + with ui.cursor() as cur: + cur.execute("SELECT 1") + expect.num(cur.fetchone()[0], 1, "backend alive") + + +def test_userinfo_in_a_configured_endpoint_is_refused(ui, expect): + """The endpoint a foreign server configures, which the bucket guard cannot see. + Two shapes: with a colon in the userinfo, and without one, which used to keep a + valid port and reach only the allow-list.""" + with ui.cursor() as cur: + for srv, endpoint in (("s3ui_py", "http://u:p@127.0.0.1:1"), ("s3ui2_py", "http://user@127.0.0.1:1"), + ("s3ok_py", "http://127.0.0.1:1")): + cur.execute(f"CREATE SERVER IF NOT EXISTS {srv} FOREIGN DATA WRAPPER pgcolumnar_parquet" + f" OPTIONS (endpoint '{endpoint}')") + cur.execute("CREATE FOREIGN TABLE ft_ui (id int) SERVER s3ui_py OPTIONS (path 's3://mybucket/x.parquet')") + cur.execute("CREATE FOREIGN TABLE ft_ui2 (id int) SERVER s3ui2_py OPTIONS (path 's3://mybucket/x.parquet')") + cur.execute("CREATE FOREIGN TABLE ft_ok (id int) SERVER s3ok_py OPTIONS (path 's3://mybucket/x.parquet')") + state, msg = _err(ui, "SELECT * FROM ft_ui") + expect.text(state, "22023", "a userinfo endpoint is refused by the parse guard (22023)") + expect.num(_has(msg, "userinfo"), 1, "and the message names userinfo") + expect.num(_has(msg, "endpoint"), 1, "and it names the ENDPOINT, not the s3:// URL, which carries none") + state, msg = _err(ui, "SELECT * FROM ft_ui2") + expect.text(state, "22023", "the user@host shape is refused by the same guard (22023, not an allow-list 42501)") + expect.num(_has(msg, "userinfo"), 1, "and its message names userinfo too") + state, msg = _err(ui, "SELECT * FROM ft_ok") + expect.num(_has(msg, "userinfo"), 0, "control: a clean endpoint is not refused as userinfo") + expect.text("yes" if state != "22023" else "no", "yes", + "control: and it gets past the parse to a connection failure, not 22023") diff --git a/test/pytest/test_partition_lifecycle.py b/test/pytest/test_partition_lifecycle.py index c7bcad322..67791e7c0 100644 --- a/test/pytest/test_partition_lifecycle.py +++ b/test/pytest/test_partition_lifecycle.py @@ -595,3 +595,46 @@ def name(t): for t in ("xnl", "xls", "xr2")), "|".join(f"55000 {name(t)} is not RANGE-partitioned on one column" for t in ("xnl", "xls", "xr2")), "a drop plan for a parent not RANGE-partitioned on one column is refused, LIST and two-column RANGE included") + + +def test_partition_create_ahead_runs_on_the_callers_own_privileges(pgc_conn, pgc_cluster, expect): + """partition_create_ahead is not SECURITY DEFINER, so its CREATE TABLE ... PARTITION + OF runs as the caller and core refuses a role that does not own the parent (#1341). + Made SECURITY DEFINER, it would run as the extension's owner and let any caller add + partitions to a table it does not own. Both roles hold USAGE and CREATE on the + schema, so ownership is all that separates them.""" + conn = pgc_conn + schema = _one(conn, "SELECT current_schema()") + with conn.cursor() as cur: + for r in ("pl_own_py", "pl_peer_py"): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} LOGIN NOSUPERUSER; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f'GRANT USAGE, CREATE ON SCHEMA "{schema}" TO {r}') + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + # The owner's control has a parent of its own (pw2), so a refusal that let the + # peer through cannot make the control fail for the peer's reason. + for t in ("pv", "pw2"): + cur.execute(f"CREATE TABLE {t} (ts timestamptz NOT NULL, v int) PARTITION BY RANGE (ts)") + cur.execute(f"ALTER TABLE {t} OWNER TO pl_own_py") + + def as_role(role, sql): + try: + with psycopg.connect(f"host=127.0.0.1 port={pgc_cluster.port} user={role} dbname=postgres", + autocommit=True) as c: + c.execute(f'SET search_path TO "{schema}", pgcolumnar, public') + return str(c.execute(sql).fetchone()[0]) + except psycopg.Error as e: + return e.sqlstate or "?" + + ahead_pv = "SELECT pgcolumnar.partition_create_ahead('pv', '1 day', 2)" + owner = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" + " WHERE oid IN ('pv'::regclass, 'pw2'::regclass)") + supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('pl_own_py', 'pl_peer_py') AND rolsuper") + expect.text(f"{owner}|{supers}", "pl_own_py|0", + "premise: pl_own owns the parent, and neither role is a superuser") + expect.text(as_role("pl_peer_py", "SELECT 1"), "1", "premise: pl_peer can open a session") + expect.text(f"{as_role('pl_peer_py', ahead_pv)}|{_kids(conn, 'pv')}", + "42501|0", "a role that does not own the parent is refused by its own privileges (42501), and creates nothing") + ahead_pw2 = "SELECT pgcolumnar.partition_create_ahead('pw2', '1 day', 2)" + expect.text(f"{as_role('pl_own_py', ahead_pw2)}|{_kids(conn, 'pw2')}", + "3|3", "a non-superuser owner creates partitions ahead on its own table") diff --git a/test/pytest/test_projections.py b/test/pytest/test_projections.py index b284143bf..1aad7783b 100644 --- a/test/pytest/test_projections.py +++ b/test/pytest/test_projections.py @@ -253,6 +253,27 @@ def test_a_bad_projection_is_refused_by_its_own_code(cat, expect, name, code, sq expect.sqlstate(exc.value, code, name) +def test_a_sort_key_with_no_ordering_is_refused(pgc_conn, expect): + """add_projection does not check the sort column's type; the writer does, when + the back-fill builds it (#1341). json has no btree comparison, so the refusal is + 0A000 with the type named, rather than a sort by a comparison that does not + exist, and nothing of the projection is left declared.""" + import psycopg + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE pjs (id int, j json) USING pgcolumnar") + cur.execute("INSERT INTO pjs SELECT g, json_build_object('k', g) FROM generate_series(1, 500) g") + try: + with pgc_conn.cursor() as cur: + cur.execute("SELECT pgcolumnar.add_projection('pjs', 'pjp', ARRAY['id','j'], ARRAY['j'])") + got = "ok" + except psycopg.Error as e: + got = e.sqlstate or "?" + expect.text(got, "0A000", "a projection sorted on a type with no ordering (json) is refused (0A000)") + with pgc_conn.cursor() as cur: + expect.num(_one(cur, "SELECT count(*) FROM pgcolumnar.projection_declaration WHERE name = 'pjp'"), 0, + "and nothing of it is declared") + + def test_a_projection_on_a_heap_table_is_refused(cat, expect): """SEPARATE, because it needs a heap table the other refusals do not, and because the refusal is about the ACCESS METHOD rather than about the arguments.""" diff --git a/test/pytest/test_rls_direct_storage.py b/test/pytest/test_rls_direct_storage.py new file mode 100644 index 000000000..692a0d5f9 --- /dev/null +++ b/test/pytest/test_rls_direct_storage.py @@ -0,0 +1,197 @@ +"""The direct-storage paths refuse a relation with RLS in force (#563): the pytest twin +of `test/rls_direct_storage.sh` (#1341). Every check carries the shell's name. + +Row-level security is applied by the REWRITER to a query's range table entry. These +functions open the relation's storage and read or write it, so no policy is ever +consulted. The fixture that matters is a role that HOLDS SELECT and INSERT and is +restricted by a policy: a role with no privilege is refused by the ACL check first, +and with the RLS guard deleted a no-privilege arm stays green. Only a granted but +restricted caller reaches the RLS check with every other refuser cleared. + +The bar is check_enable_rls(...) == RLS_ENABLED. The owner without FORCE (and a +superuser) is RLS_NONE_ENV and stays allowed; under FORCE a NON-superuser owner is +refused too, which is why an owner-only rule would not close the hole. + +REAL LOGINS, NOT `SET ROLE`, as in the shell twin. Every role holds USAGE on this +test's schema, as the shell's roles hold it on public, so a "permission denied" is +never the schema's. +""" +import os + +import pytest + +RLS, NONE, OWNER = "t_rls_py", "t_rlsnone_py", "t_owner_rls_py" +ROWS = 200 + + +def _login(cluster, role, schema, sql): + """-> (first value or None, error or None) for `sql` on a fresh login as `role`.""" + import psycopg + dsn = f"host=127.0.0.1 port={cluster.port} user={role} dbname=postgres" + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(f'SET search_path TO "{schema}", pgcolumnar, public') + cur = c.execute(sql) + row = cur.fetchone() if cur.description else None + return (row[0] if row else None), None + except psycopg.Error as e: + return None, e + + +def _verdict(cluster, schema, sql, role=RLS): + """-> rls | acl | allowed | other: , as the shell's verdict.""" + _, err = _login(cluster, role, schema, sql) + if err is None: + return "allowed" + msg = err.diag.message_primary or "" + if "row-level security" in msg: + return "rls" + if "permission denied" in msg: + return "acl" + return f"other: {err.sqlstate} {msg}" + + +def _one(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + row = cur.fetchone() + return row[0] if row else None + + +def test_the_direct_storage_paths_refuse_a_policy_restricted_caller(pgc_conn, pgc_cluster, server_dir, expect): + conn = pgc_conn + out = str(server_dir) + schema = _one(conn, "SELECT current_schema()") + with conn.cursor() as cur: + cur.execute("CREATE TABLE rls_t (id int, owner_name text, ssn text) USING pgcolumnar") + cur.execute(f"INSERT INTO rls_t SELECT g, CASE WHEN g = 1 THEN '{RLS}' ELSE 'someone' END," + f" 'ssn-'||g FROM generate_series(1,{ROWS}) g") + cur.execute("CREATE TABLE plain_t (id int, v text) USING pgcolumnar") + cur.execute(f"INSERT INTO plain_t SELECT g, 'v'||g FROM generate_series(1,{ROWS}) g") + for t in ("rls_t", "plain_t"): + cur.execute(f"SELECT pgcolumnar.add_projection('{t}','p1',ARRAY['id'],ARRAY['id'])") + cur.execute(f"SELECT pgcolumnar.rebuild_projections('{t}')") + for r in (RLS, NONE, OWNER): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} NOSUPERUSER LOGIN; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + cur.execute(f'GRANT USAGE ON SCHEMA "{schema}" TO {r}') + cur.execute(f"GRANT pg_read_server_files, pg_write_server_files TO {RLS}") + # SELECT and INSERT DELIBERATELY: this role must clear every bar but RLS. + cur.execute(f"GRANT SELECT, INSERT ON rls_t TO {RLS}") + cur.execute(f"GRANT SELECT, INSERT ON plain_t TO {RLS}") + # #567 revoked EXECUTE on the projection readers from PUBLIC; without these + # grants the projection arms stop at the grant layer and prove nothing here. + for r in (RLS, NONE, OWNER): + cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO {r}") + cur.execute(f"GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO {RLS}") + # Sources exported from rls_t BY THE OWNER, so their shape matches the import + # target: a differently shaped source fails on column counts, not on RLS. + # BEFORE RLS is enabled, so no version of the guard under test can stop them. + cur.execute(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/src.parquet')") + cur.execute(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/src.arrow')") + cur.execute(f"COPY rls_t TO '{out}/src.txt'") + cur.execute("ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_own ON rls_t FOR SELECT TO {RLS} USING (owner_name = current_user)") + cur.execute(f"CREATE POLICY p_ins ON rls_t FOR INSERT TO {RLS} WITH CHECK (owner_name = current_user)") + + def as_rls(sql): + return _login(pgc_cluster, RLS, schema, sql) + + # ---- premises, every one run BY the restricted role + expect.text(str(as_rls("SELECT 1")[0]), "1", "premise: the role can open a session") + expect.text(str(as_rls("SELECT has_table_privilege('rls_t','SELECT')")[0]), "True", + "premise: the role HOLDS select, so the ACL check cannot be what refuses") + got, _ = as_rls("SELECT count(*) FROM rls_t") + expect.num(-1 if got is None else got, 1, "the policy is in force for ordinary SQL") + expect.text(str(as_rls("SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE')")[0]), + "True", "premise: the role holds EXECUTE, so the grant layer is not what refuses") + expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, + "the owner sees every row, so the policy hides something real") + expect.text("yes" if all(os.path.exists(f"{out}/{f}") and os.path.getsize(f"{out}/{f}") > 0 + for f in ("src.parquet", "src.arrow", "src.txt")) + else "no", "yes", "the source files the import arms need exist") + + # ---- the refusals: on unfixed code each returns "allowed", which is the leak + def v(sql): + return _verdict(pgc_cluster, schema, sql) + + expect.text(v("SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')"), "rls", + "read_projection refuses a policy-restricted caller") + expect.text(v("SELECT count(*) FROM pgcolumnar.reconstruct_via_projection('rls_t','p1')"), "rls", + "reconstruct_via_projection refuses one too") + expect.text(v(f"SELECT pgcolumnar.export_parquet('rls_t','{out}/leak1.parquet')"), "rls", + "export_parquet refuses one") + expect.text(v(f"SELECT pgcolumnar.export_arrow('rls_t','{out}/leak2.arrow')"), "rls", + "export_arrow refuses one") + expect.text(v(f"SELECT pgcolumnar.parallel_export_parquet('rls_t','{out}/leak3',2)"), "rls", + "parallel_export_parquet refuses one") + expect.text(v(f"SELECT pgcolumnar.import_parquet('rls_t','{out}/src.parquet')"), "rls", + "import_parquet refuses a policy-restricted writer") + expect.text(v(f"SELECT pgcolumnar.import_arrow('rls_t','{out}/src.arrow')"), "rls", + "import_arrow refuses one") + expect.text(v(f"SELECT pgcolumnar.parallel_copy('rls_t','{out}/src.txt',2)"), "rls", + "parallel_copy refuses one") + + # The refusal must not have done the work anyway. A message is not a behaviour. + written = [f for f in ("leak1.parquet", "leak2.arrow", "leak3") if os.path.exists(f"{out}/{f}")] + expect.text(" ".join(written) or "none", "none", "no export file was written for the refused caller") + expect.num(_one(conn, "SELECT count(*) FROM rls_t"), ROWS, "and no imported row landed in the restricted table") + + # ---- a caller WITHOUT privilege sees the ACL error, not the RLS one + _, err = _login(pgc_cluster, NONE, schema, "SELECT count(*) FROM rls_t") + expect.num(1 if err is not None and "permission denied" in (err.diag.message_primary or "") else 0, 1, + "premise: the no-privilege role is told permission denied by ordinary SQL") + _, err = _login(pgc_cluster, NONE, schema, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')") + msg = (err.diag.message_primary or "") if err is not None else "" + expect.num(1 if "permission denied" in msg else 0, 1, + "and read_projection tells it the same thing, not that RLS exists") + expect.num(1 if "row-level security" in msg else 0, 0, + "so no caller without privilege learns RLS state from these functions") + + # ---- not over-refusing: a fix that refused everyone would pass the arms above + expect.text(v("SELECT count(*) FROM pgcolumnar.read_projection('plain_t','p1')"), "allowed", + "a table with NO policy is still readable by the same role") + expect.text(v(f"SELECT pgcolumnar.export_parquet('plain_t','{out}/ok1.parquet')"), "allowed", + "and still exportable by it") + # Against the LIVE count: on unfixed code the import arms leak rows into rls_t. + owner_rows = _one(conn, "SELECT count(*) FROM rls_t") + import psycopg + try: + owner_read = _one(conn, "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1')") + except psycopg.Error: + owner_read = -1 # refused: a red arm by name, not a crash + expect.num(owner_read, owner_rows, "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)") + try: + _one(conn, f"SELECT pgcolumnar.export_parquet('rls_t','{out}/owner.parquet')") + except psycopg.Error: + pass + # No file, an empty one, or one that does not read holds no rows: a refused or + # empty export is a red arm by name, not a crash (#1383 review). + exported = 0 + if os.path.exists(f"{out}/owner.parquet") and os.path.getsize(f"{out}/owner.parquet") > 0: + try: + exported = _one(conn, f"SELECT count(*) FROM pgcolumnar.read_parquet('{out}/owner.parquet') AS (id int," + " owner_name text, ssn text)") + except psycopg.Error: + exported = 0 + expect.num(exported, owner_rows, "and the owner can still export it") + + # ---- FORCE ROW LEVEL SECURITY reaches a NON-superuser owner too + with conn.cursor() as cur: + cur.execute("CREATE TABLE force_t (id int, owner_name text) USING pgcolumnar") + cur.execute("INSERT INTO force_t SELECT g, 'nobody' FROM generate_series(1,50) g") + cur.execute("SELECT pgcolumnar.add_projection('force_t','p1',ARRAY['id'],ARRAY['id'])") + cur.execute("SELECT pgcolumnar.rebuild_projections('force_t')") + cur.execute(f"ALTER TABLE force_t OWNER TO {OWNER}") + cur.execute("ALTER TABLE force_t ENABLE ROW LEVEL SECURITY") + cur.execute(f"CREATE POLICY p_none ON force_t FOR SELECT TO {OWNER} USING (owner_name = current_user)") + expect.text(_one(conn, "SELECT pg_get_userbyid(relowner) FROM pg_class WHERE oid = 'force_t'::regclass"), + OWNER, "premise: t_owner owns force_t") + got, _ = _login(pgc_cluster, OWNER, schema, "SELECT count(*) FROM pgcolumnar.read_projection('force_t','p1')") + expect.num(-1 if got is None else got, 50, "without FORCE the owner is allowed, because RLS_NONE_ENV is not RLS_ENABLED") + with conn.cursor() as cur: + cur.execute("ALTER TABLE force_t FORCE ROW LEVEL SECURITY") + _, err = _login(pgc_cluster, OWNER, schema, "SELECT count(*) FROM pgcolumnar.read_projection('force_t','p1')") + expect.num(1 if err is not None and "row-level security" in (err.diag.message_primary or "") else 0, 1, + "under FORCE the non-superuser owner IS refused, so owner-only is no substitute") diff --git a/test/pytest/test_tier_partition.py b/test/pytest/test_tier_partition.py index ad306382a..929b99a17 100644 --- a/test/pytest/test_tier_partition.py +++ b/test/pytest/test_tier_partition.py @@ -530,3 +530,63 @@ def test_tier_partition_moves_a_virtual_generated_column(pgc_own_db, server_dir, kind = _one(conn, "SELECT relkind::text FROM pg_class WHERE oid = 'cv_mar'::regclass") expect.text(f"{moved}|{kind}|{'same' if before and before == _one(conn, digest) else 'differs'}", "71|f|same", "a virtual generated column moves and reads back the same") + + +def test_tier_partition_runs_on_the_callers_own_privileges(pgc_own_db, pgc_cluster, server_dir, expect): + """tier_partition is not SECURITY DEFINER, so a role that does not own the table is + refused by its own privileges (#1341). Made SECURITY DEFINER, it would run as the + extension's owner and let any caller replace a partition it does not own. Both roles + hold every other privilege the call needs, so ownership is all that separates them.""" + conn = pgc_own_db + db = _one(conn, "SELECT current_database()") + with conn.cursor() as cur: + cur.execute("SET timezone = 'UTC'") + cur.execute("CREATE SERVER tq FOREIGN DATA WRAPPER pgcolumnar_parquet") + for r in ("tier_own_py", "tier_peer_py"): + cur.execute(f"DO $$ BEGIN CREATE ROLE {r} LOGIN NOSUPERUSER; " + "EXCEPTION WHEN duplicate_object THEN NULL; END $$") + cur.execute(f"GRANT pg_write_server_files TO {r}") + cur.execute(f"GRANT USAGE ON FOREIGN SERVER tq TO {r}") + cur.execute(f"GRANT USAGE, CREATE ON SCHEMA public TO {r}") + cur.execute(f"GRANT USAGE ON SCHEMA pgcolumnar TO {r}") + # The owner's control has a table of its own (pw), so a refusal that let the peer + # through cannot make the control fail for the peer's reason. + for t in ("po", "pw"): + _build(conn, t) + with conn.cursor() as cur: + for r in (t, f"{t}_mar", f"{t}_apr"): + cur.execute(f"ALTER TABLE {r} OWNER TO tier_own_py") + with conn.cursor() as cur: + cur.execute("GRANT SELECT ON po, po_mar TO tier_peer_py") + before = _digest(conn, "po") + + def as_role(role, sql): + try: + with psycopg.connect(f"host=127.0.0.1 port={pgc_cluster.port} user={role} dbname={db}", + autocommit=True) as c: + c.execute("SET timezone = 'UTC'") + c.execute(sql) + return "ok" + except psycopg.Error as e: + return e.sqlstate or "?" + + owners = _one(conn, "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class" + " WHERE relname IN ('po', 'po_mar', 'po_apr', 'pw', 'pw_mar', 'pw_apr')") + supers = _one(conn, "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own_py', 'tier_peer_py') AND rolsuper") + expect.text(f"{owners}|{supers}", "tier_own_py|0", + "premise: tier_own owns the parent and its partitions, and neither role is a superuser") + expect.text(as_role("tier_peer_py", "SELECT count(*) FROM po_mar"), "ok", + "premise: tier_peer can open a session and read the partition") + peer_path = server_dir / "po_peer.parquet" + expect.text(as_role("tier_peer_py", f"SELECT pgcolumnar.tier_partition('po_mar', 'tq', '{peer_path}')"), + "42501", "a role that does not own the table is refused by its own privileges (42501)") + expect.text(f"{_digest(conn, 'po')}|{_mar_shape(conn, 'po')}", f"{before}|r:pgcolumnar:{BOUND}", + "and the refused call leaves the partition local and attached, every row once") + expect.num(_one(conn, "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid" + " WHERE c.relname LIKE 'po\\_%'"), 0, "with no foreign table behind it") + # The LOCK ... IN SHARE MODE ahead of the export is the first statement to need + # more than SELECT, so the refusal comes before any file is written. + expect.text("written" if peer_path.exists() else "absent", "absent", "and the refused call wrote no export file") + expect.text(as_role("tier_own_py", f"SELECT pgcolumnar.tier_partition('pw_mar', 'tq', '{server_dir / 'pw_own.parquet'}')") + + f"|{_mar_shape(conn, 'pw')}", f"ok|f:-:{BOUND}", + "a non-superuser owner tiers its own partition") diff --git a/test/rls_direct_storage.sh b/test/rls_direct_storage.sh index 355bda4b9..e26f99424 100755 --- a/test/rls_direct_storage.sh +++ b/test/rls_direct_storage.sh @@ -58,16 +58,18 @@ psql_run "GRANT SELECT, INSERT ON plain_t TO t_rls;" # the RLS check, so they would report acl and prove nothing about this fix. psql_run "GRANT EXECUTE ON FUNCTION pgcolumnar.read_projection(regclass,text) TO t_rls;" psql_run "GRANT EXECUTE ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass,text) TO t_rls;" -psql_run "ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY;" -psql_run "CREATE POLICY p_own ON rls_t FOR SELECT TO t_rls USING (owner_name = current_user);" -psql_run "CREATE POLICY p_ins ON rls_t FOR INSERT TO t_rls WITH CHECK (owner_name = current_user);" -# A source file the role may import, produced by the owner from the open table. +# A source file the role may import, produced by the owner from the open table, +# BEFORE row-level security is enabled, so no version of the guard under test can +# stop the fixture from being made. # Exported from rls_t BY THE OWNER, so the source shape matches the import # target. A source taken from a differently shaped table makes the import arms # fail on column counts, which looks like a refusal and is not one. psql_run "SELECT pgcolumnar.export_parquet('rls_t','$OUT/src.parquet');" psql_run "SELECT pgcolumnar.export_arrow('rls_t','$OUT/src.arrow');" psql_run "COPY rls_t TO '$OUT/src.txt';" +psql_run "ALTER TABLE rls_t ENABLE ROW LEVEL SECURITY;" +psql_run "CREATE POLICY p_own ON rls_t FOR SELECT TO t_rls USING (owner_name = current_user);" +psql_run "CREATE POLICY p_ins ON rls_t FOR INSERT TO t_rls WITH CHECK (owner_name = current_user);" as_rls() { # as_rls -> the ERROR line, connfail, or noerror local out @@ -90,20 +92,26 @@ verdict() { # verdict -> rls | acl | allowed | other } # ---- premises, every one run BY the restricted role ------------------------- +# +# Two of these read the table through the columnar scan, so a product change can +# redden them, and they are checks, not premises (#1236; #1397 review): the policy's +# row count, which the custom scan dropping its quals turned into every row, and the +# owner's count. The source files are a third: the owner's export writes them, so an +# export that writes nothing reddens that check (#1383 review). check "premise: the role can open a session" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At -c 'SELECT 1;' 2>&1 | head -1)" "1" check "premise: the role HOLDS select, so the ACL check cannot be what refuses" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c "SELECT has_table_privilege('rls_t','SELECT');" 2>&1 | head -1)" "t" -check_num "premise: the policy is in force for ordinary SQL" \ +check_num "the policy is in force for ordinary SQL" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c 'SELECT count(*) FROM rls_t;' 2>&1 | head -1)" "1" check "premise: the role holds EXECUTE, so the grant layer is not what refuses" \ "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U t_rls -d "$PGC_DB" -At \ -c "SELECT has_function_privilege('pgcolumnar.read_projection(regclass,text)','EXECUTE');" 2>&1 | head -1)" "t" -check_num "premise: the owner sees every row, so the policy hides something real" \ +check_num "the owner sees every row, so the policy hides something real" \ "$(q 'SELECT count(*) FROM rls_t;')" "$ROWS" -check "premise: the source files the import arms need exist" \ +check "the source files the import arms need exist" \ "$([ -s "$OUT/src.parquet" ] && [ -s "$OUT/src.arrow" ] && [ -s "$OUT/src.txt" ] && echo yes || echo no)" "yes" # ---- the refusals ----------------------------------------------------------- @@ -166,8 +174,11 @@ check "and still exportable by it" \ owner_rows="$(q 'SELECT count(*) FROM rls_t;')" check "the OWNER can still read the RLS table's storage (RLS_NONE_ENV)" \ "$(q "SELECT count(*) FROM pgcolumnar.read_projection('rls_t','p1');" 2>&1 | head -1)" "$owner_rows" -check "and the owner can still export it" \ - "$([ -n "$(q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" 2>&1)" ] && echo ran || echo ran)" "ran" +# Measured as the rows the export holds, read back: both branches of the old test +# here echoed "ran", so a refused owner export passed it. +q "SELECT pgcolumnar.export_parquet('rls_t','$OUT/owner.parquet');" >/dev/null 2>&1 +check_num "and the owner can still export it" \ + "$([ -s "$OUT/owner.parquet" ] && q "SELECT count(*) FROM pgcolumnar.read_parquet('$OUT/owner.parquet') AS (id int, owner_name text, ssn text);" 2>&1 | head -1 || echo 0)" "$owner_rows" # ---- FORCE ROW LEVEL SECURITY reaches the owner too ------------------------- # diff --git a/test/tier_partition.sh b/test/tier_partition.sh index 03b3540d8..186370e20 100755 --- a/test/tier_partition.sh +++ b/test/tier_partition.sh @@ -399,6 +399,56 @@ else "PostgreSQL $PGC_MAJOR has no virtual generated columns" fi +# ---- the caller's own privileges (#1341) ------------------------------------- +# +# tier_partition is not SECURITY DEFINER: every step runs as the caller, so the +# owner-only steps (its LOCK ... IN SHARE MODE first) stop a role that does not own +# the table. +# Made SECURITY DEFINER, it would run as the extension's owner and let any caller +# detach, drop and replace a partition it does not own. Two roles, each holding +# every other privilege the call needs (the server-file role for the export, USAGE +# on the server and CREATE on the schema for the foreign table, and SELECT on the +# partition), so ownership is the only thing between them. +for r in tier_own tier_peer; do + psql_run "DROP ROLE IF EXISTS $r; CREATE ROLE $r LOGIN NOSUPERUSER; + GRANT pg_write_server_files TO $r; GRANT USAGE ON FOREIGN SERVER tier_pq TO $r; + GRANT USAGE, CREATE ON SCHEMA public TO $r; GRANT USAGE ON SCHEMA pgcolumnar TO $r;" +done +sqlstate_as() { # sqlstate_as ROLE SQL -> the SQLSTATE SQL fails with as ROLE, or ok + local out + out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" -d "$PGC_DB" \ + -X -At -v ON_ERROR_STOP=1 -v VERBOSITY=sqlstate -c "$2" 2>&1 >/dev/null) + [ -z "$out" ] && { echo ok; return; } + echo "$out" | sed -n 's/^\(ERROR\|FATAL\): *\([0-9A-Z]\{5\}\).*/\2/p' | head -1 +} +# The owner's control has a table of its own (pw), so a refusal that let the peer +# through cannot make the control fail for the peer's reason. +for t in po pw; do + build $t + psql_run "ALTER TABLE $t OWNER TO tier_own; ALTER TABLE ${t}_jan OWNER TO tier_own; + ALTER TABLE ${t}_feb OWNER TO tier_own;" +done +psql_run "GRANT SELECT ON po, po_jan TO tier_peer;" +pb="$(digest po)" +check "premise: tier_own owns the parent and its partitions, and neither role is a superuser" \ + "$(q "SELECT string_agg(DISTINCT pg_get_userbyid(relowner), ',') FROM pg_class WHERE relname IN ('po', 'po_jan', 'po_feb', 'pw', 'pw_jan', 'pw_feb');")|$(q "SELECT count(*) FROM pg_roles WHERE rolname IN ('tier_own', 'tier_peer') AND rolsuper;")" \ + "tier_own|0" +check "premise: tier_peer can open a session and read the partition" \ + "$(sqlstate_as tier_peer "SELECT count(*) FROM po_jan")" "ok" +check "a role that does not own the table is refused by its own privileges (42501)" \ + "$(sqlstate_as tier_peer "SELECT pgcolumnar.tier_partition('po_jan', 'tier_pq', '$W/po_peer.parquet')")" "42501" +check "and the refused call leaves the partition local and attached, every row once" \ + "$(digest po)|$(jan_shape po)" "$pb|r:pgcolumnar:$JAN_BOUND" +check_num "with no foreign table behind it" \ + "$(q "SELECT count(*) FROM pg_foreign_table t JOIN pg_class c ON c.oid = t.ftrelid WHERE c.relname LIKE 'po\\_%';")" "0" +# The first statement to need more than SELECT is the LOCK ... IN SHARE MODE ahead +# of the export, so the refusal comes before any file is written (measured on 15-19). +check "and the refused call wrote no export file" \ + "$([ -e "$W/po_peer.parquet" ] && echo written || echo absent)" "absent" +check "a non-superuser owner tiers its own partition" \ + "$(sqlstate_as tier_own "SELECT pgcolumnar.tier_partition('pw_jan', 'tier_pq', '$W/pw_own.parquet')")|$(jan_shape pw)" \ + "ok|f:-:$JAN_BOUND" + # ---- a failure after each step ---------------------------------------------- for step in export detach drop create attach; do build f