From 1daf3d6bc9f56c708edb6ffd043fbff49154840e Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 21:20:47 +0000 Subject: [PATCH 01/15] test: iceberg_dv gains its pytest twin, and a refusal must be its own guard's (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit iceberg_dv.sh gains test/pytest/test_iceberg_dv.py, one check per test under the shell's names, and two holes in the shell are closed, each measured red-first on PG17: - "a manifest/footer offset mismatch is refused (XX001)" read the SQLSTATE alone, and several guards raise XX001. With the offset match removed (the !found refusal skipped), a later XX001 refused the same file and main's shell passed 26/26. Every refusal now reads its own guard's message: refusal() answers the SQLSTATE only when the message carries the guard's phrase, and " (another refusal: ...)" otherwise. - The three "backend still up" checks read SELECT 1, which answers again after crash recovery. With the referenced_data_file check removed a NULL path reaches strcmp and the backend dies on that arm; main passed "backend still up after the DV refusals". They now also read the server log's count of killed backends (crashes/survived, as avro_manifest), and each refusal waits for the server to answer first (await_up), so a crash reddens its own arm and its section's survival check, the same names in both harnesses, and not a timing-dependent tail of collateral. The eight survivor comparisons move to check_text, so an empty oracle no longer matches an empty scan. Measured on PG17, shell then twin as shipped: seventeen mutations, each with the same reds in both (strictgt 7, anyfile/nodup/nov2/noflags/noref/nobigoff 2, the rest 1); 26/26 in each on 15-19. 25 rows dated, 1 never (the survival check after the DV applications, which no mutation crashed). Census 3884 -> 3885 never, 2286 -> 2311 dated, 6196 rows; suites_not_covered 102 -> 101; cluster_tests 2753 -> 2779; TESTS.md §268. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 26 +++ test/check_ledger_budget.txt | 22 +- test/iceberg_dv.sh | 134 +++++++---- test/pytest/TESTS.md | 38 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_iceberg_dv.py | 344 ++++++++++++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 525 insertions(+), 45 deletions(-) create mode 100644 test/pytest/test_iceberg_dv.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 218e6a82f..62f4a74d3 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2549,6 +2549,32 @@ iceberg_catalog iceberg_catalog malformed JSON is rejected (22P02), backend surv iceberg_catalog iceberg_catalog resolves the named current snapshot, not snapshots[0] 15;16;17;18;19 2026-10-21 the resolver returns the first snapshot rather than the one named current iceberg_catalog iceberg_catalog the decoy snapshot is not returned 15;16;17;18;19 2026-10-21 the resolver returns the first snapshot rather than the one named current iceberg_catalog iceberg_catalog the first snapshot has no parent (NULL) 15;16;17;18;19 never - +iceberg_dv iceberg_dv ...and the refusal names the compressed footer 15;16;17;18;19 2026-10-21 columnar_puffin.c: a compressed Puffin footer (flags bit 0) is not refused +iceberg_dv iceberg_dv ...and the refusal names the format version 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a deletion vector in a format-version 2 table is not refused +iceberg_dv iceberg_dv a DV blob declaring a compression codec is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a deletion-vector blob may declare a compression codec +iceberg_dv iceberg_dv a DV entry without referenced_data_file is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV without referenced_data_file is not refused (a NULL path reaches strcmp and the backend dies) +iceberg_dv iceberg_dv a DV whose offset+size overflows int64 is refused, not read (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a DV's offset and size are not bounded by the blob region (a wild read kills the backend) +iceberg_dv iceberg_dv a DV with run containers and a high-bucket position decodes 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies only when strictly newer than the data (> where the spec says >=) +iceberg_dv iceberg_dv a Puffin path outside the table location is refused (22023) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: ice_open_path does not hold a path under the table location +iceberg_dv iceberg_dv a bitset-container DV decodes 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies only when strictly newer than the data (> where the spec says >=) +iceberg_dv iceberg_dv a compressed Puffin footer is refused (0A000) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a compressed Puffin footer (flags bit 0) is not refused +iceberg_dv iceberg_dv a corrupted DV CRC is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a deletion vector's CRC-32 is not checked +iceberg_dv iceberg_dv a deletion vector at an equal sequence number applies (survivors == oracle) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies only when strictly newer than the data (> where the spec says >=) +iceberg_dv iceberg_dv a deletion vector in a format-version 2 table is refused (0A000) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a deletion vector in a format-version 2 table is not refused +iceberg_dv iceberg_dv a deletion vector older than the data does not apply 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies whatever its sequence number (dv->seq >= d->seq always true) +iceberg_dv iceberg_dv a manifest/footer offset mismatch is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: the Puffin footer's blob need not match the manifest's offset and length (the !found refusal skipped) +iceberg_dv iceberg_dv a record_count that disagrees with the DV cardinality is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a manifest record_count that disagrees with the DV cardinality is not refused +iceberg_dv iceberg_dv a run container overflowing its 16-bit range is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a run container whose start+length passes 0x10000 is not refused +iceberg_dv iceberg_dv a run-optimized DV (cookie 12347) decodes 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies only when strictly newer than the data (> where the spec says >=) +iceberg_dv iceberg_dv a wrong DV blob magic is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a deletion vector's magic sequence is not checked +iceberg_dv iceberg_dv an aliased duplicate DV (scheme vs no scheme) is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: the one-DV-per-data-file check compares paths without stripping the file:// scheme +iceberg_dv iceberg_dv an applicable DV supersedes position-delete files for its data file 15;16;17;18;19 2026-10-21 columnar_iceberg.c: position-delete files apply even when a DV applies (no supersede) +iceberg_dv iceberg_dv backend still up after the DV applications 15;16;17;18;19 never - +iceberg_dv iceberg_dv backend still up after the DV refusals 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV without referenced_data_file is not refused (a NULL path reaches strcmp and the backend dies) +iceberg_dv iceberg_dv backend still up after the audit arms 15;16;17;18;19 2026-10-21 columnar_puffin.c: a DV's offset and size are not bounded by the blob region (a wild read kills the backend) +iceberg_dv iceberg_dv supersede is per data file, not global 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies to every data file, not only the one it references +iceberg_dv iceberg_dv two DV blobs in one Puffin file each apply to their own data file 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a DV applies to every data file, not only the one it references +iceberg_dv iceberg_dv two DV entries for one data file are refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: two DVs for one data file are not refused iceberg_fdw_estimate iceberg_fdw_estimate a filtered scan estimates fewer rows than the whole table (selectivity applied) 15;16;17;18;19 2026-10-21 the Iceberg FDW sets rows directly, so the scan's selectivity never applies iceberg_fdw_estimate iceberg_fdw_estimate backend alive 15;16;17;18;19 never - iceberg_fdw_estimate iceberg_fdw_estimate the FDW estimates a positive row count from the manifests 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 2970fed4d..da7155f46 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 102 +suites_not_covered 101 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5439,4 +5439,22 @@ suites_not_covered 102 # never 3883 + 1 = 3884 # dated 2282 + 4 = 2286 # 3884 + 2286 == 6170 -checks_never_observed_red 3884 +# #1341, iceberg_dv seeded, with its twin (OffgridwithJD): 3884 -> 3885, and +# suites_not_covered 102 -> 101. The suite had no rows. It enters with 26 checks from +# five-major runs (all green on each of 15-19, shell and twin alike), after two fixes. +# "a manifest/footer offset mismatch is refused (XX001)" read the SQLSTATE alone, and +# several guards here raise XX001: with the offset match removed a later XX001 refused +# the same file, and main's shell passed 26/26. Every refusal now reads its own guard's +# message. And the three "backend still up" checks read SELECT 1, which answers again +# after crash recovery: with the referenced_data_file check removed a backend died on +# the arm, and main passed the survival check. They now also read the server log's +# count of killed backends, and each refusal waits for the server first, so a crash +# reddens its own arm and its section's survival check and nothing else. 25 are dated by +# seventeen mutations, each reddening the same names in both twins. 1 arrives never: +# the survival check after the DV applications, which no mutation crashed. +# +# ledger rows 6170 + 26 = 6196 +# never 3884 + 1 = 3885 +# dated 2286 + 25 = 2311 +# 3885 + 2311 == 6196 +checks_never_observed_red 3885 diff --git a/test/iceberg_dv.sh b/test/iceberg_dv.sh index d9df48ca8..5f8d80677 100755 --- a/test/iceberg_dv.sh +++ b/test/iceberg_dv.sh @@ -16,22 +16,71 @@ set -uo pipefail . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" -sqlstate_of() { - env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -qtA 2>&1 < the SQLSTATE when SQL is refused with PHRASE in its message; +# "SQLSTATE (another refusal: MESSAGE)" when a different guard refused it; "no +# error" when it ran; nothing when the server could not be asked. An arm used to +# read the SQLSTATE alone, and several guards here raise XX001: with the +# manifest/footer offset match removed, a later XX001 refused the same file and +# that arm still passed. +refusal() { + local out st msg + await_up + out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ + -d "$PGC_DB" -qtA 2>&1 <&1 < wait, at most 60 s, until the server answers. A refusal arm that kills a +# backend sends the server through crash recovery; without the wait, the next arms' +# readings fail on recovery and red as collateral, more or fewer of them by timing. +# The crash itself is still caught: survived() counts it. +await_up() { + local i + for i in $(seq 1 120); do + [ "$(q 'SELECT 1')" = 1 ] && return 0 + sleep 0.5 + done +} + +# Backends killed by a signal, from the server log: a number, or nothing when the +# log can't be read. A fresh connection answers again after crash recovery, so +# "SELECT 1" alone passed with a backend killed by a refusal arm (no +# referenced_data_file check: a NULL path reached strcmp); the survival checks +# below also require that this count has not moved. +crashes() { + local _n + _n="$(pgc_pg "grep -c 'terminated by signal' '$PGC_LOGFILE'" 2>/dev/null)" + case "$_n" in '' | *[!0-9]*) ;; *) echo "$_n" ;; esac +} +survived() { # survived BEFORE -> "1" when the server answers and no backend died since + local _up _now + _up="$(q 'SELECT 1')"; _now="$(crashes)" + if ! [[ "$1" =~ ^[0-9]+$ && "$_now" =~ ^[0-9]+$ ]]; then echo "no (crash count unreadable)" + elif [ "$_up" = 1 ] && [ "$_now" = "$1" ]; then echo 1 + else echo "no ([$_up], $(( _now - $1 )) backend crashes)"; fi +} + FX="$(dirname "${BASH_SOURCE[0]}")/fixtures/iceberg" WHD="$FX/warehouse_del" [ -f "$WHD/db/t/metadata/dvapply.metadata.json" ] || pgc_skip fixture "deletion-vector fixtures are missing" @@ -60,100 +109,103 @@ dv_ids_of() { # ---- deletion vectors are applied ------------------------------------------- # THE <= boundary: a DV at the data file's own sequence number applies (same # rule as 4a position-delete files; a strict > would wrongly exclude it) -check "a deletion vector at an equal sequence number applies (survivors == oracle)" \ +CRASHES_BEFORE="$(crashes)" +check_text "a deletion vector at an equal sequence number applies (survivors == oracle)" \ "$(dv_ids_of dvapply)" "$(dv_oracle dvapply)" # the other side: a DV strictly older than the data applies to nothing -check "a deletion vector older than the data does not apply" \ +check_text "a deletion vector older than the data does not apply" \ "$(dv_ids_of dvnoapply)" "$(dv_oracle dvnoapply)" # the spec's supersede rule: a position-delete FILE applies only when no DV # must be applied to the data file. The fixture's DV drops ordinal 1 and its # (also applicable) Parquet posdel names ordinals 1 and 3; a UNION would drop # both, supersede drops only ordinal 1. -check "an applicable DV supersedes position-delete files for its data file" \ +check_text "an applicable DV supersedes position-delete files for its data file" \ "$(dv_ids_of dvsupersede)" "$(dv_oracle dvsupersede)" # supersede is per data file: a posdel targeting a DIFFERENT data file still # applies alongside the DV -check "supersede is per data file, not global" \ +check_text "supersede is per data file, not global" \ "$(dv_ids_of dvother)" "$(dv_oracle dvother)" # roaring coverage: run-form containers plus a second 64-bit bucket -check "a DV with run containers and a high-bucket position decodes" \ +check_text "a DV with run containers and a high-bucket position decodes" \ "$(dv_ids_of dvwide)" "$(dv_oracle dvwide)" # cookie 12347 (run containers marked in the run bitset) -check "a run-optimized DV (cookie 12347) decodes" \ +check_text "a run-optimized DV (cookie 12347) decodes" \ "$(dv_ids_of dvrun)" "$(dv_oracle dvrun)" # a bitset container (cardinality above 4096) -check "a bitset-container DV decodes" \ +check_text "a bitset-container DV decodes" \ "$(dv_ids_of dvbitset)" "$(dv_oracle dvbitset)" # one Puffin file carrying two DV blobs for two data files -check "two DV blobs in one Puffin file each apply to their own data file" \ +check_text "two DV blobs in one Puffin file each apply to their own data file" \ "$(dv_ids_of dvtwo)" "$(dv_oracle dvtwo)" -check "backend still up after the DV applications" "$(q 'SELECT 1')" "1" +check "backend still up after the DV applications" "$(survived "$CRASHES_BEFORE")" "1" # ---- refusal and corruption arms -------------------------------------------- +CRASHES_BEFORE="$(crashes)" # two DV entries referencing one data file: the spec says scan results are # undefined and readers may raise; we raise rather than guess check "two DV entries for one data file are refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvdup.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvdup.metadata.json') + AS t(id bigint, region text, amount int)" "two deletion vectors reference data file")" "XX001" check "a corrupted DV CRC is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbadcrc.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbadcrc.metadata.json') + AS t(id bigint, region text, amount int)" "fails its CRC-32 check")" "XX001" check "a wrong DV blob magic is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbadmagic.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbadmagic.metadata.json') + AS t(id bigint, region text, amount int)" "has a bad magic sequence")" "XX001" # the manifest's content_offset must exactly match the footer's blob offset check "a manifest/footer offset mismatch is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvoffmismatch.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvoffmismatch.metadata.json') + AS t(id bigint, region text, amount int)" "has no blob matching the manifest's offset")" "XX001" # referenced_data_file is required for DVs check "a DV entry without referenced_data_file is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvnoref.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvnoref.metadata.json') + AS t(id bigint, region text, amount int)" "has no referenced_data_file")" "XX001" # record_count is defined as the DV's cardinality check "a record_count that disagrees with the DV cardinality is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbadcount.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbadcount.metadata.json') + AS t(id bigint, region text, amount int)" "positions but its manifest entry records")" "XX001" # deletion-vector-v1 blobs must not declare a compression codec check "a DV blob declaring a compression codec is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvcompressed.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvcompressed.metadata.json') + AS t(id bigint, region text, amount int)" "declares a compression codec")" "XX001" # a compressed Puffin footer is refused loudly, never misparsed check "a compressed Puffin footer is refused (0A000)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvflags.metadata.json') - AS t(id bigint, region text, amount int)")" "0A000" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvflags.metadata.json') + AS t(id bigint, region text, amount int)" "has a compressed Puffin footer")" "0A000" check "...and the refusal names the compressed footer" \ "$(errmsg_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvflags.metadata.json') AS t(id bigint, region text, amount int)" | grep -c "compressed")" "1" # deletion vectors are v3-only check "a deletion vector in a format-version 2 table is refused (0A000)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvv2.metadata.json') - AS t(id bigint, region text, amount int)")" "0A000" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvv2.metadata.json') + AS t(id bigint, region text, amount int)" "is a deletion vector, which format-version")" "0A000" check "...and the refusal names the format version" \ "$(errmsg_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvv2.metadata.json') AS t(id bigint, region text, amount int)" | grep -c "format-version")" "1" # a Puffin path outside the table root is stopped by the path boundary check "a Puffin path outside the table location is refused (22023)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvescape.metadata.json') - AS t(id bigint, region text, amount int)")" "22023" -check "backend still up after the DV refusals" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvescape.metadata.json') + AS t(id bigint, region text, amount int)" "is not under the table location")" "22023" +check "backend still up after the DV refusals" "$(survived "$CRASHES_BEFORE")" "1" # ---- audit arms (multi-agent adversarial audit findings) -------------------- +CRASHES_BEFORE="$(crashes)" # content_offset + content_size chosen to overflow int64: a bounds check that # adds the operands would wrap negative and read a wild pointer; testing each # operand against the file size refuses it cleanly check "a DV whose offset+size overflows int64 is refused, not read (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbigoff.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvbigoff.metadata.json') + AS t(id bigint, region text, amount int)" "lies outside the file's blob region")" "XX001" # a run container whose start+length exceeds the 16-bit container range would # fabricate positions in a neighboring container; refuse rather than corrupt check "a run container overflowing its 16-bit range is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvrunoverflow.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvrunoverflow.metadata.json') + AS t(id bigint, region text, amount int)" "overflows its 16-bit range")" "XX001" # two DV entries for one data file recorded with and without the file:// scheme: # the one-DV-per-file check strips the scheme, so the aliased pair is caught check "an aliased duplicate DV (scheme vs no scheme) is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvdupscheme.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" -check "backend still up after the audit arms" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/dvdupscheme.metadata.json') + AS t(id bigint, region text, amount int)" "two deletion vectors reference data file")" "XX001" +check "backend still up after the audit arms" "$(survived "$CRASHES_BEFORE")" "1" pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 77e562c8b..8121f453c 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -313,6 +313,7 @@ behaviour, the source of that number is named. - [265. test_iceberg_malformed.py: the Iceberg reader refuses crafted metadata cleanly, never crashing or hanging](#265-test_iceberg_malformedpy-the-iceberg-reader-refuses-crafted-metadata-cleanly-never-crashing-or-hanging) - [266. test_iceberg_name_mapping.py: an Iceberg data file with no field ids binds by name mapping](#266-test_iceberg_name_mappingpy-an-iceberg-data-file-with-no-field-ids-binds-by-name-mapping) - [267. test_iceberg_objstore.py: Iceberg tables read over object storage](#267-test_iceberg_objstorepy-iceberg-tables-read-over-object-storage) +- [268. test_iceberg_dv.py: Iceberg v3 deletion vectors apply, supersede, and refuse what they must](#268-test_iceberg_dvpy-iceberg-v3-deletion-vectors-apply-supersede-and-refuse-what-they-must) ## 1. How to read a test in here @@ -10396,3 +10397,40 @@ One check per test, each by the shell's name, every reading a value from a modul | `test_the_backend_is_still_up_after_the_s3_refusal` | backend still up after the s3 refusal | | `test_an_endpoint_not_on_the_allow_list_is_refused` | an endpoint not on objstore_allowed_endpoints is refused | | `test_the_backend_is_still_up_after_the_endpoint_refusal` | backend still up after the endpoint refusal | + +## 268. test_iceberg_dv.py: Iceberg v3 deletion vectors apply, supersede, and refuse what they must + +The twin of `test/iceberg_dv.sh`. Eight hand-crafted snapshots read the oracle's surviving ids, thirteen corrupt or unsupported ones are refused, each by its own guard, and no backend dies on the way. The warehouse is relocated into `server_dir` so its recorded root differs from where it lies. + +### Every test + +Every test makes one check under the shell's name. A refusal reads as its SQLSTATE only when the message is that guard's own (several guards raise XX001), and a failed reading is a `query_error()`. Each survival check also requires that the server log's count of backends killed by a signal has not moved since its section began, and each refusal first waits for the server to answer, so a crash reddens only its own arm and its section's survival check. + +| test | what it holds | +| --- | --- | +| `test_a_deletion_vector_at_an_equal_sequence_number_applies` | a deletion vector at an equal sequence number applies (survivors == oracle) | +| `test_a_deletion_vector_older_than_the_data_does_not_apply` | a deletion vector older than the data does not apply | +| `test_an_applicable_dv_supersedes_position_delete_files` | an applicable DV supersedes position-delete files for its data file | +| `test_dv_supersede_is_per_data_file_not_global` | supersede is per data file, not global | +| `test_a_dv_with_run_containers_and_a_high_bucket_decodes` | a DV with run containers and a high-bucket position decodes | +| `test_a_run_optimized_dv_decodes` | a run-optimized DV (cookie 12347) decodes | +| `test_a_bitset_container_dv_decodes` | a bitset-container DV decodes | +| `test_two_dv_blobs_in_one_puffin_file_each_apply_to_their_own_file` | two DV blobs in one Puffin file each apply to their own data file | +| `test_the_backend_survives_the_dv_applications` | backend still up after the DV applications | +| `test_two_dv_entries_for_one_data_file_are_refused` | two DV entries for one data file are refused (XX001) | +| `test_a_corrupted_dv_crc_is_refused` | a corrupted DV CRC is refused (XX001) | +| `test_a_wrong_dv_blob_magic_is_refused` | a wrong DV blob magic is refused (XX001) | +| `test_a_manifest_footer_offset_mismatch_is_refused` | a manifest/footer offset mismatch is refused (XX001) | +| `test_a_dv_entry_without_referenced_data_file_is_refused` | a DV entry without referenced_data_file is refused (XX001) | +| `test_a_record_count_disagreeing_with_the_dv_is_refused` | a record_count that disagrees with the DV cardinality is refused (XX001) | +| `test_a_dv_blob_declaring_a_codec_is_refused` | a DV blob declaring a compression codec is refused (XX001) | +| `test_a_compressed_puffin_footer_is_refused` | a compressed Puffin footer is refused (0A000) | +| `test_the_footer_refusal_names_the_compressed_footer` | ...and the refusal names the compressed footer | +| `test_a_dv_in_a_format_version_2_table_is_refused` | a deletion vector in a format-version 2 table is refused (0A000) | +| `test_the_v2_refusal_names_the_format_version` | ...and the refusal names the format version | +| `test_a_puffin_path_outside_the_table_is_refused` | a Puffin path outside the table location is refused (22023) | +| `test_the_backend_survives_the_dv_refusals` | backend still up after the DV refusals | +| `test_a_dv_whose_offset_plus_size_overflows_is_refused` | a DV whose offset+size overflows int64 is refused, not read (XX001) | +| `test_a_run_container_overflowing_its_range_is_refused` | a run container overflowing its 16-bit range is refused (XX001) | +| `test_an_aliased_duplicate_dv_is_refused` | an aliased duplicate DV (scheme vs no scheme) is refused (XX001) | +| `test_the_backend_survives_the_audit_arms` | backend still up after the audit arms | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index aa84b2450..07ed43164 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -906,6 +906,8 @@ guard_tests 476 # 22 checks in 22 tests, by collection over the cluster files. # cluster_tests 2732 -> 2748 for #1341's test_iceberg_objstore.py, the twin of iceberg_objstore.sh: # 16 checks in 16 tests, by collection over the cluster files. +# cluster_tests 2753 -> 2779 for #1341's test_iceberg_dv.py, the twin of iceberg_dv.sh: +# 26 checks in 26 tests, by collection over the cluster files. # cluster_tests 1264 -> 1287 for #1341's test_vector_agg_rescan_memory.py, the twin of vector_agg_rescan_memory.sh: # 23 tests, one per check, by collection over the cluster files. # cluster_tests 1287 -> 1329 for #1341's test_analyze_stats.py, the twin of analyze_stats.sh: @@ -1037,4 +1039,4 @@ guard_tests 476 # cluster_tests 2670 -> 2677 for #1637's test_catalog_filter_reads.py, the twin of catalog_filter_reads.sh: # 7 tests, one per check, by collection over the cluster files. # cluster_tests 2748 -> 2753 for #1706's five timestamp-flag tests in test_native_parquet_schema.py, by collection. -cluster_tests 2753 +cluster_tests 2779 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 5a7aa689a..8ae5ab97e 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -127,6 +127,7 @@ "iceberg_scan", "iceberg_malformed", "iceberg_objstore", + "iceberg_dv", "concurrency", "isolation", "index_only", diff --git a/test/pytest/test_iceberg_dv.py b/test/pytest/test_iceberg_dv.py new file mode 100644 index 000000000..795608f0c --- /dev/null +++ b/test/pytest/test_iceberg_dv.py @@ -0,0 +1,344 @@ +"""Iceberg v3 deletion vectors (#388 phase 4c): iceberg_scan reads a table whose current +snapshot carries Puffin deletion vectors and drops the rows they name. The pytest twin +of `test/iceberg_dv.sh`, every check by the shell's name (#1341). + +The fixtures are hand-crafted (test/fixtures/iceberg/warehouse_del): pyroaring bitmaps +in hand-built Puffin files, hand-encoded manifests, exact sequence numbers, and an +oracle cross-checked against pyiceberg and DuckDB. The warehouse is relocated so its +recorded root differs from where it lies. + +Asserted: eight snapshots read the oracle's surviving ids (the <= sequence boundary, +supersede per data file, run, bitset and high-bucket containers, two blobs in one +file); thirteen corrupt or unsupported snapshots are refused, each by its own guard; +and no backend dies on the way. + +HOW A REFUSAL IS READ. Several guards raise XX001, so a refusal counts only when its +message is that guard's own: with the manifest/footer offset match removed, a later +XX001 refused the same file and the arm, reading the SQLSTATE alone, still passed. A +refusal reads as its SQLSTATE, " (another refusal: ...)", "no error", or a +query_error() when the server could not be asked. + +HOW SURVIVAL IS READ. A fresh connection answers again after crash recovery, so +"SELECT 1" alone passed with a backend killed by a refusal arm. Each survival check +also requires that the server log's count of backends terminated by a signal has not +moved since its section began. + +ON A PRIVATE DATABASE, with the warehouse copied into `server_dir`, because the backend +reads it. Where the fixture is missing the first test fails under the shell's pgc_skip +message, and the rest decline. +""" +import json +import pathlib +import shutil +import time + +import psycopg +import pytest + +from pgc_vacuity import query_error + +DB = "pgc_iceberg_dv" +FX = pathlib.Path(__file__).resolve().parents[1] / "fixtures" / "iceberg" +WHD = FX / "warehouse_del" +COLS = "AS t(id bigint, region text, amount int)" +APPLIED = ("dvapply", "dvnoapply", "dvsupersede", "dvother", "dvwide", "dvrun", "dvbitset", "dvtwo") +REFUSED = { + "dvdup": "two deletion vectors reference data file", + "dvbadcrc": "fails its CRC-32 check", + "dvbadmagic": "has a bad magic sequence", + "dvoffmismatch": "has no blob matching the manifest's offset", + "dvnoref": "has no referenced_data_file", + "dvbadcount": "positions but its manifest entry records", + "dvcompressed": "declares a compression codec", + "dvflags": "has a compressed Puffin footer", + "dvv2": "is a deletion vector, which format-version", + "dvescape": "is not under the table location", +} +AUDIT = { + "dvbigoff": "lies outside the file's blob region", + "dvrunoverflow": "overflows its 16-bit range", + "dvdupscheme": "two deletion vectors reference data file", +} + + +def _message(exc): + return str(exc).splitlines()[0] if str(exc) else type(exc).__name__ + + +def _value(dsn, sql): + try: + with psycopg.connect(dsn, autocommit=True, connect_timeout=10) as conn: + row = conn.execute(sql).fetchone() + except psycopg.Error as exc: + return query_error(_message(exc)) + return "" if row is None or row[0] is None else row[0] + + +def _await_up(dsn): + """Wait, at most 60 s, until the server answers, as the shell's await_up: a refusal + that kills a backend sends the server through crash recovery, and the next readings + would otherwise red as collateral, by timing. survived() still counts the crash.""" + for _ in range(120): + if _value(dsn, "SELECT 1") == 1: + return + time.sleep(0.5) + + +def _refusal(dsn, sql, phrase): + """-> the SQLSTATE when sql is refused with phrase in its primary message, the + SQLSTATE with the other guard's message, "no error", or query_error().""" + _await_up(dsn) + try: + with psycopg.connect(dsn, autocommit=True, connect_timeout=10) as conn: + conn.execute(sql).fetchall() + except psycopg.Error as exc: + if not exc.sqlstate: + return query_error(_message(exc)) + msg = exc.diag.message_primary or "" + return exc.sqlstate if phrase in msg else f"{exc.sqlstate} (another refusal: {msg})" + return "no error" + + +def _names(dsn, sql, phrase): + """-> "1" when sql's refusal message carries phrase, "0" when it does not, as the + shell's grep -c over errmsg_of; query_error() when the server could not be asked.""" + _await_up(dsn) + try: + with psycopg.connect(dsn, autocommit=True, connect_timeout=10) as conn: + conn.execute(sql).fetchall() + except psycopg.Error as exc: + if not exc.sqlstate: + return query_error(_message(exc)) + return "1" if phrase in (exc.diag.message_primary or "") else "0" + return "0" + + +def _crashes(cluster): + try: + return (cluster.datadir / "server.log").read_text(errors="replace").count("terminated by signal") + except OSError as exc: + return query_error(f"crash count unreadable: {exc}") + + +def _survived(dsn, cluster, before): + up, now = _value(dsn, "SELECT 1"), _crashes(cluster) + if not isinstance(before, int) or not isinstance(now, int): + return query_error(f"crash count unreadable: {before} / {now}") + return "1" if up == 1 and now == before else f"no ([{up}], {now - before} backend crashes)" + + +@pytest.fixture(scope="module") +def dv(pgc_cluster, server_dir): + if not (WHD / "db" / "t" / "metadata" / "dvapply.metadata.json").is_file(): + yield None + return + dest = server_dir / "dv" + shutil.copytree(WHD / "db", dest / "db") + for p in [dest, *dest.rglob("*")]: + p.chmod(0o777 if p.is_dir() else 0o666) + mdir = dest / "db" / "t" / "metadata" + oracle = json.loads((WHD / "expected_deletes.json").read_text())["dv_surviving"] + + def scan(name): + return f"SELECT * FROM pgcolumnar.iceberg_scan('{mdir / (name + '.metadata.json')}') {COLS}" + + admin = pgc_cluster.dsn() + with psycopg.connect(admin, autocommit=True) as a: + a.execute(f"DROP DATABASE IF EXISTS {DB} WITH (FORCE)") + a.execute(f"CREATE DATABASE {DB}") + try: + dsn = pgc_cluster.dsn(dbname=DB) + with psycopg.connect(dsn, autocommit=True) as c: + c.execute("CREATE EXTENSION IF NOT EXISTS pgcolumnar") + r = {"oracle": {k: ",".join(str(i) for i in v) for k, v in oracle.items()}} + # the shell's order, section by section, each with its own crash baseline + before = _crashes(pgc_cluster) + for name in APPLIED: + r[name] = _value(dsn, f"SELECT string_agg(id::text, ',' ORDER BY id) FROM " + f"pgcolumnar.iceberg_scan('{mdir / (name + '.metadata.json')}') {COLS}") + r["up after applications"] = _survived(dsn, pgc_cluster, before) + before = _crashes(pgc_cluster) + for name, phrase in REFUSED.items(): + r[name] = _refusal(dsn, scan(name), phrase) + if name == "dvflags": + r["dvflags names"] = _names(dsn, scan(name), "compressed") + if name == "dvv2": + r["dvv2 names"] = _names(dsn, scan(name), "format-version") + r["up after refusals"] = _survived(dsn, pgc_cluster, before) + before = _crashes(pgc_cluster) + for name, phrase in AUDIT.items(): + r[name] = _refusal(dsn, scan(name), phrase) + r["up after audit"] = _survived(dsn, pgc_cluster, before) + yield r + finally: + with psycopg.connect(admin, autocommit=True) as a: + a.execute(f"DROP DATABASE IF EXISTS {DB} WITH (FORCE)") + + +def _declined(dv, expect): + if dv is not None: + return False + expect.cannot_run("ABSENT_FIXTURE", "the deletion-vector fixtures are missing") + return True + + +def test_a_deletion_vector_at_an_equal_sequence_number_applies(dv, expect): + if dv is None: + expect.text("missing", "present", "deletion-vector fixtures are missing") + return + expect.require_module("json", name="python3 is needed") + expect.text(str(dv["dvapply"]), dv["oracle"]["dvapply"], + "a deletion vector at an equal sequence number applies (survivors == oracle)") + + +def test_a_deletion_vector_older_than_the_data_does_not_apply(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvnoapply"]), dv["oracle"]["dvnoapply"], "a deletion vector older than the data does not apply") + + +def test_an_applicable_dv_supersedes_position_delete_files(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvsupersede"]), dv["oracle"]["dvsupersede"], + "an applicable DV supersedes position-delete files for its data file") + + +def test_dv_supersede_is_per_data_file_not_global(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvother"]), dv["oracle"]["dvother"], "supersede is per data file, not global") + + +def test_a_dv_with_run_containers_and_a_high_bucket_decodes(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvwide"]), dv["oracle"]["dvwide"], "a DV with run containers and a high-bucket position decodes") + + +def test_a_run_optimized_dv_decodes(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvrun"]), dv["oracle"]["dvrun"], "a run-optimized DV (cookie 12347) decodes") + + +def test_a_bitset_container_dv_decodes(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvbitset"]), dv["oracle"]["dvbitset"], "a bitset-container DV decodes") + + +def test_two_dv_blobs_in_one_puffin_file_each_apply_to_their_own_file(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvtwo"]), dv["oracle"]["dvtwo"], + "two DV blobs in one Puffin file each apply to their own data file") + + +def test_the_backend_survives_the_dv_applications(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["up after applications"]), "1", "backend still up after the DV applications") + + +def test_two_dv_entries_for_one_data_file_are_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvdup"]), "XX001", "two DV entries for one data file are refused (XX001)") + + +def test_a_corrupted_dv_crc_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvbadcrc"]), "XX001", "a corrupted DV CRC is refused (XX001)") + + +def test_a_wrong_dv_blob_magic_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvbadmagic"]), "XX001", "a wrong DV blob magic is refused (XX001)") + + +def test_a_manifest_footer_offset_mismatch_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvoffmismatch"]), "XX001", "a manifest/footer offset mismatch is refused (XX001)") + + +def test_a_dv_entry_without_referenced_data_file_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvnoref"]), "XX001", "a DV entry without referenced_data_file is refused (XX001)") + + +def test_a_record_count_disagreeing_with_the_dv_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvbadcount"]), "XX001", + "a record_count that disagrees with the DV cardinality is refused (XX001)") + + +def test_a_dv_blob_declaring_a_codec_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvcompressed"]), "XX001", "a DV blob declaring a compression codec is refused (XX001)") + + +def test_a_compressed_puffin_footer_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvflags"]), "0A000", "a compressed Puffin footer is refused (0A000)") + + +def test_the_footer_refusal_names_the_compressed_footer(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvflags names"]), "1", "...and the refusal names the compressed footer") + + +def test_a_dv_in_a_format_version_2_table_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvv2"]), "0A000", "a deletion vector in a format-version 2 table is refused (0A000)") + + +def test_the_v2_refusal_names_the_format_version(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvv2 names"]), "1", "...and the refusal names the format version") + + +def test_a_puffin_path_outside_the_table_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvescape"]), "22023", "a Puffin path outside the table location is refused (22023)") + + +def test_the_backend_survives_the_dv_refusals(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["up after refusals"]), "1", "backend still up after the DV refusals") + + +def test_a_dv_whose_offset_plus_size_overflows_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvbigoff"]), "XX001", "a DV whose offset+size overflows int64 is refused, not read (XX001)") + + +def test_a_run_container_overflowing_its_range_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvrunoverflow"]), "XX001", "a run container overflowing its 16-bit range is refused (XX001)") + + +def test_an_aliased_duplicate_dv_is_refused(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["dvdupscheme"]), "XX001", "an aliased duplicate DV (scheme vs no scheme) is refused (XX001)") + + +def test_the_backend_survives_the_audit_arms(dv, expect): + if _declined(dv, expect): + return + expect.text(str(dv["up after audit"]), "1", "backend still up after the audit arms") diff --git a/test/pytest/unported_suites.txt b/test/pytest/unported_suites.txt index 9a22f4629..807558ce5 100644 --- a/test/pytest/unported_suites.txt +++ b/test/pytest/unported_suites.txt @@ -20,7 +20,6 @@ fuzz_parquet harness_selftest hilbert_curve iceberg_deletes -iceberg_dv iceberg_rest iceberg_rest_scan iceberg_rest_server From 119908eb53b0ee3f5b36d5310679b1870ff4c791 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 21:43:25 +0000 Subject: [PATCH 02/15] test: logical_decoding_source gains its pytest twin, and a dead second slot no longer passes (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pytest twin of test/logical_decoding_source.sh (#754), one check per test, every check by the shell's name, on a private cluster at wal_level=logical. The shell's second slot (pgc_754b) had no heap control. Created after the 5000-row insert, it decoded nothing, and both "and it too emits no decodable change of its own" and "100x the rows does not give 100x the internal records" (0 < anything) passed: main 11/11. A heap table now writes through that slot in the same window and must decode its 5000 inserts ("CONTROL: the second slot decodes the heap table's 5000 inserts"), and the ratio check refuses a slot that carried no internal records. Under that harness mutation both are red in both twins (SAME 2). The pgc_skip message for a missing test_decoding.so was built from pg_config --pkglibdir, so the grader could not read it; it is now a literal, and the twin records the same name (with the plugin moved away on PG17: shell 1 FAIL, twin the same 1 FAIL and 12 declined). No source mutation reddens a check: the claim is that pgcolumnar writes nothing logical decoding can read. A default chunk group of 100 rows still keeps the internal records following groups (114 for 50 groups, under the 10x bound), so it dates nothing. 12/12 shell and 13/13 twin on each of 15-19. Census 3885 -> 3897 never, 2311 dated, 6208 rows (the skip name has no ledger row: it is not a check); suites_not_covered 101 -> 100; cluster_tests 2779 -> 2792; TESTS.md §269. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 12 ++ test/check_ledger_budget.txt | 24 ++- test/logical_decoding_source.sh | 15 +- test/pytest/TESTS.md | 25 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_logical_decoding_source.py | 189 ++++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 265 insertions(+), 6 deletions(-) create mode 100644 test/pytest/test_logical_decoding_source.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 62f4a74d3..66ae61070 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2902,6 +2902,18 @@ local_open_race_free local_open_race_free the partition coordinator never stats local_open_race_free local_open_race_free the partition coordinator opens O_NONBLOCK (a FIFO cannot block its open) 15;16;17;18;19 2026-10-21 the partition coordinator opens without O_NONBLOCK local_open_race_free local_open_race_free the race-free opener exists 15;16;17;18;19 never - local_open_race_free local_open_race_free the racy stat-before-open helper is gone 15;16;17;18;19 2026-10-21 the racy PgColumnarRejectNonRegularFile helper comes back +logical_decoding_source logical_decoding_source 100x the rows does not give 100x the internal records 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source CONTROL: the heap table's 50 inserts ARE decoded 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source CONTROL: the second slot decodes the heap table's 5000 inserts 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source a columnar table emits no decodable change 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source although the rows are there 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source and it too emits no decodable change of its own 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source premise: both tables hold the same 50 rows 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source premise: the 100x table really holds 100x the rows 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source premise: the server is running at wal_level=logical 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source premise: the slot decoded something at all 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source premise: the slot exists and is logical 15;16;17;18;19 never - +logical_decoding_source logical_decoding_source the slot carries pgcolumnar's internal catalog instead 15;16;17;18;19 never - maintenance_crash maintenance_crash after a crash with compact committed, the index reads the rows the scan reads 15;16;17;18;19 2026-10-16 pgcolumnar_index_fetch_tuple finds no row (columnar_tableam.c) maintenance_crash maintenance_crash after a crash with compact committed, the table is as compact left it, with the same rows 15;16;17;18;19 never - maintenance_crash maintenance_crash after a crash with compact uncommitted, the index reads the rows the scan reads 15;16;17;18;19 2026-10-16 pgcolumnar_index_fetch_tuple finds no row (columnar_tableam.c) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index da7155f46..427c1a912 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 101 +suites_not_covered 100 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5457,4 +5457,24 @@ suites_not_covered 101 # never 3884 + 1 = 3885 # dated 2286 + 25 = 2311 # 3885 + 2311 == 6196 -checks_never_observed_red 3885 +# +# #1341, logical_decoding_source seeded, with its twin (OffgridwithJD): 3885 -> 3897, +# and suites_not_covered 101 -> 100. The suite had no rows. It enters with its 12 +# checks, all green on each of 15-19 in shell and twin alike. The missing-plugin name +# pgc_skip records is now a literal both twins share, but it is a skip, not a check, so +# it has no row: CI has test_decoding and never emits it. After one fix. +# The second slot had no heap control: created after the 5000-row insert, it decoded +# nothing, and "it too emits no decodable change" and "100x the rows does not give 100x +# the internal records" (0 < anything) both passed, 11/11 on main. A heap table now writes +# through that slot in the same window and must decode its 5000 inserts, and the ratio +# check refuses a slot that carried no internal records. Under that harness mutation both +# are red in both twins. No source mutation reddens a check: the claim is that pgcolumnar +# writes nothing logical decoding can read, and a default chunk group of 100 rows still +# keeps the internal records following groups (114 for 50 groups, under the 10x bound). +# 0 are dated. 12 arrive never. +# +# ledger rows 6196 + 12 = 6208 +# never 3885 + 12 = 3897 +# dated 2311 + 0 = 2311 +# 3897 + 2311 == 6208 +checks_never_observed_red 3897 diff --git a/test/logical_decoding_source.sh b/test/logical_decoding_source.sh index c17b4b1df..8e406bf7d 100755 --- a/test/logical_decoding_source.sh +++ b/test/logical_decoding_source.sh @@ -35,7 +35,8 @@ pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" # test_decoding is stock contrib, but it is not built by default in a source # install. A missing dependency is an environment defect rather than a pass. if [ ! -f "$("$PGC_PG_CONFIG" --pkglibdir)/test_decoding.so" ]; then - pgc_skip "test_decoding" "test_decoding.so is not in $("$PGC_PG_CONFIG" --pkglibdir); build contrib/test_decoding" + # The message is the check's name, so it is a literal: the twin records the same one. + pgc_skip "test_decoding" "test_decoding.so is not installed; build contrib/test_decoding" fi check "premise: the server is running at wal_level=logical" \ @@ -100,20 +101,30 @@ check "the slot carries pgcolumnar's internal catalog instead" \ # it useless as a change stream: it does not carry the data and it does not even # carry a row count. Asserted by writing 100x the rows into a second columnar # table and showing the record count does not follow. +# +# THE SECOND SLOT GETS ITS OWN HEAP CONTROL. Without one, a slot that decoded +# nothing passed both checks below: "no decodable change" reads 0 and "not 100x +# the records" reads 0 < anything. Measured: with pgc_754b created after the +# insert, the slot carried 0 records and every check passed. psql_run "CREATE TABLE col_big (id int PRIMARY KEY, v text) USING pgcolumnar;" +psql_run "CREATE TABLE heap_big (id int PRIMARY KEY, v text) USING heap;" psql_run "SELECT pg_create_logical_replication_slot('pgc_754b', 'test_decoding');" psql_run "INSERT INTO col_big SELECT g, 'c' || g FROM generate_series(1, 5000) g;" +psql_run "INSERT INTO heap_big SELECT g, 'h' || g FROM generate_series(1, 5000) g;" psql_run "CREATE TABLE decoded_big AS SELECT data FROM pg_logical_slot_get_changes('pgc_754b', NULL, NULL);" SMALL="$(d 'table pgcolumnar.')" BIG="$(q "SELECT count(*) FROM decoded_big WHERE data LIKE '%table pgcolumnar.%';")" check "premise: the 100x table really holds 100x the rows" \ "$(q 'SELECT count(*) FROM col_big;')" "5000" +check_num "CONTROL: the second slot decodes the heap table's 5000 inserts" \ + "$(q "SELECT count(*) FROM decoded_big WHERE data LIKE '%table public.heap_big: INSERT%';")" "5000" check_num "and it too emits no decodable change of its own" \ "$(q "SELECT count(*) FROM decoded_big WHERE data LIKE '%table public.col_big:%';")" "0" echo "-- internal pgcolumnar records: $SMALL for 50 rows, $BIG for 5000" check "100x the rows does not give 100x the internal records" \ - "$([ "$BIG" -lt $(( SMALL * 10 )) ] && echo yes || echo "no ($SMALL then $BIG)")" "yes" + "$([ "${BIG:-0}" -gt 0 ] 2>/dev/null && [ "$BIG" -lt $(( SMALL * 10 )) ] 2>/dev/null && echo yes || + echo "no ($SMALL then $BIG; the second slot must carry some)")" "yes" psql_run "SELECT pg_drop_replication_slot('pgc_754');" psql_run "SELECT pg_drop_replication_slot('pgc_754b');" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 8121f453c..74bd6ff82 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -314,6 +314,7 @@ behaviour, the source of that number is named. - [266. test_iceberg_name_mapping.py: an Iceberg data file with no field ids binds by name mapping](#266-test_iceberg_name_mappingpy-an-iceberg-data-file-with-no-field-ids-binds-by-name-mapping) - [267. test_iceberg_objstore.py: Iceberg tables read over object storage](#267-test_iceberg_objstorepy-iceberg-tables-read-over-object-storage) - [268. test_iceberg_dv.py: Iceberg v3 deletion vectors apply, supersede, and refuse what they must](#268-test_iceberg_dvpy-iceberg-v3-deletion-vectors-apply-supersede-and-refuse-what-they-must) +- [269. test_logical_decoding_source.py: a columnar table is not a logical decoding source](#269-test_logical_decoding_sourcepy-a-columnar-table-is-not-a-logical-decoding-source) ## 1. How to read a test in here @@ -10434,3 +10435,27 @@ Every test makes one check under the shell's name. A refusal reads as its SQLSTA | `test_a_run_container_overflowing_its_range_is_refused` | a run container overflowing its 16-bit range is refused (XX001) | | `test_an_aliased_duplicate_dv_is_refused` | an aliased duplicate DV (scheme vs no scheme) is refused (XX001) | | `test_the_backend_survives_the_audit_arms` | backend still up after the audit arms | + +## 269. test_logical_decoding_source.py: a columnar table is not a logical decoding source + +The twin of `test/logical_decoding_source.sh` (#754). Through test_decoding slots on a private cluster at `wal_level=logical`: a heap table's inserts decode and a columnar table's do not, although its rows are there; the slot carries pgcolumnar's own catalog writes instead; and those follow groups, not rows (5000 rows give fewer than 10x the records of 50). Each slot has a heap control written in the same window, so a dead slot cannot pass as an exclusion. + +### Every test + +One check per test, each by the shell's name, every reading a value (a failed query is a `query_error()` that fails its own check) from a module fixture on a private cluster. Without the test_decoding library both harnesses fail one check, the shell's `pgc_skip` name, and the twin's other tests decline MISSING_DEPENDENCY. + +| test | what it holds | +| --- | --- | +| `test_lds_test_decoding_is_installed` | test_decoding.so is not installed; build contrib/test_decoding | +| `test_lds_server_runs_at_wal_level_logical` | premise: the server is running at wal_level=logical | +| `test_lds_slot_exists_and_is_logical` | premise: the slot exists and is logical | +| `test_lds_both_tables_hold_the_same_rows` | premise: both tables hold the same 50 rows | +| `test_lds_slot_decoded_something` | premise: the slot decoded something at all | +| `test_lds_heap_control_is_decoded` | CONTROL: the heap table's 50 inserts ARE decoded | +| `test_lds_columnar_emits_no_change` | a columnar table emits no decodable change | +| `test_lds_columnar_rows_are_there` | although the rows are there | +| `test_lds_slot_carries_internal_catalog` | the slot carries pgcolumnar's internal catalog instead | +| `test_lds_big_table_holds_its_rows` | premise: the 100x table really holds 100x the rows | +| `test_lds_second_slot_heap_control_is_decoded` | CONTROL: the second slot decodes the heap table's 5000 inserts | +| `test_lds_big_columnar_emits_no_change` | and it too emits no decodable change of its own | +| `test_lds_internal_records_do_not_follow_rows` | 100x the rows does not give 100x the internal records | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 07ed43164..630bd3f16 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -908,6 +908,8 @@ guard_tests 476 # 16 checks in 16 tests, by collection over the cluster files. # cluster_tests 2753 -> 2779 for #1341's test_iceberg_dv.py, the twin of iceberg_dv.sh: # 26 checks in 26 tests, by collection over the cluster files. +# cluster_tests 2779 -> 2792 for #1341's test_logical_decoding_source.py, the twin of logical_decoding_source.sh: +# 13 checks in 13 tests, by collection over the cluster files. # cluster_tests 1264 -> 1287 for #1341's test_vector_agg_rescan_memory.py, the twin of vector_agg_rescan_memory.sh: # 23 tests, one per check, by collection over the cluster files. # cluster_tests 1287 -> 1329 for #1341's test_analyze_stats.py, the twin of analyze_stats.sh: @@ -1039,4 +1041,4 @@ guard_tests 476 # cluster_tests 2670 -> 2677 for #1637's test_catalog_filter_reads.py, the twin of catalog_filter_reads.sh: # 7 tests, one per check, by collection over the cluster files. # cluster_tests 2748 -> 2753 for #1706's five timestamp-flag tests in test_native_parquet_schema.py, by collection. -cluster_tests 2779 +cluster_tests 2792 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 8ae5ab97e..c9fcd7ae8 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -140,6 +140,7 @@ "doc_parallel_premise", "encode_effort", "local_open_race_free", + "logical_decoding_source", "sorted_mark_rename", "tablesample", "ungrouped_vector_agg", diff --git a/test/pytest/test_logical_decoding_source.py b/test/pytest/test_logical_decoding_source.py new file mode 100644 index 000000000..d8e6a09db --- /dev/null +++ b/test/pytest/test_logical_decoding_source.py @@ -0,0 +1,189 @@ +"""pgColumnar is not a logical decoding source (#754). The pytest twin of +`test/logical_decoding_source.sh`, every check by the shell's name. + +Logical decoding reads the WAL records of heap tuples. Columnar data reaches WAL as +full-page images, which carry no tuple structure, so no change to a columnar table is +emitted. A slot on a database holding columnar tables is not silent either: it carries +pgcolumnar's own catalog writes, which follow groups and not rows. + +"Zero changes decoded" is also what a dead slot looks like, so each slot gets a heap +CONTROL written in the same window and asserted to decode. The second slot had none, +and with it created after the insert it carried nothing and every check passed. + +A PRIVATE CLUSTER at wal_level=logical: the level cannot be raised without a restart, +and the shared cluster runs at the default. +""" +import pathlib +import shutil +import tempfile + +import psycopg +import pytest + +from pgc_vacuity import query_error + +SLOT, SLOT_B = "pgc_754", "pgc_754b" + + +def _q(dsn, sql): + """-> the first column of the first row as text, or query_error() carrying the failure.""" + try: + with psycopg.connect(dsn, autocommit=True) as c: + row = c.execute(sql).fetchone() + return "" if row is None or row[0] is None else str(row[0]) + except psycopg.Error as exc: + return query_error(str(exc).splitlines()[0] if str(exc) else type(exc).__name__) + + +def _run(dsn, sql): + """-> None, or query_error() when the statement failed (the next reading then fails by name).""" + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(sql) + return None + except psycopg.Error as exc: + return query_error(str(exc).splitlines()[0] if str(exc) else type(exc).__name__) + + +def _int(v): + return int(v) if isinstance(v, str) and v.isdigit() else v + + +@pytest.fixture(scope="module") +def lds(pgc_cluster): + from pgc_cluster import Cluster, pick_port + + if not (pathlib.Path(pgc_cluster.libdir) / "test_decoding.so").is_file(): + yield {"missing": f"test_decoding.so is not in {pgc_cluster.libdir}"} + return + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-lds-")) + work.chmod(0o777) + datadir = work / "data" + datadir.mkdir() + cluster = Cluster(pgc_cluster.pg_config, "lds", datadir, pick_port(26)) + try: + cluster.initdb() + with open(datadir / "postgresql.conf", "a") as fh: + fh.write("\nwal_level=logical\nmax_replication_slots=8\nmax_wal_senders=8\n") + cluster.start() + dsn = cluster.dsn() + r = {} + _run(dsn, "CREATE EXTENSION IF NOT EXISTS pgcolumnar") + r["wal_level"] = _q(dsn, "SHOW wal_level") + _run(dsn, "CREATE TABLE heap_t (id int PRIMARY KEY, v text) USING heap") + _run(dsn, "CREATE TABLE col_t (id int PRIMARY KEY, v text) USING pgcolumnar") + # The slot is created before any row is written: a slot only carries later changes. + _run(dsn, f"SELECT pg_create_logical_replication_slot('{SLOT}', 'test_decoding')") + r["slot type"] = _q(dsn, f"SELECT slot_type FROM pg_replication_slots WHERE slot_name = '{SLOT}'") + _run(dsn, "INSERT INTO heap_t SELECT g, 'h' || g FROM generate_series(1, 50) g") + _run(dsn, "INSERT INTO col_t SELECT g, 'c' || g FROM generate_series(1, 50) g") + r["heap rows"] = _int(_q(dsn, "SELECT count(*) FROM heap_t")) + r["col rows"] = _int(_q(dsn, "SELECT count(*) FROM col_t")) + # Drained once, so every count reads the same snapshot of the slot. + _run(dsn, f"CREATE TABLE decoded AS SELECT data FROM pg_logical_slot_get_changes('{SLOT}', NULL, NULL)") + + def d(table, pattern): + return _int(_q(dsn, f"SELECT count(*) FROM {table} WHERE data LIKE '%{pattern}%'")) + + r["decoded"] = _int(_q(dsn, "SELECT count(*) FROM decoded")) + r["heap decoded"] = d("decoded", "table public.heap_t: INSERT") + r["col decoded"] = d("decoded", "table public.col_t:") + r["internal small"] = d("decoded", "table pgcolumnar.") + # The second slot, with its own heap control in the same window. + _run(dsn, "CREATE TABLE col_big (id int PRIMARY KEY, v text) USING pgcolumnar") + _run(dsn, "CREATE TABLE heap_big (id int PRIMARY KEY, v text) USING heap") + _run(dsn, f"SELECT pg_create_logical_replication_slot('{SLOT_B}', 'test_decoding')") + _run(dsn, "INSERT INTO col_big SELECT g, 'c' || g FROM generate_series(1, 5000) g") + _run(dsn, "INSERT INTO heap_big SELECT g, 'h' || g FROM generate_series(1, 5000) g") + _run(dsn, f"CREATE TABLE decoded_big AS SELECT data FROM pg_logical_slot_get_changes('{SLOT_B}', NULL, NULL)") + r["big rows"] = _q(dsn, "SELECT count(*) FROM col_big") + r["heap big decoded"] = d("decoded_big", "table public.heap_big: INSERT") + r["col big decoded"] = d("decoded_big", "table public.col_big:") + r["internal big"] = d("decoded_big", "table pgcolumnar.") + _run(dsn, f"SELECT pg_drop_replication_slot('{SLOT}')") + _run(dsn, f"SELECT pg_drop_replication_slot('{SLOT_B}')") + yield r + finally: + cluster.stop() + shutil.rmtree(work, ignore_errors=True) + + +def test_lds_test_decoding_is_installed(lds, expect): + # The shell's pgc_skip records a FAIL under this name when the plugin is missing, and + # nothing when it is there. Here the same name passes when it is there. + expect.text("missing" if "missing" in lds else "installed", "installed", + "test_decoding.so is not installed; build contrib/test_decoding") + + +def _declined(lds, expect): + if "missing" in lds: + expect.cannot_run("MISSING_DEPENDENCY", lds["missing"]) + return True + return False + + +def test_lds_server_runs_at_wal_level_logical(lds, expect): + if not _declined(lds, expect): + expect.text(lds["wal_level"], "logical", "premise: the server is running at wal_level=logical") + + +def test_lds_slot_exists_and_is_logical(lds, expect): + if not _declined(lds, expect): + expect.text(lds["slot type"], "logical", "premise: the slot exists and is logical") + + +def test_lds_both_tables_hold_the_same_rows(lds, expect): + if not _declined(lds, expect): + expect.num(lds["heap rows"], lds["col rows"], "premise: both tables hold the same 50 rows") + + +def test_lds_slot_decoded_something(lds, expect): + if not _declined(lds, expect): + n = lds["decoded"] + expect.text("yes" if isinstance(n, int) and n > 0 else f"no ({n})", "yes", + "premise: the slot decoded something at all") + + +def test_lds_heap_control_is_decoded(lds, expect): + if not _declined(lds, expect): + expect.num(lds["heap decoded"], 50, "CONTROL: the heap table's 50 inserts ARE decoded") + + +def test_lds_columnar_emits_no_change(lds, expect): + if not _declined(lds, expect): + expect.num(lds["col decoded"], 0, "a columnar table emits no decodable change") + + +def test_lds_columnar_rows_are_there(lds, expect): + if not _declined(lds, expect): + expect.num(lds["col rows"], 50, "although the rows are there") + + +def test_lds_slot_carries_internal_catalog(lds, expect): + if not _declined(lds, expect): + n = lds["internal small"] + expect.text("yes" if isinstance(n, int) and n > 0 else f"no ({n})", "yes", + "the slot carries pgcolumnar's internal catalog instead") + + +def test_lds_big_table_holds_its_rows(lds, expect): + if not _declined(lds, expect): + expect.text(lds["big rows"], "5000", "premise: the 100x table really holds 100x the rows") + + +def test_lds_second_slot_heap_control_is_decoded(lds, expect): + if not _declined(lds, expect): + expect.num(lds["heap big decoded"], 5000, "CONTROL: the second slot decodes the heap table's 5000 inserts") + + +def test_lds_big_columnar_emits_no_change(lds, expect): + if not _declined(lds, expect): + expect.num(lds["col big decoded"], 0, "and it too emits no decodable change of its own") + + +def test_lds_internal_records_do_not_follow_rows(lds, expect): + if not _declined(lds, expect): + small, big = lds["internal small"], lds["internal big"] + ok = isinstance(small, int) and isinstance(big, int) and 0 < big < small * 10 + expect.text("yes" if ok else f"no ({small} then {big}; the second slot must carry some)", "yes", + "100x the rows does not give 100x the internal records") diff --git a/test/pytest/unported_suites.txt b/test/pytest/unported_suites.txt index 807558ce5..649c15d89 100644 --- a/test/pytest/unported_suites.txt +++ b/test/pytest/unported_suites.txt @@ -25,7 +25,6 @@ iceberg_rest_scan iceberg_rest_server iceberg_rest_vended logical_decoding_cdc_recipe -logical_decoding_source logical_subscriber native_agg_deletes native_cancel From 61f7ae508ba69195b3217fa8506707ecdbce8d36 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 21:52:29 +0000 Subject: [PATCH 03/15] test: iceberg_deletes gains its pytest twin, and a crashed backend no longer passes as up (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The twin of test/iceberg_deletes.sh, one test per check under the shell's name. Fixed in both harnesses, measured red-first against main's shell on PG17: - The six "backend still up" checks read SELECT 1, which answers again after crash recovery. With abort() in place of the null-pos refusal, or the partition spec's fields guard removed (a NULL deref, signal 11), main passed the survival check once recovery had finished. They now also read the server log's count of backends killed by a signal, and every reading waits (bounded) for the server, so a crash reddens its own arm and its section's survival check rather than the two arms after it. - Every refusal now also reads its own guard's message: XX001 and 22023 are each raised by several guards here. The oracle comparisons use check_text, so an empty side is not agreement. - "the data file has 5 rows before deletes" reads pgcolumnar's own read_parquet, so it is not a premise (#1339). Twenty-three mutations each redden the same names in shell and twin on PG17; head is 49/49 in both on each of 15-19. The equal-sequence equality check is dated only with both layered guards relaxed: the per-file strict < alone passed, because the pre-skip at the oldest data file's sequence number drops that delete first. Census 3897 -> 3903 never, 2311 -> 2354 dated, 6257 rows; suites_not_covered 100 -> 99; cluster_tests 2792 -> 2841; TESTS.md §270. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 49 +++ test/check_ledger_budget.txt | 25 +- test/iceberg_deletes.sh | 174 ++++++--- test/pytest/TESTS.md | 61 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_iceberg_deletes.py | 561 ++++++++++++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 826 insertions(+), 50 deletions(-) create mode 100644 test/pytest/test_iceberg_deletes.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 66ae61070..a27f05d1f 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2549,6 +2549,55 @@ iceberg_catalog iceberg_catalog malformed JSON is rejected (22P02), backend surv iceberg_catalog iceberg_catalog resolves the named current snapshot, not snapshots[0] 15;16;17;18;19 2026-10-21 the resolver returns the first snapshot rather than the one named current iceberg_catalog iceberg_catalog the decoy snapshot is not returned 15;16;17;18;19 2026-10-21 the resolver returns the first snapshot rather than the one named current iceberg_catalog iceberg_catalog the first snapshot has no parent (NULL) 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes ...and the boundary refusal names the escape 15;16;17;18;19 2026-10-21 columnar_iceberg.c: ice_rebase does not hold a recorded path under the recorded table location +iceberg_deletes iceberg_deletes ...and the negative-pos refusal names the negative pos 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a negative position-delete pos is not refused +iceberg_deletes iceberg_deletes ...and the null-pos refusal names pos 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position-delete row with a null pos is not refused (the n1 || n2 guard skipped) +iceberg_deletes iceberg_deletes ...and the refusal names the type 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a timestamp equality-delete column is mapped to text instead of refused +iceberg_deletes iceberg_deletes a cross-spec partition-scoped delete applies to nothing (5 rows) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition-scoped equality delete applies across partition spec ids +iceberg_deletes iceberg_deletes a data file lacking partition_spec_id is refused under a partitioned delete (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a data file without partition_spec_id is not refused under a partition-scoped delete +iceberg_deletes iceberg_deletes a delete older than the data (lower seq) does not apply (5 rows) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies whatever its sequence number +iceberg_deletes iceberg_deletes a delete scoped to an empty partition deletes nothing (5 rows) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition-scoped equality delete ignores the partition value +iceberg_deletes iceberg_deletes a dotdot-escaping delete path is refused (22023) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: ice_rebase does not re-check containment after canonicalize_path +iceberg_deletes iceberg_deletes a manifest list without partition_spec_id is refused for equality deletes (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an equality delete whose manifest list carries no partition_spec_id is not refused +iceberg_deletes iceberg_deletes a never-applicable equality delete is skipped, not validated (5 rows) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an equality delete at the oldest data file's sequence number is not skipped before validation +iceberg_deletes iceberg_deletes a never-applicable equality delete without equality_ids is skipped (5 rows) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an equality delete at the oldest data file's sequence number is not skipped before validation +iceberg_deletes iceberg_deletes a null delete value matches only null data values 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a null equality-delete value matches any data value (comparing the null Datum kills the backend) +iceberg_deletes iceberg_deletes a partition spec without its fields key is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition spec without a fields array is not refused (a NULL deref kills the backend) +iceberg_deletes iceberg_deletes a partition-scoped delete with an uncomparable partition value is refused (0A000) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition-scoped equality delete with an uncomparable partition tuple is not refused +iceberg_deletes iceberg_deletes a partition-scoped equality delete applies only to its partition 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition-scoped equality delete ignores the partition value +iceberg_deletes iceberg_deletes a position delete naming a different data file deletes nothing (5 rows) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies to every data file, not only the one it names +iceberg_deletes iceberg_deletes a position delete of ordinal 0 deletes the first row (id 1 gone, 4 remain) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes a position delete with a negative pos is refused (XX001, not cast inert) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a negative position-delete pos is not refused +iceberg_deletes iceberg_deletes a position delete with a null pos is refused (XX001, not silently dropped) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position-delete row with a null pos is not refused (the n1 || n2 guard skipped) +iceberg_deletes iceberg_deletes a position delete with an inherited (null-entry) sequence number applies 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes a v1 manifest without a sequence_number column reads (default 0, 5 rows) 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes an EXISTING entry with a null sequence number is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an EXISTING (status 0) manifest entry with a null sequence number is not refused +iceberg_deletes iceberg_deletes an equality column missing from the data file errors (22023) 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes an equality delete at an equal sequence number does not apply 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an equality delete applies at an equal sequence number (both the pre-skip and the per-file strict < relaxed to <=) +iceberg_deletes iceberg_deletes an equality delete on an unsupported column type is refused (0A000) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a timestamp equality-delete column is mapped to text instead of refused +iceberg_deletes iceberg_deletes an equality delete without equality_ids is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an equality delete without equality_ids is not refused +iceberg_deletes iceberg_deletes an equality-delete path outside the table location is refused (22023) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: ice_rebase does not hold a recorded path under the recorded table location +iceberg_deletes iceberg_deletes an equality_ids value beyond int32 refuses the manifest (XX001) 15;16;17;18;19 2026-10-21 columnar_avro.c: an equality_ids value beyond int32 is not refused (truncated) +iceberg_deletes iceberg_deletes backend still up after the audit arms 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition spec without a fields array is not refused (a NULL deref kills the backend) +iceberg_deletes iceberg_deletes backend still up after the bad-sequence refusal 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes backend still up after the equality applications 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a null equality-delete value matches any data value (comparing the null Datum kills the backend) +iceberg_deletes iceberg_deletes backend still up after the equality refusals 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes backend still up after the negative-pos refusal 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes backend still up after the null-pos refusal 15;16;17;18;19 2026-10-21 columnar_iceberg.c: abort() in place of the null-pos refusal +iceberg_deletes iceberg_deletes columns beyond equality_ids are ignored for matching 15;16;17;18;19 2026-10-21 columnar_iceberg.c: no equality-delete row ever matches +iceberg_deletes iceberg_deletes equality delete drops matching rows (survivors == oracle) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: no equality-delete row ever matches +iceberg_deletes iceberg_deletes equality deletes apply when the delete column is not projected 15;16;17;18;19 2026-10-21 columnar_iceberg.c: no equality-delete row ever matches +iceberg_deletes iceberg_deletes exactly the two deleted rows were removed (5 - 2 = 3) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes multi-column equality matches all columns (AND, not OR) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an equality delete compares only its first equality column (multi-column match becomes first-column) +iceberg_deletes iceberg_deletes multiple equality-delete files apply their union 15;16;17;18;19 2026-10-21 columnar_iceberg.c: no equality-delete row ever matches +iceberg_deletes iceberg_deletes position and equality deletes combine 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes position deletes apply under column projection (id only -> 1,3,5) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes position deletes drop the listed rows (surviving == oracle) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes the data file has 5 rows before deletes 15;16;17;18;19 never - +iceberg_deletes iceberg_deletes the deleted ids (2, 4) are gone 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies only when strictly newer than the data (> where the spec says >=) +iceberg_deletes iceberg_deletes the null-region row is gone, the non-null rows remain 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a null equality-delete value matches any data value (comparing the null Datum kills the backend) +iceberg_deletes iceberg_deletes the us-partition rows survive an eu-scoped delete 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a partition-scoped equality delete ignores the partition value +iceberg_deletes iceberg_deletes the would-be-deleted ids survive when the delete is older than the data 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a position delete applies whatever its sequence number iceberg_dv iceberg_dv ...and the refusal names the compressed footer 15;16;17;18;19 2026-10-21 columnar_puffin.c: a compressed Puffin footer (flags bit 0) is not refused iceberg_dv iceberg_dv ...and the refusal names the format version 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a deletion vector in a format-version 2 table is not refused iceberg_dv iceberg_dv a DV blob declaring a compression codec is refused (XX001) 15;16;17;18;19 2026-10-21 columnar_puffin.c: a deletion-vector blob may declare a compression codec diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 427c1a912..ab9b4075c 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 100 +suites_not_covered 99 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5477,4 +5477,25 @@ suites_not_covered 100 # never 3885 + 12 = 3897 # dated 2311 + 0 = 2311 # 3897 + 2311 == 6208 -checks_never_observed_red 3897 +# +# #1341, iceberg_deletes seeded, with its twin (OffgridwithJD): 3897 -> 3903, and +# suites_not_covered 100 -> 99. The suite had no rows. It enters with 49 checks from +# five-major runs (all green on each of 15-19, shell and twin alike), after one fix. +# The six "backend still up" checks read SELECT 1, which answers again after crash +# recovery: with abort() in place of the null-pos refusal, or the partition spec's +# fields guard removed (a segfault), main's shell passed the survival check once +# recovery had finished. They now also read the server log's count of killed +# backends, and every reading waits for the server first, so a crash reddens its own +# arm and its section's survival check, not the two arms after it. Every refusal now +# also reads its own guard's message (XX001 and 22023 are shared), and the oracle +# comparisons refuse an empty side. 43 are dated by twenty-three mutations, each reddening +# the same names in both twins. The equal-sequence equality check is dated only by +# relaxing both of its layered guards: the per-file strict < alone passed, because the +# pre-skip at the oldest data file's sequence number already drops that delete. +# 6 arrive never. +# +# ledger rows 6208 + 49 = 6257 +# never 3897 + 6 = 3903 +# dated 2311 + 43 = 2354 +# 3903 + 2354 == 6257 +checks_never_observed_red 3903 diff --git a/test/iceberg_deletes.sh b/test/iceberg_deletes.sh index b9e88a875..413959ea7 100755 --- a/test/iceberg_deletes.sh +++ b/test/iceberg_deletes.sh @@ -15,22 +15,73 @@ set -uo pipefail . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" -sqlstate_of() { - env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -qtA 2>&1 < the SQLSTATE when SQL is refused with PHRASE in its message; +# "SQLSTATE (another refusal: MESSAGE)" when a different guard refused it; "no +# error" when it ran; nothing when the server could not be asked. Each guard here +# has its own message, and several share a SQLSTATE (XX001 and 22023), so an arm +# counts only its own guard's refusal. +refusal() { + local out st msg + await_up + out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ + -d "$PGC_DB" -qtA 2>&1 <&1 < wait, at most 60 s, until the server answers. A refusal arm that kills a +# backend sends the server through crash recovery; without the wait, the next arms' +# readings fail on recovery and red as collateral, more or fewer of them by timing +# (with the partition spec's fields guard removed, the backend segfaults and two +# later arms went red). The crash itself is still caught: survived() counts it. +await_up() { + local i + for i in $(seq 1 120); do + [ "$(q 'SELECT 1')" = 1 ] && return 0 + sleep 0.5 + done +} + +# Backends killed by a signal, from the server log: a number, or nothing when the +# log can't be read. A fresh connection answers again after crash recovery, so +# "SELECT 1" alone passed with a backend killed by a refusal arm: with abort() in +# place of the null-pos refusal, or the fields guard removed (a segfault), and the +# check run once recovery had finished. The survival checks below also require +# that this count has not moved since their section began. +crashes() { + local _n + _n="$(pgc_pg "grep -c 'terminated by signal' '$PGC_LOGFILE'" 2>/dev/null)" + case "$_n" in '' | *[!0-9]*) ;; *) echo "$_n" ;; esac +} +survived() { # survived BEFORE -> "1" when the server answers and no backend died since + local _up _now + await_up + _up="$(q 'SELECT 1')"; _now="$(crashes)" + if ! [[ "$1" =~ ^[0-9]+$ && "$_now" =~ ^[0-9]+$ ]]; then echo "no (crash count unreadable)" + elif [ "$_up" = 1 ] && [ "$_now" = "$1" ]; then echo 1 + else echo "no ([$_up], $(( _now - $1 )) backend crashes)"; fi +} + FX="$(dirname "${BASH_SOURCE[0]}")/fixtures/iceberg" WHD="$FX/warehouse_del" [ -f "$WHD/db/t/metadata/apply.metadata.json" ] || pgc_skip fixture "delete fixture is missing" @@ -44,7 +95,7 @@ chmod -R u+rwX "$DEST" MDIR="$DEST/db/t/metadata" # the full data, ignoring deletes, is 5 rows -- the baseline the delete removes from -check "premise: the data file has 5 rows before deletes" \ +check "the data file has 5 rows before deletes" \ "$(q "SELECT count(*) FROM pgcolumnar.read_parquet('$DEST/db/t/data/data.parquet') AS t(id bigint, region text, amount int)")" "5" @@ -55,20 +106,24 @@ o = json.load(open(sys.argv[1])) print("\n".join("%d|%s|%d" % (r["id"], r["region"], r["amount"]) for r in o["surviving"])) PY )" -check "position deletes drop the listed rows (surviving == oracle)" \ +await_up +check_text "position deletes drop the listed rows (surviving == oracle)" \ "$(q "SELECT id || '|' || region || '|' || amount FROM pgcolumnar.iceberg_scan('$MDIR/apply.metadata.json') AS t(id bigint, region text, amount int) ORDER BY id")" \ "$ORACLE" +await_up check "the deleted ids (2, 4) are gone" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/apply.metadata.json') AS t(id bigint, region text, amount int) WHERE id IN (2,4)")" "0" +await_up check "exactly the two deleted rows were removed (5 - 2 = 3)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/apply.metadata.json') AS t(id bigint, region text, amount int)")" "3" # deletes are by row ordinal, independent of column projection: reading only one # column must still drop the same rows (the ordinal is the row index, not a # column value) +await_up check "position deletes apply under column projection (id only -> 1,3,5)" \ "$(q "SELECT string_agg(id::text, ',' ORDER BY id) FROM pgcolumnar.iceberg_scan('$MDIR/apply.metadata.json') AS t(id bigint)")" \ @@ -79,9 +134,11 @@ check "position deletes apply under column projection (id only -> 1,3,5)" \ # which the spec says applies (data_seq <= delete_seq) -- the boundary that a # strict > would wrongly exclude. This arm is the other side: a delete OLDER than # the data (seq 4 < data seq 5) must NOT apply, so all five rows survive. +await_up check "a delete older than the data (lower seq) does not apply (5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/noapply.metadata.json') AS t(id bigint, region text, amount int)")" "5" +await_up check "the would-be-deleted ids survive when the delete is older than the data" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/noapply.metadata.json') AS t(id bigint, region text, amount int) WHERE id IN (2,4)")" "2" @@ -90,6 +147,7 @@ check "the would-be-deleted ids survive when the delete is older than the data" # real writers record a null entry sequence number and inherit the manifest's; # this variant's delete entry is null, inheriting seq 5 from the manifest, so it # must still apply over the seq-5 data (3 rows survive), proving inheritance. +await_up check "a position delete with an inherited (null-entry) sequence number applies" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/inherit.metadata.json') AS t(id bigint, region text, amount int)")" "3" @@ -98,6 +156,7 @@ check "a position delete with an inherited (null-entry) sequence number applies" # the delete is applicable by sequence, but its rows target another data file, so # none of data.parquet's rows may be dropped (proves the path match excludes, not # just includes -- a basename-only or always-true match would wrongly delete). +await_up check "a position delete naming a different data file deletes nothing (5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/wrongpath.metadata.json') AS t(id bigint, region text, amount int)")" "5" @@ -106,24 +165,27 @@ check "a position delete naming a different data file deletes nothing (5 rows)" # On current main a null pos is silently dropped (the `!n1 && !n2` filter) and a # negative pos is cast to a huge uint64 that matches nothing: the scan returns # the phantom row(s) with NO error. The reader must instead refuse (XX001). +CRASHES_BEFORE="$(crashes)" check "a position delete with a null pos is refused (XX001, not silently dropped)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/nullpos.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/nullpos.metadata.json') + AS t(id bigint, region text, amount int)" "has a null pos")" "XX001" check "...and the null-pos refusal names pos" \ "$(errmsg_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/nullpos.metadata.json') AS t(id bigint, region text, amount int)" | grep -c "null pos")" "1" -check "backend still up after the null-pos refusal" "$(q 'SELECT 1')" "1" +check "backend still up after the null-pos refusal" "$(survived "$CRASHES_BEFORE")" "1" +CRASHES_BEFORE="$(crashes)" check "a position delete with a negative pos is refused (XX001, not cast inert)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/negpos.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/negpos.metadata.json') + AS t(id bigint, region text, amount int)" "has a negative pos")" "XX001" check "...and the negative-pos refusal names the negative pos" \ "$(errmsg_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/negpos.metadata.json') AS t(id bigint, region text, amount int)" | grep -c "negative pos")" "1" -check "backend still up after the negative-pos refusal" "$(q 'SELECT 1')" "1" +check "backend still up after the negative-pos refusal" "$(survived "$CRASHES_BEFORE")" "1" # RISK control: pos == 0 is a LEGITIMATE delete of the first row (id 1). It must # apply, not be rejected -- a `pos <= 0` guard would wrongly turn this red. +await_up check "a position delete of ordinal 0 deletes the first row (id 1 gone, 4 remain)" \ "$(q "SELECT string_agg(id::text, ',' ORDER BY id) FROM pgcolumnar.iceberg_scan('$MDIR/zeropos.metadata.json') @@ -144,50 +206,62 @@ eq_ids_of() { AS t(id bigint, region text, amount int)" } -check "equality delete drops matching rows (survivors == oracle)" \ +CRASHES_BEFORE="$(crashes)" +await_up +check_text "equality delete drops matching rows (survivors == oracle)" \ "$(eq_ids_of eqapply)" "$(eq_oracle eqapply)" # THE boundary that distinguishes the equality rule (strict <) from the position # rule (<=): the same delete file at a sequence number EQUAL to the data's must # delete NOTHING (an equality delete never touches same-commit data) -check "an equality delete at an equal sequence number does not apply" \ +await_up +check_text "an equality delete at an equal sequence number does not apply" \ "$(eq_ids_of eqboundary)" "$(eq_oracle eqboundary)" # multi-column equality is an AND: (3,'eu') matches no row (3 is 'us'), so only # (4,'us') deletes -check "multi-column equality matches all columns (AND, not OR)" \ +await_up +check_text "multi-column equality matches all columns (AND, not OR)" \ "$(eq_ids_of eqmulti)" "$(eq_oracle eqmulti)" # only the equality_ids columns define the match; the delete file's extra amount # column carries a wrong value that must be ignored -check "columns beyond equality_ids are ignored for matching" \ +await_up +check_text "columns beyond equality_ids are ignored for matching" \ "$(eq_ids_of eqextra)" "$(eq_oracle eqextra)" # null matches null (IS NULL semantics), and ONLY null: id 6 (region NULL) goes, # id 7 ('eu') stays -check "a null delete value matches only null data values" \ +await_up +check_text "a null delete value matches only null data values" \ "$(eq_ids_of eqnull)" "$(eq_oracle eqnull)" +await_up check "the null-region row is gone, the non-null rows remain" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/eqnull.metadata.json') AS t(id bigint, region text, amount int) WHERE region IS NULL")" "0" # two delete files with different equality_ids in one manifest: both apply -check "multiple equality-delete files apply their union" \ +await_up +check_text "multiple equality-delete files apply their union" \ "$(eq_ids_of eqtwo)" "$(eq_oracle eqtwo)" # position and equality deletes in one snapshot: the skip sets merge -check "position and equality deletes combine" \ +await_up +check_text "position and equality deletes combine" \ "$(eq_ids_of eqmixed)" "$(eq_oracle eqmixed)" # the delete columns come from the table schema, not the output list: a scan # projecting only amount must still drop the id-matched rows +await_up check "equality deletes apply when the delete column is not projected" \ "$(q "SELECT string_agg(amount::text, ',' ORDER BY amount) FROM pgcolumnar.iceberg_scan('$MDIR/eqapply.metadata.json') AS t(amount int)")" \ "10,30,50" -check "backend still up after the equality applications" "$(q 'SELECT 1')" "1" +check "backend still up after the equality applications" "$(survived "$CRASHES_BEFORE")" "1" # ---- 4b: equality refusal and corruption arms ------------------------------- # content=2 with null equality_ids is corrupt metadata (the spec requires it) +CRASHES_BEFORE="$(crashes)" check "an equality delete without equality_ids is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqnoids.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqnoids.metadata.json') + AS t(id bigint, region text, amount int)" "has no equality_ids")" "XX001" # the pre-4b 'equality' fixture also lacks equality_ids, but its delete sits at # the data's own sequence number, so under the strict-< rule it can never apply # and is skipped before validation: the data reads untouched +await_up check "a never-applicable equality delete without equality_ids is skipped (5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/equality.metadata.json') AS t(id bigint, region text, amount int)")" "5" @@ -209,13 +283,16 @@ o = json.load(open(sys.argv[1])) print(",".join(str(i) for i in o["part_surviving"][sys.argv[2]])) PY } -check "a partition-scoped equality delete applies only to its partition" \ +await_up +check_text "a partition-scoped equality delete applies only to its partition" \ "$(part_ids eqpart_apply)" "$(part_oracle eqpart_apply)" +await_up check "the us-partition rows survive an eu-scoped delete" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_apply.metadata.json') AS t(id bigint, region text, grp int) WHERE region = 'us'")" "3" # a delete scoped to a partition (region=zz) that no data file is in applies to # nothing -- a legitimate no-op, not an error and not a global delete +await_up check "a delete scoped to an empty partition deletes nothing (5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_nomatch.metadata.json') AS t(id bigint, region text, grp int)")" "5" @@ -223,23 +300,24 @@ check "a delete scoped to an empty partition deletes nothing (5 rows)" \ # partition-scoped delete, would silently under-delete; refuse it (the mirror # of the delete-side has_spec_id check) check "a data file lacking partition_spec_id is refused under a partitioned delete (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_datanospec.metadata.json') - AS t(id bigint, region text, grp int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_datanospec.metadata.json') + AS t(id bigint, region text, grp int)" "carries no partition_spec_id, which a partition-scoped")" "XX001" # a delete written under a partition spec no data file uses (partition # evolution): per the spec a partitioned delete never crosses spec ids, so it # applies to nothing -- a no-op, all rows survive, never a global over-delete +await_up check "a cross-spec partition-scoped delete applies to nothing (5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_crossspec.metadata.json') AS t(id bigint, region text, grp int)")" "5" # a partition tuple the reader cannot compare exactly (a double) is refused check "a partition-scoped delete with an uncomparable partition value is refused (0A000)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_incomparable.metadata.json') - AS t(id bigint, region text, grp int)")" "0A000" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqpart_incomparable.metadata.json') + AS t(id bigint, region text, grp int)" "has a partition tuple this reader cannot compare")" "0A000" # an equality column of a type with no supported mapping (timestamp) is refused # before any file is opened check "an equality delete on an unsupported column type is refused (0A000)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqtype.metadata.json') - AS t(id bigint, region text, amount int)")" "0A000" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqtype.metadata.json') + AS t(id bigint, region text, amount int)" "is not supported")" "0A000" check "...and the refusal names the type" \ "$(errmsg_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqtype.metadata.json') AS t(id bigint, region text, amount int)" | grep -c "type")" "1" @@ -247,65 +325,69 @@ check "...and the refusal names the type" \ # data file errors loudly (the reader's missing-field-id error), never silently # skips the delete check "an equality column missing from the data file errors (22023)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqmissing.metadata.json') - AS t(id bigint, region text, amount int)")" "22023" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqmissing.metadata.json') + AS t(id bigint, region text, amount int)" "no Parquet column has field id")" "22023" # a delete-file path outside the table root is stopped by the path boundary # (22023 from the boundary, BEFORE the file is opened; if the boundary were # gone, /etc/hostname would be read and fail differently) check "an equality-delete path outside the table location is refused (22023)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqescape.metadata.json') - AS t(id bigint, region text, amount int)")" "22023" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqescape.metadata.json') + AS t(id bigint, region text, amount int)" "is not under the table location")" "22023" check "...and the boundary refusal names the escape" \ "$(errmsg_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqescape.metadata.json') AS t(id bigint, region text, amount int)" | grep -c "table location")" "1" # a path under the table root that climbs out with ".." is refused by the # containment (canonicalize collapses it and the re-check fails) check "a dotdot-escaping delete path is refused (22023)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqdotdot.metadata.json') - AS t(id bigint, region text, amount int)")" "22023" -check "backend still up after the equality refusals" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqdotdot.metadata.json') + AS t(id bigint, region text, amount int)" "escapes the table location")" "22023" +check "backend still up after the equality refusals" "$(survived "$CRASHES_BEFORE")" "1" # ---- 4b audit arms (multi-agent adversarial audit findings) ----------------- # an equality_ids value beyond int32 must refuse the manifest (XX001); a silent # 32-bit truncation would alias field 2^32+2 onto the real field 2 and delete # rows the manifest never named +CRASHES_BEFORE="$(crashes)" check "an equality_ids value beyond int32 refuses the manifest (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqbigid.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqbigid.metadata.json') + AS t(id bigint, region text, amount int)" "corrupt Avro manifest entry")" "XX001" # a manifest list omitting partition_spec_id leaves an equality delete's scope # undecidable; defaulting it to 0 (unpartitioned) would defeat the # partition-scope guard, so the reader must refuse (XX001) check "a manifest list without partition_spec_id is refused for equality deletes (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqnospec.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqnospec.metadata.json') + AS t(id bigint, region text, amount int)" "the manifest list carries no partition_spec_id")" "XX001" # a partition-specs entry matching the delete's spec id but lacking the # required "fields" key: treating it as unpartitioned would silently globalize # the delete, so it is corrupt metadata (XX001) check "a partition spec without its fields key is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqnofields.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/eqnofields.metadata.json') + AS t(id bigint, region text, amount int)" "has no fields array")" "XX001" # an equality delete that can never apply (its sequence number exceeds no data # file's) is skipped per spec, not validated: this variant's delete has an # unsupported column type (field 8, timestamp) at the data's own sequence # number, and must NOT refuse -- all five rows read +await_up check "a never-applicable equality delete is skipped, not validated (5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/eqstaletype.metadata.json') AS t(id bigint, region text, amount int)")" "5" # a v1-shaped manifest (no sequence_number column in the entry schema) defaults # every file's sequence number to 0 per spec, even for EXISTING entries -- the # opposite of badseq, whose v2 schema HAS the column and carries a null +await_up check "a v1 manifest without a sequence_number column reads (default 0, 5 rows)" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_scan('$MDIR/v1seq.metadata.json') AS t(id bigint, region text, amount int)")" "5" -check "backend still up after the audit arms" "$(q 'SELECT 1')" "1" +check "backend still up after the audit arms" "$(survived "$CRASHES_BEFORE")" "1" # ---- spec: inheritance is ADDED-only; a null seq on an EXISTING entry is bad -- # the spec inherits a null sequence number only for status-1 (ADDED) entries; an # EXISTING (status 0) entry with a null sequence number is corrupt, and silently # inheriting the too-new manifest number could keep rows a delete should remove. +CRASHES_BEFORE="$(crashes)" check "an EXISTING entry with a null sequence number is refused (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/badseq.metadata.json') - AS t(id bigint, region text, amount int)")" "XX001" -check "backend still up after the bad-sequence refusal" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$MDIR/badseq.metadata.json') + AS t(id bigint, region text, amount int)" "has no sequence number")" "XX001" +check "backend still up after the bad-sequence refusal" "$(survived "$CRASHES_BEFORE")" "1" pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 74bd6ff82..6d220aa3b 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -315,6 +315,7 @@ behaviour, the source of that number is named. - [267. test_iceberg_objstore.py: Iceberg tables read over object storage](#267-test_iceberg_objstorepy-iceberg-tables-read-over-object-storage) - [268. test_iceberg_dv.py: Iceberg v3 deletion vectors apply, supersede, and refuse what they must](#268-test_iceberg_dvpy-iceberg-v3-deletion-vectors-apply-supersede-and-refuse-what-they-must) - [269. test_logical_decoding_source.py: a columnar table is not a logical decoding source](#269-test_logical_decoding_sourcepy-a-columnar-table-is-not-a-logical-decoding-source) +- [270. test_iceberg_deletes.py: Iceberg position and equality deletes apply, scope, and refuse what they must](#270-test_iceberg_deletespy-iceberg-position-and-equality-deletes-apply-scope-and-refuse-what-they-must) ## 1. How to read a test in here @@ -10459,3 +10460,63 @@ One check per test, each by the shell's name, every reading a value (a failed qu | `test_lds_second_slot_heap_control_is_decoded` | CONTROL: the second slot decodes the heap table's 5000 inserts | | `test_lds_big_columnar_emits_no_change` | and it too emits no decodable change of its own | | `test_lds_internal_records_do_not_follow_rows` | 100x the rows does not give 100x the internal records | + +## 270. test_iceberg_deletes.py: Iceberg position and equality deletes apply, scope, and refuse what they must + +The twin of `test/iceberg_deletes.sh`. Hand-crafted snapshots exercise position deletes (the <= sequence rule, inheritance, per-file scoping, ordinal 0), equality deletes (the strict < rule, multi-column AND, nulls, unions, mixed with position deletes) and partition-scoped equality deletes, against the generator's oracle; thirteen corrupt or unsupported snapshots are refused, each by its own guard, and no backend dies on the way. The warehouse is relocated into `server_dir` so its recorded root differs from where it lies. + +### Every test + +Every test makes one check under the shell's name. A refusal reads as its SQLSTATE only when the message is that guard's own (XX001 and 22023 are shared), and a failed reading is a `query_error()`. Each survival check also requires that the server log's count of backends killed by a signal has not moved since its section began, and every reading first waits for the server to answer, so a crash reddens only its own arm and its section's survival check. + +| test | what it holds | +| --- | --- | +| `test_iceberg_deletes_the_data_file_has_five_rows` | the data file has 5 rows before deletes | +| `test_position_deletes_drop_the_listed_rows` | position deletes drop the listed rows (surviving == oracle) | +| `test_position_deleted_ids_are_gone` | the deleted ids (2, 4) are gone | +| `test_position_deletes_remove_exactly_two_rows` | exactly the two deleted rows were removed (5 - 2 = 3) | +| `test_position_deletes_apply_under_projection` | position deletes apply under column projection (id only -> 1,3,5) | +| `test_a_position_delete_older_than_the_data_does_not_apply` | a delete older than the data (lower seq) does not apply (5 rows) | +| `test_older_position_delete_leaves_its_ids` | the would-be-deleted ids survive when the delete is older than the data | +| `test_a_position_delete_inherits_a_null_sequence_number` | a position delete with an inherited (null-entry) sequence number applies | +| `test_a_position_delete_naming_another_file_deletes_nothing` | a position delete naming a different data file deletes nothing (5 rows) | +| `test_a_null_pos_is_refused` | a position delete with a null pos is refused (XX001, not silently dropped) | +| `test_the_null_pos_refusal_names_pos` | ...and the null-pos refusal names pos | +| `test_the_backend_survives_the_null_pos_refusal` | backend still up after the null-pos refusal | +| `test_a_negative_pos_is_refused` | a position delete with a negative pos is refused (XX001, not cast inert) | +| `test_the_negative_pos_refusal_names_it` | ...and the negative-pos refusal names the negative pos | +| `test_the_backend_survives_the_negative_pos_refusal` | backend still up after the negative-pos refusal | +| `test_a_position_delete_of_ordinal_zero_deletes_the_first_row` | a position delete of ordinal 0 deletes the first row (id 1 gone, 4 remain) | +| `test_an_equality_delete_drops_matching_rows` | equality delete drops matching rows (survivors == oracle) | +| `test_an_equality_delete_at_an_equal_sequence_does_not_apply` | an equality delete at an equal sequence number does not apply | +| `test_multi_column_equality_is_an_and` | multi-column equality matches all columns (AND, not OR) | +| `test_columns_beyond_equality_ids_are_ignored` | columns beyond equality_ids are ignored for matching | +| `test_a_null_equality_value_matches_only_nulls` | a null delete value matches only null data values | +| `test_the_null_region_row_is_gone` | the null-region row is gone, the non-null rows remain | +| `test_multiple_equality_delete_files_apply_their_union` | multiple equality-delete files apply their union | +| `test_position_and_equality_deletes_combine` | position and equality deletes combine | +| `test_equality_deletes_apply_to_an_unprojected_column` | equality deletes apply when the delete column is not projected | +| `test_the_backend_survives_the_equality_applications` | backend still up after the equality applications | +| `test_an_equality_delete_without_equality_ids_is_refused` | an equality delete without equality_ids is refused (XX001) | +| `test_a_never_applicable_equality_delete_without_ids_is_skipped` | a never-applicable equality delete without equality_ids is skipped (5 rows) | +| `test_a_partition_scoped_equality_delete_stays_in_its_partition` | a partition-scoped equality delete applies only to its partition | +| `test_the_us_partition_survives_an_eu_scoped_delete` | the us-partition rows survive an eu-scoped delete | +| `test_a_delete_scoped_to_an_empty_partition_deletes_nothing` | a delete scoped to an empty partition deletes nothing (5 rows) | +| `test_a_data_file_without_partition_spec_id_is_refused` | a data file lacking partition_spec_id is refused under a partitioned delete (XX001) | +| `test_a_cross_spec_partition_delete_applies_to_nothing` | a cross-spec partition-scoped delete applies to nothing (5 rows) | +| `test_an_uncomparable_partition_value_is_refused` | a partition-scoped delete with an uncomparable partition value is refused (0A000) | +| `test_an_equality_delete_on_an_unsupported_type_is_refused` | an equality delete on an unsupported column type is refused (0A000) | +| `test_the_unsupported_type_refusal_names_the_type` | ...and the refusal names the type | +| `test_an_equality_column_missing_from_the_data_file_errors` | an equality column missing from the data file errors (22023) | +| `test_an_equality_delete_path_outside_the_table_is_refused` | an equality-delete path outside the table location is refused (22023) | +| `test_the_boundary_refusal_names_the_escape` | ...and the boundary refusal names the escape | +| `test_a_dotdot_escaping_delete_path_is_refused` | a dotdot-escaping delete path is refused (22023) | +| `test_the_backend_survives_the_equality_refusals` | backend still up after the equality refusals | +| `test_an_equality_id_beyond_int32_is_refused` | an equality_ids value beyond int32 refuses the manifest (XX001) | +| `test_a_manifest_list_without_partition_spec_id_is_refused` | a manifest list without partition_spec_id is refused for equality deletes (XX001) | +| `test_a_partition_spec_without_fields_is_refused` | a partition spec without its fields key is refused (XX001) | +| `test_a_never_applicable_equality_delete_is_not_validated` | a never-applicable equality delete is skipped, not validated (5 rows) | +| `test_a_v1_manifest_without_sequence_numbers_reads` | a v1 manifest without a sequence_number column reads (default 0, 5 rows) | +| `test_the_backend_survives_the_delete_audit_arms` | backend still up after the audit arms | +| `test_an_existing_entry_with_a_null_sequence_number_is_refused` | an EXISTING entry with a null sequence number is refused (XX001) | +| `test_the_backend_survives_the_bad_sequence_refusal` | backend still up after the bad-sequence refusal | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 630bd3f16..d9713ca25 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -910,6 +910,8 @@ guard_tests 476 # 26 checks in 26 tests, by collection over the cluster files. # cluster_tests 2779 -> 2792 for #1341's test_logical_decoding_source.py, the twin of logical_decoding_source.sh: # 13 checks in 13 tests, by collection over the cluster files. +# cluster_tests 2792 -> 2841 for #1341's test_iceberg_deletes.py, the twin of iceberg_deletes.sh: +# 49 checks in 49 tests, by collection over the cluster files. # cluster_tests 1264 -> 1287 for #1341's test_vector_agg_rescan_memory.py, the twin of vector_agg_rescan_memory.sh: # 23 tests, one per check, by collection over the cluster files. # cluster_tests 1287 -> 1329 for #1341's test_analyze_stats.py, the twin of analyze_stats.sh: @@ -1041,4 +1043,4 @@ guard_tests 476 # cluster_tests 2670 -> 2677 for #1637's test_catalog_filter_reads.py, the twin of catalog_filter_reads.sh: # 7 tests, one per check, by collection over the cluster files. # cluster_tests 2748 -> 2753 for #1706's five timestamp-flag tests in test_native_parquet_schema.py, by collection. -cluster_tests 2792 +cluster_tests 2841 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index c9fcd7ae8..947c07252 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -128,6 +128,7 @@ "iceberg_malformed", "iceberg_objstore", "iceberg_dv", + "iceberg_deletes", "concurrency", "isolation", "index_only", diff --git a/test/pytest/test_iceberg_deletes.py b/test/pytest/test_iceberg_deletes.py new file mode 100644 index 000000000..532045b5e --- /dev/null +++ b/test/pytest/test_iceberg_deletes.py @@ -0,0 +1,561 @@ +"""Iceberg row-level deletes (#388 phases 4a position + 4b equality, phase 5 partition +scope): iceberg_scan reads a table whose current snapshot carries delete files and drops +the deleted rows. The pytest twin of `test/iceberg_deletes.sh`, every check by the +shell's name (#1341). + +The fixture is hand-crafted (test/fixtures/iceberg/warehouse_del): real pyarrow data +and delete Parquet files, manifests with exact sequence numbers so both ordering rules +(position: data_seq <= delete_seq; equality: data_seq < delete_seq) sit on testable +boundaries, and an oracle of the surviving rows. The warehouse is relocated so its +recorded root differs from where it lies. + +Asserted: position deletes, sequence ordering and inheritance, per-file scoping, the +ordinal-0 control; equality deletes (the strict boundary, multi-column AND, extra +columns, null matching, unions, mixed with position deletes, unprojected columns); +partition-scoped equality deletes; thirteen corrupt or unsupported snapshots refused, +each by its own guard; and no backend dies on the way. + +HOW A REFUSAL IS READ. Several guards share a SQLSTATE (XX001, 22023), so a refusal +counts only when its message is that guard's own. A refusal reads as its SQLSTATE, +" (another refusal: ...)", "no error", or a query_error() when the server +could not be asked. + +HOW SURVIVAL IS READ. A fresh connection answers again after crash recovery, so +"SELECT 1" alone passed with a backend killed by a refusal arm (abort() in place of the +null-pos refusal; the partition spec's fields guard removed, which segfaults). Each +survival check also requires that the server log's count of backends terminated by a +signal has not moved since its section began. Every reading first waits, bounded, for +the server to answer, so a crash reds its own arm and its section's survival check, not +the arms after it. + +ON A PRIVATE DATABASE, with the warehouse copied into `server_dir`, because the backend +reads it. Where the fixture is missing the first test fails under the shell's pgc_skip +message, and the rest decline. +""" +import json +import pathlib +import shutil +import time + +import psycopg +import pytest + +from pgc_vacuity import query_error + +DB = "pgc_iceberg_deletes" +FX = pathlib.Path(__file__).resolve().parents[1] / "fixtures" / "iceberg" +WHD = FX / "warehouse_del" +COLS = "AS t(id bigint, region text, amount int)" +PCOLS = "AS t(id bigint, region text, grp int)" +EQ_APPLIED = ("eqapply", "eqboundary", "eqmulti", "eqextra", "eqnull", "eqtwo", "eqmixed") +# each refusal's own guard, by the phrase its message carries +PHRASE = { + "nullpos": "has a null pos", + "negpos": "has a negative pos", + "eqnoids": "has no equality_ids", + "eqpart_datanospec": "carries no partition_spec_id, which a partition-scoped", + "eqpart_incomparable": "has a partition tuple this reader cannot compare", + "eqtype": "is not supported", + "eqmissing": "no Parquet column has field id", + "eqescape": "is not under the table location", + "eqdotdot": "escapes the table location", + "eqbigid": "corrupt Avro manifest entry", + "eqnospec": "the manifest list carries no partition_spec_id", + "eqnofields": "has no fields array", + "badseq": "has no sequence number", +} + + +def _message(exc): + return str(exc).splitlines()[0] if str(exc) else type(exc).__name__ + + +def _value(dsn, sql): + try: + with psycopg.connect(dsn, autocommit=True, connect_timeout=10) as conn: + row = conn.execute(sql).fetchone() + except psycopg.Error as exc: + return query_error(_message(exc)) + return "" if row is None or row[0] is None else row[0] + + +def _await_up(dsn): + """Wait, at most 60 s, until the server answers, as the shell's await_up: a refusal + that kills a backend sends the server through crash recovery, and the next readings + would otherwise red as collateral, by timing. _survived() still counts the crash.""" + for _ in range(120): + if _value(dsn, "SELECT 1") == 1: + return + time.sleep(0.5) + + +def _read(dsn, sql): + _await_up(dsn) + return _value(dsn, sql) + + +def _refusal(dsn, sql, phrase): + """-> the SQLSTATE when sql is refused with phrase in its primary message, the + SQLSTATE with the other guard's message, "no error", or query_error().""" + _await_up(dsn) + try: + with psycopg.connect(dsn, autocommit=True, connect_timeout=10) as conn: + conn.execute(sql).fetchall() + except psycopg.Error as exc: + if not exc.sqlstate: + return query_error(_message(exc)) + msg = exc.diag.message_primary or "" + return exc.sqlstate if phrase in msg else f"{exc.sqlstate} (another refusal: {msg})" + return "no error" + + +def _names(dsn, sql, phrase): + """-> "1" when sql's refusal message carries phrase, "0" when it does not, as the + shell's grep -c over errmsg_of; query_error() when the server could not be asked.""" + _await_up(dsn) + try: + with psycopg.connect(dsn, autocommit=True, connect_timeout=10) as conn: + conn.execute(sql).fetchall() + except psycopg.Error as exc: + if not exc.sqlstate: + return query_error(_message(exc)) + return "1" if phrase in (exc.diag.message_primary or "") else "0" + return "0" + + +def _crashes(cluster): + try: + return (cluster.datadir / "server.log").read_text(errors="replace").count("terminated by signal") + except OSError as exc: + return query_error(f"crash count unreadable: {exc}") + + +def _survived(dsn, cluster, before): + _await_up(dsn) + up, now = _value(dsn, "SELECT 1"), _crashes(cluster) + if not isinstance(before, int) or not isinstance(now, int): + return query_error(f"crash count unreadable: {before} / {now}") + return "1" if up == 1 and now == before else f"no ([{up}], {now - before} backend crashes)" + + +@pytest.fixture(scope="module") +def dl(pgc_cluster, server_dir): + if not (WHD / "db" / "t" / "metadata" / "apply.metadata.json").is_file(): + yield None + return + dest = server_dir / "del" + shutil.copytree(WHD / "db", dest / "db") + for p in [dest, *dest.rglob("*")]: + p.chmod(0o777 if p.is_dir() else 0o666) + mdir = dest / "db" / "t" / "metadata" + oracle = json.loads((WHD / "expected_deletes.json").read_text()) + + def md(name): + return mdir / (name + ".metadata.json") + + def scan(name, cols=COLS): + return f"SELECT * FROM pgcolumnar.iceberg_scan('{md(name)}') {cols}" + + def ids(name, cols=COLS): + return f"SELECT string_agg(id::text, ',' ORDER BY id) FROM pgcolumnar.iceberg_scan('{md(name)}') {cols}" + + def count(name, where="", cols=COLS): + return f"SELECT count(*) FROM pgcolumnar.iceberg_scan('{md(name)}') {cols} {where}" + + admin = pgc_cluster.dsn() + with psycopg.connect(admin, autocommit=True) as a: + a.execute(f"DROP DATABASE IF EXISTS {DB} WITH (FORCE)") + a.execute(f"CREATE DATABASE {DB}") + try: + dsn = pgc_cluster.dsn(dbname=DB) + with psycopg.connect(dsn, autocommit=True) as c: + c.execute("CREATE EXTENSION IF NOT EXISTS pgcolumnar") + r = { + "oracle": "\n".join("%d|%s|%d" % (o["id"], o["region"], o["amount"]) for o in oracle["surviving"]), + "eq oracle": {k: ",".join(str(i) for i in v) for k, v in oracle["eq_surviving"].items()}, + "part oracle": {k: ",".join(str(i) for i in v) for k, v in oracle["part_surviving"].items()}, + } + # the shell's order, section by section, each with its own crash baseline + r["data rows"] = _read(dsn, f"SELECT count(*) FROM pgcolumnar.read_parquet(" + f"'{dest / 'db' / 't' / 'data' / 'data.parquet'}') {COLS}") + r["apply rows"] = _read(dsn, f"SELECT string_agg(id || '|' || region || '|' || amount, E'\\n' ORDER BY id) " + f"FROM pgcolumnar.iceberg_scan('{md('apply')}') {COLS}") + r["apply gone"] = _read(dsn, count("apply", "WHERE id IN (2,4)")) + r["apply count"] = _read(dsn, count("apply")) + r["apply projected"] = _read(dsn, f"SELECT string_agg(id::text, ',' ORDER BY id) " + f"FROM pgcolumnar.iceberg_scan('{md('apply')}') AS t(id bigint)") + r["noapply count"] = _read(dsn, count("noapply")) + r["noapply kept"] = _read(dsn, count("noapply", "WHERE id IN (2,4)")) + r["inherit count"] = _read(dsn, count("inherit")) + r["wrongpath count"] = _read(dsn, count("wrongpath")) + before = _crashes(pgc_cluster) + r["nullpos"] = _refusal(dsn, scan("nullpos"), PHRASE["nullpos"]) + r["nullpos names"] = _names(dsn, scan("nullpos"), "null pos") + r["up after nullpos"] = _survived(dsn, pgc_cluster, before) + before = _crashes(pgc_cluster) + r["negpos"] = _refusal(dsn, scan("negpos"), PHRASE["negpos"]) + r["negpos names"] = _names(dsn, scan("negpos"), "negative pos") + r["up after negpos"] = _survived(dsn, pgc_cluster, before) + r["zeropos"] = _read(dsn, ids("zeropos")) + before = _crashes(pgc_cluster) + for name in EQ_APPLIED: + r[name] = _read(dsn, ids(name)) + if name == "eqnull": + r["eqnull nulls"] = _read(dsn, count("eqnull", "WHERE region IS NULL")) + r["eq unprojected"] = _read(dsn, f"SELECT string_agg(amount::text, ',' ORDER BY amount) " + f"FROM pgcolumnar.iceberg_scan('{md('eqapply')}') AS t(amount int)") + r["up after eq applications"] = _survived(dsn, pgc_cluster, before) + before = _crashes(pgc_cluster) + r["eqnoids"] = _refusal(dsn, scan("eqnoids"), PHRASE["eqnoids"]) + r["equality count"] = _read(dsn, count("equality")) + r["eqpart_apply"] = _read(dsn, ids("eqpart_apply", PCOLS)) + r["eqpart_apply us"] = _read(dsn, count("eqpart_apply", "WHERE region = 'us'", PCOLS)) + r["eqpart_nomatch"] = _read(dsn, count("eqpart_nomatch", "", PCOLS)) + r["eqpart_datanospec"] = _refusal(dsn, scan("eqpart_datanospec", PCOLS), PHRASE["eqpart_datanospec"]) + r["eqpart_crossspec"] = _read(dsn, count("eqpart_crossspec", "", PCOLS)) + r["eqpart_incomparable"] = _refusal(dsn, scan("eqpart_incomparable", PCOLS), PHRASE["eqpart_incomparable"]) + r["eqtype"] = _refusal(dsn, scan("eqtype"), PHRASE["eqtype"]) + r["eqtype names"] = _names(dsn, scan("eqtype"), "type") + r["eqmissing"] = _refusal(dsn, scan("eqmissing"), PHRASE["eqmissing"]) + r["eqescape"] = _refusal(dsn, scan("eqescape"), PHRASE["eqescape"]) + r["eqescape names"] = _names(dsn, scan("eqescape"), "table location") + r["eqdotdot"] = _refusal(dsn, scan("eqdotdot"), PHRASE["eqdotdot"]) + r["up after eq refusals"] = _survived(dsn, pgc_cluster, before) + before = _crashes(pgc_cluster) + r["eqbigid"] = _refusal(dsn, scan("eqbigid"), PHRASE["eqbigid"]) + r["eqnospec"] = _refusal(dsn, scan("eqnospec"), PHRASE["eqnospec"]) + r["eqnofields"] = _refusal(dsn, scan("eqnofields"), PHRASE["eqnofields"]) + r["eqstaletype count"] = _read(dsn, count("eqstaletype")) + r["v1seq count"] = _read(dsn, count("v1seq")) + r["up after audit"] = _survived(dsn, pgc_cluster, before) + before = _crashes(pgc_cluster) + r["badseq"] = _refusal(dsn, scan("badseq"), PHRASE["badseq"]) + r["up after badseq"] = _survived(dsn, pgc_cluster, before) + yield r + finally: + with psycopg.connect(admin, autocommit=True) as a: + a.execute(f"DROP DATABASE IF EXISTS {DB} WITH (FORCE)") + + +def _declined(dl, expect): + if dl is not None: + return False + expect.cannot_run("ABSENT_FIXTURE", "the delete fixture is missing") + return True + + +# ---- position deletes ------------------------------------------------------------ + +def test_iceberg_deletes_the_data_file_has_five_rows(dl, expect): + if dl is None: + expect.text("missing", "present", "delete fixture is missing") + return + expect.require_module("json", name="python3 is needed") + expect.text(str(dl["data rows"]), "5", "the data file has 5 rows before deletes") + + +def test_position_deletes_drop_the_listed_rows(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["apply rows"]), dl["oracle"], "position deletes drop the listed rows (surviving == oracle)") + + +def test_position_deleted_ids_are_gone(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["apply gone"]), "0", "the deleted ids (2, 4) are gone") + + +def test_position_deletes_remove_exactly_two_rows(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["apply count"]), "3", "exactly the two deleted rows were removed (5 - 2 = 3)") + + +def test_position_deletes_apply_under_projection(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["apply projected"]), "1,3,5", + "position deletes apply under column projection (id only -> 1,3,5)") + + +def test_a_position_delete_older_than_the_data_does_not_apply(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["noapply count"]), "5", "a delete older than the data (lower seq) does not apply (5 rows)") + + +def test_older_position_delete_leaves_its_ids(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["noapply kept"]), "2", + "the would-be-deleted ids survive when the delete is older than the data") + + +def test_a_position_delete_inherits_a_null_sequence_number(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["inherit count"]), "3", + "a position delete with an inherited (null-entry) sequence number applies") + + +def test_a_position_delete_naming_another_file_deletes_nothing(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["wrongpath count"]), "5", + "a position delete naming a different data file deletes nothing (5 rows)") + + +def test_a_null_pos_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["nullpos"]), "XX001", "a position delete with a null pos is refused (XX001, not silently dropped)") + + +def test_the_null_pos_refusal_names_pos(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["nullpos names"]), "1", "...and the null-pos refusal names pos") + + +def test_the_backend_survives_the_null_pos_refusal(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["up after nullpos"]), "1", "backend still up after the null-pos refusal") + + +def test_a_negative_pos_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["negpos"]), "XX001", "a position delete with a negative pos is refused (XX001, not cast inert)") + + +def test_the_negative_pos_refusal_names_it(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["negpos names"]), "1", "...and the negative-pos refusal names the negative pos") + + +def test_the_backend_survives_the_negative_pos_refusal(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["up after negpos"]), "1", "backend still up after the negative-pos refusal") + + +def test_a_position_delete_of_ordinal_zero_deletes_the_first_row(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["zeropos"]), "2,3,4,5", + "a position delete of ordinal 0 deletes the first row (id 1 gone, 4 remain)") + + +# ---- equality deletes -------------------------------------------------------------- + +def test_an_equality_delete_drops_matching_rows(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqapply"]), dl["eq oracle"]["eqapply"], "equality delete drops matching rows (survivors == oracle)") + + +def test_an_equality_delete_at_an_equal_sequence_does_not_apply(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqboundary"]), dl["eq oracle"]["eqboundary"], + "an equality delete at an equal sequence number does not apply") + + +def test_multi_column_equality_is_an_and(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqmulti"]), dl["eq oracle"]["eqmulti"], "multi-column equality matches all columns (AND, not OR)") + + +def test_columns_beyond_equality_ids_are_ignored(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqextra"]), dl["eq oracle"]["eqextra"], "columns beyond equality_ids are ignored for matching") + + +def test_a_null_equality_value_matches_only_nulls(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqnull"]), dl["eq oracle"]["eqnull"], "a null delete value matches only null data values") + + +def test_the_null_region_row_is_gone(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqnull nulls"]), "0", "the null-region row is gone, the non-null rows remain") + + +def test_multiple_equality_delete_files_apply_their_union(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqtwo"]), dl["eq oracle"]["eqtwo"], "multiple equality-delete files apply their union") + + +def test_position_and_equality_deletes_combine(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqmixed"]), dl["eq oracle"]["eqmixed"], "position and equality deletes combine") + + +def test_equality_deletes_apply_to_an_unprojected_column(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eq unprojected"]), "10,30,50", + "equality deletes apply when the delete column is not projected") + + +def test_the_backend_survives_the_equality_applications(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["up after eq applications"]), "1", "backend still up after the equality applications") + + +# ---- equality refusals and partition scope ---------------------------------------- + +def test_an_equality_delete_without_equality_ids_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqnoids"]), "XX001", "an equality delete without equality_ids is refused (XX001)") + + +def test_a_never_applicable_equality_delete_without_ids_is_skipped(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["equality count"]), "5", + "a never-applicable equality delete without equality_ids is skipped (5 rows)") + + +def test_a_partition_scoped_equality_delete_stays_in_its_partition(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqpart_apply"]), dl["part oracle"]["eqpart_apply"], + "a partition-scoped equality delete applies only to its partition") + + +def test_the_us_partition_survives_an_eu_scoped_delete(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqpart_apply us"]), "3", "the us-partition rows survive an eu-scoped delete") + + +def test_a_delete_scoped_to_an_empty_partition_deletes_nothing(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqpart_nomatch"]), "5", "a delete scoped to an empty partition deletes nothing (5 rows)") + + +def test_a_data_file_without_partition_spec_id_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqpart_datanospec"]), "XX001", + "a data file lacking partition_spec_id is refused under a partitioned delete (XX001)") + + +def test_a_cross_spec_partition_delete_applies_to_nothing(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqpart_crossspec"]), "5", "a cross-spec partition-scoped delete applies to nothing (5 rows)") + + +def test_an_uncomparable_partition_value_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqpart_incomparable"]), "0A000", + "a partition-scoped delete with an uncomparable partition value is refused (0A000)") + + +def test_an_equality_delete_on_an_unsupported_type_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqtype"]), "0A000", "an equality delete on an unsupported column type is refused (0A000)") + + +def test_the_unsupported_type_refusal_names_the_type(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqtype names"]), "1", "...and the refusal names the type") + + +def test_an_equality_column_missing_from_the_data_file_errors(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqmissing"]), "22023", "an equality column missing from the data file errors (22023)") + + +def test_an_equality_delete_path_outside_the_table_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqescape"]), "22023", "an equality-delete path outside the table location is refused (22023)") + + +def test_the_boundary_refusal_names_the_escape(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqescape names"]), "1", "...and the boundary refusal names the escape") + + +def test_a_dotdot_escaping_delete_path_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqdotdot"]), "22023", "a dotdot-escaping delete path is refused (22023)") + + +def test_the_backend_survives_the_equality_refusals(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["up after eq refusals"]), "1", "backend still up after the equality refusals") + + +# ---- audit arms -------------------------------------------------------------------- + +def test_an_equality_id_beyond_int32_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqbigid"]), "XX001", "an equality_ids value beyond int32 refuses the manifest (XX001)") + + +def test_a_manifest_list_without_partition_spec_id_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqnospec"]), "XX001", + "a manifest list without partition_spec_id is refused for equality deletes (XX001)") + + +def test_a_partition_spec_without_fields_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqnofields"]), "XX001", "a partition spec without its fields key is refused (XX001)") + + +def test_a_never_applicable_equality_delete_is_not_validated(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["eqstaletype count"]), "5", + "a never-applicable equality delete is skipped, not validated (5 rows)") + + +def test_a_v1_manifest_without_sequence_numbers_reads(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["v1seq count"]), "5", "a v1 manifest without a sequence_number column reads (default 0, 5 rows)") + + +def test_the_backend_survives_the_delete_audit_arms(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["up after audit"]), "1", "backend still up after the audit arms") + + +def test_an_existing_entry_with_a_null_sequence_number_is_refused(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["badseq"]), "XX001", "an EXISTING entry with a null sequence number is refused (XX001)") + + +def test_the_backend_survives_the_bad_sequence_refusal(dl, expect): + if _declined(dl, expect): + return + expect.text(str(dl["up after badseq"]), "1", "backend still up after the bad-sequence refusal") diff --git a/test/pytest/unported_suites.txt b/test/pytest/unported_suites.txt index 649c15d89..8749f6f3a 100644 --- a/test/pytest/unported_suites.txt +++ b/test/pytest/unported_suites.txt @@ -19,7 +19,6 @@ fuzz_listing fuzz_parquet harness_selftest hilbert_curve -iceberg_deletes iceberg_rest iceberg_rest_scan iceberg_rest_server From 44ec07adf1dd8ac2d8f0d640d4c4e99056caae27 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 22:00:34 +0000 Subject: [PATCH 04/15] test: logical_subscriber gains its pytest twin, and a subscription that applied nothing no longer passes (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The twin of test/logical_subscriber.sh, on two private clusters (a heap publisher at wal_level=logical, a columnar subscriber), one check per test, each by the shell's name. Four holes, each measured red-first on main's shell (PG17): - With the insert-only subscription disabled before its UPDATE, main passed 20/20: "skipped by design" and "no apply errors" read after a fixed sleep. An INSERT after the UPDATE must now arrive before either is read. - With no UPDATE sent to the wedge publication, "the error does NOT say only 'row locking'" and "the wedge is real" passed. The first now needs the replication error in the subscriber's log; the second reads "publisher -> subscriber". - With every lock statement failing on a missing table, the three "is not told about replication" checks passed. They now need the generic row-locking error first. - "premise: the second subscription synced before the UPDATE" is reddened by COPY failing in a logical worker, so it is not a premise (#1339) and is dated. The lock loop is unrolled into literal checks so the grader reads each name. Five mutations (hintnoname, nogate, overgate, applyins, syncins) redden the same names in both twins; clean 20/20 on 15-19 in both, and 8/8 head repeats on 17. 20 rows, 16 dated, 4 never. Census 3903 -> 3907 never, 2354 -> 2370 dated, 6277 rows; suites_not_covered 99 -> 98; cluster_tests 2841 -> 2861; TESTS.md §271. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 20 ++ test/check_ledger_budget.txt | 24 +- test/logical_subscriber.sh | 52 +++- test/pytest/TESTS.md | 32 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_logical_subscriber.py | 333 +++++++++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 450 insertions(+), 17 deletions(-) create mode 100644 test/pytest/test_logical_subscriber.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index a27f05d1f..431b86e2f 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2963,6 +2963,26 @@ logical_decoding_source logical_decoding_source premise: the server is running a logical_decoding_source logical_decoding_source premise: the slot decoded something at all 15;16;17;18;19 never - logical_decoding_source logical_decoding_source premise: the slot exists and is logical 15;16;17;18;19 never - logical_decoding_source logical_decoding_source the slot carries pgcolumnar's internal catalog instead 15;16;17;18;19 never - +logical_subscriber logical_subscriber an ordinary SELECT ... FOR NO KEY UPDATE is not told about replication 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock raises the replication error for every row lock, not only the apply worker's +logical_subscriber logical_subscriber an ordinary SELECT ... FOR NO KEY UPDATE keeps the generic message 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock raises the replication error for every row lock, not only the apply worker's +logical_subscriber logical_subscriber an ordinary SELECT ... FOR SHARE is not told about replication 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock raises the replication error for every row lock, not only the apply worker's +logical_subscriber logical_subscriber an ordinary SELECT ... FOR SHARE keeps the generic message 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock raises the replication error for every row lock, not only the apply worker's +logical_subscriber logical_subscriber an ordinary SELECT ... FOR UPDATE is not told about replication 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock raises the replication error for every row lock, not only the apply worker's +logical_subscriber logical_subscriber an ordinary SELECT ... FOR UPDATE keeps the generic message 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock raises the replication error for every row lock, not only the apply worker's +logical_subscriber logical_subscriber insert-only: initial sync copied every row 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_multi_insert raises in a logical worker, so initial sync's COPY fails +logical_subscriber logical_subscriber insert-only: no apply errors 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_insert raises in a logical worker, so streamed INSERTs fail +logical_subscriber logical_subscriber insert-only: streamed INSERTs applied 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_insert raises in a logical worker, so streamed INSERTs fail +logical_subscriber logical_subscriber insert-only: the UPDATE is skipped by design, not applied 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_insert raises in a logical worker, so streamed INSERTs fail +logical_subscriber logical_subscriber it retries rather than giving up, so the subscription is wedged 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock never takes the IsLogicalWorker() branch, so the apply worker gets the generic row-locking error +logical_subscriber logical_subscriber premise: the lock fixture really is columnar 15;16;17;18;19 never - +logical_subscriber logical_subscriber premise: the publisher has logical decoding on 15;16;17;18;19 never - +logical_subscriber logical_subscriber premise: the subscriber cluster answers 15;16;17;18;19 never - +logical_subscriber logical_subscriber premise: the subscriber loaded pgcolumnar 15;16;17;18;19 never - +logical_subscriber logical_subscriber the error does NOT say only 'row locking is not supported' 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock never takes the IsLogicalWorker() branch, so the apply worker gets the generic row-locking error +logical_subscriber logical_subscriber the error names logical replication, not row locking 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_lock never takes the IsLogicalWorker() branch, so the apply worker gets the generic row-locking error +logical_subscriber logical_subscriber the hint names the insert-only publication 15;16;17;18;19 2026-10-21 columnar_tableam.c: the apply worker's hint stops naming CREATE PUBLICATION ... WITH (publish = 'insert') +logical_subscriber logical_subscriber the second subscription synced before the UPDATE 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_multi_insert raises in a logical worker, so initial sync's COPY fails +logical_subscriber logical_subscriber the wedge is real: the row is still unchanged on the subscriber 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_multi_insert raises in a logical worker, so initial sync's COPY fails maintenance_crash maintenance_crash after a crash with compact committed, the index reads the rows the scan reads 15;16;17;18;19 2026-10-16 pgcolumnar_index_fetch_tuple finds no row (columnar_tableam.c) maintenance_crash maintenance_crash after a crash with compact committed, the table is as compact left it, with the same rows 15;16;17;18;19 never - maintenance_crash maintenance_crash after a crash with compact uncommitted, the index reads the rows the scan reads 15;16;17;18;19 2026-10-16 pgcolumnar_index_fetch_tuple finds no row (columnar_tableam.c) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index ab9b4075c..5b34b545e 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 99 +suites_not_covered 98 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5498,4 +5498,24 @@ suites_not_covered 99 # never 3897 + 6 = 3903 # dated 2311 + 43 = 2354 # 3903 + 2354 == 6257 -checks_never_observed_red 3903 +# +# #1341, logical_subscriber seeded, with its twin (OffgridwithJD): 3903 -> 3907, +# and suites_not_covered 99 -> 98. The suite had no rows. It enters with 20 checks +# from five-major runs (all green on each of 15-19, shell and twin alike), after four fixes. +# With the insert-only subscription disabled before its UPDATE, main's shell passed 20/20: +# "skipped by design" and "no apply errors" read after a fixed sleep, and nothing applied +# is nothing applied wrongly. An INSERT after the UPDATE must now arrive first. With no +# UPDATE sent, "does NOT say only 'row locking'" and "the wedge is real" passed; the first +# now needs the replication error in the log, the second reads publisher -> subscriber. +# With the lock statements failing on a missing table, the three "is not told about +# replication" checks passed; they now need the generic error first. And the second +# subscription's sync check lost its premise prefix: COPY failing in a logical worker +# reddens it. The lock loop is unrolled so the twin grader reads each name. +# 16 are dated by five mutations, each reddening the same names in both twins. 4 +# arrive never. +# +# ledger rows 6257 + 20 = 6277 +# never 3903 + 4 = 3907 +# dated 2354 + 16 = 2370 +# 3907 + 2370 == 6277 +checks_never_observed_red 3907 diff --git a/test/logical_subscriber.sh b/test/logical_subscriber.sh index 40b350889..190163813 100755 --- a/test/logical_subscriber.sh +++ b/test/logical_subscriber.sh @@ -96,12 +96,22 @@ psql_run "INSERT INTO src SELECT g, 'v'||g FROM generate_series(11,15) g;" >/dev for _ in $(seq 1 30); do [ "$(SUB 'SELECT count(*) FROM src')" = "15" ] && break; sleep 1; done check "insert-only: streamed INSERTs applied" "$(SUB 'SELECT count(*) FROM src')" "15" +# An INSERT committed after the UPDATE: once it has arrived, the UPDATE before it in the +# stream has been processed. A fixed sleep proved nothing. With the subscription disabled +# before the UPDATE, or with streamed INSERTs failing, both checks below used to pass: +# nothing had been applied, so nothing had been applied wrongly. psql_run "UPDATE src SET v = 'changed' WHERE id = 1;" >/dev/null 2>&1 -sleep 3 +psql_run "INSERT INTO src VALUES (16, 'after the update');" >/dev/null 2>&1 +for _ in $(seq 1 30); do [ "$(SUB 'SELECT count(*) FROM src WHERE id = 16')" = "1" ] && break; sleep 1; done +SENTINEL="$(SUB 'SELECT count(*) FROM src WHERE id = 16')" +after_sentinel() { # reading -> the reading, or "no reading" when the sentinel never arrived + if [ "$SENTINEL" = "1" ]; then echo "$1" + else echo "no reading (the INSERT after the UPDATE never arrived: [$SENTINEL]; reading [$1])"; fi +} check "insert-only: the UPDATE is skipped by design, not applied" \ - "$(SUB "SELECT v FROM src WHERE id = 1")" "v1" + "$(after_sentinel "$(SUB "SELECT v FROM src WHERE id = 1")")" "v1" check "insert-only: no apply errors" \ - "$(SUB "SELECT coalesce(sum(apply_error_count),0)::text FROM pg_stat_subscription_stats" | tail -1)" "0" + "$(after_sentinel "$(SUB "SELECT coalesce(sum(apply_error_count),0)::text FROM pg_stat_subscription_stats" | tail -1)")" "0" # ---- 2. the wedge: a publication carrying UPDATE ---------------------------- # Asserted from the subscriber's log, because the failure happens in the apply worker and @@ -112,7 +122,7 @@ psql_run "CREATE TABLE upd (id int PRIMARY KEY, v text); SUB "CREATE TABLE upd (id int PRIMARY KEY, v text) USING pgcolumnar;" >/dev/null 2>&1 SUB "CREATE SUBSCRIPTION s_all CONNECTION 'host=127.0.0.1 port=$PGC_PORT user=postgres dbname=$PGC_DB' PUBLICATION p_all;" >/dev/null 2>&1 for _ in $(seq 1 30); do [ "$(SUB 'SELECT count(*) FROM upd')" = "5" ] && break; sleep 1; done -check "premise: the second subscription synced before the UPDATE" \ +check "the second subscription synced before the UPDATE" \ "$(SUB 'SELECT count(*) FROM upd')" "5" psql_run "UPDATE upd SET v = 'changed' WHERE id = 1;" >/dev/null 2>&1 @@ -125,12 +135,17 @@ log_has() { pgc_pg "grep -c '$1' '$SUB_LOG' 2>/dev/null || echo 0" | tail -1; } check "the error names logical replication, not row locking" \ "$([ "$(log_has 'logical replication cannot apply UPDATE or DELETE')" -ge 1 ] && echo yes || echo no)" "yes" +# Absence counts only once the replication error is there: with no UPDATE sent, an +# unreadable log, or a wedge that never happened, this used to pass. check "the error does NOT say only 'row locking is not supported'" \ - "$([ "$(log_has 'row locking is not supported')" = 0 ] && echo yes || echo no)" "yes" + "$(if [ "$(log_has 'logical replication cannot apply UPDATE or DELETE')" -lt 1 ]; then + echo "no reading (the subscriber log holds no replication error)" + elif [ "$(log_has 'row locking is not supported')" = 0 ]; then echo yes; else echo no; fi)" "yes" check "the hint names the insert-only publication" \ "$([ "$(log_has 'publish = ')" -ge 1 ] && echo yes || echo no)" "yes" +# Read as "publisher -> subscriber": with no UPDATE sent, the subscriber's v1 used to pass. check "the wedge is real: the row is still unchanged on the subscriber" \ - "$(SUB "SELECT v FROM upd WHERE id = 1")" "v1" + "$(q "SELECT v FROM upd WHERE id = 1") -> $(SUB "SELECT v FROM upd WHERE id = 1")" "changed -> v1" # Unbounded is the whole point, so assert the SECOND attempt rather than a counter that # may not have flushed. Core re-sends the same transaction about every 5 seconds. for _ in $(seq 1 20); do @@ -152,13 +167,24 @@ psql_run "CREATE TABLE lk (id int primary key, v text) USING pgcolumnar; INSERT INTO lk VALUES (1, 'a');" >/dev/null 2>&1 check "premise: the lock fixture really is columnar" \ "$(q "SELECT amname FROM pg_class c JOIN pg_am a ON a.oid=c.relam WHERE c.relname='lk'")" "pgcolumnar" -for lk in "FOR UPDATE" "FOR SHARE" "FOR NO KEY UPDATE"; do - out=$(psql_run "SELECT * FROM lk WHERE id = 1 $lk" 2>&1) - check "an ordinary SELECT ... $lk keeps the generic message" \ - "$(grep -c 'row locking is not supported' <<<"$out")" "1" - check "an ordinary SELECT ... $lk is not told about replication" \ - "$(grep -c 'logical replication cannot apply' <<<"$out")" "0" -done +# lk_said MODE -> the error an ordinary SELECT ... MODE raised. lk_replication MODE -> how +# often it mentions replication, but only when it raised the generic message: a statement +# that failed some other way (a missing table, a dead server) used to pass "is not told +# about replication" because nothing was said at all. Unrolled so each check's name is a +# literal the twin grader can read. +lk_said() { psql_run "SELECT * FROM lk WHERE id = 1 $1" 2>&1; } +lk_generic() { grep -c 'row locking is not supported' <<<"$(lk_said "$1")"; } +lk_replication() { + local out; out="$(lk_said "$1")" + if grep -q 'row locking is not supported' <<<"$out"; then grep -c 'logical replication cannot apply' <<<"$out" + else echo "no reading (no generic error: $(head -1 <<<"$out"))"; fi +} +check "an ordinary SELECT ... FOR UPDATE keeps the generic message" "$(lk_generic 'FOR UPDATE')" "1" +check "an ordinary SELECT ... FOR UPDATE is not told about replication" "$(lk_replication 'FOR UPDATE')" "0" +check "an ordinary SELECT ... FOR SHARE keeps the generic message" "$(lk_generic 'FOR SHARE')" "1" +check "an ordinary SELECT ... FOR SHARE is not told about replication" "$(lk_replication 'FOR SHARE')" "0" +check "an ordinary SELECT ... FOR NO KEY UPDATE keeps the generic message" "$(lk_generic 'FOR NO KEY UPDATE')" "1" +check "an ordinary SELECT ... FOR NO KEY UPDATE is not told about replication" "$(lk_replication 'FOR NO KEY UPDATE')" "0" SUB "DROP SUBSCRIPTION s_ins;" >/dev/null 2>&1 SUB "DROP SUBSCRIPTION s_all;" >/dev/null 2>&1 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 6d220aa3b..d650153f3 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -316,6 +316,7 @@ behaviour, the source of that number is named. - [268. test_iceberg_dv.py: Iceberg v3 deletion vectors apply, supersede, and refuse what they must](#268-test_iceberg_dvpy-iceberg-v3-deletion-vectors-apply-supersede-and-refuse-what-they-must) - [269. test_logical_decoding_source.py: a columnar table is not a logical decoding source](#269-test_logical_decoding_sourcepy-a-columnar-table-is-not-a-logical-decoding-source) - [270. test_iceberg_deletes.py: Iceberg position and equality deletes apply, scope, and refuse what they must](#270-test_iceberg_deletespy-iceberg-position-and-equality-deletes-apply-scope-and-refuse-what-they-must) +- [271. test_logical_subscriber.py: columnar as a logical replication apply target](#271-test_logical_subscriberpy-columnar-as-a-logical-replication-apply-target) ## 1. How to read a test in here @@ -10520,3 +10521,34 @@ Every test makes one check under the shell's name. A refusal reads as its SQLSTA | `test_the_backend_survives_the_delete_audit_arms` | backend still up after the audit arms | | `test_an_existing_entry_with_a_null_sequence_number_is_refused` | an EXISTING entry with a null sequence number is refused (XX001) | | `test_the_backend_survives_the_bad_sequence_refusal` | backend still up after the bad-sequence refusal | + +## 271. test_logical_subscriber.py: columnar as a logical replication apply target + +The twin of `test/logical_subscriber.sh` (#435). A heap publisher and a columnar subscriber, on two private clusters: an INSERT-only publication syncs and streams every row, a publication carrying UPDATE raises the extension's own error (naming logical replication and the insert-only way out) and the apply worker retries it forever, and an ordinary SELECT ... FOR UPDATE / SHARE / NO KEY UPDATE keeps the generic row-locking message. + +### Every test + +One check per test, each by the shell's name, every reading a value from a module fixture. The apply worker's error is read from the subscriber's server log. A reading that would pass on an absence (a skipped UPDATE, a message not said) is taken only once the event it depends on is observed: an INSERT after the UPDATE has arrived, the replication error is in the log, or the statement raised the generic error. + +| test | what it holds | +| --- | --- | +| `test_the_publisher_has_logical_decoding_on` | premise: the publisher has logical decoding on | +| `test_the_subscriber_cluster_answers` | premise: the subscriber cluster answers | +| `test_the_subscriber_loaded_pgcolumnar` | premise: the subscriber loaded pgcolumnar | +| `test_insert_only_initial_sync_copied_every_row` | insert-only: initial sync copied every row | +| `test_insert_only_streamed_inserts_applied` | insert-only: streamed INSERTs applied | +| `test_insert_only_the_update_is_skipped_by_design` | insert-only: the UPDATE is skipped by design, not applied | +| `test_insert_only_no_apply_errors` | insert-only: no apply errors | +| `test_the_second_subscription_synced_before_the_update` | the second subscription synced before the UPDATE | +| `test_the_apply_error_names_logical_replication` | the error names logical replication, not row locking | +| `test_the_apply_error_does_not_say_only_row_locking` | the error does NOT say only 'row locking is not supported' | +| `test_the_apply_hint_names_the_insert_only_publication` | the hint names the insert-only publication | +| `test_the_wedged_row_is_unchanged_on_the_subscriber` | the wedge is real: the row is still unchanged on the subscriber | +| `test_the_apply_worker_retries_so_the_subscription_is_wedged` | it retries rather than giving up, so the subscription is wedged | +| `test_the_lock_fixture_is_columnar` | premise: the lock fixture really is columnar | +| `test_select_for_update_keeps_the_generic_message` | an ordinary SELECT ... FOR UPDATE keeps the generic message | +| `test_select_for_update_is_not_told_about_replication` | an ordinary SELECT ... FOR UPDATE is not told about replication | +| `test_select_for_share_keeps_the_generic_message` | an ordinary SELECT ... FOR SHARE keeps the generic message | +| `test_select_for_share_is_not_told_about_replication` | an ordinary SELECT ... FOR SHARE is not told about replication | +| `test_select_for_no_key_update_keeps_the_generic_message` | an ordinary SELECT ... FOR NO KEY UPDATE keeps the generic message | +| `test_select_for_no_key_update_is_not_told_about_replication` | an ordinary SELECT ... FOR NO KEY UPDATE is not told about replication | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index d9713ca25..7fd6669d8 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -912,6 +912,8 @@ guard_tests 476 # 13 checks in 13 tests, by collection over the cluster files. # cluster_tests 2792 -> 2841 for #1341's test_iceberg_deletes.py, the twin of iceberg_deletes.sh: # 49 checks in 49 tests, by collection over the cluster files. +# cluster_tests 2841 -> 2861 for #1341's test_logical_subscriber.py, the twin of logical_subscriber.sh: +# 20 checks in 20 tests, by collection over the cluster files. # cluster_tests 1264 -> 1287 for #1341's test_vector_agg_rescan_memory.py, the twin of vector_agg_rescan_memory.sh: # 23 tests, one per check, by collection over the cluster files. # cluster_tests 1287 -> 1329 for #1341's test_analyze_stats.py, the twin of analyze_stats.sh: @@ -1043,4 +1045,4 @@ guard_tests 476 # cluster_tests 2670 -> 2677 for #1637's test_catalog_filter_reads.py, the twin of catalog_filter_reads.sh: # 7 tests, one per check, by collection over the cluster files. # cluster_tests 2748 -> 2753 for #1706's five timestamp-flag tests in test_native_parquet_schema.py, by collection. -cluster_tests 2841 +cluster_tests 2861 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 947c07252..1a182819d 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -142,6 +142,7 @@ "encode_effort", "local_open_race_free", "logical_decoding_source", + "logical_subscriber", "sorted_mark_rename", "tablesample", "ungrouped_vector_agg", diff --git a/test/pytest/test_logical_subscriber.py b/test/pytest/test_logical_subscriber.py new file mode 100644 index 000000000..006b852bd --- /dev/null +++ b/test/pytest/test_logical_subscriber.py @@ -0,0 +1,333 @@ +"""Columnar as a logical replication APPLY TARGET (#435, #1341). The pytest twin of +`test/logical_subscriber.sh`, every check by the shell's name. + +A heap table on a publisher, a columnar mirror on a subscriber. Two facts are pinned: +an INSERT-only publication works end to end (initial sync and streamed INSERTs take no +tuple lock), and a publication that carries UPDATE raises OUR error, which names logical +replication and the way out, retried by the apply worker forever. An ordinary +SELECT ... FOR UPDATE / SHARE / NO KEY UPDATE on a columnar table keeps the generic +row-locking message: the replication one is gated on the apply worker. + +HOW THE READINGS ARE MADE FAIL-CLOSED, as in the shell: +- an INSERT committed after the insert-only UPDATE must arrive before "skipped by design" + and "no apply errors" are read, so a subscription that applied nothing cannot pass them; +- "does NOT say only 'row locking'" is read only once the replication error is in the + subscriber's log, and "is not told about replication" only once the statement raised + the generic error, so an absence that is no evidence cannot pass; +- the wedge reads "publisher -> subscriber", so an UPDATE never sent cannot pass as one + never applied. + +PRIVATE CLUSTERS, two of them: the publisher at wal_level=logical, and a subscriber whose +server log holds the apply worker's error, which never reaches a client. +""" +import pathlib +import shutil +import tempfile +import time + +import psycopg +import pytest + +from pgc_vacuity import query_error + +ROUND_TRIP = "the logical subscriber round trip" +REPL_ERR = "logical replication cannot apply UPDATE or DELETE" +GENERIC = "row locking is not supported" + + +def _q(cluster, sql): + """-> the first column of the last statement's first row, as text; or query_error().""" + try: + with psycopg.connect(cluster.dsn(), autocommit=True, connect_timeout=5) as c: + cur = None + for stmt in [s for s in sql.split(";") if s.strip()]: + cur = c.execute(stmt) + row = cur.fetchone() if cur is not None and cur.description else None + return "" if row is None or row[0] is None else str(row[0]) + except psycopg.Error as exc: + return query_error(str(exc).splitlines()[0] if str(exc) else type(exc).__name__) + + +def _run(cluster, *sqls): + """Setup statements, as the shell runs them with their output dropped. A failure is + left for the readings after it to report by name.""" + try: + with psycopg.connect(cluster.dsn(), autocommit=True, connect_timeout=5) as c: + for sql in sqls: + c.execute(sql) + except psycopg.Error: + pass + + +def _as_psql(exc): + """-> a raised error's lines as psql prints them, so the shell's grep -c counts match.""" + d = exc.diag + lines = [f"ERROR: {d.message_primary or str(exc)}"] + if d.message_detail: + lines.append(f"DETAIL: {d.message_detail}") + if d.message_hint: + lines.append(f"HINT: {d.message_hint}") + return "\n".join(lines) + + +def _said(cluster, sql): + """-> the error a statement raised, as psql prints it, or "" when it raised nothing. + Compared only with fixed counts, never with another reading.""" + try: + with psycopg.connect(cluster.dsn(), autocommit=True, connect_timeout=5) as c: + c.execute(sql).fetchall() + return "" + except psycopg.Error as exc: + raised = exc + return _as_psql(raised) + + +def _lines(text, needle): + return sum(1 for line in text.splitlines() if needle in line) + + +def _log_has(log, needle): + """-> how many lines of the subscriber's log hold `needle`, or query_error().""" + try: + return _lines(log.read_text(errors="replace"), needle) + except OSError as exc: + return query_error(f"subscriber log unreadable: {exc}") + + +def _wait(cluster, sql, want, tries=30): + for _ in range(tries): + if _q(cluster, sql) == want: + break + time.sleep(1) + return _q(cluster, sql) + + +def _conf(datadir, *lines): + with open(datadir / "postgresql.conf", "a") as fh: + fh.write("\n" + "\n".join(lines) + "\n") + + +def _yes_if(n, at_least): + return "yes" if isinstance(n, int) and n >= at_least else f"no ({n})" + + +def _lock_readings(pub, mode): + out = _said(pub, f"SELECT * FROM lk WHERE id = 1 {mode}") + generic = _lines(out, GENERIC) + told = (_lines(out, "logical replication cannot apply") if generic + else f"no reading (no generic error: {out.splitlines()[0] if out else 'the statement raised nothing'})") + return generic, told + + +@pytest.fixture(scope="module") +def ls(pgc_cluster): + from pgc_cluster import Cluster, pick_port + + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-lsub-")) + work.chmod(0o777) + pub = sub = None + r = {} + try: + try: + pub_port, sub_port = pick_port(26), pick_port(27) + except RuntimeError as exc: + yield {"no port": f"no free port for the subscriber cluster ({exc})"} + return + (work / "pub").mkdir() + (work / "sub").mkdir() + pub = Cluster(pgc_cluster.pg_config, "lsub-pub", work / "pub", pub_port) + pub.initdb() + _conf(work / "pub", "wal_level=logical", "max_wal_senders=10", "max_replication_slots=10", + "max_locks_per_transaction=256") + pub.start() + sub = Cluster(pgc_cluster.pg_config, "lsub-sub", work / "sub", sub_port) + sub.initdb() + _conf(work / "sub", "wal_level=logical", "max_locks_per_transaction=256") + try: + sub.start() + except RuntimeError: + pass # the premise below reports it by name + log = work / "sub" / "server.log" + conninfo = f"host=127.0.0.1 port={pub.port} user=postgres dbname=postgres" + + r["wal_level"] = _q(pub, "SHOW wal_level") + r["sub answers"] = _q(sub, "SELECT 1") + r["sub loaded"] = _q(sub, "CREATE EXTENSION IF NOT EXISTS pgcolumnar; " + "SELECT count(*) FROM pg_extension WHERE extname='pgcolumnar'") + _run(pub, "CREATE EXTENSION IF NOT EXISTS pgcolumnar") + + # ---- 1. the working pattern: publish inserts only ---- + _run(pub, "CREATE TABLE src (id int PRIMARY KEY, v text)", + "INSERT INTO src SELECT g, 'v'||g FROM generate_series(1,10) g", + "CREATE PUBLICATION p_ins FOR TABLE src WITH (publish = 'insert')") + _run(sub, "CREATE TABLE src (id int PRIMARY KEY, v text) USING pgcolumnar") + _run(sub, f"CREATE SUBSCRIPTION s_ins CONNECTION '{conninfo}' PUBLICATION p_ins") + r["initial sync"] = _wait(sub, "SELECT count(*) FROM src", "10") + _run(pub, "INSERT INTO src SELECT g, 'v'||g FROM generate_series(11,15) g") + r["streamed"] = _wait(sub, "SELECT count(*) FROM src", "15") + # the INSERT after the UPDATE proves the UPDATE before it was processed + _run(pub, "UPDATE src SET v = 'changed' WHERE id = 1") + _run(pub, "INSERT INTO src VALUES (16, 'after the update')") + sentinel = _wait(sub, "SELECT count(*) FROM src WHERE id = 16", "1") + + def after_sentinel(reading): + return reading if sentinel == "1" else \ + f"no reading (the INSERT after the UPDATE never arrived: [{sentinel}]; reading [{reading}])" + r["skipped"] = after_sentinel(_q(sub, "SELECT v FROM src WHERE id = 1")) + r["apply errors"] = after_sentinel( + _q(sub, "SELECT coalesce(sum(apply_error_count),0)::text FROM pg_stat_subscription_stats")) + + # ---- 2. the wedge: a publication carrying UPDATE ---- + _run(pub, "CREATE TABLE upd (id int PRIMARY KEY, v text)", + "INSERT INTO upd SELECT g, 'v'||g FROM generate_series(1,5) g", + "CREATE PUBLICATION p_all FOR TABLE upd") + _run(sub, "CREATE TABLE upd (id int PRIMARY KEY, v text) USING pgcolumnar") + _run(sub, f"CREATE SUBSCRIPTION s_all CONNECTION '{conninfo}' PUBLICATION p_all") + r["second synced"] = _wait(sub, "SELECT count(*) FROM upd", "5") + _run(pub, "UPDATE upd SET v = 'changed' WHERE id = 1") + for _ in range(25): + if isinstance(_log_has(log, REPL_ERR), int) and _log_has(log, REPL_ERR) >= 1: + break + time.sleep(1) + repl = _log_has(log, REPL_ERR) + r["names replication"] = _yes_if(repl, 1) + if not (isinstance(repl, int) and repl >= 1): + r["not only generic"] = "no reading (the subscriber log holds no replication error)" + else: + g = _log_has(log, GENERIC) + r["not only generic"] = "yes" if g == 0 else f"no ({g})" + r["hint"] = _yes_if(_log_has(log, "publish = "), 1) + r["wedge"] = f"{_q(pub, 'SELECT v FROM upd WHERE id = 1')} -> {_q(sub, 'SELECT v FROM upd WHERE id = 1')}" + for _ in range(20): + n = _log_has(log, REPL_ERR) + if isinstance(n, int) and n >= 2: + break + time.sleep(2) + r["retries"] = _yes_if(_log_has(log, REPL_ERR), 2) + + # ---- 3. the gate must not widen ---- + _run(pub, "CREATE TABLE lk (id int primary key, v text) USING pgcolumnar", + "INSERT INTO lk VALUES (1, 'a')") + r["lk am"] = _q(pub, "SELECT amname FROM pg_class c JOIN pg_am a ON a.oid=c.relam WHERE c.relname='lk'") + for mode in ("FOR UPDATE", "FOR SHARE", "FOR NO KEY UPDATE"): + r[mode] = _lock_readings(pub, mode) + + _run(sub, "DROP SUBSCRIPTION s_ins") + _run(sub, "DROP SUBSCRIPTION s_all") + yield r + finally: + for c in (sub, pub): + if c is not None: + c.stop() + shutil.rmtree(work, ignore_errors=True) + + +def _runnable(ls, expect): + if "no port" in ls: + expect.cannot_run("UNAVAILABLE_ENDPOINT", ls["no port"], name="the logical subscriber round trip") + return False + return True + + +def test_the_publisher_has_logical_decoding_on(ls, expect): + if _runnable(ls, expect): + expect.text(ls["wal_level"], "logical", "premise: the publisher has logical decoding on") + + +def test_the_subscriber_cluster_answers(ls, expect): + if _runnable(ls, expect): + expect.text(ls["sub answers"], "1", "premise: the subscriber cluster answers") + + +def test_the_subscriber_loaded_pgcolumnar(ls, expect): + if _runnable(ls, expect): + expect.text(ls["sub loaded"], "1", "premise: the subscriber loaded pgcolumnar") + + +def test_insert_only_initial_sync_copied_every_row(ls, expect): + if _runnable(ls, expect): + expect.text(ls["initial sync"], "10", "insert-only: initial sync copied every row") + + +def test_insert_only_streamed_inserts_applied(ls, expect): + if _runnable(ls, expect): + expect.text(ls["streamed"], "15", "insert-only: streamed INSERTs applied") + + +def test_insert_only_the_update_is_skipped_by_design(ls, expect): + if _runnable(ls, expect): + expect.text(ls["skipped"], "v1", "insert-only: the UPDATE is skipped by design, not applied") + + +def test_insert_only_no_apply_errors(ls, expect): + if _runnable(ls, expect): + expect.text(ls["apply errors"], "0", "insert-only: no apply errors") + + +def test_the_second_subscription_synced_before_the_update(ls, expect): + if _runnable(ls, expect): + expect.text(ls["second synced"], "5", "the second subscription synced before the UPDATE") + + +def test_the_apply_error_names_logical_replication(ls, expect): + if _runnable(ls, expect): + expect.text(ls["names replication"], "yes", "the error names logical replication, not row locking") + + +def test_the_apply_error_does_not_say_only_row_locking(ls, expect): + if _runnable(ls, expect): + expect.text(ls["not only generic"], "yes", "the error does NOT say only 'row locking is not supported'") + + +def test_the_apply_hint_names_the_insert_only_publication(ls, expect): + if _runnable(ls, expect): + expect.text(ls["hint"], "yes", "the hint names the insert-only publication") + + +def test_the_wedged_row_is_unchanged_on_the_subscriber(ls, expect): + if _runnable(ls, expect): + expect.text(ls["wedge"], "changed -> v1", "the wedge is real: the row is still unchanged on the subscriber") + + +def test_the_apply_worker_retries_so_the_subscription_is_wedged(ls, expect): + if _runnable(ls, expect): + expect.text(ls["retries"], "yes", "it retries rather than giving up, so the subscription is wedged") + + +def test_the_lock_fixture_is_columnar(ls, expect): + if _runnable(ls, expect): + expect.text(ls["lk am"], "pgcolumnar", "premise: the lock fixture really is columnar") + + +def test_select_for_update_keeps_the_generic_message(ls, expect): + if _runnable(ls, expect): + expect.text(str(ls["FOR UPDATE"][0]), "1", "an ordinary SELECT ... FOR UPDATE keeps the generic message") + + +def test_select_for_update_is_not_told_about_replication(ls, expect): + if _runnable(ls, expect): + expect.text(str(ls["FOR UPDATE"][1]), "0", + "an ordinary SELECT ... FOR UPDATE is not told about replication") + + +def test_select_for_share_keeps_the_generic_message(ls, expect): + if _runnable(ls, expect): + expect.text(str(ls["FOR SHARE"][0]), "1", "an ordinary SELECT ... FOR SHARE keeps the generic message") + + +def test_select_for_share_is_not_told_about_replication(ls, expect): + if _runnable(ls, expect): + expect.text(str(ls["FOR SHARE"][1]), "0", + "an ordinary SELECT ... FOR SHARE is not told about replication") + + +def test_select_for_no_key_update_keeps_the_generic_message(ls, expect): + if _runnable(ls, expect): + expect.text(str(ls["FOR NO KEY UPDATE"][0]), "1", + "an ordinary SELECT ... FOR NO KEY UPDATE keeps the generic message") + + +def test_select_for_no_key_update_is_not_told_about_replication(ls, expect): + if _runnable(ls, expect): + expect.text(str(ls["FOR NO KEY UPDATE"][1]), "0", + "an ordinary SELECT ... FOR NO KEY UPDATE is not told about replication") diff --git a/test/pytest/unported_suites.txt b/test/pytest/unported_suites.txt index 8749f6f3a..483e7ff52 100644 --- a/test/pytest/unported_suites.txt +++ b/test/pytest/unported_suites.txt @@ -24,7 +24,6 @@ iceberg_rest_scan iceberg_rest_server iceberg_rest_vended logical_decoding_cdc_recipe -logical_subscriber native_agg_deletes native_cancel native_cluster From 4d3ca68d8e3d25d9db95124e772486f600ae96e5 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 22:23:32 +0000 Subject: [PATCH 05/15] test: logical_decoding_cdc_recipe gains its pytest twin, and a rollback that undid nothing no longer passes (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The twin of test/logical_decoding_cdc_recipe.sh, every check by the shell's name, one check per test, on a private cluster at wal_level=logical. Like the shell, it extracts the CDC recipe from docs/user-guide.md at run time and executes it. Two holes, both measured on PG17 against main's shell, fixed in both harnesses: - The rollback arm read the capture count and the row only after ROLLBACK, so a rolled-back INSERT that never ran passed both arms. With it made a duplicate key, so the block stopped there, main passed 35/35. A reading inside the transaction, before the rollback, now requires the row both captured and present ("1:1"). Under the same mutation it is red in shell and twin. - "and it saw the same values the table holds" compared two readings with check. With the server stopped before it, "" = "" passed. It is check_text now. The missing-plugin pgc_skip name is a literal the twin records, and the seven recipe-contains premises are unrolled into literal names the grader can read. Five mutations date seven checks, the same names red in both twins on PG17: pgcolumnar_tuple_delete marking nothing (1), the update writing its new version with every column null (1), the update leaving the old version live (3), and two mutations of the guide's recipe: the trigger not firing on DELETE (2), and every non-DELETE captured as an INSERT (4). 36/36 shell and 37/37 twin on each of 15-19. Census 3907 -> 3936 never, 2370 -> 2377 dated, 6313 rows (the skip name has no ledger row: it is not a check); suites_not_covered 98 -> 97; cluster_tests 2861 -> 2898; TESTS.md §272. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 36 ++ test/check_ledger_budget.txt | 22 +- test/logical_decoding_cdc_recipe.sh | 44 +- test/pytest/TESTS.md | 49 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + .../test_logical_decoding_cdc_recipe.py | 400 ++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 542 insertions(+), 15 deletions(-) create mode 100644 test/pytest/test_logical_decoding_cdc_recipe.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 431b86e2f..fa43889f7 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2951,6 +2951,42 @@ local_open_race_free local_open_race_free the partition coordinator never stats local_open_race_free local_open_race_free the partition coordinator opens O_NONBLOCK (a FIFO cannot block its open) 15;16;17;18;19 2026-10-21 the partition coordinator opens without O_NONBLOCK local_open_race_free local_open_race_free the race-free opener exists 15;16;17;18;19 never - local_open_race_free local_open_race_free the racy stat-before-open helper is gone 15;16;17;18;19 2026-10-21 the racy PgColumnarRejectNonRegularFile helper comes back +logical_decoding_cdc_recipe logical_decoding_cdc_recipe CONTROL: the columnar table itself still emits nothing 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe a DELETE captures the OLD values, which are gone from the table 15;16;17;18;19 2026-10-21 docs/user-guide.md: the recipe's trigger fires on INSERT OR UPDATE only, not DELETE +logical_decoding_cdc_recipe logical_decoding_cdc_recipe a bulk load writes exactly one capture row per loaded row 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe a rolled-back transaction captures nothing 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe a row trigger on a COLUMNAR table fires once per inserted row 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe an UPDATE captures exactly one row 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_update leaves the old row version live +logical_decoding_cdc_recipe logical_decoding_cdc_recipe and it is recorded as UPDATE with the NEW value 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_update writes its new row version with every column null +logical_decoding_cdc_recipe logical_decoding_cdc_recipe and it saw the same values the table holds 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe and leaves no row in the columnar table either 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe and the DELETE as a DELETE 15;16;17;18;19 2026-10-21 docs/user-guide.md: the recipe's trigger fires on INSERT OR UPDATE only, not DELETE +logical_decoding_cdc_recipe logical_decoding_cdc_recipe and the rolled-back row is not in the stream 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe and the row really is gone from the columnar table 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_delete marks nothing deleted +logical_decoding_cdc_recipe logical_decoding_cdc_recipe inside the transaction the row was captured and is in the columnar table, before the rollback 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe nor the storage's internal INSERT 15;16;17;18;19 2026-10-21 docs/user-guide.md: the recipe's capture records every non-DELETE change as an INSERT +logical_decoding_cdc_recipe logical_decoding_cdc_recipe one decoded capture row per captured change 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: FOR ALL TABLES really does include pgcolumnar's internal tables 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: and it is the whole block, not a fragment 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: events is a COLUMNAR table 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: events_cdc is a HEAP table 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [CREATE FUNCTION events_capture] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [CREATE TABLE events] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [CREATE TABLE events_cdc] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [CREATE TRIGGER events_cdc_t] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [OLD.id] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [TG_OP = 'DELETE'] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extracted recipe contains [USING pgcolumnar] 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the extraction is not empty 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the guide is where this suite expects it 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the identical insert COMMITTED does capture, so the arm above is real 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the same matcher finds the INSERTs, so a 1 above is a real match 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the server is running at wal_level=logical 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the slot decoded something at all 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe premise: the slot exists and is logical 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe the capture table IS carried by the slot 15;16;17;18;19 never - +logical_decoding_cdc_recipe logical_decoding_cdc_recipe the decoded stream carries the UPDATE as an UPDATE 15;16;17;18;19 2026-10-21 columnar_tableam.c: pgcolumnar_tuple_update leaves the old row version live +logical_decoding_cdc_recipe logical_decoding_cdc_recipe the update did NOT leak the storage's internal DELETE 15;16;17;18;19 never - logical_decoding_source logical_decoding_source 100x the rows does not give 100x the internal records 15;16;17;18;19 never - logical_decoding_source logical_decoding_source CONTROL: the heap table's 50 inserts ARE decoded 15;16;17;18;19 never - logical_decoding_source logical_decoding_source CONTROL: the second slot decodes the heap table's 5000 inserts 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 5b34b545e..122207061 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 98 +suites_not_covered 97 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5518,4 +5518,22 @@ suites_not_covered 98 # never 3903 + 4 = 3907 # dated 2354 + 16 = 2370 # 3907 + 2370 == 6277 -checks_never_observed_red 3907 +# +# #1341, logical_decoding_cdc_recipe seeded, with its twin (OffgridwithJD): 3907 -> 3936, +# and suites_not_covered 98 -> 97. The suite had no rows. It enters with its 36 checks from five-major +# runs (all green on each of 15-19, shell and twin alike), after two fixes. The +# missing-plugin name pgc_skip records is a skip, not a check, so it has no row: CI has +# test_decoding and never emits it. The rollback arm read the capture count +# and the row after ROLLBACK only, so a rolled-back INSERT that never ran passed both arms: +# with it made a duplicate key, main passed 35/35. A reading inside the transaction, before +# the rollback, now requires the row captured and present. And "it saw the same values the +# table holds" compared two readings with check, so with the server gone "" = "" passed; it +# is check_text. The recipe-contains premises are unrolled into literal names. Three +# table-AM mutations and two mutations of the guide's own recipe date checks, each red by the +# same names in both twins. 7 are dated. 29 arrive never. +# +# ledger rows 6277 + 36 = 6313 +# never 3907 + 29 = 3936 +# dated 2370 + 7 = 2377 +# 3936 + 2377 == 6313 +checks_never_observed_red 3936 diff --git a/test/logical_decoding_cdc_recipe.sh b/test/logical_decoding_cdc_recipe.sh index 830e0222b..4b7dea0c4 100755 --- a/test/logical_decoding_cdc_recipe.sh +++ b/test/logical_decoding_cdc_recipe.sh @@ -47,7 +47,8 @@ max_wal_senders=8" pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" if [ ! -f "$("$PGC_PG_CONFIG" --pkglibdir)/test_decoding.so" ]; then - pgc_skip "test_decoding" "test_decoding.so is not in $("$PGC_PG_CONFIG" --pkglibdir); build contrib/test_decoding" + # The message is the check's name, so it is a literal: the twin records the same one. + pgc_skip "test_decoding" "test_decoding.so is not installed; build contrib/test_decoding" fi check "premise: the server is running at wal_level=logical" \ @@ -81,12 +82,21 @@ printf '%s\n' "$RECIPE" > "$RECIPE_FILE" # produced nothing. check "premise: the extraction is not empty" \ "$([ -n "$RECIPE" ] && echo nonempty || echo EMPTY)" "nonempty" -for want in "CREATE TABLE events" "CREATE TABLE events_cdc" \ - "CREATE FUNCTION events_capture" "CREATE TRIGGER events_cdc_t" \ - "USING pgcolumnar" "TG_OP = 'DELETE'" "OLD.id"; do - check "premise: the extracted recipe contains [$want]" \ - "$(grep -cF "$want" "$RECIPE_FILE" | awk '{print ($1>0)?"yes":"no"}')" "yes" -done +# Unrolled so each name is a literal the twin's grader can read. +check "premise: the extracted recipe contains [CREATE TABLE events]" \ + "$(grep -qF "CREATE TABLE events" "$RECIPE_FILE" && echo yes || echo no)" "yes" +check "premise: the extracted recipe contains [CREATE TABLE events_cdc]" \ + "$(grep -qF "CREATE TABLE events_cdc" "$RECIPE_FILE" && echo yes || echo no)" "yes" +check "premise: the extracted recipe contains [CREATE FUNCTION events_capture]" \ + "$(grep -qF "CREATE FUNCTION events_capture" "$RECIPE_FILE" && echo yes || echo no)" "yes" +check "premise: the extracted recipe contains [CREATE TRIGGER events_cdc_t]" \ + "$(grep -qF "CREATE TRIGGER events_cdc_t" "$RECIPE_FILE" && echo yes || echo no)" "yes" +check "premise: the extracted recipe contains [USING pgcolumnar]" \ + "$(grep -qF "USING pgcolumnar" "$RECIPE_FILE" && echo yes || echo no)" "yes" +check "premise: the extracted recipe contains [TG_OP = 'DELETE']" \ + "$(grep -qF "TG_OP = 'DELETE'" "$RECIPE_FILE" && echo yes || echo no)" "yes" +check "premise: the extracted recipe contains [OLD.id]" \ + "$(grep -qF "OLD.id" "$RECIPE_FILE" && echo yes || echo no)" "yes" check "premise: and it is the whole block, not a fragment" \ "$([ "$(grep -c ';' "$RECIPE_FILE")" -ge 4 ] && echo yes || echo no)" "yes" @@ -111,7 +121,9 @@ check "premise: the slot exists and is logical" \ psql_run "INSERT INTO events VALUES (1,'sale',100), (2,'refund',50), (3,'sale',75);" check_num "a row trigger on a COLUMNAR table fires once per inserted row" \ "$(q "SELECT count(*) FROM events_cdc WHERE op = 'INSERT';")" "3" -check "and it saw the same values the table holds" \ +# check_text, not check: with the server gone both readings are empty, and "" = "" +# passed this arm (#418). +check_text "and it saw the same values the table holds" \ "$(q "SELECT string_agg(id||':'||kind||':'||amount, ',' ORDER BY id) FROM events_cdc WHERE op='INSERT';")" \ "$(q "SELECT string_agg(id||':'||kind||':'||amount, ',' ORDER BY id) FROM events;")" @@ -142,9 +154,19 @@ check_num "and the row really is gone from the columnar table" \ # ---- 3. the capture is transactional ----------------------------------------- CAP_BEFORE="$(q "SELECT count(*) FROM events_cdc;")" -psql_run "BEGIN; - INSERT INTO events VALUES (99,'rolledback',1); - ROLLBACK;" +# Read INSIDE the transaction before rolling it back, so the two arms below have +# something to undo. Without this reading, a rolled-back INSERT that never ran +# (it failed, or was never sent) left both arms green: nothing captured, no row. +# Measured: with the INSERT made a duplicate key, so the block stopped at it, the +# suite passed 35/35. +INSIDE="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ + -d "$PGC_DB" -qAt -v ON_ERROR_STOP=1 -c "BEGIN;" \ + -c "INSERT INTO events VALUES (99,'rolledback',1);" \ + -c "SELECT ((SELECT count(*) FROM events_cdc) - ${CAP_BEFORE:-0}) || ':' || + (SELECT count(*) FROM events WHERE id = 99);" \ + -c "ROLLBACK;" 2>&1)" +check "inside the transaction the row was captured and is in the columnar table, before the rollback" \ + "$INSIDE" "1:1" check_num "a rolled-back transaction captures nothing" \ "$(q "SELECT count(*) FROM events_cdc;")" "$CAP_BEFORE" check_num "and leaves no row in the columnar table either" \ diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index d650153f3..b98a776b7 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -317,6 +317,7 @@ behaviour, the source of that number is named. - [269. test_logical_decoding_source.py: a columnar table is not a logical decoding source](#269-test_logical_decoding_sourcepy-a-columnar-table-is-not-a-logical-decoding-source) - [270. test_iceberg_deletes.py: Iceberg position and equality deletes apply, scope, and refuse what they must](#270-test_iceberg_deletespy-iceberg-position-and-equality-deletes-apply-scope-and-refuse-what-they-must) - [271. test_logical_subscriber.py: columnar as a logical replication apply target](#271-test_logical_subscriberpy-columnar-as-a-logical-replication-apply-target) +- [272. test_logical_decoding_cdc_recipe.py: the documented CDC recipe captures, transacts and decodes](#272-test_logical_decoding_cdc_recipepy-the-documented-cdc-recipe-captures-transacts-and-decodes) ## 1. How to read a test in here @@ -10552,3 +10553,51 @@ One check per test, each by the shell's name, every reading a value from a modul | `test_select_for_share_is_not_told_about_replication` | an ordinary SELECT ... FOR SHARE is not told about replication | | `test_select_for_no_key_update_keeps_the_generic_message` | an ordinary SELECT ... FOR NO KEY UPDATE keeps the generic message | | `test_select_for_no_key_update_is_not_told_about_replication` | an ordinary SELECT ... FOR NO KEY UPDATE is not told about replication | + +## 272. test_logical_decoding_cdc_recipe.py: the documented CDC recipe captures, transacts and decodes + +The twin of `test/logical_decoding_cdc_recipe.sh` (#754). The CDC recipe is extracted from docs/user-guide.md at run time and executed on a private cluster at `wal_level=logical`: a row trigger on a columnar table fires once per row and sees the table's values; an UPDATE is captured as one UPDATE, never as the storage's delete and insert; a DELETE captures the OLD row; the capture is transactional (read inside the transaction before the rollback, so the rollback has something to undo); the capture table decodes through a test_decoding slot while the columnar table emits nothing; FOR ALL TABLES includes pgcolumnar's internal tables; and a bulk load writes one capture row per loaded row. + +### Every test + +One check per test, each by the shell's name, every reading a value (a failed query is a `query_error()` that fails its own check) from a module fixture on a private cluster. Without the test_decoding library both harnesses fail one check, the shell's `pgc_skip` name, and the twin's other tests decline MISSING_DEPENDENCY. + +| test | what it holds | +| --- | --- | +| `test_ldcr_test_decoding_is_installed` | test_decoding.so is not installed; build contrib/test_decoding | +| `test_ldcr_server_runs_at_wal_level_logical` | premise: the server is running at wal_level=logical | +| `test_ldcr_guide_is_where_expected` | premise: the guide is where this suite expects it | +| `test_ldcr_extraction_is_not_empty` | premise: the extraction is not empty | +| `test_ldcr_recipe_contains_events` | premise: the extracted recipe contains [CREATE TABLE events] | +| `test_ldcr_recipe_contains_events_cdc` | premise: the extracted recipe contains [CREATE TABLE events_cdc] | +| `test_ldcr_recipe_contains_capture_function` | premise: the extracted recipe contains [CREATE FUNCTION events_capture] | +| `test_ldcr_recipe_contains_trigger` | premise: the extracted recipe contains [CREATE TRIGGER events_cdc_t] | +| `test_ldcr_recipe_contains_using_pgcolumnar` | premise: the extracted recipe contains [USING pgcolumnar] | +| `test_ldcr_recipe_contains_delete_branch` | premise: the extracted recipe contains [TG_OP = 'DELETE'] | +| `test_ldcr_recipe_contains_old_id` | premise: the extracted recipe contains [OLD.id] | +| `test_ldcr_recipe_is_the_whole_block` | premise: and it is the whole block, not a fragment | +| `test_ldcr_events_is_columnar` | premise: events is a COLUMNAR table | +| `test_ldcr_events_cdc_is_heap` | premise: events_cdc is a HEAP table | +| `test_ldcr_slot_exists_and_is_logical` | premise: the slot exists and is logical | +| `test_ldcr_trigger_fires_once_per_row` | a row trigger on a COLUMNAR table fires once per inserted row | +| `test_ldcr_trigger_saw_the_table_values` | and it saw the same values the table holds | +| `test_ldcr_update_captures_one_row` | an UPDATE captures exactly one row | +| `test_ldcr_update_recorded_with_new_value` | and it is recorded as UPDATE with the NEW value | +| `test_ldcr_update_leaks_no_delete` | the update did NOT leak the storage's internal DELETE | +| `test_ldcr_update_leaks_no_insert` | nor the storage's internal INSERT | +| `test_ldcr_delete_captures_old_values` | a DELETE captures the OLD values, which are gone from the table | +| `test_ldcr_deleted_row_is_gone` | and the row really is gone from the columnar table | +| `test_ldcr_inside_the_transaction_the_row_was_captured` | inside the transaction the row was captured and is in the columnar table, before the rollback | +| `test_ldcr_rollback_captures_nothing` | a rolled-back transaction captures nothing | +| `test_ldcr_rollback_leaves_no_row` | and leaves no row in the columnar table either | +| `test_ldcr_committed_insert_captures` | premise: the identical insert COMMITTED does capture, so the arm above is real | +| `test_ldcr_slot_decoded_something` | premise: the slot decoded something at all | +| `test_ldcr_columnar_table_emits_nothing` | CONTROL: the columnar table itself still emits nothing | +| `test_ldcr_capture_table_is_carried` | the capture table IS carried by the slot | +| `test_ldcr_one_decoded_row_per_capture` | one decoded capture row per captured change | +| `test_ldcr_rolled_back_row_not_in_stream` | and the rolled-back row is not in the stream | +| `test_ldcr_stream_carries_update` | the decoded stream carries the UPDATE as an UPDATE | +| `test_ldcr_stream_carries_delete` | and the DELETE as a DELETE | +| `test_ldcr_matcher_finds_inserts` | premise: the same matcher finds the INSERTs, so a 1 above is a real match | +| `test_ldcr_publication_includes_internal_tables` | premise: FOR ALL TABLES really does include pgcolumnar's internal tables | +| `test_ldcr_bulk_load_one_capture_per_row` | a bulk load writes exactly one capture row per loaded row | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 7fd6669d8..ff47042be 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -914,6 +914,8 @@ guard_tests 476 # 49 checks in 49 tests, by collection over the cluster files. # cluster_tests 2841 -> 2861 for #1341's test_logical_subscriber.py, the twin of logical_subscriber.sh: # 20 checks in 20 tests, by collection over the cluster files. +# cluster_tests 2861 -> 2898 for #1341's test_logical_decoding_cdc_recipe.py, the twin of logical_decoding_cdc_recipe.sh: +# 37 checks in 37 tests, by collection over the cluster files. # cluster_tests 1264 -> 1287 for #1341's test_vector_agg_rescan_memory.py, the twin of vector_agg_rescan_memory.sh: # 23 tests, one per check, by collection over the cluster files. # cluster_tests 1287 -> 1329 for #1341's test_analyze_stats.py, the twin of analyze_stats.sh: @@ -1045,4 +1047,4 @@ guard_tests 476 # cluster_tests 2670 -> 2677 for #1637's test_catalog_filter_reads.py, the twin of catalog_filter_reads.sh: # 7 tests, one per check, by collection over the cluster files. # cluster_tests 2748 -> 2753 for #1706's five timestamp-flag tests in test_native_parquet_schema.py, by collection. -cluster_tests 2861 +cluster_tests 2898 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 1a182819d..74197946b 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -143,6 +143,7 @@ "local_open_race_free", "logical_decoding_source", "logical_subscriber", + "logical_decoding_cdc_recipe", "sorted_mark_rename", "tablesample", "ungrouped_vector_agg", diff --git a/test/pytest/test_logical_decoding_cdc_recipe.py b/test/pytest/test_logical_decoding_cdc_recipe.py new file mode 100644 index 000000000..6a44a6c9d --- /dev/null +++ b/test/pytest/test_logical_decoding_cdc_recipe.py @@ -0,0 +1,400 @@ +"""The documented CDC recipe works (#754). The pytest twin of +`test/logical_decoding_cdc_recipe.sh`, every check by the shell's name. + +docs/user-guide.md offers a workaround for "a columnar table is not a logical decoding +source": capture rows into a heap table with a row trigger, and decode that. The +recipe's SQL is EXTRACTED FROM THE GUIDE AT RUN TIME and executed, so the guide is the +thing under test; premises assert what the extraction contains, not merely that it ran. + +Asserted: the trigger fires on a columnar table and sees the table's values; an UPDATE +is captured as one UPDATE, never as the storage's delete and insert; a DELETE captures +the OLD row; the capture is transactional, read inside the transaction before it is +rolled back so the rollback has something to undo; the capture table decodes through a +slot while the columnar table emits nothing; a publication FOR ALL TABLES includes +pgcolumnar's internal tables; and a bulk load writes one capture row per loaded row. + +A PRIVATE CLUSTER at wal_level=logical: the level cannot be raised without a restart, +and the shared cluster runs at the default. +""" +import pathlib +import re +import shutil +import tempfile + +import psycopg +import pytest + +from pgc_vacuity import query_error + +GUIDE = pathlib.Path(__file__).resolve().parents[2] / "docs" / "user-guide.md" +SLOT = "pgc_cdc" + + +def _err(exc): + return query_error(str(exc).splitlines()[0] if str(exc) else type(exc).__name__) + + +def _q(dsn, sql, params=None): + """-> the first column of the first row as text, or query_error() carrying the failure.""" + try: + with psycopg.connect(dsn, autocommit=True) as c: + row = c.execute(sql, params).fetchone() + return "" if row is None or row[0] is None else str(row[0]) + except psycopg.Error as exc: + return _err(exc) + + +def _run(dsn, sql): + """-> None, or query_error() when the statement failed (the next reading then fails by name).""" + try: + with psycopg.connect(dsn, autocommit=True) as c: + c.execute(sql) + return None + except psycopg.Error as exc: + return _err(exc) + + +def _int(v): + return int(v) if isinstance(v, str) and re.fullmatch(r"-?\d+", v) else v + + +def _extract(text): + """The first ```sql fence inside the CDC section, as the shell's awk reads it.""" + out, in_sec, in_sql, seen = [], False, False, False + for line in text.splitlines(): + if not in_sec: + in_sec = line.startswith("## Capture changes for replication or CDC") + continue + if not in_sql and line.startswith("## "): + break + if not in_sql and not seen and line == "```sql": + in_sql = seen = True + continue + if in_sql and line == "```": + break + if in_sql: + out.append(line) + return "\n".join(out) + + +@pytest.fixture(scope="module") +def cdc(pgc_cluster): + from pgc_cluster import Cluster, pick_port + + if not (pathlib.Path(pgc_cluster.libdir) / "test_decoding.so").is_file(): + yield {"missing": f"test_decoding.so is not in {pgc_cluster.libdir}"} + return + r = {"guide": "found" if GUIDE.is_file() else f"missing ({GUIDE})"} + recipe = _extract(GUIDE.read_text()) if GUIDE.is_file() else "" + r["recipe"] = recipe + work = pathlib.Path(tempfile.mkdtemp(prefix="pgc-ldcr-")) + work.chmod(0o777) + datadir = work / "data" + datadir.mkdir() + cluster = Cluster(pgc_cluster.pg_config, "ldcr", datadir, pick_port(27)) + try: + cluster.initdb() + with open(datadir / "postgresql.conf", "a") as fh: + fh.write("\nwal_level=logical\nmax_replication_slots=8\nmax_wal_senders=8\n") + cluster.start() + dsn = cluster.dsn() + _run(dsn, "CREATE EXTENSION IF NOT EXISTS pgcolumnar") + r["wal_level"] = _q(dsn, "SHOW wal_level") + # The guide's own SQL, as the shell's psql -f runs it: statement errors do not stop it. + if recipe: + _run(dsn, recipe) + am = "SELECT am.amname FROM pg_class c JOIN pg_am am ON am.oid = c.relam WHERE c.relname = %s" + r["events am"] = _q(dsn, am, ("events",)) + r["cdc am"] = _q(dsn, am, ("events_cdc",)) + _run(dsn, f"SELECT pg_create_logical_replication_slot('{SLOT}', 'test_decoding')") + r["slot type"] = _q(dsn, f"SELECT slot_type FROM pg_replication_slots WHERE slot_name = '{SLOT}'") + + def n(sql): + return _int(_q(dsn, sql)) + + # 1. the trigger fires on a columnar table + _run(dsn, "INSERT INTO events VALUES (1,'sale',100), (2,'refund',50), (3,'sale',75)") + r["inserted"] = n("SELECT count(*) FROM events_cdc WHERE op = 'INSERT'") + r["seen"] = _q(dsn, "SELECT string_agg(id||':'||kind||':'||amount, ',' ORDER BY id) " + "FROM events_cdc WHERE op='INSERT'") + r["held"] = _q(dsn, "SELECT string_agg(id||':'||kind||':'||amount, ',' ORDER BY id) FROM events") + # 2. an UPDATE arrives as one UPDATE + before_ins = n("SELECT count(*) FROM events_cdc WHERE op='INSERT'") + before_del = n("SELECT count(*) FROM events_cdc WHERE op='DELETE'") + _run(dsn, "UPDATE events SET amount = 150 WHERE id = 1") + r["updates"] = n("SELECT count(*) FROM events_cdc WHERE op='UPDATE'") + r["update row"] = _q(dsn, "SELECT op||':'||id||':'||amount FROM events_cdc WHERE op='UPDATE'") + r["dels after update"] = (n("SELECT count(*) FROM events_cdc WHERE op='DELETE'"), before_del) + r["ins after update"] = (n("SELECT count(*) FROM events_cdc WHERE op='INSERT'"), before_ins) + # a DELETE captures the OLD row + _run(dsn, "DELETE FROM events WHERE id = 2") + r["delete row"] = _q(dsn, "SELECT op||':'||id||':'||kind||':'||amount FROM events_cdc WHERE op='DELETE'") + r["gone"] = n("SELECT count(*) FROM events WHERE id = 2") + # 3. the capture is transactional: read inside the transaction, then roll back + cap_before = n("SELECT count(*) FROM events_cdc") + try: + with psycopg.connect(dsn) as c: + c.execute("INSERT INTO events VALUES (99,'rolledback',1)") + row = c.execute("SELECT ((SELECT count(*) FROM events_cdc) - %s) || ':' || " + "(SELECT count(*) FROM events WHERE id = 99)", + (cap_before if isinstance(cap_before, int) else 0,)).fetchone() + r["inside"] = str(row[0]) + c.rollback() + except psycopg.Error as exc: + r["inside"] = _err(exc) + r["after rollback"] = (n("SELECT count(*) FROM events_cdc"), cap_before) + r["row 99"] = n("SELECT count(*) FROM events WHERE id = 99") + _run(dsn, "INSERT INTO events VALUES (98,'committed',1)") + r["after commit"] = (n("SELECT count(*) FROM events_cdc"), + cap_before + 1 if isinstance(cap_before, int) else cap_before) + # 4. the capture table decodes, and the columnar table still does not + _run(dsn, f"CREATE TABLE decoded AS SELECT data FROM pg_logical_slot_get_changes('{SLOT}', NULL, NULL)") + + def d(pattern): # LIKE, as the shell's d() + return _int(_q(dsn, "SELECT count(*) FROM decoded WHERE data LIKE %s", (f"%{pattern}%",))) + + def dq(text): # strpos, as the shell's dq() + return _int(_q(dsn, "SELECT count(*) FROM decoded WHERE strpos(data, %s) > 0", (text,))) + + r["decoded"] = n("SELECT count(*) FROM decoded") + r["col decoded"] = d("table public.events:") + r["cdc decoded"] = d("table public.events_cdc: INSERT") + r["captured"] = n("SELECT count(*) FROM events_cdc") + r["rolledback decoded"] = d("rolledback") + r["op update"] = dq("op[text]:'UPDATE'") + r["op delete"] = dq("op[text]:'DELETE'") + r["op insert"] = dq("op[text]:'INSERT'") + # the guide's warning about the pgcolumnar schema + _run(dsn, "CREATE PUBLICATION pgc_all FOR ALL TABLES") + r["pub internal"] = n("SELECT count(*) FROM pg_publication_tables " + "WHERE pubname='pgc_all' AND schemaname='pgcolumnar'") + _run(dsn, "DROP PUBLICATION pgc_all") + # the guide's cost claim + bulk_before = n("SELECT count(*) FROM events_cdc") + _run(dsn, "INSERT INTO events SELECT g, 'bulk', g FROM generate_series(1000, 6000) g") + bulk_after = n("SELECT count(*) FROM events_cdc") + r["bulk"] = (bulk_after - bulk_before if isinstance(bulk_after, int) and isinstance(bulk_before, int) + else f"no reading ({bulk_before} then {bulk_after})") + _run(dsn, f"SELECT pg_drop_replication_slot('{SLOT}')") + yield r + finally: + cluster.stop() + shutil.rmtree(work, ignore_errors=True) + + +def test_ldcr_test_decoding_is_installed(cdc, expect): + # The shell's pgc_skip records a FAIL under this name when the plugin is missing, and + # nothing when it is there. Here the same name passes when it is there. + expect.text("missing" if "missing" in cdc else "installed", "installed", + "test_decoding.so is not installed; build contrib/test_decoding") + + +def _declined(cdc, expect): + if "missing" in cdc: + expect.cannot_run("MISSING_DEPENDENCY", cdc["missing"]) + return True + return False + + +def _has(cdc, text): + return "yes" if text in cdc["recipe"] else "no" + + +def _gt(v, floor): + return "yes" if isinstance(v, int) and v > floor else f"no ({v})" + + +def test_ldcr_server_runs_at_wal_level_logical(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["wal_level"], "logical", "premise: the server is running at wal_level=logical") + + +def test_ldcr_guide_is_where_expected(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["guide"], "found", "premise: the guide is where this suite expects it") + + +def test_ldcr_extraction_is_not_empty(cdc, expect): + if not _declined(cdc, expect): + expect.text("nonempty" if cdc["recipe"] else "EMPTY", "nonempty", "premise: the extraction is not empty") + + +def test_ldcr_recipe_contains_events(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "CREATE TABLE events"), "yes", + "premise: the extracted recipe contains [CREATE TABLE events]") + + +def test_ldcr_recipe_contains_events_cdc(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "CREATE TABLE events_cdc"), "yes", + "premise: the extracted recipe contains [CREATE TABLE events_cdc]") + + +def test_ldcr_recipe_contains_capture_function(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "CREATE FUNCTION events_capture"), "yes", + "premise: the extracted recipe contains [CREATE FUNCTION events_capture]") + + +def test_ldcr_recipe_contains_trigger(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "CREATE TRIGGER events_cdc_t"), "yes", + "premise: the extracted recipe contains [CREATE TRIGGER events_cdc_t]") + + +def test_ldcr_recipe_contains_using_pgcolumnar(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "USING pgcolumnar"), "yes", + "premise: the extracted recipe contains [USING pgcolumnar]") + + +def test_ldcr_recipe_contains_delete_branch(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "TG_OP = 'DELETE'"), "yes", + "premise: the extracted recipe contains [TG_OP = 'DELETE']") + + +def test_ldcr_recipe_contains_old_id(cdc, expect): + if not _declined(cdc, expect): + expect.text(_has(cdc, "OLD.id"), "yes", "premise: the extracted recipe contains [OLD.id]") + + +def test_ldcr_recipe_is_the_whole_block(cdc, expect): + if not _declined(cdc, expect): + n = cdc["recipe"].count(";") + expect.text("yes" if n >= 4 else f"no ({n} statements)", "yes", + "premise: and it is the whole block, not a fragment") + + +def test_ldcr_events_is_columnar(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["events am"], "pgcolumnar", "premise: events is a COLUMNAR table") + + +def test_ldcr_events_cdc_is_heap(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["cdc am"], "heap", "premise: events_cdc is a HEAP table") + + +def test_ldcr_slot_exists_and_is_logical(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["slot type"], "logical", "premise: the slot exists and is logical") + + +def test_ldcr_trigger_fires_once_per_row(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["inserted"], 3, "a row trigger on a COLUMNAR table fires once per inserted row") + + +def test_ldcr_trigger_saw_the_table_values(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["seen"], cdc["held"], "and it saw the same values the table holds") + + +def test_ldcr_update_captures_one_row(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["updates"], 1, "an UPDATE captures exactly one row") + + +def test_ldcr_update_recorded_with_new_value(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["update row"], "UPDATE:1:150", "and it is recorded as UPDATE with the NEW value") + + +def test_ldcr_update_leaks_no_delete(cdc, expect): + if not _declined(cdc, expect): + got, want = cdc["dels after update"] + expect.num(got, want, "the update did NOT leak the storage's internal DELETE") + + +def test_ldcr_update_leaks_no_insert(cdc, expect): + if not _declined(cdc, expect): + got, want = cdc["ins after update"] + expect.num(got, want, "nor the storage's internal INSERT") + + +def test_ldcr_delete_captures_old_values(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["delete row"], "DELETE:2:refund:50", + "a DELETE captures the OLD values, which are gone from the table") + + +def test_ldcr_deleted_row_is_gone(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["gone"], 0, "and the row really is gone from the columnar table") + + +def test_ldcr_inside_the_transaction_the_row_was_captured(cdc, expect): + if not _declined(cdc, expect): + expect.text(cdc["inside"], "1:1", + "inside the transaction the row was captured and is in the columnar table, before the rollback") + + +def test_ldcr_rollback_captures_nothing(cdc, expect): + if not _declined(cdc, expect): + got, want = cdc["after rollback"] + expect.num(got, want, "a rolled-back transaction captures nothing") + + +def test_ldcr_rollback_leaves_no_row(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["row 99"], 0, "and leaves no row in the columnar table either") + + +def test_ldcr_committed_insert_captures(cdc, expect): + if not _declined(cdc, expect): + got, want = cdc["after commit"] + expect.num(got, want, "premise: the identical insert COMMITTED does capture, so the arm above is real") + + +def test_ldcr_slot_decoded_something(cdc, expect): + if not _declined(cdc, expect): + expect.text(_gt(cdc["decoded"], 0), "yes", "premise: the slot decoded something at all") + + +def test_ldcr_columnar_table_emits_nothing(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["col decoded"], 0, "CONTROL: the columnar table itself still emits nothing") + + +def test_ldcr_capture_table_is_carried(cdc, expect): + if not _declined(cdc, expect): + expect.text(_gt(cdc["cdc decoded"], 0), "yes", "the capture table IS carried by the slot") + + +def test_ldcr_one_decoded_row_per_capture(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["cdc decoded"], cdc["captured"], "one decoded capture row per captured change") + + +def test_ldcr_rolled_back_row_not_in_stream(cdc, expect): + if not _declined(cdc, expect): + expect.text(str(cdc["rolledback decoded"]), "0", "and the rolled-back row is not in the stream") + + +def test_ldcr_stream_carries_update(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["op update"], 1, "the decoded stream carries the UPDATE as an UPDATE") + + +def test_ldcr_stream_carries_delete(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["op delete"], 1, "and the DELETE as a DELETE") + + +def test_ldcr_matcher_finds_inserts(cdc, expect): + if not _declined(cdc, expect): + expect.text(_gt(cdc["op insert"], 1), "yes", + "premise: the same matcher finds the INSERTs, so a 1 above is a real match") + + +def test_ldcr_publication_includes_internal_tables(cdc, expect): + if not _declined(cdc, expect): + expect.text(_gt(cdc["pub internal"], 0), "yes", + "premise: FOR ALL TABLES really does include pgcolumnar's internal tables") + + +def test_ldcr_bulk_load_one_capture_per_row(cdc, expect): + if not _declined(cdc, expect): + expect.num(cdc["bulk"], 5001, "a bulk load writes exactly one capture row per loaded row") diff --git a/test/pytest/unported_suites.txt b/test/pytest/unported_suites.txt index 483e7ff52..1862624e6 100644 --- a/test/pytest/unported_suites.txt +++ b/test/pytest/unported_suites.txt @@ -23,7 +23,6 @@ iceberg_rest iceberg_rest_scan iceberg_rest_server iceberg_rest_vended -logical_decoding_cdc_recipe native_agg_deletes native_cancel native_cluster From dd70f3652d1f2833fda7253789ab710a7f2daea1 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 22:34:42 +0000 Subject: [PATCH 06/15] test: iceberg_rest gains its pytest twin, and a refusal must be its own guard's (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The twin of test/iceberg_rest.sh, every check by the shell's name, one check per test. It runs a private cluster restarted with the bearer token in the postmaster environment (a good one, a wrong one, none) against the shell's own REST fixture. Two holes in the shell, each measured red-first on PG17 and fixed in both harnesses: - The 42501 arms read the SQLSTATE alone, and two guards raise 42501. With the allow-list refusing everything and the link-local refusal gone (denyall_noll), "a link-local catalog host is refused even if allow-listed" passed on the allow-list's 42501; with every address read as link-local and the allow-list gone (llall_noallow), "a catalog endpoint off the allow-list is refused" passed on the link-local guard's. Every refusal now reads its own guard's message. - "backend still up after the REST refusals" read SELECT 1, which answers again after crash recovery. With the backend aborting on the catalog's 404 (crash404), main passed it. It now also reads the server log's count of killed backends. Mutants (PG17), SAME reds in shell and twin: tokenlog 1, tokenurl 2, no401 2, no404 1, nocfgobj 1, nocap 1, crash404 2, noauth 4, denyall_noll 8, llall_noallow 8, denyall 8. Clean 14/14 in both on each of 15-19. Seeded: 14 rows, 12 dated, 2 never (the premises). Census 3936 -> 3938 never, 2377 -> 2389 dated, 6327 rows; suites_not_covered 97 -> 96; cluster_tests 2898 -> 2912; TESTS.md §273. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 14 ++ test/check_ledger_budget.txt | 22 +- test/iceberg_rest.sh | 86 +++++-- test/pytest/TESTS.md | 26 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_iceberg_rest.py | 344 ++++++++++++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 470 insertions(+), 28 deletions(-) create mode 100644 test/pytest/test_iceberg_rest.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index fa43889f7..c40be3cf5 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2711,6 +2711,20 @@ iceberg_objstore iceberg_objstore name mapping reads an id-less file over s3 (5 iceberg_objstore iceberg_objstore position deletes read over s3 (surviving == filesystem oracle) 15;16;17;18;19 2026-10-21 columnar_objstore_module.c: os_check_endpoint_allowed refuses every endpoint, the listed one included iceberg_objstore iceberg_objstore premise: sigv4 fixture server is up 15;16;17;18;19 never - iceberg_objstore iceberg_objstore the position-deleted ids are gone over s3 15;16;17;18;19 2026-10-21 columnar_objstore_module.c: os_check_endpoint_allowed refuses every endpoint, the listed one included +iceberg_rest iceberg_rest a catalog endpoint off the allow-list is refused (42501) 15;16;17;18;19 2026-10-21 objstore/columnar_objstore_module.c: every address reads as link-local and the allow-list is gone +iceberg_rest iceberg_rest a link-local catalog host is refused even if allow-listed (42501) 15;16;17;18;19 2026-10-21 objstore/columnar_objstore_module.c: the allow-list refuses every endpoint and the link-local refusal is gone +iceberg_rest iceberg_rest a non-object catalog config is refused, not walked (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: a non-object /v1/config body is not refused +iceberg_rest iceberg_rest a wrong bearer token is refused (28000) 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: a catalog 401/403 is not mapped to 28000 +iceberg_rest iceberg_rest an anonymous request to a token-protected catalog is refused (28000) 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: a catalog 401/403 is not mapped to 28000 +iceberg_rest iceberg_rest an over-cap catalog response is refused (54000) 15;16;17;18;19 2026-10-21 objstore/columnar_objstore_module.c: os_slurp_body does not refuse an advertised length over the cap +iceberg_rest iceberg_rest an unknown table surfaces a clean not-found (42P01) 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: a catalog 404 is not mapped to 42P01 +iceberg_rest iceberg_rest backend still up after the REST refusals 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: the backend aborts on a catalog 404 (abort() before the 42P01 refusal) +iceberg_rest iceberg_rest loadTable resolves the current metadata-location 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: no Authorization header is ever sent +iceberg_rest iceberg_rest premise: rest fixture server is up 15;16;17;18;19 never - +iceberg_rest iceberg_rest premise: the PG server log holds lines to search 15;16;17;18;19 never - +iceberg_rest iceberg_rest the loadTable request carried an Authorization header 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: the bearer token also travels in the request URL's query string +iceberg_rest iceberg_rest the token never appears in the server (PG) log 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: rest_get_json logs the bearer token (elog LOG) +iceberg_rest iceberg_rest the token value never appears in the catalog request log 15;16;17;18;19 2026-10-21 columnar_iceberg_rest.c: the bearer token also travels in the request URL's query string iceberg_rest_server iceberg_rest_server a catalog_uri option on the USER MAPPING is rejected (HV00D) 15;16;17;18;19 never - iceberg_rest_server iceberg_rest_server a half OAuth2 credential (id, no secret) is refused before any request (28000) 15;16;17;18;19 never - iceberg_rest_server iceberg_rest_server a link-local oauth_token_uri is refused by the allow-list (42501) 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 122207061..1980c7293 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 97 +suites_not_covered 96 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5536,4 +5536,22 @@ suites_not_covered 97 # never 3907 + 29 = 3936 # dated 2370 + 7 = 2377 # 3936 + 2377 == 6313 -checks_never_observed_red 3936 +# +# #1341, iceberg_rest seeded, with its twin (OffgridwithJD): 3936 -> 3938, and +# suites_not_covered 97 -> 96. The suite had no rows. It enters with 14 checks from +# five-major runs (all green on each of 15-19, shell and twin alike), after two fixes. +# The 42501 arms read the SQLSTATE alone, and two guards raise 42501: with the allow-list +# refusing everything and the link-local refusal gone, "a link-local catalog host is +# refused" passed on the allow-list's 42501, and with every address read as link-local +# and the allow-list gone, "a catalog endpoint off the allow-list is refused" passed on +# the link-local guard's. Every refusal now reads its own guard's message. And "backend +# still up after the REST refusals" read SELECT 1, which answers again after crash +# recovery: with the backend aborting on the catalog's 404, main passed it. It now also +# reads the server log's count of killed backends. 12 are dated by eleven mutations, +# each reddening the same names in both twins. 2 arrive never: the two premises. +# +# ledger rows 6313 + 14 = 6327 +# never 3936 + 2 = 3938 +# dated 2377 + 12 = 2389 +# 3938 + 2389 == 6327 +checks_never_observed_red 3938 diff --git a/test/iceberg_rest.sh b/test/iceberg_rest.sh index 8383f5d34..a97ac65b2 100755 --- a/test/iceberg_rest.sh +++ b/test/iceberg_rest.sh @@ -41,12 +41,42 @@ rest_teardown() { } trap rest_teardown EXIT INT TERM -sqlstate_of() { - env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -qtA 2>&1 < "