From d0c9e2c354925baadf9302e8fa8259d3f8e2f3da Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 21 Oct 2026 21:25:43 +0000 Subject: [PATCH] test: iceberg_malformed gains its pytest twin, and a crash or an empty read no longer passes as a refusal (#1341) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pytest twin of test/iceberg_malformed.sh, one test per check by the shell's name, plus two that mirror its skip names. Two fixes to the shell, measured on PG17 at origin/main: - Every refusal arm read only XX001, which later parse steps also raise. With the S_ISREG guard in PgColumnarOpenLocalRegularFile removed, a FIFO opens O_NONBLOCK and reads as an empty file, and four of the seven FIFO arms still passed. Each arm now also requires its refusal's own message. - Every "backend still up" check was a bare SELECT 1, which answers again after crash recovery. With a NULL path segfaulting the backend and the probe 3s later, both survival checks passed. They now read the server log's count of killed backends since their own arm, and wait out a recovery so the arms after a crash are not collateral. Seven source mutations each redden the same names in shell and twin. The array-valued "fields" arm stays never: on a build without asserts the schema walker still refuses through av_schema_err with the guard removed. Census 3853 -> 3868 never, 2243 -> 2259 dated, 6127 rows; suites_not_covered 105 -> 104; cluster_tests 2677 -> 2710; TESTS.md ยง265. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 31 ++ test/check_ledger_budget.txt | 22 +- test/iceberg_malformed.sh | 138 +++++--- test/pytest/TESTS.md | 45 +++ test/pytest/expected_tests.txt | 4 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_iceberg_malformed.py | 446 ++++++++++++++++++++++++++ test/pytest/unported_suites.txt | 1 - 8 files changed, 642 insertions(+), 46 deletions(-) create mode 100644 test/pytest/test_iceberg_malformed.py diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index a9fdef60b..10900f8bb 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2560,6 +2560,37 @@ iceberg_fdw_projection iceberg_fdw_projection one column : FDW projection matche iceberg_fdw_projection iceberg_fdw_projection select one, filter another : FDW projection matches iceberg_scan 15;16;17;18;19 2026-10-21 the Iceberg FDW's projection mask ignores columns only the scan clauses reference iceberg_fdw_projection iceberg_fdw_projection three columns : FDW projection matches iceberg_scan 15;16;17;18;19 never - iceberg_fdw_projection iceberg_fdw_projection whole row : FDW projection matches iceberg_scan 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed a NULL data-file path is refused in iceberg_data_files (XX001, not a crash) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a manifest entry with a NULL file_path is not refused +iceberg_malformed iceberg_malformed a NULL data-file path is refused in iceberg_scan (XX001, not a crash) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a manifest entry with a NULL file_path is not refused +iceberg_malformed iceberg_malformed a NULL manifest_path is refused, not a crash (XX001) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: ice_open_path does not refuse a NULL recorded path +iceberg_malformed iceberg_malformed a dangling current-schema-id is refused (XX001), not silently stale-read 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a dangling current-schema-id resolves to the legacy top-level schema +iceberg_malformed iceberg_malformed a manifest-list that is a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file +iceberg_malformed iceberg_malformed a null manifest-list min_sequence_number decodes as NULL, not 0 (#686) 15;16;17;18;19 2026-10-21 columnar_avro.c: read_manifest_list decodes a null min_sequence_number as 0 +iceberg_malformed iceberg_malformed a null manifest-list sequence_number decodes as NULL, not 0 15;16;17;18;19 2026-10-21 columnar_avro.c: read_manifest_list decodes a null sequence_number as 0 +iceberg_malformed iceberg_malformed an array-valued schema fields element is refused, not crashed (XX001) 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the FIFO refusals 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the dangling-schema refusal 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the lister refusal 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the malformed-schema refusal 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the manifest-list FIFO refusal 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the null-manifest_path refusal 15;16;17;18;19 2026-10-21 columnar_iceberg.c: ice_open_path does not refuse a NULL recorded path +iceberg_malformed iceberg_malformed backend still up after the null-sequence refusal 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed backend still up after the scan refusal 15;16;17;18;19 2026-10-21 columnar_iceberg.c: a manifest entry with a NULL file_path is not refused +iceberg_malformed iceberg_malformed control: a concrete manifest seq scans all 5 rows (seq-3 delete is older than seq-5 data) 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: a legacy-only-schema table still reads (5 rows) 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: a present file_path lists one data file 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: a present manifest-list sequence_number decodes to its value 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: a present min_sequence_number decodes to its value (#686) 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: a resolvable current-schema-id reads the correct amount column 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: a well-formed field object still decodes (0 entries) 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed control: the legacy table binds amount correctly (10..50) 15;16;17;18;19 never - +iceberg_malformed iceberg_malformed iceberg_current_snapshot on a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file +iceberg_malformed iceberg_malformed iceberg_data_files on a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file +iceberg_malformed iceberg_malformed iceberg_scan on a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file +iceberg_malformed iceberg_malformed iceberg_scan refuses a null manifest seq an ADDED entry must inherit (XX001, not a silent wrong row) 15;16;17;18;19 2026-10-21 columnar_iceberg.c: an added entry inherits a null manifest sequence number as 0 +iceberg_malformed iceberg_malformed read_avro_manifest on a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file +iceberg_malformed iceberg_malformed read_manifest_list on a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file +iceberg_malformed iceberg_malformed read_parquet on a FIFO is refused, not a hang (XX001) 15;16;17;18;19 2026-10-21 columnar_objstore.c: PgColumnarOpenLocalRegularFile drops its S_ISREG guard, so a FIFO opens O_NONBLOCK and reads as an empty file iceberg_name_mapping_memory iceberg_name_mapping_memory and the mapped values are still correct 15;16;17;18;19 2026-10-21 the name-mapping parse binds every name but region, so region reads NULL on every row;the name-mapping parse ignores every entry's names, so nothing binds iceberg_name_mapping_memory iceberg_name_mapping_memory core's own parse is the larger part of the scan's peak 15;16;17;18;19 2026-10-21 the name-mapping parse ignores every entry's names, so nothing binds iceberg_name_mapping_memory iceberg_name_mapping_memory the large arm produced three readings 15;16;17;18;19 2026-10-21 the name-mapping parse ignores every entry's names, so nothing binds diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 07074f0c9..66efc310d 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -15,7 +15,7 @@ # # Adding a check to a suite that is already covered does not move it, which is # what makes it safe to bound. -suites_not_covered 105 +suites_not_covered 104 # # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become # one. @@ -5374,4 +5374,22 @@ suites_not_covered 105 # never 3850 + 3 = 3853 # dated 2239 + 4 = 2243 # 3853 + 2243 == 6096 -checks_never_observed_red 3853 +# #1341, iceberg_malformed seeded, with its twin (OffgridwithJD): 3853 -> 3868, +# and suites_not_covered 105 -> 104. The suite had no rows. It enters with 31 +# checks from five-major runs (all green on each of 15-19, shell and twin alike), after +# two fixes measured on PG17 at origin/main. Every refusal arm read only XX001, which +# later parse steps also raise: with the regular-file guard removed, a FIFO opened +# O_NONBLOCK reads as an empty file and four of the seven FIFO arms still passed. Each +# arm now also names its refusal's message. And every "backend still up" check was a +# bare SELECT 1, which answers again after crash recovery: with a NULL path segfaulting +# the backend and the probe 3s later, both survival checks passed. They now read the +# server log's crash count since their own arm, and wait out a recovery. 16 are dated +# by seven mutations, each reddening the same names in both twins; 15 arrive never. +# The array-valued "fields" arm stays never: on a build without asserts the schema +# walker still refuses through av_schema_err, so dropping the object check passes all 31. +# +# ledger rows 6096 + 31 = 6127 +# never 3853 + 15 = 3868 +# dated 2243 + 16 = 2259 +# 3868 + 2259 == 6127 +checks_never_observed_red 3868 diff --git a/test/iceberg_malformed.sh b/test/iceberg_malformed.sh index 872dd228a..588f62063 100755 --- a/test/iceberg_malformed.sh +++ b/test/iceberg_malformed.sh @@ -5,8 +5,10 @@ # whoever authored the table, and is called by an honest pg_read_server_files # role that named only the trusted metadata pointer. A crafted manifest must be # REFUSED cleanly, never crash the backend. Each arm has a control so a vacuous -# pass is visible, and a "backend still up" probe after every refusal proves the -# refusal was an error, not a segfault. Fixtures are hand-crafted +# pass is visible, each refusal names its own message (the SQLSTATE alone is +# shared with later parse errors), and a "backend still up" probe after every +# refusal reads the server log's crash count, so a segfault followed by crash +# recovery does not pass as a refusal. Fixtures are hand-crafted # (test/fixtures/iceberg/warehouse_malformed) because no writer emits these # shapes; see gen_malformed_fixture.py. # @@ -16,31 +18,69 @@ set -uo pipefail . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" -sqlstate_of() { +# refusal SQL REGEX LABEL -> " (LABEL)" when SQL raised an error whose message +# matches REGEX, ": " for any other error, or "no error". Every +# arm here refuses with XX001, and so do later parse steps: a FIFO read as an empty +# file failed a parse with the same XX001, so the SQLSTATE alone passed with the +# guard removed. The arm now names the refusal it wants. +refusal_of() { # stdin: psql output -> the refusal reading + local re="$1" label="$2" line state msg + local out; out="$(cat)" + line="$(sed -n 's/^ERROR: \([0-9A-Z]\{5\}\): /\1|/p' <<<"$out" | head -1)" + if [ -z "$line" ]; then + grep -qE 'server closed the connection|connection to server' <<<"$out" \ + && echo "no refusal: the connection was lost" || echo "no error" + return + fi + state="${line%%|*}"; msg="${line#*|}" + if grep -qE -- "$re" <<<"$msg"; then echo "$state ($label)"; else echo "$state: $msg"; fi +} +refusal() { env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ - -d "$PGC_DB" -qtA 2>&1 <&1 <&1 </dev/null)" + case "$_n" in '' | *[!0-9]*) ;; *) echo "$_n" ;; esac +} +survived() { # survived BEFORE -> "1" when the server answers and no backend died since BEFORE + local _up _now _i + _up="$(q 'SELECT 1')"; _now="$(crashes)" + if ! [[ "$1" =~ ^[0-9]+$ && "$_now" =~ ^[0-9]+$ ]]; then echo "no (crash count unreadable)"; return; fi + if [ "$_up" = 1 ] && [ "$_now" = "$1" ]; then echo 1; return; fi + for _i in $(seq 1 60); do [ "$(q 'SELECT 1' 2>/dev/null)" = 1 ] && break; sleep 0.5; done + echo "no ([$_up], $(( _now - $1 )) backend crashes)" } # Release a backend that is blocked in open(2) on a FIFO (main only): opening the @@ -70,10 +110,12 @@ chmod -R u+rwX "$DEST" # backend (an object-container Assert / OOB heap read). It must now be a # clean malformed-schema error, and the control must still decode. # ========================================================================= -check "an array-valued schema \"fields\" element is refused, not crashed (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.read_avro_manifest('$DEST/arrayfield/evil.avro')")" \ - "XX001" -check "backend still up after the malformed-schema refusal" "$(q 'SELECT 1')" "1" +B="$(crashes)" +check "an array-valued schema fields element is refused, not crashed (XX001)" \ + "$(refusal "SELECT * FROM pgcolumnar.read_avro_manifest('$DEST/arrayfield/evil.avro')" \ + 'unsupported or malformed Avro schema in manifest' "malformed schema")" \ + "XX001 (malformed schema)" +check "backend still up after the malformed-schema refusal" "$(survived "$B")" "1" check "control: a well-formed field object still decodes (0 entries)" \ "$(q "SELECT count(*) FROM pgcolumnar.read_avro_manifest('$DEST/arrayfield/good.avro')")" \ "0" @@ -84,12 +126,16 @@ check "control: a well-formed field object still decodes (0 entries)" \ # be a clean DATA_CORRUPTED refusal; the control (file_path present) reads. # ========================================================================= NP="$DEST/nullpath/db/t/metadata/t.metadata.json" +B="$(crashes)" check "a NULL data-file path is refused in iceberg_data_files (XX001, not a crash)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_data_files('$NP')")" "XX001" -check "backend still up after the lister refusal" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_data_files('$NP')" \ + 'has no data file path' "no data file path")" "XX001 (no data file path)" +check "backend still up after the lister refusal" "$(survived "$B")" "1" +B="$(crashes)" check "a NULL data-file path is refused in iceberg_scan (XX001, not a crash)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_scan('$NP') AS t(id bigint)")" "XX001" -check "backend still up after the scan refusal" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_scan('$NP') AS t(id bigint)" \ + 'has no data file path' "no data file path")" "XX001 (no data file path)" +check "backend still up after the scan refusal" "$(survived "$B")" "1" check "control: a present file_path lists one data file" \ "$(q "SELECT count(*) FROM pgcolumnar.iceberg_data_files('$DEST/nullpath_ctl/db/t/metadata/t.metadata.json')")" \ "1" @@ -101,9 +147,11 @@ check "control: a present file_path lists one data file" \ # author-controlled, so this is reachable. Refuse it as DATA_CORRUPTED. # ========================================================================= NMP="$DEST/nullmanifestpath/db/t/metadata/t.metadata.json" +B="$(crashes)" check "a NULL manifest_path is refused, not a crash (XX001)" \ - "$(sqlstate_of "SELECT * FROM pgcolumnar.iceberg_data_files('$NMP')")" "XX001" -check "backend still up after the null-manifest_path refusal" "$(q 'SELECT 1')" "1" + "$(refusal "SELECT * FROM pgcolumnar.iceberg_data_files('$NMP')" \ + 'a manifest path in .* is null' "null manifest path")" "XX001 (null manifest path)" +check "backend still up after the null-manifest_path refusal" "$(survived "$B")" "1" # ========================================================================= # #1 a non-regular file (FIFO) named on any Iceberg-reachable open must be @@ -113,31 +161,32 @@ check "backend still up after the null-manifest_path refusal" "$(q 'SELECT 1')" FIFODIR="$PGC_WORKDIR/fifo"; mkdir -p "$FIFODIR" FIFO="$FIFODIR/x"; mkfifo "$FIFO" +B="$(crashes)" # Site A -- ice_slurp_text, FIFO as the metadata argument (three entry points) check "iceberg_current_snapshot on a FIFO is refused, not a hang (XX001)" \ - "$(sqlstate_or_hang "SELECT * FROM pgcolumnar.iceberg_current_snapshot('$FIFO')")" \ - "XX001"; fifo_release "$FIFO" + "$(refusal_or_hang "SELECT * FROM pgcolumnar.iceberg_current_snapshot('$FIFO')")" \ + "XX001 (not a regular file)"; fifo_release "$FIFO" check "iceberg_data_files on a FIFO is refused, not a hang (XX001)" \ - "$(sqlstate_or_hang "SELECT * FROM pgcolumnar.iceberg_data_files('$FIFO')")" \ - "XX001"; fifo_release "$FIFO" + "$(refusal_or_hang "SELECT * FROM pgcolumnar.iceberg_data_files('$FIFO')")" \ + "XX001 (not a regular file)"; fifo_release "$FIFO" check "iceberg_scan on a FIFO is refused, not a hang (XX001)" \ - "$(sqlstate_or_hang "SELECT * FROM pgcolumnar.iceberg_scan('$FIFO') AS t(id bigint)")" \ - "XX001"; fifo_release "$FIFO" + "$(refusal_or_hang "SELECT * FROM pgcolumnar.iceberg_scan('$FIFO') AS t(id bigint)")" \ + "XX001 (not a regular file)"; fifo_release "$FIFO" # Site C -- av_slurp_file, FIFO as the argument (both entry points) check "read_avro_manifest on a FIFO is refused, not a hang (XX001)" \ - "$(sqlstate_or_hang "SELECT * FROM pgcolumnar.read_avro_manifest('$FIFO')")" \ - "XX001"; fifo_release "$FIFO" + "$(refusal_or_hang "SELECT * FROM pgcolumnar.read_avro_manifest('$FIFO')")" \ + "XX001 (not a regular file)"; fifo_release "$FIFO" check "read_manifest_list on a FIFO is refused, not a hang (XX001)" \ - "$(sqlstate_or_hang "SELECT * FROM pgcolumnar.read_manifest_list('$FIFO')")" \ - "XX001"; fifo_release "$FIFO" + "$(refusal_or_hang "SELECT * FROM pgcolumnar.read_manifest_list('$FIFO')")" \ + "XX001 (not a regular file)"; fifo_release "$FIFO" # Site D -- pq_source_open_cfg via the single-file parquet API (sibling) check "read_parquet on a FIFO is refused, not a hang (XX001)" \ - "$(sqlstate_or_hang "SELECT * FROM pgcolumnar.read_parquet('$FIFO') AS t(id bigint)")" \ - "XX001"; fifo_release "$FIFO" + "$(refusal_or_hang "SELECT * FROM pgcolumnar.read_parquet('$FIFO') AS t(id bigint)")" \ + "XX001 (not a regular file)"; fifo_release "$FIFO" -check "backend still up after the FIFO refusals" "$(q 'SELECT 1')" "1" +check "backend still up after the FIFO refusals" "$(survived "$B")" "1" # Site B -- ice_slurp_bin, FIFO named as the manifest-list INSIDE metadata.json. # realpath() on the FIFO succeeds and passes ice_open_path's containment, so only @@ -153,10 +202,11 @@ cat > "$MDJSON" < 2670 for #1341's test_parquet_export_stats.py, the twin of parquet_export_stats.sh: # 162 tests, one per check, by collection over the cluster files. +# cluster_tests 2677 -> 2710 for #1341's test_iceberg_malformed.py, the twin of iceberg_malformed.sh: +# 31 checks in 33 tests, by collection over the cluster files. # cluster_tests 2670 -> 2677 for #1637's test_catalog_filter_reads.py, the twin of catalog_filter_reads.sh: # 7 tests, one per check, by collection over the cluster files. -cluster_tests 2677 +cluster_tests 2710 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index f6c63c000..d89e8525b 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -124,6 +124,7 @@ "iceberg_name_mapping_memory", "iceberg_catalog", "iceberg_scan", + "iceberg_malformed", "concurrency", "isolation", "index_only", diff --git a/test/pytest/test_iceberg_malformed.py b/test/pytest/test_iceberg_malformed.py new file mode 100644 index 000000000..5b966150b --- /dev/null +++ b/test/pytest/test_iceberg_malformed.py @@ -0,0 +1,446 @@ +"""The Iceberg reader refuses crafted metadata, manifests and manifest lists cleanly, +never crashing or hanging the backend (#388 phase 4a, #644). The pytest twin of +`test/iceberg_malformed.sh`, every check by the shell's name (#1341). + +The hand-crafted fixtures under test/fixtures/iceberg/warehouse_malformed are copied +into `server_dir`, so the recorded root differs from the actual one and every read +goes through the same path rebasing the honest read path uses. Each refusal is read +as " (