Repository navigation
test: iceberg_malformed gains its pytest twin, and a crash or an empty read no longer passes as a refusal (#1341) - #1638
Conversation
jdatcmd
left a comment
There was a problem hiding this comment.
Approved at 2d9ecc4. Both claims in the message reproduce, measured here on PG17 at origin/main a1b04ee.
The S_ISREG guard removed (if (false && !S_ISREG(st.st_mode)) in PgColumnarOpenLocalRegularFile):
- main's shell: 28 + 3 failed. Only the three JSON-path FIFO arms fail (
got [22P02]). - port's shell: 24 + 7 failed. All seven FIFO arms fail, for example
got [XX001: columnar: not an Avro object-container file (bad magic)] want [XX001 (not a regular file)]. - twin: 7 failed, the same seven.
The NULL data-file path check removed (columnar_iceberg.c:808), which crashes the scan:
- main's shell: 5 failed. Two of them are collateral after the crash: "control: a present file_path lists one data file" and the null-manifest_path arm both read
got []. - port's shell: 3 failed, and only the right ones. "backend still up after the scan refusal" reads
no ([], 1 backend crashes), and every arm after it passes. - twin: 3 failed, the same names.
Unmutated: shell 31/31 and twin 33 passed, on PG15 and PG19.
Ledger: 31 rows, all shell check names, no skip names (unlike #1640).
Non-blocking: the twin's two skip-mirror tests record premises the shell does not have. "premise: the dangling-schema metadata parses as JSON" can fail only on a broken file, since json is in the standard library. That is fine as a premise; I mention it only because the test name suggests it guards a dependency.
|
@OffgridwithJD #1638, #1634, #1635 and #1636 are approved on their current heads, but GitHub refuses the merge: main moved to 7f78a62 when #1622, #1628 and #1629 merged, and all four conflict in check_ledger_budget.txt, TESTS.md and expected_tests.txt ( |
…y read no longer passes as a refusal (#1341) The pytest twin of test/iceberg_malformed.sh, one test per check by the shell's name, plus two that mirror its skip names. Two fixes to the shell, measured on PG17 at origin/main: - Every refusal arm read only XX001, which later parse steps also raise. With the S_ISREG guard in PgColumnarOpenLocalRegularFile removed, a FIFO opens O_NONBLOCK and reads as an empty file, and four of the seven FIFO arms still passed. Each arm now also requires its refusal's own message. - Every "backend still up" check was a bare SELECT 1, which answers again after crash recovery. With a NULL path segfaulting the backend and the probe 3s later, both survival checks passed. They now read the server log's count of killed backends since their own arm, and wait out a recovery so the arms after a crash are not collateral. Seven source mutations each redden the same names in shell and twin. The array-valued "fields" arm stays never: on a build without asserts the schema walker still refuses through av_schema_err with the guard removed. Census 3853 -> 3868 never, 2243 -> 2259 dated, 6127 rows; suites_not_covered 105 -> 104; cluster_tests 2677 -> 2710; TESTS.md §265. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2d9ecc4 to
d0c9e2c
Compare
jdatcmd
left a comment
There was a problem hiding this comment.
Re-approved at d0c9e2c, a restate onto main 5a94757. Against my approval at 2d9ecc4, the shell, the twin, the ledger rows, test_compare_to_bash.py and unported_suites.txt carry the identical change. Only the budget, expected_tests.txt and TESTS.md moved, and the census agrees with the tsv at this head. Merging once CI is green on this head.
@jdatcmd
iceberg_malformedgains its pytest twin, with one test per check by the shell's name (#1341). It is part of a stack withiceberg_dv,iceberg_name_mappingandiceberg_objstore, and will be restated as those land.Two holes in the shell, measured on PG17 at origin/main
1. Every refusal arm read only XX001. Later parse steps raise the same SQLSTATE. Mutant
nofiforemoves theS_ISREGguard inPgColumnarOpenLocalRegularFile, so a FIFO opensO_NONBLOCKand reads as an empty file. Under it, 4 of the 7 FIFO arms still passed (read_avro_manifest,read_manifest_list,read_parquetand the manifest-list FIFO). Each arm now names its refusal's own message:XX001 (not a regular file),(no data file path),(null manifest path),(null sequence to inherit),(dangling schema id)and(malformed schema). Anything else reads as<SQLSTATE>: <message>. The same tightening dates the lister's NULL-path arm: undernullfp, the lister was refused by ice_open_path's later null check, which raises the same XX001.2. Every "backend still up" check was a bare
SELECT 1, which answers again after crash recovery. I rannullmpandnullfp, where a NULL path segfaults the backend (the server log showsterminated by signal), with each probe delayed 3s. Both survival checks passed. Each survival check now compares the server log's crash count with the one taken just before its own arm, and an unreadable log fails it. After a crash it waits out the recovery, so the arms after it aren't collateral. With the same 3s delay, both now go red.Measured (PG17, shell then twin as shipped, each mutant built with
make clean)nofifo: the regular-file guard droppedblockopen: the original bug, a blocking open and no guardnullfp: a NULL file_path not refusednullmp: a NULL recorded path not refuseddangling/seqinherit/seqzero/minseqzeroarrayobj: an array-valuedfieldselement passes the object checkarrayobjstays green: on a build without asserts, the schema walker still refuses throughav_schema_errwith the same message. That arm staysnever. It's only visible under asserts or ASAN.The check name
an array-valued schema "fields" element …lost its quotes. The grader read\"as the end of the name.Counts
suites_not_covered: 113 → 112.cluster_tests: 2454 → 2487.2d9ecc43: guards 467 passed; collection 2487; ledger gate rc=0; self-test PASSED;compare_to_bash33 literal matches, every bash property covered.🤖 Generated with Claude Code