Repository navigation
test: native_parquet_multifile gets its pytest twin (#1341) - #1498
Conversation
test_native_parquet_pushdown.py carries all 36 checks by the shell's name in four tests, each writing its file with pyarrow and declining under the shell's own pgc_skip name when pyarrow is absent. Under each of seven mutations of the FDW's skip decision, the twin reddens exactly the shell's set of checks: no group ever skipped, the inverted-interval guard dropped, float > and >= pushed, a NaN equality pushed, INT64 statistics not decoded, col < const skipping the groups that match, and an equality skipping every group. The last two are what date the oracle arms: a skip that only stops saving work cannot change an answer. Each test creates its own foreign server, because a server is database-wide and pgc_conn isolates tests by schema only. COMPLETE gains native_parquet_pushdown, and its literal tables join the loop-reader guard's list. TESTS.md section 169, cluster_tests 730 -> 734. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y seven skip mutations (#1341) The suite had no rows. suites_not_covered 197 -> 196; census 4263 rows / 3002 never / 1261 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…faults they name (#1341) test_native_parquet_flba.py carries all 15 checks by the shell's name in three tests. Under each of eight mutations of the FLBA and DECIMAL read path, the twin reddens exactly the shell's set of checks. Two arms change, in both twins, each measured red first: - "pyarrow uuid reads as uuid" read only the smallest value, the all-zero uuid, which every byte order leaves unchanged. With the reader reversing a uuid's bytes it stayed green. It now reads both values in order. - "crafted out-of-range scale is rejected, not decoded" took any error as REJECTED. With the bind guard (pq_decimal_bounds_ok) removed, the decoder's own scale check still refused the file, as XX001 "could not decode Parquet column 0" after decoding began, and the arm stayed green. It now requires the bind's refusal, 42804. COMPLETE gains native_parquet_flba. TESTS.md section 170, cluster_tests 734 -> 737. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x read-path mutations (#1341) The suite had no rows. suites_not_covered 196 -> 195; census 4278 rows / 3006 never / 1272 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_native_parquet_multifile.py carries all 32 checks by the shell's name in three tests. Under each of nine mutations of the directory walk and glob expansion, the twin reddens exactly the shell's set of checks: an empty directory and an unmatched glob reading as nothing, an unstat-able path dropped, a FIFO opened, _ and . names walked, a symlinked directory descended, the files unsorted, no depth bound, and every extension read. A read that could block runs on its own connection in a thread with a deadline, as the shell runs psql under timeout. A cancel is no substitute: a backend blocked in open(2) on a FIFO resumes the open after a signal. On a timeout the twin opens the FIFO for writing, which releases the reader. The FIFO arm is its own test, so it can decline under the shell's "the FIFO case" when mkfifo is unavailable. COMPLETE gains native_parquet_multifile. TESTS.md section 171, cluster_tests 737 -> 740. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by nine walk and glob mutations (#1341) The suite had no rows. suites_not_covered 195 -> 194; census 4310 rows / 3010 never / 1300 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd
left a comment
There was a problem hiding this comment.
Approve, at 40ebcfc (its own commits 1871e47 and 40ebcfc, on #1497). This time I ran both twins: pyarrow is now in my pytest env.
My own mutant, different from your nine: the glob path drops its candidate filter (if (true || pq_path_is_candidate(...)) after glob()), so a directory that matches is read as a file. The directory walk keeps its filter. PG17:
- shell:
glob skips a directory match, reads the files: got [QUERY_ERROR.1] want [225b83ef…]; - pytest: the same arm, red:
… is not a regular file ….
That row arrives never in this PR, and this mutant dates it, if you want to fold it in.
A finding, not blocking. "glob matching only directories errors" stayed green under the same mutant. Its error came from reading the directory (is not a regular file), not from the refusal it names, pattern … matched no regular files. It's the "any error counts" shape you checked for on the empty-directory arm. Requiring matched no regular files (or its SQLSTATE) would make it see the refusal. Since it's one of the four that arrive never, the follow-up can date it too.
|
@OffgridwithJD, on your next step: I haven't started on the #1339 custom-scan entries you posted, so they're yours. So we don't overlap, here's where I am:
If you want any of those, say so and I'll leave it. |
|
@OffgridwithJD, merging #1498 now: approved at 40ebcfc, 15/15 green, CLEAN. |
@jdatcmd The last suite on item 74's list, stacked on #1497 (#1489 → #1493 → #1496 → #1497 → this). This PR's own commits are the top two.
The twin, name for name
All 32 checks by the shell's name, in three tests. Under each of nine mutations of the directory walk and glob expansion (
columnar_parquet_reader.c), the twin reddens exactly the shell's set:_/.names are walkedOn the "any error counts" arms: I checked whether removing a refusal let a downstream error keep them green, as on #1497. It doesn't here: with the empty-directory refusal gone, "empty directory errors" is red.
Blocking reads. The shell bounds the FIFO and symlink-loop reads with
timeoutaround psql. The twin runs them on their own connection, in a thread with a deadline. A cancel is no substitute, because a backend blocked inopen(2)on a FIFO resumes the open after a signal (SA_RESTART). On a timeout the twin opens the FIFO for writing, which releases the reader, so the mutant run finishes and the cluster tears down. The FIFO arm is its own test, so it can decline under the shell'scheck_skipname, "the FIFO case", where mkfifo is unavailable.Five-major runs at head: 32/32 for the shell and 32/32 for the twin, on each of 15–19.
Ledger
Census 4310 / 3010 never / 1300 dated (on #1497's 4278 / 3006 / 1272), with suites_not_covered 195 → 194. 28 of the 32 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 4 arrive never: the three glob arms (subset, directory match skipped, only directories) and the schema mismatch.
Collection gives
cluster_tests 737 -> 740, and TESTS.md gets §171. The grader is 34/34 by name, and the guards pass 438/438.That clears item 74's list. Next I'll take the #1339 custom-scan audit entries I posted, unless you've started on them.
🤖 Generated with Claude Code