Skip to content

test: native_parquet_multifile gets its pytest twin (#1341) - #1498

Merged
jdatcmd merged 6 commits into
mainfrom
test/1341-native-parquet-multifile-twin
Oct 16, 2026
Merged

jdatcmd merged 6 commits into
mainfrom
test/1341-native-parquet-multifile-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd The last suite on item 74's list, stacked on #1497 (#1489 → #1493 → #1496 → #1497 → this). This PR's own commits are the top two.

The twin, name for name

All 32 checks by the shell's name, in three tests. Under each of nine mutations of the directory walk and glob expansion (columnar_parquet_reader.c), the twin reddens exactly the shell's set:

mutation reds, shell = twin
an empty directory resolves to nothing 2
an unmatched glob resolves to nothing 1
an unstat-able path is dropped 2: both dangling-symlink arms
a FIFO is a candidate 1, and the read blocks
_/. names are walked 2
a symlinked directory is descended 2
the files are unsorted 1
no depth bound 1
every extension is read 22

On the "any error counts" arms: I checked whether removing a refusal let a downstream error keep them green, as on #1497. It doesn't here: with the empty-directory refusal gone, "empty directory errors" is red.

Blocking reads. The shell bounds the FIFO and symlink-loop reads with timeout around psql. The twin runs them on their own connection, in a thread with a deadline. A cancel is no substitute, because a backend blocked in open(2) on a FIFO resumes the open after a signal (SA_RESTART). On a timeout the twin opens the FIFO for writing, which releases the reader, so the mutant run finishes and the cluster tears down. The FIFO arm is its own test, so it can decline under the shell's check_skip name, "the FIFO case", where mkfifo is unavailable.

Five-major runs at head: 32/32 for the shell and 32/32 for the twin, on each of 15–19.

Ledger

Census 4310 / 3010 never / 1300 dated (on #1497's 4278 / 3006 / 1272), with suites_not_covered 195 → 194. 28 of the 32 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 4 arrive never: the three glob arms (subset, directory match skipped, only directories) and the schema mismatch.

Collection gives cluster_tests 737 -> 740, and TESTS.md gets §171. The grader is 34/34 by name, and the guards pass 438/438.

That clears item 74's list. Next I'll take the #1339 custom-scan audit entries I posted, unless you've started on them.

🤖 Generated with Claude Code

jdatcmd and others added 6 commits October 16, 2026 17:18
test_native_parquet_pushdown.py carries all 36 checks by the shell's name
in four tests, each writing its file with pyarrow and declining under the
shell's own pgc_skip name when pyarrow is absent. Under each of seven
mutations of the FDW's skip decision, the twin reddens exactly the shell's
set of checks: no group ever skipped, the inverted-interval guard dropped,
float > and >= pushed, a NaN equality pushed, INT64 statistics not decoded,
col < const skipping the groups that match, and an equality skipping every
group. The last two are what date the oracle arms: a skip that only stops
saving work cannot change an answer.

Each test creates its own foreign server, because a server is
database-wide and pgc_conn isolates tests by schema only.

COMPLETE gains native_parquet_pushdown, and its literal tables join the
loop-reader guard's list. TESTS.md section 169, cluster_tests 730 -> 734.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y seven skip mutations (#1341)

The suite had no rows. suites_not_covered 197 -> 196; census 4263 rows /
3002 never / 1261 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…faults they name (#1341)

test_native_parquet_flba.py carries all 15 checks by the shell's name in
three tests. Under each of eight mutations of the FLBA and DECIMAL read
path, the twin reddens exactly the shell's set of checks.

Two arms change, in both twins, each measured red first:
- "pyarrow uuid reads as uuid" read only the smallest value, the all-zero
  uuid, which every byte order leaves unchanged. With the reader reversing
  a uuid's bytes it stayed green. It now reads both values in order.
- "crafted out-of-range scale is rejected, not decoded" took any error as
  REJECTED. With the bind guard (pq_decimal_bounds_ok) removed, the
  decoder's own scale check still refused the file, as XX001 "could not
  decode Parquet column 0" after decoding began, and the arm stayed green.
  It now requires the bind's refusal, 42804.

COMPLETE gains native_parquet_flba. TESTS.md section 170, cluster_tests
734 -> 737.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x read-path mutations (#1341)

The suite had no rows. suites_not_covered 196 -> 195; census 4278 rows /
3006 never / 1272 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_native_parquet_multifile.py carries all 32 checks by the shell's name
in three tests. Under each of nine mutations of the directory walk and glob
expansion, the twin reddens exactly the shell's set of checks: an empty
directory and an unmatched glob reading as nothing, an unstat-able path
dropped, a FIFO opened, _ and . names walked, a symlinked directory
descended, the files unsorted, no depth bound, and every extension read.

A read that could block runs on its own connection in a thread with a
deadline, as the shell runs psql under timeout. A cancel is no substitute:
a backend blocked in open(2) on a FIFO resumes the open after a signal. On a
timeout the twin opens the FIFO for writing, which releases the reader. The
FIFO arm is its own test, so it can decline under the shell's "the FIFO
case" when mkfifo is unavailable.

COMPLETE gains native_parquet_multifile. TESTS.md section 171,
cluster_tests 737 -> 740.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by nine walk and glob mutations (#1341)

The suite had no rows. suites_not_covered 195 -> 194; census 4310 rows /
3010 never / 1300 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve, at 40ebcfc (its own commits 1871e47 and 40ebcfc, on #1497). This time I ran both twins: pyarrow is now in my pytest env.

My own mutant, different from your nine: the glob path drops its candidate filter (if (true || pq_path_is_candidate(...)) after glob()), so a directory that matches is read as a file. The directory walk keeps its filter. PG17:

  • shell: glob skips a directory match, reads the files: got [QUERY_ERROR.1] want [225b83ef…];
  • pytest: the same arm, red: … is not a regular file ….

That row arrives never in this PR, and this mutant dates it, if you want to fold it in.

A finding, not blocking. "glob matching only directories errors" stayed green under the same mutant. Its error came from reading the directory (is not a regular file), not from the refusal it names, pattern … matched no regular files. It's the "any error counts" shape you checked for on the empty-directory arm. Requiring matched no regular files (or its SQLSTATE) would make it see the refusal. Since it's one of the four that arrive never, the follow-up can date it too.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, on your next step: I haven't started on the #1339 custom-scan entries you posted, so they're yours. So we don't overlap, here's where I am:

If you want any of those, say so and I'll leave it.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, merging #1498 now: approved at 40ebcfc, 15/15 green, CLEAN.

@jdatcmd
jdatcmd merged commit 8898163 into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants