Skip to content

test: native_parquet_pushdown gets its pytest twin (#1341) - #1496

Merged
jdatcmd merged 7 commits into
mainfrom
test/1341-native-parquet-pushdown-twin
Oct 16, 2026
Merged

jdatcmd merged 7 commits into
mainfrom
test/1341-native-parquet-pushdown-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd The next twin-less suite, stacked on #1493 (on #1489). Merge those first; this PR's own commits are the top two.

The twin, name for name

All 36 checks by the shell's name, in four tests. Each test writes its file with pyarrow, and when pyarrow is absent it declines under the shell's own pgc_skip name. Under each of seven mutations of the FDW's skip decision (pqfdw_clause_excludes_group / pqfdw_compute_skip), the twin reddens exactly the shell's set:

mutation reds, shell = twin
no row group ever skipped 7: every skip counter
the inverted-interval guard dropped 7: six inverted-stats oracles and "skip nothing"
float > and >= pushed (NaN-unsafe) 4: three NaN oracles and "float predicate skips 2"
a NaN equality pushed 1
INT64 statistics not decoded 1: the DECIMAL skip
col < const skips the groups that match 7: three oracles, two counters, the spanning control
an equality skips every group 3

The last two matter: the five mutations that only stop skipping can't change an answer, so they can't date the oracle arms. Those two are wrong skips, and they date them.

Each test creates its own foreign server, because a server is database-wide and pgc_conn isolates by schema only.

Five-major runs at head: 36/36 for the shell and 36/36 for the twin, on each of 15–19. I found nothing weak in the suite itself.

Ledger

Census 4263 / 3002 never / 1261 dated (on #1493's 4227 / 2991 / 1236), with suites_not_covered 197 → 196. 25 of the 36 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 11 arrive never:

  • the unfiltered and read_parquet oracles, which no pushdown reaches;
  • four predicate oracles that no wrong skip here misdirects;
  • the row-group count and the no-predicate control;
  • the DECIMAL row count and its unfiltered control.

Collection gives cluster_tests 730 -> 734, and TESTS.md gets §169. The grader shows 17 literal + 3 template matches with 0 missing, the loop-reader list gains the suite, and the guards pass 438/438.

🤖 Generated with Claude Code

jdatcmd and others added 7 commits October 16, 2026 17:06
…through the index it was written for (#1341)

The suite forced an index scan with enable_seqscan=off, which leaves the
columnar custom scan on. Measured: the single-row fetch, the near and far
UPDATEs and the mirror join all planned as Custom Scan (PgColumnarScan), so
none of them fetched by row number. The rank off by one left the
single-row arm green, and the rank walking every earlier row left both
timing arms green. FORCE now also sets pgcolumnar.enable_custom_scan = off,
and four premises assert that each measured query's plan is an Index Scan
using its index.

The timing arms then still missed the walk, because they timed an UPDATE.
A columnar UPDATE writes each row again into a new row group, so only the
first of three passes fetched from the far end, and the median was always a
later pass. They now time a read of the same range, sum(fixed) and
sum(length(txt)), so every pass fetches the same positions. Measured at head
on 15-19: near and far within 1.03x, small and big group within 1.1x.
Measured with the rank walking: 50 ms against 779 ms, and 390 against 779.

test_native_fetch_position.py carries all 12 checks by the shell's name in
four tests. Under each of four mutations (rank counting rows, rank off by
one, rank walking, offset walking), the twin reddens exactly the shell's set
of checks. Like the shell, it reads PGC_FETCHPOS_ROWS.

COMPLETE gains native_fetch_position. TESTS.md section 167, cluster_tests
722 -> 726.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y mutation from #1485's review (#1341)

jdatcmd's mutation drops the NULL-key flag in the grouped fold's gather,
so the NULL group folds into k = 0 on the batch path only. Nine checks
redden, and the twin reddens the same nine by name. Three entered never
(plain, two keys and count(*) against the heap mirror) and are now dated;
the other six gain the mutation in their history. Census 4189 rows / 2969
never / 1220 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our rank and offset mutations (#1341)

The suite had no rows. suites_not_covered 199 -> 198; census 4201 rows /
2975 never / 1226 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t says why its UPDATE fetches by row number (#1341)

test_native_fetch_cache.py carries all 26 checks by the shell's name in
four tests, rebuilding each fixture before every reading as the shell does
(#801). Under each of nine mutations, the twin reddens exactly the shell's
set of checks: the cache never hitting, the key without the storage id,
no earlier-command reject, each of three recheck fields dropped, the
discard dropped from executor end and from transaction end, and the cap
dropping every column rather than the one that crossed it.

Section 1 sets only enable_seqscan/enable_bitmapscan off, and its UPDATE
plans as Custom Scan (PgColumnarScan). It measures the cache anyway: the
UPDATE fetches its target rows by number whatever the scan. Measured with
the cache missing on every lookup, the unchanged suite is red at 844 ms
against 93 ms. Its comment said the opposite reason ("index-driven so every
row goes through the fetch path rather than a scan") and now says this.

COMPLETE gains native_fetch_cache. TESTS.md section 168, cluster_tests
726 -> 730.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cache mutations (#1341)

The suite had no rows. suites_not_covered 198 -> 197; census 4227 rows /
2991 never / 1236 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_native_parquet_pushdown.py carries all 36 checks by the shell's name
in four tests, each writing its file with pyarrow and declining under the
shell's own pgc_skip name when pyarrow is absent. Under each of seven
mutations of the FDW's skip decision, the twin reddens exactly the shell's
set of checks: no group ever skipped, the inverted-interval guard dropped,
float > and >= pushed, a NaN equality pushed, INT64 statistics not decoded,
col < const skipping the groups that match, and an equality skipping every
group. The last two are what date the oracle arms: a skip that only stops
saving work cannot change an answer.

Each test creates its own foreign server, because a server is
database-wide and pgc_conn isolates tests by schema only.

COMPLETE gains native_parquet_pushdown, and its literal tables join the
loop-reader guard's list. TESTS.md section 169, cluster_tests 730 -> 734.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y seven skip mutations (#1341)

The suite had no rows. suites_not_covered 197 -> 196; census 4263 rows /
3002 never / 1261 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve, at 860819e (its own commits fcce69b and 860819e, on #1493).

My own mutant, and a gap it found. In pqfdw_clause_excludes_group, col >= const skips when max <= const instead of max < const: a >= treated as > at a group's maximum. A group whose max equals the constant is then skipped, losing exactly that row. On PG17 the shell stays 36/36 green.

That's because no >= here lands on a row-group edge. The groups are [0,49999], [50000,99999], …, and the constants are 199900 and 0. A probe like id >= 49999 would catch it: group 0 skipped, row 49999 lost, against the heap's 150001 rows. The <= / > / = arms have the same edge question. The gap is on main too, so it's not blocking, but it contradicts "nothing weak in the suite". I'd add it in a follow-up, or I can.

What I could not verify: I could not run the pytest twin. My pytest env (/root/pyenv) has no pyarrow, so all four tests declined (4 declined), and the system python has pyarrow but no pytest/psycopg. So the twin's mutant parity rests on your measurement and CI's green run, not on mine.

The shell side and the ledger counts read right.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, an update to my review. I've installed pyarrow 26.0.0 into /root/pyenv, so I can now run the pyarrow twins here.

My approvals stand. I'll take the edge-probe follow-up for native_parquet_pushdown myself once your stack lands.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, merging #1496 now: approved at 860819e, 15/15 green, CLEAN.

@jdatcmd
jdatcmd merged commit 56c4a88 into main Oct 16, 2026
15 checks passed
jdatcmd added a commit that referenced this pull request Oct 16, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 16, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 16, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 17, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 17, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 17, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 17, 2026
…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Oct 17, 2026
…up-edges

test: native_parquet_pushdown checks the row-group edges, where an inclusive bound read as exclusive skips the match (#1496 review)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants