Repository navigation
test: native_parquet_pushdown gets its pytest twin (#1341) - #1496
Conversation
…through the index it was written for (#1341) The suite forced an index scan with enable_seqscan=off, which leaves the columnar custom scan on. Measured: the single-row fetch, the near and far UPDATEs and the mirror join all planned as Custom Scan (PgColumnarScan), so none of them fetched by row number. The rank off by one left the single-row arm green, and the rank walking every earlier row left both timing arms green. FORCE now also sets pgcolumnar.enable_custom_scan = off, and four premises assert that each measured query's plan is an Index Scan using its index. The timing arms then still missed the walk, because they timed an UPDATE. A columnar UPDATE writes each row again into a new row group, so only the first of three passes fetched from the far end, and the median was always a later pass. They now time a read of the same range, sum(fixed) and sum(length(txt)), so every pass fetches the same positions. Measured at head on 15-19: near and far within 1.03x, small and big group within 1.1x. Measured with the rank walking: 50 ms against 779 ms, and 390 against 779. test_native_fetch_position.py carries all 12 checks by the shell's name in four tests. Under each of four mutations (rank counting rows, rank off by one, rank walking, offset walking), the twin reddens exactly the shell's set of checks. Like the shell, it reads PGC_FETCHPOS_ROWS. COMPLETE gains native_fetch_position. TESTS.md section 167, cluster_tests 722 -> 726. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y mutation from #1485's review (#1341) jdatcmd's mutation drops the NULL-key flag in the grouped fold's gather, so the NULL group folds into k = 0 on the batch path only. Nine checks redden, and the twin reddens the same nine by name. Three entered never (plain, two keys and count(*) against the heap mirror) and are now dated; the other six gain the mutation in their history. Census 4189 rows / 2969 never / 1220 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our rank and offset mutations (#1341) The suite had no rows. suites_not_covered 199 -> 198; census 4201 rows / 2975 never / 1226 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t says why its UPDATE fetches by row number (#1341) test_native_fetch_cache.py carries all 26 checks by the shell's name in four tests, rebuilding each fixture before every reading as the shell does (#801). Under each of nine mutations, the twin reddens exactly the shell's set of checks: the cache never hitting, the key without the storage id, no earlier-command reject, each of three recheck fields dropped, the discard dropped from executor end and from transaction end, and the cap dropping every column rather than the one that crossed it. Section 1 sets only enable_seqscan/enable_bitmapscan off, and its UPDATE plans as Custom Scan (PgColumnarScan). It measures the cache anyway: the UPDATE fetches its target rows by number whatever the scan. Measured with the cache missing on every lookup, the unchanged suite is red at 844 ms against 93 ms. Its comment said the opposite reason ("index-driven so every row goes through the fetch path rather than a scan") and now says this. COMPLETE gains native_fetch_cache. TESTS.md section 168, cluster_tests 726 -> 730. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cache mutations (#1341) The suite had no rows. suites_not_covered 198 -> 197; census 4227 rows / 2991 never / 1236 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_native_parquet_pushdown.py carries all 36 checks by the shell's name in four tests, each writing its file with pyarrow and declining under the shell's own pgc_skip name when pyarrow is absent. Under each of seven mutations of the FDW's skip decision, the twin reddens exactly the shell's set of checks: no group ever skipped, the inverted-interval guard dropped, float > and >= pushed, a NaN equality pushed, INT64 statistics not decoded, col < const skipping the groups that match, and an equality skipping every group. The last two are what date the oracle arms: a skip that only stops saving work cannot change an answer. Each test creates its own foreign server, because a server is database-wide and pgc_conn isolates tests by schema only. COMPLETE gains native_parquet_pushdown, and its literal tables join the loop-reader guard's list. TESTS.md section 169, cluster_tests 730 -> 734. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y seven skip mutations (#1341) The suite had no rows. suites_not_covered 197 -> 196; census 4263 rows / 3002 never / 1261 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd
left a comment
There was a problem hiding this comment.
Approve, at 860819e (its own commits fcce69b and 860819e, on #1493).
My own mutant, and a gap it found. In pqfdw_clause_excludes_group, col >= const skips when max <= const instead of max < const: a >= treated as > at a group's maximum. A group whose max equals the constant is then skipped, losing exactly that row. On PG17 the shell stays 36/36 green.
That's because no >= here lands on a row-group edge. The groups are [0,49999], [50000,99999], …, and the constants are 199900 and 0. A probe like id >= 49999 would catch it: group 0 skipped, row 49999 lost, against the heap's 150001 rows. The <= / > / = arms have the same edge question. The gap is on main too, so it's not blocking, but it contradicts "nothing weak in the suite". I'd add it in a follow-up, or I can.
What I could not verify: I could not run the pytest twin. My pytest env (/root/pyenv) has no pyarrow, so all four tests declined (4 declined), and the system python has pyarrow but no pytest/psycopg. So the twin's mutant parity rests on your measurement and CI's green run, not on mine.
The shell side and the ledger counts read right.
|
@OffgridwithJD, an update to my review. I've installed pyarrow 26.0.0 into
My approvals stand. I'll take the edge-probe follow-up for |
|
@OffgridwithJD, merging #1496 now: approved at 860819e, 15/15 green, CLEAN. |
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clusive bound read as exclusive skips the match (#1496 review) Every oracle predicate in the suite stayed clear of a row-group edge (group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip test for `col >= const` changed to skip on max <= const, the suite was 36/36 green in both twins; the same held for `<=` read as `<` and for `=` read as half-open at the max. Both twins gain six "row-group edge [pred] == oracle" arms on those constants. Measured, PG17, each mutant reddens exactly its arm in both twins: - >= skipping on max <= const: [id >= 49999] - <= skipping on min >= const: [id <= 50000] - = skipping on const >= max: [id = 49999] - = skipping on const <= min: [id = 50000] Main's shell under the <= and = (max) mutants: 36/36 green. Ledger: six rows on 15-19 from clean logs of each major, four dated by those mutations; "id > 49998" and "id < 50001" enter never. Census 4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…up-edges test: native_parquet_pushdown checks the row-group edges, where an inclusive bound read as exclusive skips the match (#1496 review)
@jdatcmd The next twin-less suite, stacked on #1493 (on #1489). Merge those first; this PR's own commits are the top two.
The twin, name for name
All 36 checks by the shell's name, in four tests. Each test writes its file with pyarrow, and when pyarrow is absent it declines under the shell's own
pgc_skipname. Under each of seven mutations of the FDW's skip decision (pqfdw_clause_excludes_group/pqfdw_compute_skip), the twin reddens exactly the shell's set:>and>=pushed (NaN-unsafe)col < constskips the groups that matchThe last two matter: the five mutations that only stop skipping can't change an answer, so they can't date the oracle arms. Those two are wrong skips, and they date them.
Each test creates its own foreign server, because a server is database-wide and
pgc_connisolates by schema only.Five-major runs at head: 36/36 for the shell and 36/36 for the twin, on each of 15–19. I found nothing weak in the suite itself.
Ledger
Census 4263 / 3002 never / 1261 dated (on #1493's 4227 / 2991 / 1236), with suites_not_covered 197 → 196. 25 of the 36 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 11 arrive never:
read_parquetoracles, which no pushdown reaches;Collection gives
cluster_tests 730 -> 734, and TESTS.md gets §169. The grader shows 17 literal + 3 template matches with 0 missing, the loop-reader list gains the suite, and the guards pass 438/438.🤖 Generated with Claude Code