Skip to content

test: native_parquet_pushdown checks the row-group edges, where an inclusive bound read as exclusive skips the match (#1496 review) - #1501

Merged
jdatcmd merged 1 commit into
mainfrom
test/1496-parquet-row-group-edges
Oct 17, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
test/1496-parquet-row-group-edges

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

The follow-up I offered on #1496.

What was missing

Every oracle predicate in native_parquet_pushdown stays clear of a row-group edge: group 0 ends at 49999, group 1 starts at 50000. An inclusive bound read as exclusive skips the group that holds the match, and only a constant on the edge can see it. On main, both twins stayed 36/36 green under each of these pqfdw_clause_excludes_group mutants:

  • col >= c skipping on max <= c;
  • col <= c skipping on min >= c;
  • col = c skipping on c >= max.

The fix (both twins)

Six "row-group edge [pred] == oracle" arms: id >= 49999, id > 49998, id <= 50000, id < 50001, id = 49999, id = 50000.

Measured (PG17)

mutant red, shell = twin
>= skips on max <= c [id >= 49999]
<= skips on min >= c [id <= 50000]
= skips on c >= max [id = 49999]
= skips on c <= min [id = 50000]

Each mutant reddens exactly its arm, and nothing else.

Matrix:

  • both twins 42/42 on PG15–19 (pytest with pyarrow, PGC_REQUIRE=pyarrow);
  • harness_selftest 1246;
  • collection 737;
  • guards 438/438.

Ledger

  • Rows: six new rows on 15–19, from a clean log of each major.
  • Dated: four, by the mutants above.
  • Never: "id > 49998" and "id < 50001", because no natural off-by-one over-skips them.
  • Census: 4310 → 4316 rows, 3010 → 3012 never, 1300 → 1304 dated. The gate returns rc 0, ceiling 194 → 194.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approve, at a9d0acd. Thanks for the follow-up; these are the edges my #1496 missed.

Measured on PG17, each mutant build's source fingerprint checked against head:

mutant (pqfdw_clause_excludes_group) main's suite this PR
none 36/36 42/42
>= skips on max <= c 36/36 green red: row-group edge [id >= 49999] == oracle, alone
= skips on c >= max 36/36 green red: row-group edge [id = 49999] == oracle, alone

The twin passes 4/4 and compare_to_bash covers every property. The tsv gives 4316 / 3012 / 1304, which equals the paragraph.

It's CONFLICTING now: #1499 and #1500 are in or landing, and both touch the ledger. So it needs a re-anchor on whatever main is when you merge. The row changes stay the same.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, restated onto #1490: now cf918a7. Bookkeeping only; details are on #1490.

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd One line, from the restate. In test/check_ledger_budget.txt at 27c7972, line 3635 of your #1496-review paragraph was overwritten by an arithmetic line:

# an inclusive bound read as exclusive skips the group holding the match. Four are
#     dated         1370 + 4 = 1374
# and "id < 50001" enter never, as no natural off-by-one over-skips them.

At a9d0acd, which I approved, that line read:

# dated by that mutation for >=, <=, and = at the max and at the min; "id > 49998"

It looks like the line-replace aimed at the dated arithmetic row also caught the prose line that begins "# dated by". The numbers themselves are right: 3078 + 2 = 3080, 1370 + 4 = 1374, 4448 + 6 = 4454, and 3080 + 1374 == 4454. With the prose line back, I'll approve.

🤖 Generated with Claude Code

…clusive bound read as exclusive skips the match (#1496 review)

Every oracle predicate in the suite stayed clear of a row-group edge
(group 0 ends at 49999, group 1 starts at 50000). With the FDW's skip
test for `col >= const` changed to skip on max <= const, the suite was
36/36 green in both twins; the same held for `<=` read as `<` and for
`=` read as half-open at the max.

Both twins gain six "row-group edge [pred] == oracle" arms on those
constants. Measured, PG17, each mutant reddens exactly its arm in both
twins:
- >= skipping on max <= const: [id >= 49999]
- <= skipping on min >= const: [id <= 50000]
- =  skipping on const >= max: [id = 49999]
- =  skipping on const <= min: [id = 50000]
Main's shell under the <= and = (max) mutants: 36/36 green.

Ledger: six rows on 15-19 from clean logs of each major, four dated by
those mutations; "id > 49998" and "id < 50001" enter never. Census
4310 -> 4316 rows, 3010 -> 3012 never, 1300 -> 1304 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the test/1496-parquet-row-group-edges branch from 27c7972 to 63970d0 Compare October 17, 2026 02:43
@jdatcmd

jdatcmd commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, fixed at 63970d00, rebased onto main after #1490 and #1510 merged. Thanks, that was a real bug in my re-anchoring script, not one bad line.

The cause. reanchor.py rewrites the census rows with re.sub(r"^#\s+dated.*$", …), and the same for ledger rows and never. Your prose line begins "# dated by that mutation…", so it matched and was overwritten. The three patterns now require the arithmetic form N ± d = M. Tested: # dated 1370 + 4 = 1374 matches, and # dated by that mutation for >=, … and # never reddened by anything do not.

The paragraph is restored, and I diffed it against a9d0acd8, the version you approved. The two are identical except the header's census (3010 → 3012 then, 3078 → 3080 now) and the four arithmetic rows. Gate rc 0 (census 3080 agrees), guards 440, test_mutation_ledger and harness_deps 64 passed.

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approved at 63970d0. The prose line is back: "# dated by that mutation for >=, <=, and = at the max and at the min; "id > 49998"". The range-diff against 27c7972 shows only that line changed. It is based on b92a244, whose census is 3078, so 3078 -> 3080 and 4448 + 6 = 4454 still hold. CI was still running when I looked (11 passed, 4 pending).

🤖 Generated with Claude Code

@jdatcmd
jdatcmd merged commit 5351aba into main Oct 17, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants