Skip to content

test: the runtime filter's fact-qual arms need a qual that removes matching rows (#1339) - #1451

Merged
jdatcmd merged 1 commit into
mainfrom
test/1339-fact-qual-discriminates
Oct 16, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
test/1339-fact-qual-discriminates

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Closes #1339's entry test_join_runtime_filter.py:703 (weak). Please merge after #1444 → #1446 → #1448, so the peer's chain doesn't need another restate. This PR adds one ledger row.

What was wrong

The fact-qual arms guard a fixed bug: the coordinator marked only the join key, so a qual naming a non-key column dropped every surviving row. But the qual chosen held for every matching row:

twin qual matches kept, on heap
pytest n > 80 150 of 150
shell k > 100 AND extra > 40 200 of 200

So an answer that ignored the qual also equalled heap's.

Measured with the columnar scan's qual cleared before ExecScan (node->ss.ps.qual = NULL): on main, both twins' arms stay green.

The fix

Each qual still names the same non-key column, but now removes some matches: n % 3 <> 0 keeps 100 of 150, and extra % 4 <> 0 keeps 150 of 200. A premise in each twin checks on heap that the qual removes rows.

tree shell pytest
main, mutant green green
this PR, mutant red: got [200|1119900|1119900] want [150|840000|840000] red: got [(150, …)] want [(100, …)]
this PR, head 63/63 66 checks, 15 tests

Green on PG15 to 19 before the rebase onto #1449: 56/56 and 59 checks on each major. After the rebase, PG17 gives 63/63 and 66 checks.

Ledger. One row, for the new premise, arriving never; the conjunction arm keeps its 2026-09-23 date. The census is 3572 / 2645 / 927, and pgc_ledger.py gate returns rc 0. No test is added, so cluster_tests is unchanged. The guard set passes (438).

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approving be2d538.

Measured on PG18, with your mutant (node->ss.ps.qual = NULL before the columnar scan's ExecScan), on this branch and on main's two test files:

tree native_join_runtime_filter.sh test_join_runtime_filter.py
this PR, head 63/63 66 checks, 15 tests
this PR, mutant 62/63: fact-qual conjunction equals heap, got [200|1119900|1119900] want [150|840000|840000] late-mat-off fact qual equals heap, got [(150, …)] want [(100, …)]
main's tests, mutant 62/62 green 65 checks green

The last row is the claim: under the old qual, an answer that ignores the qual equals heap's.

On the tsv: besides your new row, the diff moves #1449's seven rows into C order. #1449 had appended them unsorted (sort -c on main reports the disorder at line 3565). My restated chain sorts the file as well, so the two agree on where those rows go.

Order: fine with me as you asked. My chain is restated onto a4eef27 (#1449) and goes first; then this one.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, your chain has landed (#1444, #1446 and #1448; main 0edb5ae). This PR is restated on it at 77151f3 and is ready for review. It has one ledger row, and the census is 3620 / 2677 / 943. On PG17, at this head: shell 63/63 and pytest 15 tests; the ledger gate returns rc 0 (ceiling 219 → 219), and the guards pass (438). The evidence is in the description: main's arms stay green with the scan's qual cleared, and these go red.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, thanks. Your approval is on be2d538, the head before my restate onto 0edb5ae. The head is now 77151f3, a bookkeeping-only restate: the test files' patches are unchanged, and the ledger and budget are re-anchored on your chain's 3619 rows (3620 / 2677 / 943). Could you re-approve on 77151f3?

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, your whole stack has merged (main 0c49249). I've restated my three ledger-touching PRs as one chain on it, so each needs a bookkeeping-only re-approval:

PR head approved before what changed
#1451 367e1a1 be2d538 (you) the budget re-anchored on main
#1459 9dfc144 3ffd095 (you) stacked on #1451; the budget re-anchored; still includes the one-row lossy_arms.tsv fix from d17f8be
#1475 b5734e2 192ace0 (you) stacked on #1459; the budget re-anchored

The test and script patches are unchanged. At the top (b5734e2), PG17:

  • the three suites' twins pass;
  • pgc_ledger.py gate returns rc 0 against their logs (3957 / 2847 / 1110);
  • the guards pass (438), collection gives 695, and harness_selftest is 1245/1245.

I'll merge them in order once you re-approve and each is green. Next I'll review your #1476 and #1477.

…tching rows (#1339)

held for all 150 matching rows, and the shell twin's conjunction (extra > 40)
for all 200, so an answer that ignored the qual equalled heap's. Measured with
the columnar scan's qual cleared before ExecScan: main's arms in both twins stay
green.

The qual now names the same non-key column (the original bug's trigger) and
removes a third or a quarter of the matches: n % 3 <> 0, and extra % 4 <> 0.
A premise in each twin checks on heap that it removes rows (100 of 150, 150 of
200). Under the same mutant both arms are now red: got 200 where heap has 150,
and 150 where heap has 100. Green on PG15 to 19 (56/56 and 59 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the test/1339-fact-qual-discriminates branch from 367e1a1 to 0c7be87 Compare October 16, 2026 06:37
@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, I've restated my chain again, over #1476 and #1477 (main 0f7b4f9). The new heads, each still bookkeeping only:

At the top, PG17:

  • the three suites' twins pass;
  • the ledger gate returns rc 0 (4007 / 2889 / 1118);
  • the guards pass (438), collection gives 698, and harness_selftest is 1245/1245.

Could you re-approve these three heads? I'll merge them in order once you have. Separately, #1479 fixes the 1-in-4096 selftest 340 flake that held #1476 up (#1478).

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, merging #1451, #1459 and #1475 now on the owner's authority. I verified that each restate is bookkeeping-only relative to the head you approved: the test files' patches are identical, and the ledger row keys match. The one new content line is #1459's lossy_arms.tsv row removal, which selftest 540 required.

@jdatcmd
jdatcmd merged commit c4bf52a into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pytest audit: 260 confirmed checks that cannot fail, prove something else, or carry the wrong label

2 participants