Skip to content

test: native_groupagg's upper-bound and NULL-key arms can fail, and a fixture fact is an arm (#1339) - #1459

Merged
jdatcmd merged 3 commits into
mainfrom
test/1339-groupagg-bounds
Oct 16, 2026
Merged

jdatcmd merged 3 commits into
mainfrom
test/1339-groupagg-bounds

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Closes #1339's second test_native_groupagg.py entry (:168, :595, :847). Please merge after #1451 and the #1450 → #1458 stack. It re-dates two ledger rows and adds a budget paragraph, so landing it later costs the peer no restate.

What was wrong, and the mutants that prove it

finding defect mutant main this PR
:595 inaccurate "and it is an UPPER bound" compared the true count with the literal 12, not with the bound produced the bucket bound's + 2.0 dropped (10 for 11 true groups) green (only the accuracy arm reddens) red: no (11 groups over a bound of 10)
:847 weak the NULL-key arm read only the answer (4 groups) the bound without the NULL group (*withNull *= one) and the runtime cap off green: the node took the query and answered 4 red in both twins: got [1|4] want [0|4]
:168 mislabelled "premise: the pair loaded in full" can be reddened by a repository change (an INSERT path that loses rows) — — now an arm (#1236); pytest only, since the shell has no such check

The fixes. The upper-bound arm compares with the bound measured. The NULL-key arm also requires that the node declined (under_cap 3 = 0), alongside the 4 groups.

A mistake of mine, caught by measuring. My first pytest version of the NULL-key check tested whether the marker was a member of EXPLAIN's JSON plan, which is a list of dicts. That test is always false, so the check passed under the mutant. It now searches the plan's text, and the mutant reddens it.

Evidence

  • Head, PG15 to 19: native_groupagg.sh 94/94, and test_native_groupagg.py 102 checks over 20 tests, on each major.
  • The mutants, on PG17: the rows above. Under the +2 mutant, pytest stops at the accuracy arm, which comes first in the same test, so its upper-bound arm isn't reached there. The shell shows it red.
  • Ledger: no row is added or removed. The two changed arms are re-dated by these mutants, never → 2026-10-16. The census is 3619 / 2674 / 945, and pgc_ledger.py gate returns rc 0 (ceiling 219 → 219); orphan-scan reports 0 orphans.
  • Counts: collection gives 660, unchanged. The guard set passes (438).

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approve.

I reproduced both mutants on my own PG18 build (~/pg/18-r) at 3ffd0959, in a scratch worktree, and restored the source after each.

NULL key (*withNull *= one, plus state->maxGroups = INT_MAX). I ran the pytest twin with -k "null_key or bound_is_accurate":

tests result
this PR red: got '1|4' want '0|4'
main's test_native_groupagg.py 10/10 green

Upper bound (floor((hi - lo) / width), without the + 2.0). I ran the shell:

shell "the bound is accurate …" "and it is an UPPER bound …"
this PR FAIL [10] FAIL no (11 groups over a bound of 10)
main FAIL [10] PASS: the vacuity you describe

So both re-datings stand, and both are from the product, not the harness.

Three observations, none of them blocking:

  1. As you say, the pytest upper-bound arm can't be dated from pytest under this mutant, because expect stops at the accuracy arm first. The ledger row it re-dates is the shell's, so that's fine. If a later change lets that test continue past a failure, though, actual <= bound with bound = None would raise TypeError instead of giving a clean red.
  2. Searching the json.dumps text for MARKER is right, and your own mutant run shows it isn't vacuous. MARKER is plain ASCII, so JSON escaping can't hide it.
  3. On merge order: my new #1461 (pg_upgrade twin, stacked on #1458) touches no ledger or budget file. Your checks_never_observed_red restate doesn't need to account for it, only for #1451 and the #1450 → #1458 stack, as you wrote.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, CI's suites (PG 17) and (PG 18) failed on 3ffd095, and it's my miss. Selftest 540 found the upper-bound arm repaired: it now prints "no (N groups over a bound of M)". It then refused the stale row in test/lossy_arms.tsv, with "a repaired arm must have its row removed, so the list only shrinks". My matrix ran the suite, the twin and the guards, but not harness_selftest.

Fixed in d17f8be: that one row removed, and nothing else changed. harness_selftest is 1245/1245 locally. Could you re-approve on d17f8be? I've also added harness_selftest to my matrix script, so a check whose failure output changes is run past selftest 540 before I push.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, a reminder, in case these got lost under the stack. Two of mine need only a re-approval on their current heads:

Both are green.

jdatcmd and others added 3 commits October 16, 2026 06:33
…tching rows (#1339)

held for all 150 matching rows, and the shell twin's conjunction (extra > 40)
for all 200, so an answer that ignored the qual equalled heap's. Measured with
the columnar scan's qual cleared before ExecScan: main's arms in both twins stay
green.

The qual now names the same non-key column (the original bug's trigger) and
removes a third or a quarter of the matches: n % 3 <> 0, and extra % 4 <> 0.
A premise in each twin checks on heap that it removes rows (100 of 150, 150 of
200). Under the same mutant both arms are now red: got 200 where heap has 150,
and 150 where heap has 100. Green on PG15 to 19 (56/56 and 59 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fixture fact is an arm (#1339)

Three #1339 findings in test_native_groupagg.py:

- :595, inaccurate. "and it is an UPPER bound" compared the true group count
  with the literal 12, not with the bound the planner produced. With the bound's
  two partial buckets dropped (10 for 11 true groups), only the accuracy arm
  failed; this one still said yes. Both twins now compare with the bound
  measured: red under that mutant, "11 groups over a bound of 10".
- :847, weak. The NULL-key arm read only the answer, 4 groups. With the bucket
  bound leaving out the NULL group and the runtime cap off, the node took the
  query and answered 4, so the arm passed. Both twins now also require that the
  node declined: red under that mutant, "1|4" for "0|4".
- :168, mislabelled. "premise: the pair loaded in full" can be reddened by a
  repository change (an INSERT path that loses rows), so it is an arm (#1236).
  pytest only; the shell has no such check.

Both changed shell arms are re-dated by these mutants. Green on PG15 to 19:
94/94 and 102 checks. A first pytest version of the NULL-key check tested
membership in EXPLAIN's JSON list and was always false; the mutant run caught
it, and the check now searches the plan's text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…now prints what it measured (#1339)

The arm now reports "no (N groups over a bound of M)" on failure, so selftest 540
found it repaired and refused the stale row: "a repaired arm must have its row
removed, so the list only shrinks". harness_selftest is 1245/1245 with the row
removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the test/1339-groupagg-bounds branch from 9dfc144 to 32c7635 Compare October 16, 2026 06:37
@jdatcmd
jdatcmd merged commit 6a8e107 into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pytest audit: 260 confirmed checks that cannot fail, prove something else, or carry the wrong label

2 participants