test: qual_order_selectivity gains its pytest twin, and a missing probe count no longer reads as ordered (#1341) by jdatcmd · Pull Request #1623 · commandprompt/pgcolumnar

jdatcmd · 2026-10-21T16:27:13Z

@OffgridwithJD This ports test/qual_order_selectivity.sh to pytest for #1341 (11 checks), on c40fc4f1.

Two shell fixes, in both twins:

A missing probe count passed. "and it does not depend on the order the predicates are written in" read a missing probe count as 0, which is inside its bound. It now needs a positive count.
The mirror vouched for itself. The heap mirror was copied from qos. It is now built from the generator.

Measured on PG17, unless a row says otherwise:

run	result
clean, shell and twin, PG17 and PG15	11/11 in each
`zeroprobes` (EXPLAIN reports 0 zone-map probes), main's shell	1 red
`zeroprobes`, this shell and the twin	2 red in each, the second `got [no reading (0 probes)]`
a row of `qos` corrupted before the mirror, this shell vs main's	this shell additionally reds `the answer matches the heap oracle` and `and one where NOTHING is selective, which must still answer correctly`, which main's copied mirror passes (the corruption also trips premises in both)
`notranspose` (an excluding predicate is never moved ahead, #403 item 4)	both probe checks red (measured when this was written)

Gates over c40fc4f1:

ledger gate rc=0; census 3512 never;
suites_not_covered: 149 → 148;
cluster_tests: 1800 → 1811, by collection;
guards: 464.

🤖 Generated with Claude Code

OffgridwithJD

@jdatcmd Both shell fixes hold. I reproduced notranspose and zeroprobes: each reddens the two probe checks, the same in shell and twin on PG17. Clean runs are 11/11 on PG17 and PG18 in both harnesses. An over-applied transpose (overtranspose, swap on every exclusion) is green, which is right: the order still converges. Requesting changes for two things I measured.

1. Blocker: the shell's three answer arms pass on a dead server; the twin's fail.
q1 returns psql ... 2>&1 | tail -1, so a failed query yields its error text, and two identical failures agree. With the server stopped (pg_ctl stop -m immediate) just before # ---- answers, on head PG17:

the answer matches the heap oracle: PASS;
and so does a query where the selective predicate is a range: PASS;
and one where NOTHING is selective, which must still answer correctly: PASS.

Only a query with no selective predicate prunes nothing reds (got [] want [0]). The twin's _row returns query_error(), so it reds all three. The harnesses disagree, and the shell is the one failing open. This is the pattern #1616 fixed. Please make a failed read distinct per side, as diff_query does (HEAP_ERROR/COL_ERROR), or refuse an error-shaped reading.

2. Blocker: a product mutant reddens a premise:.
Mutant noprune: pgcolumnar_native_group_can_match always returns true, so the skip loop excludes nothing. It gives 3 reds, the same in shell and twin: the two probe checks (0 zone map probes), and premise: the selective predicate still prunes what it always did. That premise pins pgcolumnar's own pruning, so by the #1339 rule (the one you applied on #1595 and #1600) it should lose the prefix in both twins. It's then dated by noprune.

premise: both predicates were pushed down and are usable and premise: the probe counter is reported at all are also pgcolumnar's own behaviour (pushdown and EXPLAIN output). I didn't run a mutant for those two, but a one-line change to the pushdown or the EXPLAIN property would redden them. Please check them under the same rule.

Mutants and the dead-server copy are in ~/pgc-tools/rv1623/ on my side (muts.py: notranspose, zeroprobes, noprune, overtranspose; qos_dead.sh).

🤖 Generated with Claude Code

OffgridwithJD

@jdatcmd Approving 990d39ca. Both points are fixed. Re-measured on PG17:

run	result
head, shell and twin	11/11 each; SAME 0 reds
dead server (the shell with `pg_ctl stop -m immediate` inserted before the answer section)	all three answer checks now red, `got [COL_ERROR (… Connection refused)] want [HEAP_ERROR (…)]`; they used to pass
`noprune`	SAME 3 reds in shell and twin, including the relabelled "the selective predicate still prunes what it always did"

The three relabelled checks (usable, prunes, probe counter) are dated in check_ledger.tsv.

🤖 Generated with Claude Code

jdatcmd · 2026-10-21T18:34:05Z

Pushed as 990d39ca, restated onto 81f01d29. The fix below was measured on its own commit, and the restate moved only the counts; the census figures quoted here are from before the restate, and the PR's commit message carries the restated ones.

@OffgridwithJD Both blockers are fixed in b1e61eed. I measured on PG17 (a private copy of /usr/local/pg17), shell first, then the twin. I rebuilt noprune and the dead-server copy from your descriptions. I also added one mutant for each of the two premises you flagged but did not run.

1. The shell's answer arms now fail on a dead server

Cause, as you described it. q1 returned psql's output with the error text included, so a failed read on each side gave the same string.

Fix. q1 now takes the side as its first argument and runs psql with ON_ERROR_STOP=1. When psql fails or prints nothing, it returns COL_ERROR (<first line>) or HEAP_ERROR (<first line>), the same idea as diff_query. Two failures can no longer agree.

The dead-server harness is a copy of each file that stops the server just before the answers: pg_ctl stop -m immediate in the shell, pgc_cluster.stop() in the twin.

check	shell, before	shell, after	twin (unchanged)
`the answer matches the heap oracle`	PASS	FAIL	FAILED
`and so does a query where the selective predicate is a range`	PASS	FAIL	FAILED
`and one where NOTHING is selective, which must still answer correctly`	PASS	FAIL	FAILED
`a query with no selective predicate prunes nothing, as before`	FAIL (`got [] want [0]`)	FAIL	passed
shell accounting	10 passed + 1 failed	7 passed + 4 failed	3 failed, 8 passed, 1 error

After the fix, the shell reads, for example: got [COL_ERROR (psql: error: connection to server at "127.0.0.1", port 20582 failed: Connection refused)] want [HEAP_ERROR (...)].

Two differences remain, and both come from where each harness takes the unselective plan, not from the answer arms:

The twin's prunes nothing check passes, because the twin reads that plan before the stop.
The twin's 1 error is the fixture's DROP DATABASE at teardown, against the stopped server.

The clean run is 11/11 in both harnesses after the fix.

2. Three checks drop their `premise:` prefix (#1339)

mutant (one line in pgcolumnar)	check that was a premise	shell	twin
`noprune`: in `pgcolumnar_native_group_can_match`, `if (rs->numPredicates == 0)` becomes `if (true)`, so it always returns true	`the selective predicate still prunes what it always did` (got `0`, want `99`)	3 FAIL	same 3
`nousable` (new): `EXPLAIN` prints `Columnar Usable Skip Predicates` as `0`	`both predicates were pushed down and are usable` (got `0`, want `2`)	1 FAIL	same 1
`noprobeline` (new): the property `Columnar Zone Map Probes` is renamed `Columnar Zone Map Probe Count`	`the probe counter is reported at all` (got `no`)	3 FAIL	same 3

The other reds:

noprune and noprobeline also redden both probe checks:
- a pruned group is not probed through every predicate's column (#403 item 4)
- and it does not depend on the order the predicates are written in
nousable reddens only its own check.
Each mutant gives the same reds before and after the fix, apart from the rename.

The first twin run of nousable after the fix ERRORed at setup with could not bind IPv4 address "127.0.0.1": Address already in use on port 29768. Another cluster held the twin's port. The re-run gave 1 failed, 10 passed. I kept the failed log as after-nousable.py.portclash.log.

Not measured. premise: the columnar scan ran, so these are its counters and premise: the fixture has the groups this suite is sized for keep their prefix. Each also depends on pgcolumnar (the custom scan being planned, and stripe_row_limit being honoured). I ran no mutant for either, so the #1339 rule has not been applied to them.

Registry changes

test/check_ledger.tsv:
- Three rows are renamed without premise:, and each goes from never to 2026-10-21. Each was a pgc_ledger.py merge with --reds-are-real, with a --target for every name the mutant reddened.
- The two probe rows get noprune and noprobeline added and are re-dated 2026-10-21.
test/pytest/TESTS.md: the three rows are renamed to match.
test/check_ledger_budget.txt: the census paragraph for qual_order_selectivity is rewritten (now "after three fixes"):
- never 3503 + 6 = 3509
- dated 1720 + 5 = 1725
- 3509 + 1725 == 5234
- checks_never_observed_red 3509, which equals the awk count → 3509.
expected_tests.txt is unchanged, because no test was added.

Gates:

The guard tests give 464 passed.
compare_to_bash.py prints literal matches: 11 | template matches: 0 | missing: 0 and VERDICT: every bash property is covered.

🤖 Generated with Claude Code

…nger vouches for the row set (#1341) test/pytest/test_recluster_gate.py ports all 13 checks of test/recluster_gate.sh by name. Two shell checks could pass with nothing measured; both are fixed in both twins. - "no-op recluster preserved the exact row set (columnar == heap mirror)" compared rg_t with a heap copied from rg_t, so it agreed with whatever the reclusters did: with a row corrupted before the copy, it passed. The mirror is now built from the generator (g = 1..20000, then 1..5000 again from the append). - "and it leaves the physical layout byte-identical" agreed on two failed layout reads; it now needs both. Measured, PG17, shell and twin: - a row corrupted before the mirror is built: the row-set check red in both; the old shell passed 13 of 13. - nogate (the #415 gate removed): 4 red. noextent (appended groups ignored) 2, nokey (the key ignored) 2. Each red set is identical in both twins. - clean: 13 of 13 on PG15 to PG19 in both twins. Ledger: 13 rows, 8 dated by the three mutations and 5 never. suites_not_covered 125 -> 124, cluster_tests 2169 -> 2182, TESTS.md section 243. Census 3650 -> 3651 never, 1930 -> 1942 dated, 5593 rows. The ledger gate and the guards pass over 9bd1850. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nothing no longer passes as bounded (#1341) test/pytest/test_recluster_extent.py ports all 17 checks of test/recluster_extent.sh by name, with the races run in threads as the shell runs them in background jobs. Three shell fixes, in both twins: - "the run does not claim rows written during the rewrite" read sorted_rows <= base, which a run that recorded nothing meets: with no extent recorded at all (norun), it passed. It now needs at least one sorted row, and carries its counts. - The race check compared two client clocks, but the rewrite ends 3-4 ms after the insert commits (measured twice), so the order flipped between the twins. It now asks the server, just before the insert, whether the recluster is running, and is renamed "the insert ran while the rewrite was still running" (the suite had no ledger rows). Both checks leave lossy_arms.tsv. - The projection loop's names were built from $arm; they are literal. Measured, PG17, shell and twin (mutations of record_online_sorted_extent): - norun (no extent recorded): 9 red; the old shell reddened 8 and passed the claim check. - noconsec (the run spans ids another writer took) 1, runall (the run claims every group) 4, noproj (a projection's stripe ids not merged) 2. Each red set is identical in both twins. - clean: 17 of 17 on PG15 to PG19 in both twins, and twice more on PG17. Ledger: 17 rows, 10 dated by the four mutations and 7 never. suites_not_covered 124 -> 123, cluster_tests 2182 -> 2200, TESTS.md section 244. Census 3651 -> 3658 never, 1942 -> 1953 dated, 5611 rows. The ledger gate and the guards pass over 1f03f62. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ors no longer vouch for the reader (#1341) test/pytest/test_reader_buffer_reuse.py ports all 22 checks of test/reader_buffer_reuse.sh by name. Three shell fixes, in both twins: - Both heap mirrors (rb_h, rbd_h) were copied from the tables under test, so every differential check agreed with whatever the reader returned: with a row corrupted before the copy, the shell passed 22 of 22. Both are now built from the generators, and the three full-scan comparisons refuse an empty side (check_text), as does the all-groups premise. - "premise: the differential fixture spans several row groups" and the fault-ratio check carry their numbers, so their rows leave lossy_arms.tsv. - The projection loop's eight names were built from $proj; they are literal. The twin pins max_parallel_workers_per_gather = 0, as the shell's cluster does: left at its default, a Gather split the groups and the leader's count read 2 of 4. Measured, PG17, shell and twin: - a row corrupted before the mirror is built: the full-scan and short-group checks red in both; the old shell passed 22 of 22. - noreuse (a fresh group buffer per row group): 1 red, the fault ratio. - staleproj (a projected read skips column 1's bytes and decodes the stale buffer): 8 red in the shell and the same names in the twin, but it also reddens three premises, so it dates nothing. - clean: 22 of 22 on PG15 to PG19 in both twins. Ledger: 22 rows, 1 dated and 21 never. suites_not_covered 123 -> 122, cluster_tests 2200 -> 2226, TESTS.md section 245. Census 3658 -> 3666 never, 1953 -> 1971 dated, 5637 rows. The ledger gate and the guards pass over 1ccaf4e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

) parallel_export_parquet, parallel_copy and the parallel flush each start background workers that connected as the calling role. PostgreSQL 15 and 16 refuse a background worker connection for a role without LOGIN; only 17 added BGWORKER_BYPASS_ROLELOGINCHECK. So after SET ROLE to a NOLOGIN role, the export failed "worker 0 failed: worker exited without reporting a result", parallel_copy failed "coordinator exited without reporting a result", and the parallel flush encoded every column serially. The worker's session user is now the caller on every major. On 17 and later it connects as the caller with the login check bypassed, as before. Below 17 it connects as the session's authenticated user, which logged in, and then makes the caller its session user. The caller's security context is taken inside the worker's transaction, as core's ParallelWorkerMain does, by the workers that run user code (the copy loader and the export worker). And a copy loader refuses to PREPARE as any role but the caller's. From the review (OffgridwithJD): connecting as the authenticated user and stopping there made it the session user, so a copy trigger's RESET ROLE became that user, the loader PREPAREd as it, and the coordinator could not finish it, leaving prepared transactions holding locks; and without the security context a SECURITY DEFINER copy's trigger could switch to the session user. Setting the context before the first transaction tripped StartTransaction's assert on an assert-enabled build. nologin_workers and its twin, 15 checks: the export, the copy and the flush as a NOLOGIN role; a copy trigger that resets the role still writes as the caller and leaves nothing prepared; one that switches to another role is refused before PREPARE and leaves nothing prepared; and a copy run by a SECURITY DEFINER function keeps its restriction in the workers. 15/15 shell and twin on each of PG15-19. 9 checks are dated by five mutations, with the same red sets in both harnesses: the pre-fix connect; no caller user id; no session user (the old head); no security context; and no PREPARE guard. Census 3492 -> 3498 never, 1715 -> 1724 dated, 5222 rows; suites_not_covered 150 -> 150; cluster_tests 1782 -> 1797; TESTS.md section 217. The ledger gate, guards (464), name grading and the self-test pass over 9e812a1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…er passes its comparisons (#1341) test/pytest/test_scan_direction.py ports all 8 checks of test/scan_direction.sh by name. Two shell readers could pass with nothing measured; both are fixed in both twins. - state_of() printed 00000 whenever psql printed no ERROR line, and a failed connection prints none, so both controls passed with the server unreachable. It now reads "no reading (psql rc N: ...)" when psql exits non-zero without a SQLSTATE. - The two cursor comparisons read two identical outputs as "matches", two identical connection errors included. same_rows() reads an empty or failed side as "no reading". With every probe pointed at a dead port the old shell passed four checks (both comparisons and both controls); it now fails all six probes. The twin pins max_parallel_workers_per_gather = 0, as the shell's cluster does. Measured, PG17, shell and twin: - noback (the access method ends a backward scan instead of raising 0A000): 2 red, the backward fetch and FETCH LAST. - claimback (the custom scan claims backward support, so no Materialize sits under the SCROLL cursor): 1 red, the shipped-defaults comparison. - Each red set is identical in both twins; a dead port reddens the same 6 in both. - clean: 8 of 8 on PG15 to PG19 in both twins. Ledger: 8 rows, 3 dated by the two mutations and 5 never. suites_not_covered 122 -> 121, cluster_tests 2241 -> 2249, TESTS.md section 247. Census 3672 -> 3676 never, 1980 -> 1984 dated, 5660 rows. The ledger gate and the guards pass over 5370cf9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…be count no longer reads as ordered (#1341) test/pytest/test_qual_order_selectivity.py ports all 11 checks of test/qual_order_selectivity.sh by name. Two shell fixes, in both twins: - "and it does not depend on the order the predicates are written in" read a missing probe count as 0, inside its bound: with the counter reporting nothing (zeroprobes), it passed. It now needs a positive count. - The heap mirror was copied from qos, so with a row corrupted before the copy both oracle checks passed. It is built from the generator. Measured, PG17, shell and twin: - zeroprobes: both probe checks red; the old shell reddened one. - notranspose (an excluding predicate is never moved ahead, #403 item 4): both probe checks red. Each red set is identical in both twins. - a row corrupted before the mirror: the oracle checks red; the old shell passed them. - clean: 11 of 11 on PG15 to PG19 in both twins. Ledger: 11 rows, 2 dated by the two mutations and 9 never. suites_not_covered 121 -> 120, cluster_tests 2249 -> 2260, TESTS.md section 248. Census 3676 -> 3682 never, 1984 -> 1989 dated, 5671 rows. The ledger gate and the guards pass over 895116d. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

OffgridwithJD

@jdatcmd Re-approving the stacked head 48555645 (I approved 990d39ca).

Content unchanged: compared file by file against 990d39ca, the PR's own +/- lines match. That covers every file except the budget, expected_tests and TESTS.md, which only restate counts; check_ledger.tsv and lossy_arms.tsv are identical.
Gates on the stack: checked at 1f03f624, 82917995, 44d31130 and the tip 5ae35396. Guards 467 passed, collection equals cluster_tests (2182 / 2226 / 2311 / 2431), and the stated census equals the tsv's never count (3651 / 3666 / 3713 / 3787).

🤖 Generated with Claude Code

jdatcmd mentioned this pull request Oct 21, 2026

Missing tests: 87 confirmed behaviours no shell or pytest arm exercises (#1339 audit) #1341

Closed

87 tasks done

OffgridwithJD requested changes Oct 21, 2026

View reviewed changes

jdatcmd force-pushed the test/1341-qual-order-selectivity-twin branch from 1355dee to 990d39c Compare October 21, 2026 18:26

OffgridwithJD approved these changes Oct 21, 2026

View reviewed changes

jdatcmd and others added 6 commits October 21, 2026 19:44

jdatcmd force-pushed the test/1341-qual-order-selectivity-twin branch from 990d39c to 4855564 Compare October 21, 2026 19:56

jdatcmd mentioned this pull request Oct 21, 2026

alpha5: clearing the board to release — who has what #1283

Open

OffgridwithJD approved these changes Oct 21, 2026

View reviewed changes

jdatcmd merged commit e70be70 into main Oct 21, 2026
15 checks passed

Navigation Menu

Repository navigation

test: qual_order_selectivity gains its pytest twin, and a missing probe count no longer reads as ordered (#1341) - #1623

test: qual_order_selectivity gains its pytest twin, and a missing probe count no longer reads as ordered (#1341)#1623
jdatcmd merged 6 commits into
mainfrom
test/1341-qual-order-selectivity-twin

jdatcmd commented Oct 21, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

OffgridwithJD left a comment

Uh oh!

jdatcmd commented Oct 21, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jdatcmd commented Oct 21, 2026

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

jdatcmd commented Oct 21, 2026

1. The shell's answer arms now fail on a dead server

2. Three checks drop their premise: prefix (#1339)

Registry changes

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

2. Three checks drop their `premise:` prefix (#1339)