test: parallel gains its pytest twin, and a parallel check that ran serially no longer passes (#1341) by jdatcmd · Pull Request #1672 · commandprompt/pgcolumnar

jdatcmd · 2026-10-22T02:57:03Z

@OffgridwithJD, this ports test/parallel.sh to test/pytest/test_parallel.py for #1341. All 13 checks are ported by name. Three shell checks could pass with the property broken. Each is fixed in both twins, and each fix adds one named check.

What was fixed

"par no-filter" ran serially. It was a bare count(*). The vectorized aggregate answers that without a Gather, so this check never reached the parallel scan. When every participant read every row group, main's shell reddened the other four par checks and left this one green. The query now returns rows. A new check, "par queries all plan a Gather", requires all five par queries to plan a Gather over a PgColumnarScan.
The 25 row groups were assumed. If the writer ignores stripe_row_limit, the table has 1 row group. With 1 group, a claim that skips every other group returns the right rows, and main's shell passed 10 of 10. A new check, "par table has 25 row groups to claim", counts the groups.
A serial setting that did not take went unnoticed. setg swallows its error. If max_parallel_workers_per_gather = 0 failed, the three serial checks ran in parallel and main's shell passed 10 of 10. A new check, "serial queries plan no Gather", catches this.

The queries are now defined once, so the plan checks read the same queries that the comparisons ran. A failed EXPLAIN counts for neither plan check, so a dead server cannot satisfy either one.

Measured on PG17

The mutants that change pgcolumnar are below the first rule. The rows marked "harness edit" change the test file only, so each one applies to one twin.

edit or mutant	main's shell	this shell	twin
every participant reads every row group (`columnar_reader.c`: claims from `rowGroupIndex++`, not the shared counter; D6e)	4 red, "par no-filter" green	5 red: the five par checks	the same 5
the writer ignores `stripe_row_limit` (`columnar_write_state.c`: `if (false && opts.stripeRowLimitSet)`)	0 red, 10/10	1 red: "par table has 25 row groups to claim"	the same 1
ignores `stripe_row_limit` and a claim skips every other group	0 red, 10/10	1 red: "par table has 25 row groups to claim"	the same 1
a claim skips every other group (`pg_atomic_fetch_add_u32(counter, 2)`)	4 red, "par no-filter" green	5 red: the five par checks	the same 5
the partial path is never added (`add_partial_path` dropped)	1 red	2 red: "parallel plan has Gather", "par queries all plan a Gather"	the same 2
the scan is named `ColumnarScan`, as before #428	1 red	3 red: "parallel plan has PgColumnarScan", "par queries all plan a Gather", "serial queries plan no Gather"	the same 3
a serial scan reads every other row group (`rowGroupIndex += 2`)	3 red	3 red: the three serial checks	the same 3
harness edit: the serial setting misspelled (`max_parallel_workers_per_gathr`)	0 red, 10/10	1 red: "serial queries plan no Gather"	twin's own edit: the same 1
harness edit: "par no-filter" put back to `count(*)`	(main's own query)	1 red: "par queries all plan a Gather" (got 4, want 5)	twin's own edit: the same 1

No mutant crashed the backend, so the red sets have no collateral reds.

The one red that each fix turned from green:

"par no-filter", when every participant reads every row group: FAIL par no-filter: got [467a38dfe69386b7e1ce31f04385c9a4] want [6cc72b3097d8320360f0e5e1af20d78f]
"par queries all plan a Gather", when the harness edit restores main's count(*): FAIL par queries all plan a Gather: got [4] want [5]
"par table has 25 row groups to claim", when the writer ignores stripe_row_limit: FAIL par table has 25 row groups to claim: got [1] want [25]
"serial queries plan no Gather", when the serial setting does not take: FAIL serial queries plan no Gather: got [0] want [3]

Gates

Clean runs: 13 of 13 on PG15, 16, 17, 18 and 19, in both the shell and the twin.
Ledger: 13 rows. All 13 are dated by five mutations: the writer ignoring stripe_row_limit, every participant reading every row group, the partial path never added, the scan renamed, and a serial scan reading every other group. 0 rows are never red. Never-red census: 3853 -> 3853. Total rows: 6096 -> 6109.
pgc_ledger.py gate against origin/main: rc 0, suites_not_covered 105 -> 104. orphan-scan: 0 orphans.
Guards: 474 passed. test_harness_deps.py and test_capability_decline.py with --pg-config: 30 passed. Capability-import scan: [].
cluster_tests 2677 -> 2690. The value is the collected count: 2690.
compare_to_bash.py: 13 literal matches. Verdict: every bash property is covered.
harness_selftest.sh on PG17: 1248 of 1248. The first run failed no suite pipes a captured string into an early-exit reader, because the new plans_with helper used echo "$plan" | grep -q. That helper now uses [[ ... == *...* ]]. After the change, the shell was rerun clean on all five majors and under every mutant, and the results were identical.
shellcheck -S error: clean.

TESTS.md section 265.

Re-checked by me before pushing (PG17, main 5a94757): the writer ignoring stripe_row_limit (both if (opts.stripeRowLimitSet) sites in columnar_write_state.c made false &&): main's shell 10/10; this shell 12 + 1 failed, "par table has 25 row groups to claim: got [1] want [25]"; the twin fails the same test. Two of the reds come from harness edits rather than product mutants ("par queries all plan a Gather" and "serial queries plan no Gather"); the PR body above says which.

🤖 Generated with Claude Code

OffgridwithJD

@jdatcmd Approving 64769599. Measured on PG17 with ~/pg/17-p: main's shell, this head's shell, then the twin as shipped.

run	main's shell	this shell	twin
clean (also PG18)	—	13/13	13/13
`everyreads`: the parallel claim reads `rowGroupIndex++`, so every participant reads every group	4 red, "par no-filter" PASS	5 red: the five par checks	SAME 5
`ignorestripe`: both `stripeRowLimitSet` sites `false &&`	10/10	1 red: "par table has 25 row groups to claim"	SAME 1
`skipfirst` (mine): each parallel claim is one past the counter, so group 0 is never read	—	5 red	SAME 5
harness: `Q_NOFILTER` back to `count(*)`	—	1 red: "par queries all plan a Gather: got [4] want [5]"	—
harness: `max_parallel_workers_per_gathr`	—	1 red: "serial queries plan no Gather: got [0] want [3]"	—
harness: server stopped before the plan check	—	5 red: both plan counts read 0, and the serial diffs read `QUERY_ERROR.n` against `QUERY_ERROR.m`, never equal	—

Both red-first claims hold, the plan checks fail closed on a dead server, and the suite has no premise: checks a product mutant could redden.

🤖 Generated with Claude Code

OffgridwithJD

@jdatcmd Re-approving the restated head d5f9ade1 (I approved 64769599). The PR's added lines are identical in parallel.sh (48), test_parallel.py (149), check_ledger.tsv (13) and test_compare_to_bash.py (1); unported_suites.txt only drops the stem. Only the budget and counters moved. The earlier measurements stand.

🤖 Generated with Claude Code

jdatcmd · 2026-10-22T03:24:39Z

Restated as a stack: now on #1671 (it merges after it), registries regenerated over that head with the same tooling. The suite and twin are unchanged; the ledger gate passed (rc=0) and the guards passed (476). Head d5f9ade.

…1339) Fixes #1339 test_projection_drop_column.py:113 and :119, test_projection_parallel.py:217, test_projection_privilege.py:259, test_projection_scan_io.py:260 and test_projections.py:459 and :600, each in its shell twin too. For each finding, a one-line mutation of pgcolumnar was first run on main, where both twins stayed green (or reddened the wrong name). After the fix, the same mutation reddens the target by the same name in both twins. The pytest twin stops at a test's first red, so it can show fewer names than the shell suite. - #1339 test_projection_drop_column.py:113 (weak). The stranger arms read only 42501, so deleting the unlocked ownership check in columnar_tableam.c passed (main: 8/8 shell, twin green). New arm in both: while the owner holds SHARE, a stranger is refused at once. Under the mutation it waits out lock_timeout: "a non-owner is refused before it waits for the table lock" got 55P03. - #1339 test_projection_drop_column.py:119 (mislabelled, twin mismatch). "premise: and the two refusals are the same error" came after the two constant arms, so it could never run red; the shell suite did not have it. With both ownership checks deleted, main's twin reddened "a non-owner learns nothing". The check loses the prefix, moves first, and joins the shell suite. Under that mutation both twins now redden "and the two refusals are the same error, so neither leaks" (2BP01 against 42501). - #1339 test_projection_parallel.py:217 (mislabelled). With parallel_workers = 1 on the projection path, main reddened "premise: EXPLAIN ANALYZE launched two workers" in both twins (and the rows= premise in the shell suite). Both lose the prefix; the same mutation reddens them under their new names. - #1339 test_projection_privilege.py:259 (twin mismatch). The twin asserted 42501 (and 0A000 under the policy) twice on one error. With read_projection's aclcheck_error replaced by elog, main's shell reddened "and THAT refusal is the C check" and the twin reddened "a role with EXECUTE but no SELECT is refused read_projection". The twin's first arm now asks only "refused", as the shell does. The shell's reconstruct arms now read the SQLSTATE, as the twin did: main's shell stayed green with reconstruct's aclcheck_error replaced. Under each of the three mutations (both elogs, and the RLS refusal raised as 42501), the two twins now redden identical names. - #1339 test_projection_scan_io.py:260 (weak). With the one-page floor deleted, an empty projection is priced at 0 pages and is still chosen (main green in both). New arm in both: "and its scan is not priced at zero pages" (PG17: total cost 5.00, and 0.00 under the mutation). - #1339 test_projections.py:459 (weak). With the executor never opening the projection while EXPLAIN still names it, main was green in both. New arm in both reads EXPLAIN ANALYZE: the filter sees fewer rows than the base holds (5980 against 17980 on PG17; 17980 of 17980 under the mutation). - #1339 test_projections.py:600 (inaccurate). EXPLAIN shows that the count plans pmp (Columnar Projection: pmp), so the triage's guess that it reads the base did not hold. But nothing asserted it. With projection pages multiplied by 1000000, so the count plans the base, every :600 arm stayed green on main. New arm in both: "the old-snapshot count is planned as a projection scan". The triage's candidate mutation (GetLatestSnapshot() for the projection read only) stays green: the base liveness cache still uses the query snapshot. With both on GetLatestSnapshot(), main's shell reddened "old snapshot projection scan does not see post-snapshot rows" but the twin reddened "session A responded post-commit". The twin's two timeout arms now read whether session A answered. The shell's poll no longer accepts an ERROR that echoes the token as an answer: with session A unable to see pm, main's shell reddened the counts but never "session A opened snapshot". It does now, in both twins. Ledger: projection_drop_column is seeded (11 rows; suites_not_covered 104 -> 103). 19 rows are dated by ten mutations, 2 are renamed, and 14 are new. checks_never_observed_red 3868 -> 3863 (6141 rows = 3863 never + 2278 dated). Clean on 15-19 for every touched suite and twin. Gates: the ledger gate passes for all five suites, compare_to_bash finds every bash property covered for all five pairs, and the guard set passes (476 passed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… stale stored digest no longer passes (#1339) #1339 listed "premise: a tdigest stored in a columnar table still answers" as mislabelled: nothing compared the stored digests, so a pgColumnar storage change could redden that premise alone, or pass it. hll's stored sketch has had a byte-for-byte arm; tdigest had none. New arm, in both harnesses: "a tdigest stored in a columnar table comes back byte for byte". The daily digests are built once into a heap table and stored from there, because a tdigest depends on the order its inputs arrive in; digests rebuilt from heap could differ for reasons that are not storage. The union-within-0.1% check keeps its premise label: a storage change that altered a digest now reddens the arm first, the suite's rule for its bounds. Measured on PG17: - Mutant: a reader returning each long value (over 512 bytes) as the previous long value read (columnar_reader.c). Main's suite: 11 + 1 failed, only "an hll stored in a columnar table comes back byte for byte"; every tdigest check passed. This branch: 11 + 2 failed, the new arm "got [f] want [t]" and the hll arm; the twin fails the same two names (read from its own records). - Rejected mutants: flipping the last byte of a long value, or zeroing its second half, makes tdigest's output function take the server down, so every later arm fails as collateral. Not used for dating. - Clean: shell 13/13 on PG15 to PG18, 7 + 6 skipped on PG19 (no hll there); the twin passes on each; the new arm passes on all five. - compare_to_bash: 15 literal matches, 0 missing. Guards 474 passed. Ledger: 6141 + 1 = 6142 rows, 3863 never (unchanged), 2278 -> 2279 dated. suites_not_covered and cluster_tests unchanged. Stacked on #1674; the ledger gate passes, and guards 476 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… the file's first read no longer passes (#1341) test/pytest/test_parallel_copy_dedup.py ports all 17 checks of test/parallel_copy_dedup.sh by name. Two properties the shell's header promises could break with main's shell still green; both are fixed in both twins. - The changed file differed from the first in its first bytes, and the whole file fitted in the coordinator's first 256 KiB read. With the fingerprint taken from the first read alone, main's shell passed 14 of 14. The changed file now differs only in its last row, 25,000 rows put that row past the first read (a premise checks this), and a new check, "the records hold the SHA-256 of each loaded file's bytes", compares the table's records with the server's sha256() of each file. - The header says a refused load "says what it skipped rather than returning a silent 0", but no check read the NOTICE. With the NOTICE removed, main's shell passed 14 of 14. A new check reads it. - "premise: without dedup a repeated load doubles the table, ..." and "premise: and each of those loads reported its rows" lose their prefix under #1339: dedup is pgcolumnar's own default, and forcing it on reddens both. Measured, PG17, shell and twin: - fingerprint of the first read alone: 4 red; main's shell 0. - the skip's NOTICE never raised: 1 red; main's shell 0. - dedup forced on: 4 red. The refused load committed anyway: 2 red. A fingerprint matched in any table: 3 red. The refused load reporting the rows it discarded: 1 red. Each red set is identical in both twins. - the fixture shrunk to 20,000 rows: 1 red, the size premise. - clean: 17 of 17 on PG15 to PG19 in both twins. Ledger: 17 rows, 13 dated by the 6 mutations and 4 never. suites_not_covered 74 -> 73, cluster_tests 3222 -> 3239, TESTS.md section 296. Census 4132 -> 4136 never, 2818 -> 2831 dated, 6967 rows. The ledger gate and the guards pass over eb4aa69. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… no longer passes as one that survived (#1341) test/pytest/test_parallel_copy.py ports all 81 checks of test/parallel_copy.sh by name. Ten shell checks could pass with the property broken; all are fixed in both twins. - Five survival checks read "SELECT 1", which answers again after crash recovery. With one backend killed by SIGSEGV before each, main's shell passed 81 of 81. They now also need the server log's count of killed backends unmoved since the suite began (crashes() and survived(), from avro_manifest.sh). - "atomic: a loader failure fails the whole load" and "single table: a bad row fails the whole load" took any message with "failed" in it, and every worker failure reads "worker N failed: ...". With the loader's message dropped, main's shell passed 81 of 81. They now need the loader's own cause after "failed: ". - The nine reconstruct checks compared the per-range loads with a single columnar COPY of the same file, which shares every defect of the COPY path. With multi_insert dropping the row whose id is 1, all nine passed on main. They now compare with the generator that wrote the file. - Check names built from loop variables and helper labels are literal at the call site, so the three suite-local check_ helpers are gone. test_compare_to_bash.py asserted those three existed; that arm now asserts no suite defines its own check_ helper, and it reddens on main's parallel_copy.sh. Measured, PG17, shell and twin, each red set identical in both: - a backend killed before each survival check: 5 red; main's shell 0. - the loader's message dropped: 2 red; main's shell 0. - multi_insert dropping id 1: 19 red, 9 of them reconstruct checks; main's shell 10. - a split offset on the newline: 23. The split degraded to one range: 5. The S_ISREG refusal removed: 2. The guard on max_prepared_transactions removed, bulk_parallel_writer booting on, the FIFO opener blocking, workers < 1 accepted, the key-type restriction removed: 1 each. - clean: 81 of 81 on PG15 to PG19 in both twins. Ledger: 81 rows, 49 dated by the 11 mutations and 32 never. suites_not_covered 73 -> 72, cluster_tests 3239 -> 3320, TESTS.md section 297. Census 4136 -> 4168 never, 2831 -> 2880 dated, 7048 rows. The ledger gate and the guards pass over 7e8c10f. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…erially no longer passes (#1341) test/pytest/test_parallel.py ports all 13 checks of test/parallel.sh by name. Three shell checks could pass with the property broken; each is fixed in both twins, and each fix adds one named check. - "par no-filter" was a bare count(*). The vectorized aggregate answers that with no Gather, so the check ran serially. With every participant reading every row group, main's shell reddened the other four par checks and left this one green. It now returns rows, and "par queries all plan a Gather" holds all five par queries to a Gather. - The table was assumed to have 25 row groups. With the writer ignoring stripe_row_limit it had 1, and a claim that skips every other group passed main's shell 10 of 10. "par table has 25 row groups to claim" now counts them. - setg swallows its error, so a serial setting that did not take ran the three serial checks in parallel, and main's shell passed 10 of 10. "serial queries plan no Gather" now says so. The queries are named once, so the plan checks read the queries the comparisons ran. Measured, PG17, shell and twin: - every participant reads every row group: 5 red (the par checks); main's shell 4, without "par no-filter". - the writer ignores stripe_row_limit: 1 red (the row-group count); main's shell 0. With a claim that skips every other group as well: the same 1 red; main's shell 0. - the serial setting misspelled: 1 red (the serial plan check); main's shell 0. "par no-filter" put back to count(*): 1 red (the par plan check). - a claim that skips every other group: 5 red. The partial path never added: 2 red. The scan named ColumnarScan, as before #428: 3 red. A serial scan reading every other row group: 3 red. Each red set is identical in both twins. - clean: 13 of 13 on PG15 to PG19 in both twins. Ledger: 13 rows, 13 dated by the 5 mutations and 0 never. suites_not_covered 72 -> 71, cluster_tests 3320 -> 3333, TESTS.md section 298. Census 4168 -> 4168 never, 2880 -> 2893 dated, 7061 rows. The ledger gate and the guards pass over 9ae8951. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jdatcmd · 2026-10-22T19:00:26Z

Restated again, onto #1671's new head (main 9ab6a4c underneath), because the registries conflicted with main. The registries are regenerated over that head by the same tooling as before; the suite and twin are unchanged; the ledger gate and the guards pass (see the commit message). Re-approval needed. Head f953f5a.

OffgridwithJD

@jdatcmd Re-approved at f953f5a6, on #1671's 9ae8951f. The diff outside the bookkeeping hashes the same as the approved d5f9ade1's. The tsv is sorted, and its census (7061 / 4168 / 2893) matches the budget.

🤖 Generated with Claude Code

OffgridwithJD approved these changes Oct 22, 2026

View reviewed changes

jdatcmd force-pushed the test/1341-parallel-twin branch from 6476959 to d5f9ade Compare October 22, 2026 03:19

OffgridwithJD approved these changes Oct 22, 2026

View reviewed changes

OffgridwithJD mentioned this pull request Oct 22, 2026

test: parallel_vector_agg gains its pytest twin, and two failed arms no longer agree (#1341) #1673

Merged

OffgridwithJD mentioned this pull request Oct 22, 2026

Missing tests: 87 confirmed behaviours no shell or pytest arm exercises (#1339 audit) #1341

Closed

87 tasks done

jdatcmd and others added 5 commits October 22, 2026 18:31

jdatcmd force-pushed the test/1341-parallel-twin branch from d5f9ade to f953f5a Compare October 22, 2026 19:00

OffgridwithJD approved these changes Oct 22, 2026

View reviewed changes

jdatcmd merged commit 5fc7dd3 into main Oct 23, 2026
15 checks passed

Navigation Menu

Repository navigation

test: parallel gains its pytest twin, and a parallel check that ran serially no longer passes (#1341) - #1672

test: parallel gains its pytest twin, and a parallel check that ran serially no longer passes (#1341)#1672
jdatcmd merged 5 commits into
mainfrom
test/1341-parallel-twin

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

OffgridwithJD left a comment

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jdatcmd commented Oct 22, 2026

What was fixed

Measured on PG17

Gates

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants