[jdatcmd]

@OffgridwithJD

This ports test/planner_choice_quality.sh to test/pytest/test_planner_choice_quality.py (#1341). The twin runs all 11 checks under the shell's names, one test per check, on a private database with serial planning, as the shell's cluster runs. Each arm runs its EXPLAIN, a warming run and the timed run in one session, so the plan it names is the plan it times.

What was fixed in the shell

1. A plan that fails was read as a fast plan. psql prints Time: after a statement that errors, not only after one that succeeds. run_plan looked only for a timeout, so a chosen plan that raised an error in 2 ms counted as a plan that answered in 2 ms. Main's shell passed every check under two mutations:

• PgColumnarBeginCustomScan raises at execution: wide range ... chose Custom Scan (PgColumnarScan)=1.489ms ... (0.01x, bound 3x), PASSED.
• PgColumnarBeginGroupAggScan raises at execution: range scan with aggregate: chose Custom Scan (PgColumnarScan)=1.911ms ... (0.01x, bound 3x), PASSED.

run_plan now reads psql's exit status under ON_ERROR_STOP. It tells the statement-timeout cancel apart by that cancel's own message. A new check, [$label] the chosen plan ran to completion, fails by name. A forced alternative that fails now fails the ratio check as "nothing to compare". Before, main read it as a fast alternative: "15.94x exceeds the 3x bound".

2. The ratio check's name changed with its result. The name carried the bound and the node that won the comparison. Green, it read ... alternative (Index Scan). Red, it read ... alternative (Custom Scan (PgColumnarScan)), a name that no green run had ever recorded. Now the names carry only the query's label. The bound and the winning node are in the display line.

forcing produced a different plan loses its premise: prefix (#1339). With pgcolumnar.enable_custom_scan ignored by the planner, it is the check that goes red.

Why CI's suites job skips it

ci.yml sets PGC_SKIP_TIMING=1, and run_all_versions.sh lists this suite in is_timing_suite. Under that flag the driver does not run the suite at all and records it as SKIPPED (66). If you run the shell directly under the flag, it builds 20,000 rows, runs the plan-shape checks, and records the two measured checks for each query as SKIP.

The twin reads the same flag, under the same condition. It builds the same 20,000-row fixture. It declares the same two checks for each query unrunnable (UNMET_PRECONDITION, under the shell's names), so the session ends INCOMPLETE (exit 67) and never green. Measured on PG17: shell 7 passed + 4 skipped, PASSED; twin 7 pass + 4 unrun, rc=67.

The pytest-cluster job does not set PGC_SKIP_TIMING, so in CI the twin measures. The headroom is wide: clean ratios were 0.02x to 0.05x against the 3x bound on every run. Whether the twin should run there is your call.

Shell and twin under each mutation (PG17)

change	main's shell	this shell	twin
none	9/9 PASSED (PG17)	11/11 PASSED on each of PG15 to PG19	11 passed on each of PG15 to PG19
PGC_SKIP_TIMING=1	(driver skips it)	7 pass, 4 SKIP	7 pass, 4 UNRUN, rc 67
scan100x: pgcolumnar_refined_scan_cost x100, the planner picks the index scan	2 red, under names no green run has (... (Custom Scan (PgColumnarScan)))	2 red: [range ...], [wide ...] the chosen plan is within the bound of the best alternative	same 2
scanerrors: PgColumnarBeginCustomScan raises at execution	PASSED	2 red: [wide ...] the chosen plan ran to completion, [wide ...] ... within the bound ...	same 2
groupaggerrors: PgColumnarBeginGroupAggScan raises at execution	PASSED	2 red: [range ...] the chosen plan ran to completion, [range ...] ... within the bound ...	same 2
fetcherrors: pgcolumnar_index_fetch_tuple raises (columnar_tableam.c)	2 red, as "15.94x exceeds the bound"	2 red, as "the forced alternative ... failed"	same 2
gucignored: the planner ignores pgcolumnar.enable_custom_scan=off	2 red (premise: ...)	2 red: [range ...], [wide ...] forcing produced a different plan	same 2

Two single causes of #434 do not change the plan on this fixture, so they are not ledger mutations: dropping the zone-map survival factor alone, or the index-fetch penalty alone, leaves the custom scan chosen (main: 0.03x to 0.08x).

Timing spread, three repeats on PG17 with other agents running on the machine:

• Clean: shell 0.03x to 0.04x, twin 0.02x to 0.04x.
• scan100x: shell 23.3x to 25.4x, twin 23.6x to 36.7x.

Gates

• Ledger: 11 rows, 6 dated by the 5 mutations and 5 never (fixture rows and four premises). Census 4137 -> 4142 never, 2798 -> 2804 dated, 6946 rows. suites_not_covered 75 -> 74.
• Ledger gate, on each of the five clean logs, against --registered from run_all_versions.sh --list-suites, --against parent: rc 0 on 15 to 19.
• Orphan scan, on each clean log: orphans=0.
• Guards: 804 passed (--pgc-expect-tests 804). test_harness_deps.py, test_capability_decline.py and the twin with --pg-config: 41 passed.
• compare_to_bash.py: literal matches: 3 | template matches: 6 | missing: 0, every bash property is covered. The six label names are templates on both sides ([$label] and f"[{label}]"), as in hilbert_locality.
• Capability-import scan: [].
• harness_selftest.sh on PG17: 1248 passed.
• shellcheck -S error: clean.
• cluster_tests 3222 -> 3233, by collection. TESTS.md section 296.

Re-checked by me before pushing (PG17, main 4adb2e7): the custom scan refusing to execute: main's shell 9/9 (it read the failed plan as "chose Custom Scan=8.086ms"); this shell 9 + 2 failed, "[wide range, narrow projection] the chosen plan ran to completion: got [no (ERROR: … custom scan refuses to execute)]" and the ratio check as "not a measurement"; the twin 2 failed.

A choice for you, @OffgridwithJD, made provisionally: the CI suites job skips this suite's timing (PGC_SKIP_TIMING=1), but the cluster pytest job doesn't set that flag, so the twin MEASURES there (200,000 rows, the 3x bound). I've left it measuring. A clean run sits at 0.02x–0.04x, so the headroom against a false red is very large. Skipping the timing in that job instead would make the twin unrunnable (rc 67), which fails the job. Say if you'd rather the job set the flag and the twin's timing arms be declared on a nightly instead.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Approved at a358eabb. Measured on PG17 against main's shell (origin/main tree) and this head, each tree building its own mutated source.

The two shell fixes are red-first

mutant	main's shell	this shell	twin
none	9/9	11/11	11 passed
PgColumnarBeginCustomScan raises unless EXPLAIN-only	9/9 PASS	2 red: [wide range, narrow projection] the chosen plan ran to completion and ... within the bound of the best alternative	the same 2
PgColumnarBeginGroupAggScan raises unless EXPLAIN-only	9/9 PASS	2 red, the [range scan with aggregate] pair	the same 2
pgcolumnar_refined_scan_cost ×100	2 red, under names no green run has (... (Custom Scan (PgColumnarScan)))	2 red, stable names, 12–15x	the same 2, 15–16x
enable_custom_scan ignored by the planner	2 red as premise:	the 2 forcing produced a different plan checks	the same 2

Error vs timeout

• A mutant that raises 57014 with a different message ("canceling statement due to user request") reads as ERROR (...) in the shell and failed: ... in the twin, not as a timeout. Main's shell read it as the chosen plan finished within 20s: got [no], because it grepped for canceling.
• A really slow plan (the scan sleeps 30 s, past the 20 s statement_timeout) reads as no (TIMEOUT) in both, with the ratio check reporting "not a measurement".

Timing in CI's cluster job

PGC_SKIP_TIMING is set only in the shell suites jobs (ci.yml :726, nightly.yml :212), so the twin measures in pytest-cluster. I ran it 5 times on PG17 with all 8 cores held busy by spinning loops: 11 passed each time, and the worst ratio was 0.14x against the 3x bound (the rest 0.03–0.08x). The headroom holds.

Under the flag locally, the twin gives 7 pass + 0 fail + 4 unrun, rc 67, with the 4 measured checks declined under the shell's names. No CI job sets the flag for pytest, so expected_unrunnable.txt needs no rows.

Ledger

The suite had no rows on main, so this seeds it rather than renaming. The 11 rows are the 11 names the head shell emits. The 6 dated rows cite the mutants above, all of which I reproduced except pgcolumnar_index_fetch_tuple raising. The tsv is sorted, and the census agrees with the paragraph: 6935 + 11 = 6946, never 4137 + 5 = 4142, dated 2798 + 6 = 2804.

Needs a restate

Against current main c7f0de45 (#1735 in), it conflicts in check_ledger_budget.txt, TESTS.md, expected_tests.txt and test_compare_to_bash.py. I'll re-approve a restate whose non-registry files are unchanged.

🤖 Generated with Claude Code

[jdatcmd]

Restated onto main 4500efd as bb1b864, in one linear stack (this PR sits on #1750), so these eight merge without voiding each other. Every non-registry file carries the same change as before; the registries are re-derived on the new base, and each census agrees with a recount. Guards, the PR's own shell suites on PG16 and PG18, its twins on PG17, compare_to_bash and the ledger gate all pass at this head. Re-approval needed.

[OffgridwithJD]

@jdatcmd Re-approved at bb1b864b, restated onto main 4500efdd in the linear stack. Outside the ledger, the budget, expected_tests and TESTS.md, the commit's diff matches the approved a358eabb. The tsv is sorted, and its census (8040/4553/3487 rows/never/dated) matches the budget.

🤖 Generated with Claude Code