fix: a second cancel during the parallel flush's error cleanup frees what the workers pinned (#1745) by OffgridwithJD · Pull Request #1746 · commandprompt/pgcolumnar

OffgridwithJD · 2026-10-22T16:55:00Z

@jdatcmd Fixes #1745. Standalone on main 9be30954, not on the frozen stack.

The defect, measured

With pgcolumnar.parallel_flush on, an error during the dispatcher's wait runs pflush_error_cleanup under PG_ENSURE_ERROR_CLEANUP: terminate the workers, wait for them, then unpin the output segments they published. The wait checks for interrupts. A second cancel landing in it raised out of the cleanup before the unpin loop, so the segments stayed pinned until postmaster restart, and the second cancel was reported instead of the first.

The scenario: seven cheap int columns publish within a second, and one zstd-19 text column keeps one worker encoding for many seconds. The first cancel is a 2 s statement_timeout; the second is pg_cancel_backend at 3.5 s, while the cleanup waits for the encoding worker. The suite's cluster runs dynamic_shared_memory_type = mmap, so segments are files in its own pg_dynshmem, counted from SQL.

run (PG17 probe)	new segments left	error reported
single cancel, main	0	57014 statement timeout
double cancel, main	4	57014 user request
double cancel, this PR	0	57014 statement timeout

A correction to the issue

The issue said a SIGTERMed worker exits promptly. It doesn't. The encode path has no interrupt check, so a terminated worker exits only after the column it is encoding: about 15 s here, and minutes at zstd 22. A single cancel already waits that long; the single-cancel insert returned when the encoding worker finished.

The fix (`src/columnar_write_state.c`)

HOLD_INTERRUPTS() around the terminate-and-wait. Given the above, it adds no latency over a single cancel. It only stops a second cancel, or a terminate (which fires once the cleanup returns), from abandoning the cleanup. I chose this over fix: a second cancel during a remote export's cleanup keeps the export's own error (#1741) #1744's copy-and-rethrow helper: the helper keeps the first error, but a cleanup cut short still skips the unpin loop, so the leak would stay.
The unpin loop frees any slot with a valid outHandle, whatever state the worker reached, since every worker has exited by then.
The worker records its handle before pinning. One that died in between leaves an unpinned segment that is already gone (dsm_attach returns NULL).

Tests: `parallel_flush_optin` arm 4 and its twin

The twin builds a private cluster for the two startup-only settings. Two premises: a worker is still encoding at the second cancel, and outputs were published by then. Two checks: the first cancel's error is reported (57014: canceling statement due to statement timeout), and no new segment survives.

run	shell	twin
main, PG15–19	7 + 2 failed on each (both checks)	the same 2 names on each
this PR, PG15–19	9/9 on each	9/9 on each
`nohold` (the cleanup without `HOLD_INTERRUPTS`), PG17	the same 2	the same 2

nologin_workers (15/15) and sorted_pathkeys (113/113) pass on PG17 with the fix. The build is warning-free on PG15–19.

Bookkeeping: 4 ledger rows; the 2 checks are dated by nohold and the 2 premises arrive never. Census: never 3884 → 3886, dated 2286 → 2288, rows 6170 → 6174. suites_not_covered stays 102, cluster_tests 2753 → 2757, plus §262 rows in TESTS.md and a CHANGELOG Fixed entry.

Cost: about 20 s per harness, all of it the encoding worker finishing after the cancels.

Out of scope, noted: on the HAPPY path, a worker that pinned its output and then failed (state FAILED) is never collected or unpinned; only the error cleanup now frees it. Worth a one-line follow-up.

🤖 Generated with Claude Code

jdatcmd

@OffgridwithJD Requesting changes on #1746 at c5b22d15.

The C fix works: I reproduced the leak on main and it is gone at this head, under every interrupt pattern I tried. The test is the problem. Arm 4 cannot run as root, and CI's suites job runs as root, so suites (PG 17) and suites (PG 18) are red on this PR.

Blocking: arm 4's restart does not happen as root

The suite sets dynamic_shared_memory_type = 'mmap' and then restarts with a bare pg_ctl:

env PATH="$PGC_BINDIR:$PATH" pg_ctl -D "$PGC_PGDATA" restart -w -o "-p $PGC_PORT" >/dev/null 2>&1 || true

As root, pg_ctl refuses to run, and || true hides the refusal:

$ sudo env PATH=".../pg17/bin:$PATH" pg_ctl -D <data> restart -w -o "-p 15771"; echo rc=$?
pg_ctl: cannot be run as root
...
rc=1

So the server keeps POSIX shared memory, pg_dynshmem stays empty, and arm 4 counts 0 segments whatever the code does. The premise catches it, so the suite fails closed. But the leak check passes with nothing to read: on main as root it passed with 0 new segments.

CI shows the same failure on PG 17 (job 119962833253):

FAIL  premise: the cheap workers published their segments before the second cancel: got [no (new segments=[0])] want [yes]

Suggested fix (one line): restart through pgc_pg, as pgc_setup and the teardown already do:

pgc_pg "pg_ctl -D '$PGC_PGDATA' -l '$PGC_LOGFILE' restart -w -o '-p $PGC_PORT'" >/dev/null 2>&1 || true

I applied this line only in my worktree and ran the suite as root. Main's code goes red for the right reason, and this head goes green:

run (PG17, as root)	result
main code, PR test as pushed	7 + 2 failed: `premise: the cheap workers published...` (`new segments=[0]`) and `reports the first cancel` (`user request`). `leaves no worker segment pinned` passed.
this head, PR test as pushed	8 + 1 failed (the same premise)
main code, restart through `pgc_pg`	7 + 2 failed: `reports the first cancel` (got `user request`), `leaves no worker segment pinned` (got `4 new segments`)
this head, restart through `pgc_pg`	9/9
`nohold` (`HOLD_INTERRUPTS`/`RESUME_INTERRUPTS` removed), restart through `pgc_pg`	shell 7 + 2 failed, twin 7 + 2 failed, the same two names

The twin builds its own cluster as postgres, so it is not affected. Run serially, the twin fails 7 + 2 on main's code (the same two names: user request, 4 new segments) and passes 9/9 at this head. compare_to_bash.py: 9 literal matches, 0 missing.

The fix, measured outside the suite (PG17, mmap cluster, 4 workers)

I counted the files in pg_dynshmem before and after each run.

scenario	main: left / error	this head: left / error
2 s timeout, cancel at 3.5 s	4 / `user request`	0 / `statement timeout`
2 s timeout, cancels at 2.8, 3.5 and 4.2 s	4 / `user request`	0 / `statement timeout`
2 s timeout, terminate at 3.5 s	4 / FATAL 57P01	0 / ERROR 57014, then FATAL 57P01
cancel the moment the dispatcher starts waiting (0 workers connected), again 20 ms later	3, 4, 3 in three runs / `user request`	0, 0, 0 / `user request`
a single cancel at 2.5 s	0, returned at 18.3 s	0, returned at 23.4 s
a single terminate at 2.5 s	0, gone at 18.9 s	0, gone at 22.0 s

The cancel-on-start probe finds the dispatcher from another session by its wait event, IPC/BgworkerShutdown. That event is spelled with a lower-case w on PG17.

Error-handling rules

HOLD_INTERRUPTS: it covers only the terminate-and-wait. The wait lasts until each terminated worker finishes the column it is encoding. That is the same bound main already has for a single cancel (18.3 s) and for a terminate, where proc_exit holds interrupts anyway (18.9 s). With a second signal, this head returned at 23.0 s and 24.8 s; main returned at about 4 s but leaked 4 segments. This wait is bounded by one column's encode time, not by a constant: at zstd 22 a session stays uncancellable for that long. The PR body says so. I did not test postmaster death, which WaitForBackgroundWorkerShutdown returns on; that comes from reading the source.
FlushErrorState: this PR adds none. The only one in the area is the worker's own PG_CATCH, which does not re-throw.
ereport(ERROR) inside the cleanup callback: the callback calls dsm_attach and dsm_unpin_segment, and both can raise. One case is reachable on main and on this head; it is not introduced here. The collect loop unpins a segment while it still has it mapped. If anything raises before the loop invalidates the handle (I injected an ereport(ERROR) right after the unpin, for worker 0 only), the callback then raises can't attach the same segment more than once. That error replaces the original one and cuts the cleanup short: 3 segments left, the same on main. Invalidating the handle right after dsm_unpin_segment fixed it in my worktree: the injected error was reported and 0 segments were left; a clean run left 0 with correct rows. This belongs in a follow-up issue, not in this PR.

Not covered by the test (non-blocking)

The second bullet of the fix frees any slot with a valid handle, not only a DONE one. No test covers it. I put the state == PFLUSH_DONE && condition back, and the shell and the twin both passed 9/9. #1748's injection is what exercises it.

Other observations

With -n 2, the twin errored in all 3 of my runs: could not bind IPv4 address "127.0.0.1": Address already in use. Both xdist workers build the module-scoped dc cluster on pick_port(41). Serial runs are clean, and CI runs serially.
Registries add up. The ledger file has 6170 → 6174 rows, 3884 → 3886 never and 2286 → 2288 dated. The 4 row names match the twin's and the shell's names. cluster_tests goes 2753 → 2757. The pyarrow import scan prints [].

Logs: logs/1746/ (parent-shell.log, head-shell-root.log, *-restartfix.log, *-twin-n0.*, mutant-nohold-*, mutant-doneonly-*, probe-main-*, probe-head-*, ci-suites-pg17.log).

🤖 Generated with Claude Code

…he collect loop frees what it did not collect (#1747) A worker pinned its OUTPUT segment and only then ran table_close, PopActiveSnapshot and CommitTransactionCommand before writing PFLUSH_DONE. An ERROR in between reached its PG_CATCH, which wrote PFLUSH_FAILED; the dispatcher's collect loop attaches and unpins only DONE slots, and the error cleanup does not run on the normal path, so the segment stayed pinned to postmaster restart. Measured on PG17 main with an ERROR injected right after dsm_pin_segment (a measurement-only patch gated by an environment variable given to the test postmaster; not committed): a 300000-row, 8-column parallel flush logged the "worker N failed, completing its columns serially" WARNING for 4 workers on each of 2 stripes, returned the right rows, and left 8 new segments in pg_dynshmem. In production the window raises only through out-of-memory or another preloaded library's transaction callback: pgcolumnar's own pre-commit callbacks find nothing pending in the worker. The worker now builds its record in TopMemoryContext, commits, then creates, fills, names, pins and detaches the segment and writes DONE: nothing that can raise follows the pin. The collect loop also frees any slot it did not collect that names a segment. With the same injection placed after the new pin, the 8 FAILED workers leave 0 segments and the rows are right. parallel_flush_optin gains arm 5 and its twin: the worker's source with comments blanked must hold only the detach between dsm_pin_segment and the DONE write. Red on #1746's head in both harnesses; green 10/10 on PG15-19; the worker committing after the pin again reddens it by name in both. nologin_workers 15/15 and sorted_pathkeys 113/113 on PG17. Census: never 4490 -> 4490, dated 3262 -> 3263, rows 7752 -> 7753; cluster_tests 4031 -> 4032; TESTS.md §262 row; CHANGELOG Fixed entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… from the table under test no longer vouches for it (#1341) Ports test/native_fold_deferral.sh to test/pytest/test_native_fold_deferral.py, all 13 checks by the shell's name, one test per check, on a private database. Three holes in the shell, each measured red-first on PG17 against main and fixed in both harnesses: - The heap mirror was `CREATE TABLE fd_heap AS SELECT * FROM fd_t`, read through the scan under test. With the writer storing one p1 value off by one (writecorrupt), the fold, the row path and the mirror all agreed: main passed 13/13. The mirror is now written from the generator, and writecorrupt reddens both heap-mirror checks in both twins. - The answers were read with `psql ... | tail -1` and compared with check. With the server stopped before the correctness section, all three answer checks passed on empty strings. Readings now run with ON_ERROR_STOP and come back empty on failure; the checks are check_text, and the counts go to check_num raw instead of defaulting to 0. Stopped server: only the three checks read before the stop pass. - Three premises are reddened by a one-line pgcolumnar change: the fold's ANALYZE line (foldno), the EXPLAIN group total (badtotal), the writer's stripe limit (nostripe). They lost the prefix (#1339). Eight mutations on PG17 each redden the same names in shell and twin: nodefer 3, deferall 1, nocursor 3, dropsurv 4, writecorrupt 2, foldno 1, badtotal 2, nostripe 1. Head is 13/13 in both on each of 15-19. Seeded: 13 rows, 9 dated, 4 never. Census 4192 -> 4196 never, 2952 -> 2961 dated, 7157 rows; suites_not_covered 69 -> 68; cluster_tests 3416 -> 3429; TESTS.md §301. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d through the reader no longer vouches for it (#1341) Ports test/native_lazy_slot.sh to test/pytest/test_native_lazy_slot.py, all 10 checks by the shell's name, one test per check. Two holes in the shell, each measured red-first on main's shell on PG17 and fixed in both harnesses: - The heap mirror was a copy of lz_c (INSERT INTO lz_h SELECT * FROM lz_c), read through the reader under test. With the writer storing c40 off by one (writec40), main passed 10/10. It is now written from the generator, and writec40 reddens 4 checks. - Five comparisons read `q ... | tail -1` and compared with check, so two failed reads agreed, and a timing whose statement failed read as fast. With the server stopped after setup, 6 of 10 passed on main. A new qv helper names the side in its failure, the comparisons use check_text, and a failed timing is "no reading": all 10 now fail. Mutants on PG17, the same failing checks by name in shell and twin: writec40 4, stale (the fetch no longer clears the slot) 2, eager (decode the whole row) 1, short (one column too few) 6. An earlier stale mutant that kept only PgColumnarSlotStoreDeferred's clear was inert, because the fetch clears the slot first; the faithful one drops both. Head is 10/10 in both on PG15-19. Seeded: 10 rows, 7 dated, 3 never. Census 4196 -> 4199 never, 2961 -> 2968 dated, 7167 rows; suites_not_covered 68 -> 67; cluster_tests 3429 -> 3439; TESTS.md §302. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… counted nothing no longer passes (#1341) Ports test/native_metadata_flush.sh to test/pytest/test_native_metadata_flush.py, every check by the shell's name, one test per check (#1341). Four holes, each measured on PG17 against main's shell, which passed all six checks under each: - A witness that counted nothing passed. With the flush session never counting an open (zerocount, opens=0 for every flush), "narrow and wide open the same number" (0 = 0) and "a small constant" (0 <= 6) both passed. A new check wants at least the three tables every flush writes; zerocount now reddens it. - The heap mirror was copied from the columnar table, so a writer that stored 1777 as 1778 (writecorrupt) agreed with itself. The mirror is now written from the generator, and the sum check reddens: got 40020020, want 40020000. - Nothing showed the zone rows were read. The reader rules vectors out by value after decoding, with or without a usable zone map, so Vectors Skipped alone cannot say. A new check wants a vector ruled out by its zone row alone (Skipped minus Ruled Out by Value); zone rows with no min/max (zonenull) redden it. - "premise: the flush emits the metadata-open witness" reads pgcolumnar's own behaviour: a flush that never begins its session (nosession) reddens it. It lost the prefix (#1339). Mutants on PG17, the same names red in shell and twin: noreuse (#445 reverted) 2, zerocount 1, writecorrupt 1, zonenull 1, nosession 4. Head is 8/8 in both on each of 15-19. Seeded: 8 rows, 6 dated, 2 never (the row count and the range read against the mirror). Census 4199 -> 4201 never, 2968 -> 2974 dated, 7175 rows; suites_not_covered 67 -> 66; cluster_tests 3439 -> 3447; TESTS.md §303. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d counter no longer reads as 0 (#1341) Every counter in the shell was read through `${x:-0}`, and four checks want exactly 0. With the server stopped before the feature-off arm, "with the feature off nothing is filtered early" and "a volatile qual is refused the two-pass path" passed on main (PG17). The counters are always printed, so a missing line is a failed reading: counter() now returns "no reading (...)" when the EXPLAIN fails or prints no such line, the feature-off readings run under ON_ERROR_STOP, and check_num refuses them. With the server stopped both checks now fail. Four premises a one-line pgcolumnar change reddens lost the prefix (#1339): "the query runs on the columnar custom scan" (noscan), the two "nothing was pruned" checks (noscan: their counters are not printed), and "the same shape without the volatile term does use it" (nocount, noscan, droprow). The twin test_native_late_materialization.py makes every check by the shell's name, one per test, on a private database with serial plans (a parallel plan prints per-worker averages). Every reading is a value: a failed query is a query_error(), an unprinted counter is "no reading", never 0. Measured on PG17, the same failing checks by name in shell and twin: nocount 2, noscan 7, droprow 3, skipkeep 1, nocursor 1, novolguard 1. matall (a rejected row's payload built anyway, still counted) reddens nothing: no counter reports per-row materialization, and the answers are the same by design. Head is 14/14 in both on each of 15-19. Seeded: 14 rows, 8 dated, 6 never. Census 4201 -> 4207 never, 2974 -> 2982 dated, 7189 rows; suites_not_covered 66 -> 65; cluster_tests 3447 -> 3461; TESTS.md §304. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nd a writer mutant dates four rows (#1691 review) jdatcmd's #1691 review: three premises read the columnar table n, and dropseven (pgcolumnar_tuple_insert drops the row whose id is 7, one line) reddened all three. The two fixture premises ("the fixture loaded every row", "the needle is selective") now read the heap mirror h, which pgcolumnar does not write, so no pgcolumnar change reddens them. "heap agrees on how many rows match" compares n with h and is not a premise: the prefix is dropped and the row is renamed, not removed. dropseven on PG17, shell and twin: the same 7 names red, the two fixture premises green (7 pass + 7 fail = 14 in both). It dates four rows that were never: "heap agrees on how many rows match", "the executor still reports the rows the scan filtered", "and the answers are the same with it off" and "and the volatile query still matches the heap mirror". Clean: 14/14 in both on PG17. Census for this suite: 14 rows, 12 dated + 2 never (was 8 + 6): never -4, dated +4. The header's "nothing pruned at the vector level" premise is a check since #1691. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… that drops every free row no longer passes (#1341) Every reconcile check counted only the injected stale free_space row, so a reconcile that dropped EVERY free row, leaking all reusable space, passed all 11 checks on main (overreconcile). A plain compact, which does not reuse, must now keep the legitimate free ranges the earlier compact_rewrite left (count and bytes), with a premise that one exists. And "row count intact" compared two readings with plain check: with the server stopped before it, main passed it on two empty strings. It is check_num now. The reuse-cycle loop is unrolled so its two check names are literals the grader reads. New twin test/pytest/test_native_reclaim_reconcile.py, one check per test under the shell's names, on a private database; readings are values (query_error()), row sets compared as an md5 over sorted rows. Mutants on PG17, same names red in shell and twin: noreconcile 4, overreconcile 1. 13/13 shell and twin on each of 15-19. Seeded: 13 rows, 5 dated, 8 never. Census 4203 -> 4211 never, 2986 -> 2991 dated, 7202 rows; suites_not_covered 65 -> 64; cluster_tests 3461 -> 3474; TESTS.md §305. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nd nofree dates it (#1692 review) jdatcmd's #1692 review: "premise: a legitimate free range exists before the plain compact" reads pgcolumnar's own free list, which holds ranges only because compact_rewrite recorded them. nofree (record_free_space drops its insert, one line in columnar_metadata.c) leaks every freed range and reddened only that premise, so a real leak read as "the fixture was wrong". It is renamed "compact_rewrite recorded the free ranges it left" (the row is kept, renamed) and dated by nofree. nofree on PG17: shell "FAIL compact_rewrite recorded the free ranges it left: got [no (0|0)] want [yes]", 12 + 1; the twin fails the same name, 12 pass + 1 fail. Clean: 13/13 in both. The next check still compares against a non-empty legit_before. Census for this suite: 13 rows, 6 dated + 7 never (was 5 + 8): never -1, dated +1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…longer passes as refused (#1341) Ports test/native_page_offset_bound.sh to test/pytest/test_native_page_offset_bound.py, every check by the shell's name, one test per check. Three holes in the shell, each measured red-first on PG17 against main and fixed in both harnesses: - Every refusal arm counted any psql failure as refused. With the whole-group containment guard removed (noguardwhole) the backend segfaulted and the arms still passed; with the projected guard removed (noguardproj) a later XX001 ("row group 1 is inconsistent with its column chunks") passed the projected overrun arm. Each arm now wants XX001 with the guard's own message, "lies outside row group". - Survival was a bare SELECT 1, which answers again once crash recovery has finished. With the probe 4s late, main passed both whole-group survival arms with the backend killed. It now also reads the server log's count of backends killed by a signal since the arm began. - Nothing read an intact table, and every arm wants a refusal. A guard over-applied to refuse every chunk (overall), or a legal chunk ending exactly at its group end (offby1), passed all 10 checks on main. Two controls now read rows behind an OFFSET 0 fence on each path; an aggregate over an intact table is answered without decoding a chunk, which is why the first version of these controls stayed green. Mutants on PG17, same failing checks by name in shell and twin: noguardwhole 6, noguardproj 2, offby1 1, overall 2. Without the whole-group guard the read either segfaults or fails decoding, depending on where it lands (three runs: shell 4-6, twin 6); its three refusal checks are red every time. 12/12 shell and twin on each of 15-19. Seeded: 12 rows, 10 dated, 2 never (the projected survival arms). Census 4210 -> 4212 never, 2992 -> 3002 dated, 7214 rows; suites_not_covered 64 -> 63; cluster_tests 3474 -> 3486; TESTS.md §306. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lated param no longer passes (#1341) Ports test/native_param_pushdown.sh to test/pytest/test_native_param_pushdown.py, every check by the shell's name, one test per check, and closes three holes in the shell, each measured on PG17 against main and fixed in both harnesses. - The PARAM_EXEC guard was untested. With it removed (freezeexec) main passed 5/5: Begin freezes before any outer row sets the param, so the frozen value is its initial one and "ts >= <that>" prunes nothing; the lower-bound correlated answer stayed 3002. The arm now reads the scan's Usable Skip Predicates (must be 0), asks the upper bound "ts <= thr.lo" (77000), where the frozen value prunes every group, and checks the plan rescans the columnar scan per outer row. - A value frozen once and reused by a later execution (staleparam) passed 5/5: every arm ran one execution per session. Two executions of one generic plan in one session now each count (2001) and prune (18 of 20) with their own value. - Survival was a bare SELECT 1: with a backend SIGKILLed before it, main passed. It now also requires the server log's crash count to be unchanged. Mutants on PG17, same failing checks by name in shell and twin: shiftparam 4, noparam 2, freezeexec 2, staleparam 2. The harness kill reds "backend alive" in both. 10/10 shell and twin on each of 15-19. Seeded: 10 rows, 6 dated, 4 never. Census 4212 -> 4216 never, 3002 -> 3008 dated, 7224 rows; suites_not_covered 63 -> 62; cluster_tests 3486 -> 3496; TESTS.md §307. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…wrong order no longer passes (#1341) Ports test/native_sort_by.sh to test/pytest/test_native_sort_by.py, every check by the shell's name, one test per check, and fixes two holes in the shell, each measured red-first against main's shell on PG17 (PG18 for the virtual arm): - The physical order was never read. Group skipping on host was the only evidence of a sort, so a rewrite that applied only the first key (firstonly: ts left unordered) passed 16/16. The rows are now read in scan order and every adjacent pair is compared on (host, ts) ("the scan order follows the declared key (host, ts)"), behind a premise that the inserted order is not already sorted, and the explicit-column sort must lay the rows out as the declared key does. - Every refusal counted any psql failure. A backend aborting in place of the no-key refusal (keyabort) passed its own check; a crash in place of the virtual-column refusal (virtabort) passed 18/18 on PG18; a dropped declared column given the generic message (persgeneric) passed 16/16. Each refusal now wants its guard's SQLSTATE and a phrase of its message, every reading after a refusal waits for the server, and "backend still up after the refusals" reads the server log's crash count. The empty and single-row sorts read psql's exit status. Mutants on PG17, the same checks red by name in shell and twin: firstonly 1, persgeneric 1, keyabort 2, nosort 2, wrongcol 4, desc 4; virtabort on PG18 2. Clean: shell 20/20 (+1 skip) on 15-17 and 22/22 on 18-19, twin 22/22 on each. Before PG18 the virtual arm is recorded as a skip, and its two twin tests decline (expected_unrunnable.txt). 23 rows, 9 dated, 14 never. Census 4216 -> 4230 never, 3008 -> 3017 dated, 7247 rows; suites_not_covered 62 -> 61; cluster_tests 3496 -> 3518; TESTS.md §308. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e index stale no longer passes (#1341) Ports test/native_rewrite.sh to test/pytest/test_native_rewrite.py, every check by the shell's name, one test per check, and fixes four holes in the shell, each measured on PG17 against main and fixed in both harnesses: - The threshold was untested from both sides. With every group that has a live row rewritten (overrewrite), or min_deleted_fraction ignored (nothreshold), main passed 17/17. Two new checks: "a second rewrite with nothing deleted rewrites no group" and "a group deleted below the threshold is not rewritten". - vacuum's index rebuild was untested. With pgcolumnar_compact_relation's reindex skipped (noreindex), main passed 17/17: over the whole table the stale entries land on every live row once, so a count and a hash through the index both still matched. A narrow index range read does not (id 4097 read back as 6146). New check: "an index range read after vacuum returns the rows asked for". - Four counts compared two readings with plain check, and "unique constraint still enforced" took any psql failure as enforcement. With the server stopped after the rewrite, four checks passed on main. Counts are check_num, and the refusal wants 23505 and n_id_uq's own message. Mutants on PG17, same failing names in shell and twin: droprow 6, noindex 5, overrewrite 2, nothreshold 1, noreindex 1. With the server stopped after the rewrite, both harnesses fail 15 and pass the 5 read before it. 20/20 shell and twin on each of 15-19. Seeded: 20 rows, 9 dated, 11 never. Census 4230 -> 4241 never, 3017 -> 3026 dated, 7267 rows; suites_not_covered 61 -> 60; cluster_tests 3518 -> 3538; TESTS.md §309. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ector's first value no longer passes (#1341) The suite checked each zone-map bound for presence and width only, and every column of its fixture ascended, so a vector's first value was always its minimum. A writer that never raised a vector's max past its first value (nomaxupd), never lowered its min (nominupd), or never folded a chunk max past its first vector (foldmax) stored bounds of the right width, and main's shell passed 18/18 under each, on PG17. - v is now a permutation of 1..6150 (its values, nulls and counts unchanged), with a premise that it is not ascending within a vector. - Four checks decode each bound of id, k and v (zm_le reads the stored native bytes) and compare it with the heap's own for its group and vector: every vector's min, every vector's max, every whole-chunk min, every whole-chunk max. - "value_count sums (v, nullable)", "null_count sums (v, nullable)" and "whole-chunk sum(id) matches heap" compared two readings with check: with the server stopped they passed on two empty strings. They are check_num now; a stopped server fails every check read after it. The twin test_native_zonemap.py asserts all 23 checks by the shell's names, one per test. Mutants on PG17, the same names red in shell and twin: nomaxupd 2, nominupd 2, foldmax 1, nominmax 5. Clean 23/23 in both on each of 15-19. Seeded: 23 rows, 5 dated, 18 never. Census 4241 -> 4259 never, 3026 -> 3031 dated, 7290 rows; suites_not_covered 60 -> 59; cluster_tests 3538 -> 3561; TESTS.md §310. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…orage id is unread (#1697 review) jdatcmd's #1697 review: when the storage id could not be read, the twin fell back to `WHERE storage_id = NULL`, which matches no row, and a count of bad rows over no rows is the 0 nine checks want. The shell has no fallback: its empty $SID is a syntax error, so each check fails. Each zone-map reading now goes through _zint, which returns sid's own failure instead of querying; _bounds_match and the after-drop count short-circuit the same way, so expect.num refuses each one, as under a stopped server. nullsid (pgcolumnar_relation_storageid returns NULL for every relation, one line in columnar_vacuum.c) on PG17: - before: shell 3 passed + 20 failed; twin 12 pass + 6 fail, with nine names passing in the twin that fail in the shell ("id (int4) min width = 4 bytes", "k (int8) max width = 8 bytes", "min/max present for all columns", "per-vector value+null = 1024", "per-vector value_count folds into chunk (k)", "sum null for non-int4 columns", "sum present for int4 columns", "whole-chunk value+null = group rows", "zone maps cleaned on drop"); - after: the same 20 names red in both (comm prints nothing either way). Clean: 23/23 in both. No check name or ledger row changes; nullsid is not used to date rows (every reading is unreadable under it, so it says nothing about what each measures). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…red across scans no longer passes (#1341) "nested columnar scans each hold their own handle" compared only the join's count, which a shared handle still answers correctly: with one static zone-map session shared across scans (the first implementation of #744), main's shell passed it. It is now the witness, each of the join's two scans reporting its own groups opened once, and the count is a check of its own, "nested columnar scans return the right answer". The three scans around the one that errors in a subtransaction must also each probe exactly their own groups. Two premises a one-line pgcolumnar change reddens lost the prefix (#1339): the probe count (probetwice, noprobe) and the middle scan's error (excludeall). A new check holds zs2 to more than one group, which a writer ignoring stripe_row_limit fails. Six mutants on PG17 redden the same names in both twins: shared 5, reopen 5, probetwice 4, noprobe 7, excludeall 8, nostripe 1. With the server stopped after setup, only the check read before it passes. The re-resolved index oid that columnar_metadata.c documents (idxagain) stays 13/13: no witness here counts it. 13/13 shell and twin on each of 15-19. Seeded: 13 rows, 12 dated, 1 never. Census 4259 -> 4260 never, 3031 -> 3043 dated, 7303 rows; suites_not_covered 59 -> 58; cluster_tests 3561 -> 3574; TESTS.md §311. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d optx100 dates it (#1698 review) jdatcmd's #1698 review: "premise: the fixture really has more than one chunk group" reads pgcolumnar's own row_group catalog, the writer's output, so it can redden for a writer bug; zs2's twin check had already lost the prefix for that reason. It is renamed "the fixture has more than one chunk group" (the row is kept, renamed) and dated by optx100 (columnar_write_state.c: the writer multiplies the table's stripe_row_limit by 100, one line). optx100 on PG17: shell "FAIL the fixture has more than one chunk group: got [no] want [yes]" and "FAIL zs2 has more than one chunk group", 11 + 2; the twin fails the same 2 names. Clean: 13/13 in both. Census for this suite: 13 rows, 13 dated + 0 never (was 12 + 1): never -1, dated +1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ver pruned no longer passes (#1341) Ports test/native_zonemap_narrow.sh to test/pytest/test_native_zonemap_narrow.py, every check by the shell's name, one check per test. Nothing showed a group was skipped. The pruning check read only the answer to c1 > 19000, which is 1000 rows whether or not a group was removed: with group skipping reading the zone map but never excluding a group (noexclude), or never consulting it (nozm), main's shell passed all four checks on PG17. Two checks now read the scan's own "Columnar Chunk Groups Removed by Filter" under EXPLAIN ANALYZE, which must be 9 of the 10 groups on the wide table and on the narrow one. Both fail under each mutant, the same names in shell and twin. The width check compared two counts with -le and errored to "no" on a blank one; it now refuses an unread count as "no reading" rather than comparing it, so its lossy_arms.tsv row is removed. With the server stopped after setup, all six checks fail in both harnesses. Mutants on PG17, the same failing checks by name in shell and twin: noexclude 2, nozm 2, wholegroup (#314 reverted) 1, overprune 5. Head is 6/6 in both on each of 15-19. Seeded: 6 rows, 6 dated, 0 never. Census 4259 -> 4259 never, 3044 -> 3050 dated, 7309 rows; suites_not_covered 58 -> 57; cluster_tests 3574 -> 3580; TESTS.md §312. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ulls as values no longer passes (#1341) Ports test/native_vecdecode.sh to test/pytest/test_native_vecdecode.py, every check by the shell's name, one test per check. Three holes in the shell, each measured on PG17 against main and fixed in both harnesses: - The fixture had no nulls. With the fold ignoring the validity bitmap (nonull), main passed 24/24. A nullable column w (NULL on every seventh row) now runs through the fold ("the fold answers count and sum of a nullable column over a skipping range": 51|2444 against heap's 44|2606 under nonull) and through a scalar parity check, behind a heap premise that the range holds nulls. - Missing counters read as 0. "decoded plus skipped is the group's vectors" passed as 0 + 0 = 0 with the plans failing, and two aggregate parities compared two empty readings: with the server stopped before the plans, main passed 3. Every counter must now be read, the parities use check_text, and the same stop fails all 28. - Five "premise:" checks read pgcolumnar's own behaviour: nofold reddens the three "reaches the fold" checks, decodeall and noskip the fold arm's decode count, and noskip "the narrow range skips vectors". All lost the prefix (#1339). "a ruled-out vector is not decoded" now names both readings when it fails, so its lossy_arms.tsv row is removed. The poison check ("the fold does not count rows from a vector it never decoded") is dated by holeread: the fold ignores the skip mask, and the mutant zero-fills the hole, so the read is deterministic on a non-assert build too. Mutants on PG17, the same failing checks by name in shell and twin: decodeall 3, holeread 1, nonull 1, nofold 4, noskip 3. Head is 28/28 in both on each of 15-19. Seeded: 28 rows, 10 dated, 18 never. Census 4259 -> 4277 never, 3050 -> 3060 dated, 7337 rows; suites_not_covered 57 -> 56; cluster_tests 3580 -> 3608; TESTS.md §313. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…can and optdiv100 date eight rows (#1700 review) jdatcmd's #1700 review: four premises read the plan pgcolumnar produces, and noscan (pgcolumnar.enable_custom_scan off by default, one line; the C initializer too, or an assert build traps) reddens all four, as it did the same check in #1691. They lose the prefix (rows renamed, not removed): "the narrow plan is a columnar custom scan", "the wide plan is a columnar custom scan", "no whole group is pruned on the narrow query" and "the wide range skips none, so the arms really differ". The review asked to apply the same test to "one row group": optdiv100 (the writer divides the table's stripe_row_limit by 100, one line in columnar_write_state.c) reddens it (got [51] want [1]), so it loses the prefix too. The ground-first order stays, so a failure still names its cause. PG17, shell = twin by name: - noscan: 15 red in both (twin: 3 recorded fails + 12 named VacuityErrors); dates the four renamed checks plus "EXPLAIN reports Columnar Vectors Decoded", "and it is not stuck at zero when a full scan decodes everything" and "control: and the same counter reads every vector when nothing is skipped", all never before. - optdiv100: 6 red in both; dates "one row group, so every difference is per-vector". - clean: 28/28 in both. Census for this suite: 28 rows, 18 dated + 10 never (was 10 + 18): never -8, dated +8. The heap-side nulls check stays a premise: it reads h. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t be its own guard's (#1341) The refusal arms read only the SQLSTATE, and four of the six refusals share 22023 (absent id, count mismatch, id-less file, duplicate requested id). With the "file carries no field ids" guard removed (noanyid), an id-less file fell to the absent-id refusal, also 22023, and main's shell passed 11/11. Each arm now wants its own guard's primary message, read with VERBOSITY verbose in the shell and diag.message_primary in the twin. Survival was a bare SELECT 1, which answers again after crash recovery: with the absent-id refusal turned into abort() (abortabsent) it passed, and the crash took five later refusal arms with it. Survival now also reads the server log's count of killed backends, and every reading first waits for the server, so a crash fails its own arm and survival only. The twin's writer runs as a subprocess, so the module imports no pyarrow, and every test declines under the shell's skip name before it asks for the fixture. Measured on PG17, each giving the same failing names in shell and twin: noanyid 1, abortabsent 2, noscalar 1, offbyone 5, bypos 6. With the server stopped before the projection arms, all 11 fail. Clean: 11/11 shell and twin on each of 15-19. Seeded: 11 rows, 9 dated, 2 never. Census 4269 -> 4271 never, 3068 -> 3077 dated, 7348 rows; suites_not_covered 56 -> 55; cluster_tests 3608 -> 3619; TESTS.md §314. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… through the reader under test no longer passes (#1341) Every codec was compared with c_none.parquet read back through read_parquet, the reader under test. A defect after decompression -- in the page decode every codec shares -- corrupted both sides alike: with one decoded byte flipped in every page over 4 KB, the same way for every codec (postcorrupt), main's shell passed 9/9 on PG17. Each file now decodes to the generator's rows, built in SQL from the same generator the crafter wrote them from; postcorrupt fails five checks in both harnesses (and the gzip sum reads 199990033, not 199990000). Each file's codec is now read from its own metadata. The crafter printed it to stderr and nothing read it: with every file written uncompressed, main passed 9/9; the three codec premises now fail. Survival read a bare SELECT 1. With the crafted-size guard aborting (hdrabort) and the probe 3 s late, main passed it; it now also reads the server log's crash count and fails as "no ([1], 1 backend crashes)". The fixtures move into test/craft_parquet_codecs.py, which the twin runs as a subprocess so its fixture never imports pyarrow; each twin test declines under the shell's skip name first. The gzip sum is now check_num against 199990000, so two failed reads can't agree, and the codec and FDW loops are unrolled into literal names. postcorrupt, nocodec, mapcodec and hdrabort each fail the same checks by name in shell and twin on PG17 (5, 4, 1, 2). Head is 14/14 in both on each of 15-19. Seeded: 14 rows, 7 dated, 7 never (the four codec premises read pyarrow's metadata, and the FDW counts hold under every mutation). Census 4271 -> 4278 never, 3077 -> 3084 dated, 7362 rows; suites_not_covered 55 -> 54; cluster_tests 3619 -> 3633; TESTS.md §315. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…le read through the reader under test no longer passes (#1341) The twin, test/pytest/test_native_parquet_units.py, makes each of the shell's 20 checks under its name, on a private database, and declines under the shell's pyarrow skip name before it asks for the fixture. Two holes in the shell, each measured red-first on PG17 against main, fixed in both harnesses: - The pushdown oracle was the two-group millis file read back through read_parquet, the reader under test. With every row after a page's first taking the next dictionary entry (dictshift), the scan and the mirror shifted alike and main passed 19/19. The oracle is now the generator's hourly series, built in SQL, and a new check, "the millis file reads back exactly as written", compares the whole file with it. Under dictshift both fail, the same names in shell and twin. - The nanos truncation checks compared each file with a second file read the same way, with plain check. With the server stopped before them, main passed both on two empty reads. They now want the toward-zero microseconds as literals, through check_text, and fail with the server stopped. The per-unit and per-tag loops are unrolled, so every name is a literal the grader reads. Mutants on PG17, each failing the same checks by name in shell and twin: msasus (MILLIS read as micros) 9, nsasus (NANOS read as micros) 4, floordiv (nanos truncated toward -inf) 1, dictshift 2. Head passes 20/20 in both harnesses on each of 15-19. Not tested, because there is nothing to mutate: the reader skips isAdjustedToUTC, and a timestamptz column decodes exactly as a timestamp one, so there is no UTC adjustment for a check to catch. Seeded: 20 rows, 13 dated, 7 never. Census 4278 -> 4285 never, 3084 -> 3097 dated, 7382 rows; suites_not_covered 54 -> 53; cluster_tests 3633 -> 3653; TESTS.md §316. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ard (#1341) test/pytest/test_objstore_http_read.py is the twin of test/objstore_http_read.sh: 24 checks under the shell's names, one test per check. Holes on main's shell, each passing 18/18 on PG17, now red in both harnesses: - arm C took the SQLSTATE alone, so a dropped Range-ignored guard passed on 08P01 from "unexpected HTTP status 200" (noguard200). Each refusal now needs its own guard's message as well. - no fixture sent a short 206 body, cut a body off mid-transfer, or dropped a keep-alive connection: noshort, eofok, noretry and alwaysretry passed. The fixture server gains /short/, /trunc/, /dropka/ and /hangup/ modes, inert unless a path names them. - no crash check: a survival check now counts "terminated by signal" after waiting for the server, and every reading after a possible crash waits for the server first, so one arm's crash can't redden the next. Ledger: 24 rows, 16 dated by ten mutants (SAME reds in shell and twin on PG17), 8 never. Census 4355 -> 4363 never, 3266 -> 3282 dated, 7621 -> 7645 rows; suites_not_covered 40 -> 39; cluster_tests 3895 -> 3919. In the stack, three twin test names collided with names earlier ports define (test_the_fixture_server_is_up, test_the_cancel_latency_is_bounded, test_the_fixture_is_written): they are now test_the_http_read_*, in the twin and in its TESTS.md rows. No check name changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1341) Port test/objstore_sink_write.sh to test/pytest/test_objstore_sink_write.py, one test per shell check under its literal name, on a private cluster restarted with AWS_* and the SigV4 fixture server. Holes in the shell, measured red-first on PG17 before the fix: - the fixture completed an upload from every stored part, ignoring the list, so a client dropping the last part from it passed 30/30. It now assembles the listed parts, each checked against its ETag, and a failed complete leaves the upload open, as S3 does. - no check made a part fail, so a refused part read as stored passed 30/30. objstore_http_server.py gains a __part_status__ hook; four checks want the part check's own 58030, no key and no open upload. - abort was read from the DELETE count, so an abort naming the wrong upload passed 30/30. The server keeps __open_uploads__ and two checks want none open. - the Arrow arm read only export_arrow > 0, green with the object's last byte missing; the object is imported back from the bucket. - every oracle hashed the columnar table the export scanned; they are the generators now. Each refusal wants its own guard's message, and the cancel arm lets in-flight requests land before reading the prefix. The pair is declared INCOMPLETE: the two opt-in real-S3 integration arms (PGC_S3_INTEGRATION_ENDPOINT, never set in CI) are not ported. Eight mutants on PG17, the same reds in shell and twin; 35/35 on 15-19. Ledger 35 rows (11 dated, 24 never); never 4363 -> 4387, dated 3282 -> 3293, snc 39 -> 38, cluster_tests 3919 -> 3954. In the stack, the twin test test_the_sigv4_fixture_server_is_up collided with the one #1718 defines: it is now test_the_sink_write_sigv4_fixture_server_is_up, in the twin and its TESTS.md row. No check name changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1722) On cancel, pexport_cleanup terminates each parallel_export worker, and SIGTERM ends it in FATAL. FATAL calls proc_exit without unwinding, so the writers' PG_CATCH never reached PgColumnarSinkAbort, and each worker's multipart upload stayed open on the store, billed until a lifecycle rule reaps it. The sink layer now keeps the remote sinks with an upload in progress on a list and aborts any still listed from a before_shmem_exit callback. The callback is registered after InitPostgres's ShutdownPostgres, so it runs while the transaction and the module's handles are intact. Finish (after a complete that succeeded) and abort unlist a sink, and transaction end empties the list. The SIGTERM can land mid-request, with a part's body still on the connection, so the module's abort now reconnects before its DELETE. Without that, the DELETE was read as more of the part (the request log shows the part's PUT complete and no abort for its upload), and 4 of 20 cancel runs of the twin still left one upload open. With it, 0 of 15. objstore_sink_write's cancel arm gains "cancel: the server holds no open upload under the prefix". It was red on main's code on all five majors, in the shell and the twin alike (4 uploads open under cx/). With the fix it passes 36/36 on 15-19 in both. Dropping the before_shmem_exit registration reddens it by name in both twins, which dates its row. Census: never 4387 -> 4387, dated 3293 -> 3294, rows 7680 -> 7681; cluster_tests 3954 -> 3955. Stacked on #1721, whose fixture lists open uploads. Fixes #1722 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…old a terminated export worker (#1727 review) jdatcmd's #1727 review: os_wait passed no WL_TIMEOUT, so WaitLatchOrSocket ignored the 5 s slice (timeout=-1) and a silent store never counted a quiet slice; the 60 s quiet limit was dead code. Under proc_exit interrupts are held, so a parallel_export worker that SIGTERM ended stayed in its exit-callback upload abort until the store answered. os_wait now passes WL_TIMEOUT. objstore_sink_write gains "terminate: export workers exit within 75 s while the store is silent": SIGSTOP the fixture mid-upload, terminate the workers, time their exit. - red first on #1727's head, PG17, both twins: "got [slow (4 workers, 4 left after 91s)]" (twin: 90s), which dates the row; - with the fix: 61 s on each of PG15-19, 38/38 in both twins. pg_ctl stop -m fast with the store silent mid-export: 60 s with the fix; without it the stop did not finish within 150 s (rc=1). The coordinator's addition: jdatcmd saw "cancel: the server holds no open upload under the prefix: got [1] want [0]" flake in CI on #1723's head, an upload whose POST ?uploads was still in flight when the cancel landed (#1743's window). The cancel arm now cancels only once the fixture's __open_uploads__ shows all four workers' uploads, with its own check ("all four workers' uploads were open before the cancel"), so the open-upload check pins exactly what #1722 fixes. That check is not a premise: nomultipart (the sink's flush loop never runs, so no export opens a multipart upload; one line in columnar_objstore_module.c) reddens it by name in both twins on PG17 ("got [no (0 open)] want [yes]"; the same 11 names red in shell and twin), which dates it. PG17 at the fix: 10 runs of shell and twin, 0 flakes (38/38 each). noreg (the before_shmem_exit registration dropped) still reddens "cancel: the server holds no open upload under the prefix: got [4] want [0]" in both twins. Census: +2 rows, never 4350 -> 4350, dated 3117 -> 3119; cluster_tests 3756 -> 3758; two TESTS.md rows in objstore_sink_write's section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nted out no longer passes it (#1341) test/pytest/test_native_fetch_interrupt.py ports all 3 checks of test/native_fetch_interrupt.sh by name. It reads src/columnar_reader.c in Python, needs no cluster, and is listed in NO_CLUSTER. One shell fix, in both twins: - Every check read the function bodies with their comments in them. With pgcolumnar_fetch_row's guard written as /* CHECK_FOR_INTERRUPTS(); */ (cficomment), or folded into the comment block above it (cfiml), main's shell passed 3 of 3: the #212 regression the suite exists to catch. With the lookup's uncached PgColumnarReadRowGroupList call replaced by NIL and a comment naming it (rglcomment), main passed 3 of 3 too. Both bodies now have their comments blanked, multi-line ones included, with their newlines kept so the placement check still orders lines of code. Measured on PG17, main's shell, this shell and the twin: - cficomment: main 0 red; 2 red here, the same 2 in the twin. cfiml: main 0 red; the same 2. cfigone (the guard removed): main 2 red; the same 2. cfilate (the guard moved past the lookup call): main 1 red; the same 1. rglcomment: main 0 red; 1 red here, the same 1 in the twin. - clean: 3 of 3 on PG15 to PG19 in the shell, and in the twin. Ledger: 3 rows, all dated, none never. suites_not_covered 38 -> 37, guard_tests 806 -> 809, TESTS.md section 332. Census 4387 -> 4387 never, 3296 -> 3299 dated, 7686 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s3:// no longer passes it (#1341) test/pytest/test_objstore_module.py ports all 30 checks of test/objstore_module.sh by name, from one module fixture that takes a pristine copy of the installed module and puts it back in its finally. Two shell fixes, in both twins: - The remote-read checks took any of seven phrases, "is not supported" among them. With the module no longer handling s3:// (s3unhandled), gs:// (gsunhandled) or, built with OpenSSL, https:// (httpsunhandled), each read said so and main's shell passed 30 of 30. Each URL's refusal is now pinned by SQLSTATE and message (28000 naming AWS_ENDPOINT_URL or AWS_ACCESS_KEY_ID; for https:// 42501 from the allow-list when the module links libssl, else 0A000), and the globs by 28000 naming AWS_ENDPOINT_URL. The suite clears AWS_* before starting its cluster, so an ambient one cannot turn a refusal into a connection attempt; the twin declines those checks when its own environment carries one. - "a local path still works" counted a columnar table, which never reaches read_parquet's remote dispatch. With every local read_parquet refused as a missing file (localrefused), main passed 30 of 30. It now reads back, through read_parquet, a file export_parquet wrote. The start guard's failure was recorded under a name built from two paths, which no twin could carry; it is now "a stash restored over a missing module is a module", the paths in its detail. The twin ports the guard and reads that check every run (one test beyond the 30); the shell records it only when it fails, so it gets no ledger row. Measured on PG17, main's shell, this shell and the twin: - s3unhandled, gsunhandled, httpsunhandled, localrefused: main 0 red each; 1 red here each, the same name in the twin. - absentunsupported (a missing module reported as an unhandled scheme): main 2 red; the same 2 here and in the twin. triedfirst (the loader's verdict cached before the load can raise): 1 red, the same in all three. brokenswallowed (a present but unloadable module reported as not installed): 2 red, the same in all three. - the twin's start guard: debris beside a valid module removed, 31 passed; a 19-byte module with garbage at .away, the guard's check red by name. - clean: 30 of 30 on PG15 to PG19 in the shell, 31 of 31 in the twin. Ledger: 30 rows, 8 dated by the seven mutations and 22 never. suites_not_covered 37 -> 36, cluster_tests 3957 -> 3988, TESTS.md section 333. Census 4387 -> 4409 never, 3299 -> 3307 dated, 7716 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…st be the one stored (#1341) test/phase2.sh round-tripped each of four codecs and never read which one was stored: the native conversion dropped the 2.2 catalog's check and left $code computed for nothing. With the session's pgcolumnar.compression ignored (every write zstd), or with no block codec ever applied, all 42 checks passed. Each codec now reads its payload column's block_codec. The filter checks held with the zone maps ruling out nothing, because exact selection returns the same rows after decoding every vector. A range inside the last of three vectors must now decode one of them, read as Vectors Decoded (Vectors Skipped still says 2 with the zone maps off). The suite moves onto lib.sh's check, as smoke.sh did: its own check called pgc_record directly, which the twin's grader cannot read. test/pytest/test_phase2.py is the twin, one check per test by the shell's name, on a private database; the per-codec tests are parametrized under the shell's own $codec template. Census 4409 -> 4434 never, 3307 -> 3329 dated, 7716 -> 7763 rows; suites_not_covered 36 -> 35; cluster_tests 3988 -> 4035; TESTS.md section 334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… a savepoint leaves nothing (#1341) test/phase3.sh's savepoint arm rolled back only completed statements. Executor end had already flushed their buffers under the subtransaction, so the abort's xmin verdict hid them and the savepoint's own discard was never reached (#1441). With the subxact abort discarding neither pending writes nor pending deletes, all 32 checks passed. An INSERT whose 2001st row divides by zero, and a DELETE that fails at a = 9, are now each rolled back to a savepoint and must leave nothing, in the transaction and after it. Measured: the mutants leave 2000 rows visible, and rows 2..8 deleted. The delete arm said its range straddled a vector boundary and nothing read one: with 16384-row vectors all 32 passed. The zone maps' maxima are now read. A DELETE whose RETURNING reads the table in a block that commits into one that aborts must see the rows deleted so far, as heap does. With the savepoint commit not handing its delete buffers and written marks to the parent, the abort left the read's marks attributed to a finished block, no later read wrote them again, and the reads summed 19999 against heap's 14950. The statement's end still wrote every mark, so only the reads showed it. The suite moves onto lib.sh's check, as smoke.sh did: its own check called pgc_record directly, which the twin's grader cannot read. test/pytest/test_phase3.py is the twin, one check per test by the shell's name, on a private database. Census 4434 -> 4465 never, 3329 -> 3335 dated, 7763 -> 7800 rows; suites_not_covered 35 -> 34; cluster_tests 4035 -> 4072; TESTS.md section 335. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…wrong row no longer passes (#1341) test/pytest/test_phase4.py is the twin of test/phase4.sh, every check by the shell's name. Fixed in both, test-first (each mutant below passed main's shell): - A refusal passed on any failure. A unique check raising XX000 instead of 23505 passed both duplicate arms (dirtyerr: 38/38 on main), and a backend that died on the duplicate passed "uq cross-stmt dup" as rejected (bufcrash). Every refusal now asserts its SQLSTATE: 23505, 23502, 23514. - The point reads ran with only enable_seqscan off, so they planned the columnar custom scan and never reached pgcolumnar_index_fetch_tuple. An index fetch handing back the next live row's values passed every hash read and the btree point (fetchshift: 37/38 on main, only "btree range first" red). The reads now run under the plan checks' settings (custom scan and index-only scan off), and a new check, "deleted range plan is index scan", asserts the delete-vector reads plan a plain index scan. - The suite's wrappers called pgc_record directly, which a twinned suite may not (test_compare_to_bash.py's extractor arm). They record through pgc_pass and pgc_fail now; a failure still shows the measured value. Seeded: 39 rows, 12 dated by four mutations (the same names in both twins), 27 never. Census 4465 + 3335 == 7800 -> 4492 + 3347 == 7839, suites_not_covered 34 -> 33, cluster_tests 4072 -> 4111, TESTS.md section 336. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… its own (#1723 review) jdatcmd's #1723 review: "uq unchanged" was dated by bufcrash (a unique check that finds a buffered row calls abort()), but it reddened only as a crash's collateral: the next count landed during crash recovery ("got [] want [3]"), and after recovery the count is right. It is re-dated, per the review's suggestion, by dirtyskip: pgcolumnar_index_fetch_tuple returns false to a dirty snapshot (one line in columnar_tableam.c), so the unique check never sees the existing row and the server stays up. dirtyskip on PG17, shell = twin by name, 6 red: "uq cross-stmt dup", "uq within-stmt dup", "uq unchanged", "uq final count", "uq2 dup", "uq2 count" (33 + 6 in both). It also dates "uq final count" and "uq2 count", never before. Clean: 39/39 in both. (The shell needs PGUSER=postgres off CI, as before.) Census for this suite: 39 rows, 14 dated + 25 never (was 12 + 27): never -2, dated +2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…longer passes (#1341) test/pytest/test_phase6.py is the twin of test/phase6.sh, every check by the shell's name. Fixed in both, test-first (each mutant below passed main's shell): - Every aggregate and scan check compared vectorization on with vectorization off. Both read the same columnar decode, and 21 of the 39 queries plan identically either way (measured on 17: every filtered aggregate, every fallback aggregate, every row scan), so those compared a path with itself. A float decode that flipped a mantissa bit of each vector's first value passed 43/43 (floatbit); a varlena decode that changed one byte per vector reddened only "min/max text" (textw). Each query now also runs against schema `oracle`, heap tables of the same names filled by the same generator statements and the same DELETE, and all three readings must agree. - The suite's wrappers called pgc_record directly, which a twinned suite may not (test_compare_to_bash.py's extractor arm). They record through pgc_pass and pgc_fail now; a failure still shows all three readings. Seeded: 43 rows, 5 dated by two mutations (the same names in both twins), 38 never. Census 4490 + 3349 == 7839 -> 4528 + 3354 == 7882, suites_not_covered 33 -> 32, cluster_tests 4111 -> 4154, TESTS.md section 337. phase6.sh was the last suite calling pgc_record directly, so in the stack the guard premise "some suite does call pgc_record directly, so this is not vacuous" had nothing left to count. It is now a planted control ("the detector finds a planted direct pgc_record call"), which fails with the detector broken; the arm it protects ("no suite calling pgc_record directly has a twin") still fails when a twinned suite calls pgc_record directly (planted in phase6.sh: got phase6). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ts session no longer passes (#1341) test/pytest/test_cancel_decode.py ports all 11 checks of test/cancel_decode.sh by name, on a private database built once by a module fixture in the shell's order. Three shell fixes, in both twins: - The three after-the-cancel checks ("the session is usable", "the table still reads correctly", "the cancelled predicate returns its true answer") each opened a new psql, so they read a backend that had never been cancelled. With a flag the group load sets around its decode, which a cancel leaves set, the next scan in that backend raised an ERROR (stalestate) or called abort() (stalecrash), and main's shell passed 10 of 10 under both. One session now takes a cancel and then answers all three; the rows read also sums a column. - Nothing looked for a crash. A new check counts "terminated by signal" in the server log once the server answers again: a fresh SELECT 1 answers after crash recovery and cannot see one. - A cancel was any output containing "canceling statement". It is now SQLSTATE 57014 with "canceling statement due to statement timeout" (psql -v VERBOSITY=verbose), and anything else is reported as what came back. The ratio and absolute-bound arms now carry both latencies when they fail. Measured on PG17, main's shell, this shell and the twin: - stalestate: main 0 red; 1 red here (the predicate's answer), the same 1 in the twin. stalecrash: main 0; 2 (+ no backend crashed), the same 2. - loadcrash (abort() where the group load takes a pending cancel): main 7; 8, the same 8. noload (the per-chunk check removed) and nogroup (that and the decode macro): main 1; 1, the same 1 (the latency ratio, about 520 ms against 110 ms). - clean: 11 of 11 on PG15 to PG19 in the shell, and in the twin. Still a ratio between two latencies read back to back, so the suite keeps its scheduling: not a timing suite, no PGC_SKIP_TIMING guard, and no row in expected_unrunnable.txt. Ledger: 11 rows, 8 dated, 3 never (the two group counts and the scan outlasting the timeout, which no product mutation here moves). suites_not_covered 32 -> 31, cluster_tests 4154 -> 4165, TESTS.md section 338. Census 4528 -> 4531 never, 3354 -> 3362 dated, 7893 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t of its loop no longer passes (#1341) test/pytest/test_decode_interrupts.py ports all 29 checks of test/decode_interrupts.sh by name: the per-decoder pair is parametrized over the shell's own $f loop. It reads src/columnar_encoding.c and src/columnar_reader.c in Python, needs no cluster, and is listed in NO_CLUSTER. Two shell fixes, in both twins: - Every check read the source with its comments in it. With decode_for's COLUMNAR_DECODE_INTERRUPT(i) written as a comment (forcomment), or the macro's own CHECK_FOR_INTERRUPTS() (macrocomment), main's shell passed 29 of 29. Both files are now read with comments blanked, multi-line ones included, newlines kept. - Presence in a function stood for presence in its loop. A function's range ran to the next definition, and "checks for interrupts" counted a check anywhere in it; the two placement arms took anything after the loop's header line, and the two reader arms counted the call anywhere in the function. Main passed 29 of 29 under each of: decode_for's check hoisted in front of its loop, where it runs once per vector (forhoist), the same for bitunpack, decode_delta, decode_gorilla, decode_dod, decode_alp and decode_dict (hoist:*), decode_rle's check moved below the run's value loop (rleafter), decode_fsst_shared's below its decode loop (fsstafter), and either reader check moved in front of its loop (loadhoist, rowhoist). A range now ends at the function's closing brace, and a check counts only inside the braces of a for, while or foreach body, or of the named loop where placement is the point, walked a character at a time. Measured on PG17, the same failures by name in shell and twin under every one of 30 mutants; main's shell was 0 red under the 13 above, and 1 to 3 red under the other 17 (renames, the stride, the loop headers). Clean: 29 of 29 on PG15 to PG19 in the shell, and in the twin. Three comments that described decode_interrupts.sh's removed checks_in and its two piped-awk arms in the present tense now say #1341 retired them. Ledger: 29 rows, all dated, none never. suites_not_covered 31 -> 30, guard_tests 809 -> 838, TESTS.md section 339. Census 4531 -> 4531 never, 3362 -> 3391 dated, 7922 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…list too late no longer passes (#1341) Ports test/decode_skip_interrupts.sh to test/pytest/test_decode_skip_interrupts.py: all 5 checks under the shell's names, one test per check. The skip-bomb footer is written in Python as the shell's heredoc writes it, into server_dir; the statement runs in a daemon thread abandoned after 30 s, as the shell's KILL abandons psql. Four holes, each passing on main's shell (PG17): - Survival was a fresh SELECT 1, which answers once crash recovery is over: with abort() in the list arm it passed. It now counts "terminated by signal" in the server log since the scenario began. - The cancel was any 57014: a guard refusing the list under the cancel code passed. It now wants the timeout's own message. - With PgColumnarThriftSkip's interrupt check moved into the struct arm (per field, not per value), the cancel still came after a whole 4.3e9-element list (7 s), inside the 30 s cap, and the shape count still read 1: main passed 4/4. A new check wants the cancel within 3 s of a now 1 s timeout, and the Thrift shape counts before the switch, as av_skip's does. - Both shape counts read comments: with av_skip's check commented out, main passed 4/4. Comments are blanked first. Census 4531 -> 4531 never, 3391 -> 3396 dated, 7922 -> 7927 rows; suites_not_covered 30 -> 29; cluster_tests 4165 -> 4170. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…loop interrupt no longer passes (#1341) Ports test/native_cancel.sh to test/pytest/test_native_cancel.py: all 4 checks under the shell's names, one test per check, in a private database (the row-group premise counts a catalog every test shares). The timeout is still a quarter of the server's own load time (#1481), in both. Two holes, each passing on main's shell (PG17): - The table had two columns, so load_group's per-chunk interrupt check fell at half the load. With COLUMNAR_DECODE_INTERRUPT a no-op, the cancel landed there (90 ms against a 172 ms load and a 106 ms limit) and main passed 4/4, the opposite of what its header claimed. The table is now one column of 16M values, so every per-chunk check falls before the decode; the same mutant now reads 173 ms against 168. - The timeout's error was matched by message alone. It is now pinned by SQLSTATE too, on every attempt: the cancel is the quickest of three that each raised it, and an attempt refused during crash recovery is not timed, so it cannot read as an early cancel. Census 4531 -> 4533 never, 3396 -> 3398 dated, 7927 -> 7931 rows; suites_not_covered 29 -> 28; cluster_tests 4170 -> 4174. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…sting and its read (#1633) pgcolumnar_recluster_online listed the groups it would retire under one snapshot, locked them, read every row a LATER snapshot saw, and retired only the groups it listed. A row committed in between, including while recluster waited for another session's group lock, sat in a group it never listed: it was read and rewritten, and its own group was kept, so the row appeared twice. The rewrite now reads only the groups it listed (a binary search over their row ranges). A row written in the window stays where it is and is counted as appended; a row deleted in the window is invisible to the read snapshot and stays deleted. Reading under the listing's snapshot instead would have written those deleted rows back, so that direction was not taken. snapshot to between the listing and the group locks, so recluster_extent's concurrent writer lands in the whole window. No second mechanism. A DELETE now joins the insert inside the hold, and two checks are new in both harnesses: "no row was rewritten and also kept" and "a row deleted during the rewrite stays deleted". Measured on PG17, shell then twin as shipped: - head: 20/20 and 20/20. - the fix reverted (main's read, hold in the window): the same 4 reds in both, 1599000 rows against 1549000 and 50000 duplicates. - reading under the listing snapshot: the same 3 reds, all 1000 deleted rows back. - over-applied (retire every group the read snapshot sees, appended ones too): the same 3 reds, the 50000 inserted rows lost. Clean: recluster_extent 20/20 in both on PG15-19. Every other suite that calls recluster, shell and twin on PG17 and PG18: recluster_gate, native_recluster, hilbert_cluster, column_extent, autovacuum, maintenance_crash, maintenance_rollback, native_ownership, native_reclaim, native_reclaim_frag, eager_ordering_record, sort_status, sorted_mark_rename, vm_clear_on_renumber, vacuum_lock_privilege; native_scale (shell only) on PG17. All green. Ledger: 2 rows added and dated, "every row is present afterwards" dated. Census 4533 -> 4532 never, 3398 -> 3401 dated, 7933 rows; suites_not_covered unchanged; cluster_tests 4174 -> 4176. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ead the oracle, and three unmeasured claims are measured (#1339) Fixes the #1339 entries for test_differential.py (:312, :360, :519, :541, :704, :840) and differential.sh :349. - :349 [twin_mismatch]. Five shell arms compared EMPTY with EMPTY: "c_int eq" (600 lands on a NULL row), "c_ztext is null" (never NULL), "textbloom collate-mismatch" ('k100' is in no row), "empty scan" and "wide nomatch". With the row reader returning nothing (noscan), main's shell passed all five. The shell gains the twin's present-value controls, and collate-mismatch now probes a present value; under noscan each control is red in both twins. - :519 / :541 [mislabelled, inaccurate]. Six twin premises read t_col, so noscan reddened them as "premises": they now read the t_heap oracle. "The all-null column spans several chunk groups" reads pgcolumnar's layout and loses its prefix (ignoring chunk_group_row_limit reddened it). "Exactly one hundred rows are null, so one whole group is" counted nulls only; under biggroups it still read 100. The null count is now a heap premise, and the whole group is counted in zone_map: "allnull chunk: exactly one chunk group of the column is entirely null". - :360 [twin_mismatch]. The twin's ordered-oracle premise read t_col; it now reads generate_series as pgc_check_ordered_oracle does, with the shell's two other names ("agrees with itself", "the set oracle is order-blind by design"). - :704 [inaccurate]. With every vector stored as encoding NONE (forcenone), both "enc+nocompress" arms stayed green: "enc+nocompress: the column is stored encoded, with no block codec" reads column_chunk. - :840 [weak]. With the bloom never consulted (bloommaybe), every textbloom arm stayed green: "textbloom absent: the bloom removes chunk groups min/max alone cannot" compares groups removed for an absent in-range probe with the bloom on and off. - :312 [vacuous]. "{col} sum/avg tolerance is tight enough to fail on a real decode error" compared FLOAT_RTOL with a literal; removed. Decode errors are caught exactly by "{col} project". Measured on PG17, shell then twin: forcenone, bloommaybe 1 red each, SAME; biggroups 7, SAME; noscan reddens no premise and every new control. Head is 218/218 and 89/89 on each of 15-19. 12 ledger rows, 10 dated. Census 4532 -> 4534 never, 3401 -> 3411 dated, 7945 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ays (#1341) harness_selftest tests the shell harness itself (lib.sh and portlib.sh: the port-squatter identity guard, start retries, the suite list's rules), not pgcolumnar, so a pytest twin of it would re-test bash library code from Python. Agreed on #1341 (option 1): declare it, rather than leave it in unported_suites.txt forever or exempt it one way. test_compare_to_bash.py gains SHELL_HARNESS = {stem: reason}. The unported guard accepts a declared stem without a twin; red first: with the stem simply deleted from unported_suites.txt, "every registered suite without a pytest twin is declared unported" failed naming harness_selftest. A new guard, test_a_shell_harness_suite_is_declared_both_ways, holds the declaration the other way. Each arm was planted and reddened by name: - a declared stem gains test_<stem>.py: "a shell-harness suite has no pytest twin" - a declared stem is not in the runner's SUITES: "and every shell-harness stem is a registered suite" - a declared stem is also listed unported: "and no shell-harness stem is also declared unported" - an empty reason: "and every shell-harness stem says why" harness_selftest leaves unported_suites.txt, which empties its a-m half. guard_tests 474 -> 475 by collection; guards 475 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e shell's names, and a daemon that converts a zorder table no longer passes (#1339) Fixes the eleven open #1339 entries in test_hilbert_cluster.py and test_hilbert_locality.py, each measured red-first on PG17 with a one-line product mutant, in both harnesses where the twin can see it. - :298, :646, :1499, :1520 and locality :614 [mislabelled]: every `premise:` that a one-line pgcolumnar change reddens loses the prefix, in both harnesses: 49 records in the shell (42 in hilbert_cluster, 7 in hilbert_locality). Each verb's "ran without raising" and "moved its table" (hrun/crun and _verb), the established verbs' S1 refusals, both hand-driven references, the daemon's view of av_hi, s4a/s4b/s5/s6t, and "the Z-order and Hilbert partitions differ". Under the mutants used here, no `premise:` row goes red any more. - :301 [inaccurate]: "and {new}'s state ... is the one {old} raises" compared two readings with nothing requiring {old} to have raised. With the shared column cap removed (nocap), cluster and cluster_hilbert both returned noerror on nine columns, and the comparison passed in both harnesses. It now compares only a raised SQLSTATE: under nocap it is red, the same 6 names in shell and twin. The twin's S1 test is split into one check per test, so a red premise no longer hides the comparison after it. - :1362 [weak, MISSING]: every table the daemon handled was a hilbert one. A daemon that reclustered everything on the hilbert curve (avallhil) passed 181/181 and 45/45. A zorder table, av_zd, is now left decayed for it. It must fold av_zd's tail and keep it zorder on its key: under avallhil that is the one red, in both. - :854 [twin_mismatch]: "(a2) and it folded the appended tail back in" shared a test with six other checks, so once the first went red it was never evaluated: under a no-op recluster_hilbert (rhnoop) the shell named it and the twin did not. The (a2) test is split into one check per test, and under rhnoop both now name it. - locality :733 [twin_mismatch]: the shell gains the twin's "parallelism is off on the connection every arm runs through". - locality :792 [twin_mismatch]: the twin's refusal branch records the shell's four unrunnable names, not one anonymous decline. - :1647 (#1339 cross-file 7): "the catalog and the install script still agree on the symbol set" compared a "same" token. Both harnesses now compare the two lists, so an empty side is refused and a failure names the symbols. Head is 187/187 and 69/69 (hilbert_cluster), 66/66 and 18/18 (hilbert_locality) on PG17 and PG18. compare_to_bash: 0 missing for both. cluster_tests 2748 -> 2772. The suites have no ledger rows, so the census is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…sory_lock_class, catalog_delete_index and test_connection checks measure what they name (#1339) Fixes #1339 entries, each measured red-first on PG17 against main: - test_index_fetch_penalty_width.py :192 [mislabelled]: "the penalty is what moves this plan" and the narrow/wide ladder arms read pgcolumnar's costing; with the penalty switched off in the source the first went red. They lose "premise:" in both twins. :199 [twin_mismatch]: the shell's ladder started at 20, so it read the wide table's flip as 0 rows on every run and its ordering arm passed for the wrong reason; it gains the twin's rungs 1, 2, 5, 10 and the wide-table arm. - test_index_fetch_penalty_crossover.py :101 [mislabelled]: with ANALYZE sampling the first int4 column halved, "premise: the 50000-row estimate is exact" went red in both twins; it loses the prefix and is dated. :125 [weak]: "both paths return the same aggregate" compared the two sums with each other and never saw the plan; with the index path left unforced, main passed 7/7. Both sums now meet the generator, and "and the index-forced path really fetches by index" reds that mutation in both. - test_index_am_support.py :187 [weak]: counts only, and no plan. With an index fetch returning the next row, main passed btree, gist and spgist; md5 of the ids now reds gist, hash and spgist (btree reads ids from the index itself), and "<am> answers through the index it built" asserts the plan. - test_format_downgrade.py :214 [weak]: "and fetches from it by index" read only the SQLSTATE, so a fetch that returned NULL (the #1137 defect) passed; with the fetch returning NULL, main's twin passed it. Both twins now compare the value with its generator, and the twin reds that mutation. The shell needs root (runuser) and was not run here. - test_advisory_lock_class.py :205 [twin_mismatch]: the twin accepted one held tag where the shell wants all; with the holder taking only the first, main's twin passed 7/7, and now reds. :153 stays a premise: a session-scoped unique-key lock leaves it green in both twins, because the discoverer's backend ends. - test_catalog_delete_index.py :455 [twin_mismatch]: the twin-only premise "the vacuumed table holds rows" (a columnar count) is removed; "the vacuum walked this table's groups" holds it in both twins. - test_connection.py :44 [inaccurate]: also checks a table USING pgcolumnar is columnar. :111 [mislabelled]: with the upper-path hook planning no vectorized aggregate, "premise: the vector aggregate engaged" went red; it loses the prefix. :128, :202 [weak]: with one shared schema and a cluster root without the worker's slot, main passed 13/13; "the schema is test-private" now wants it empty and "its data directory was made for worker <id>" reads the root, and both red. Clean: index_fetch_penalty_width 9/9, crossover 8/8, index_am_support 24/24, advisory_lock_class 18/18 (shell) and their twins on each of 15-19; test_connection 13/13 and test_format_downgrade 2/2 (18 checks) on PG17. Census 4534 -> 4538 never, 3411 -> 3412 dated, 7950 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jdatcmd's review of #1680: the twin read idx_sum on one cursor and took the "really fetches by index" plan through _plan(), which forces the index path itself, so dropping custom_scan=off from the read still passed (1 passed). The read and its EXPLAIN now share one SET list, IDX_FORCE, on the same cursor, as the shell's IDX_FORCE feeds both. With that SET dropped from the list, the twin fails "and the index-forced path really fetches by index: got 'Custom Scan' want 'Index Scan'"; at head it passes 8/8 on PG17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…map (#1738) The cross-type ordering proc for (timestamp, timestamptz) reads a chunk group's min/max wall time as an instant in the session zone, and across a spring-forward gap that is not order-preserving: in New York on 2024-03-10 the nonexistent 02:30 is 07:30Z while 03:00 is 07:00Z. A group whose max was 03:00 was removed for `ts > '2024-03-10 07:15+00'::timestamptz` although its 02:30 row matched, and a group whose min was 02:30 for `ts < ...07:15+00` although its 03:00 row matched. pgcolumnar_make_predicates now drops that one pair; the reverse pair converts the constant once and the date pairs carry no zone, so both still skip. native_skip.sh and its twin gain fourteen checks, each against a heap built by the same generator in a New York session: the five operators across the gap (with a premise each that heap finds rows), the fall-back hour and the reverse pair as parity controls, and two skip controls. Main's reader fails the five gap arms on PG15-19 in shell and twin alike; restoring the cross-type proc for the pair fails the same five; refusing the reverse pair too fails only "the reverse pair, tz against a timestamp constant, still skips groups". native_skip has no ledger rows, so none are added. cluster_tests 2753 -> 2767. Fixes #1738. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t's own error (#1741) A parallel_export_parquet to s3:// cancelled again while the first cancel's cleanup was deleting its parts failed with "XX000: errstart was not called" instead of 57014. PG_ENSURE_ERROR_CLEANUP ran the cleanup with the first cancel's error still on the error stack; the second cancel raised inside the remote DELETE's wait, and the module's best-effort catch swallowed it with FlushErrorState(), which empties the whole stack. The re-throw then had nothing to report. PgColumnarCleanupAndRethrow (columnar_sink.c) copies the handled error out, empties the stack, runs the cleanup (dropping anything it raises), and re-throws the copy with ReThrowError. The export's guarded region spells out PG_ENSURE_ERROR_CLEANUP's shape and calls it from its PG_CATCH; the sink abort in the Parquet, Arrow and _SUCCESS writers goes through PgColumnarSinkAbortAndRethrow for the same reason. No HOLD_INTERRUPTS: the cleanup waits on the store, and holding interrupts would make a cancelled export to a dead endpoint unstoppable for the module's 60 s quiet limit per key. New suite objstore_export_double_cancel and its twin. The fixture server gains a __delete_delay__ hook, so the second cancel lands while a cleanup DELETE is provably outstanding. Red on main on PG15-19, shell and twin alike: the export reports XX000 and the server log gains one "errstart was not called"; green on PG15-19 with the fix. Census: never 4538 -> 4543, dated 3412 -> 3414, rows 7950 -> 7957; suites_not_covered stays 28 (a new suite, registered with its rows); cluster_tests 4214 -> 4221; TESTS.md §342. In the stack, the twin test test_the_sigv4_fixture_server_is_up collided with the one #1718 defines: it is now test_the_double_cancel_sigv4_fixture_server_is_up, in the twin and its TESTS.md row. No check name changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…what the workers pinned (#1745) With pgcolumnar.parallel_flush on, an error during the dispatcher's wait runs pflush_error_cleanup under PG_ENSURE_ERROR_CLEANUP: terminate the workers, wait for them, then unpin every output segment they published. The wait checks for interrupts, so a second cancel landing in it raised out of the cleanup before the unpin loop. The segments stayed pinned until postmaster restart, and the second cancel was reported in place of the first. Measured on main, PG15-19, shell and twin alike: a parallel flush cancelled by a 2s statement_timeout, then by pg_cancel_backend while one worker still encodes a zstd-19 text column, left 4 new segments in pg_dynshmem and reported "57014: canceling statement due to user request". A single cancel leaves none. The terminate-and-wait now runs under HOLD_INTERRUPTS. A terminated worker exits only at its next interrupt check, after the column it is encoding, so a single cancel already waits that long (measured: the single-cancel insert returned when the encoding worker finished); holding interrupts adds no latency, it only stops a second cancel from abandoning the cleanup. The unpin loop now frees any slot with a valid outHandle, whatever state the worker reached, and the worker records the handle before pinning: one that died in between left an unpinned segment that is already gone (dsm_attach returns NULL). parallel_flush_optin gains arm 4 and its twin: the suite's cluster runs dynamic_shared_memory_type = mmap so the segments are counted from SQL in its own pg_dynshmem; the twin builds a private cluster for the same settings. Two premises (a worker still encoding at the second cancel; outputs published by then) and two checks (the first cancel's error is reported; no new segment survives). Red on main on PG15-19 in both harnesses; green 9/9 with the fix; the cleanup without HOLD_INTERRUPTS reddens the two checks by name in both. Census: never 4543 -> 4545, dated 3414 -> 3416, rows 7957 -> 7961; suites_not_covered stays 28; cluster_tests 4221 -> 4225; TESTS.md §262 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e at its unpin (#1746 review, #1753) jdatcmd's review of #1746: the suite restarted its cluster with a bare pg_ctl, which refuses to run as root (CI's suites job), and `|| true` hid it, so the cluster kept posix segments and arm 4 counted 0 whatever the code did. It now restarts through pgc_pg and says so out loud when the restart fails. As root on PG17: main's code fails "a twice-cancelled parallel flush leaves no worker segment pinned" (4 new segments) and "reports the first cancel" (user request), and this head passes 10/10. #1753, folded in at jdatcmd's suggestion: the collect loop unpinned a DONE worker's output segment and kept its handle until after the detach, so an error in between sent pflush_error_cleanup to attach it again; "can't attach the same segment more than once" replaced the original error and left 3 segments pinned (an uncommitted, env-gated ERROR after the first unpin, PG17). The handle is now forgotten in the statement after the unpin; with the same injection the original error is reported and 0 segments are left. Arm 5 reads that statement from the source with comments blanked. The handle invalidated after the detach again reddens it alone in both twins and dates its row. Green on PG15-19, shell 10/10 and twin 10/10. Census: rows 7961 -> 7962, never 4545, dated 3416 -> 3417; cluster_tests 4225 -> 4226. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…he collect loop frees what it did not collect (#1747) A worker pinned its OUTPUT segment and only then ran table_close, PopActiveSnapshot and CommitTransactionCommand before writing PFLUSH_DONE. An ERROR in between reached its PG_CATCH, which wrote PFLUSH_FAILED; the dispatcher's collect loop attaches and unpins only DONE slots, and the error cleanup does not run on the normal path, so the segment stayed pinned to postmaster restart. Measured on PG17 main with an ERROR injected right after dsm_pin_segment (a measurement-only patch gated by an environment variable given to the test postmaster; not committed): a 300000-row, 8-column parallel flush logged the "worker N failed, completing its columns serially" WARNING for 4 workers on each of 2 stripes, returned the right rows, and left 8 new segments in pg_dynshmem. In production the window raises only through out-of-memory or another preloaded library's transaction callback: pgcolumnar's own pre-commit callbacks find nothing pending in the worker. The worker now builds its record in TopMemoryContext, commits, then creates, fills, names, pins and detaches the segment and writes DONE: nothing that can raise follows the pin. The collect loop also frees any slot it did not collect that names a segment. With the same injection placed after the new pin, the 8 FAILED workers leave 0 segments and the rows are right. parallel_flush_optin gains arm 6 and its twin: the worker's source with comments blanked must hold only the detach between dsm_pin_segment and the DONE write. Red on #1746's head in both harnesses; green 10/10 on PG15-19; the worker committing after the pin again reddens it by name in both. nologin_workers 15/15 and sorted_pathkeys 113/113 on PG17. Census: never 4545 -> 4545, dated 3417 -> 3418, rows 7962 -> 7963; cluster_tests 4226 -> 4227; TESTS.md §262 row; CHANGELOG Fixed entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…PG_TRY (#1748 review) jdatcmd's review of #1748: arm 6 (arm 5 before #1753 took that number) read from dsm_pin_segment to the DONE write, so a commit moved AFTER the DONE write ("commit at the end, as it used to be") passed it, though an ERROR there lands in PG_CATCH, which overwrites DONE with FAILED and leaks the pinned segment. The stretch now runs to the end of the PG_TRY block and must be "dsm_detach(outseg); pg_atomic_write_u32(&me->state, PFLUSH_DONE);" alone. Measured on PG17: with the commit moved after the DONE write, the suite as pushed passes 10/10, and this one fails the renamed check by name in both twins (9 + 1). The commit moved back before the DONE write (commitlast) still reddens it, so the renamed row keeps its date. Green on PG15-19, shell 10/10 and twin 10/10. Counts unchanged: rows 7963, never 4545, dated 3418; cluster_tests 4227. The review's first point (arm 4 red as root) is fixed in #1746's review commit, which this PR sits on. Folded onto #1746's review commit, whose #1753 check took arm 5, this arm is arm 6 and the suite has 11 checks: shell 11/11 and twin 11/11 on PG15-19, and 11/11 in both on PG17 run as root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jdatcmd

@OffgridwithJD Approving #1746 at e90e8ade.

Both points from my last review are fixed. Arm 4 now restarts through pgc_pg, so it measures as root. Arm 5 (#1753's handle reset) goes red on its own when the handle is forgotten late, in both harnesses. I reviewed only this PR's two commits (4ade3d3d, e90e8ade), against #1744's head 5743c3c0. All runs are on PG17, against my own prefix copy, as root, one run at a time. The twin ran serially (-n 0).

Runs

run	shell	twin
this head	`10 passed + 0 failed`	`10 pass + 0 fail + 0 unrun = 10`
base code (`5743c3c0`'s `src/`), this head's tests	`7 passed + 3 failed`	`7 pass + 3 fail`
`nohold`: `HOLD_INTERRUPTS`/`RESUME_INTERRUPTS` removed	`8 passed + 2 failed`	`8 pass + 2 fail`
`lateforget`: the handle forgotten after `dsm_detach`, as before	`9 passed + 1 failed`	`9 pass + 1 fail`

Blocking item from my last review: fixed

Arm 4 counted nothing as root, so the leak check passed on main's code. Now, as root, the base code fails it by name in both harnesses:

FAIL  a twice-cancelled parallel flush reports the first cancel: got [57014: canceling statement due to user request] want [57014: canceling statement due to statement timeout]
FAIL  a twice-cancelled parallel flush leaves no worker segment pinned: got [4 new segments] want [0 new segments]
FAIL  the collect loop forgets a worker's handle in the statement after it unpins the segment: got [obase = (char *) dsm_segment_address(outseg);] want [slots[i].outHandle = DSM_HANDLE_INVALID;]

The twin names the same three, and its leak reading is got = '4 new segments'. At this head the shell log has no RESTART FAILED line, and both premises pass. nohold still reddens the two dated arm-4 rows by name in both harnesses (4 new segments, user request).

Arm 5 (#1753) is red first

lateforget moves slots[i].outHandle = DSM_HANDLE_INVALID; back after dsm_detach(outseg);, which is the base's placement. The first nine checks (arms 1 to 4) pass it. Only arm 5 fails, by name, in both harnesses:

FAIL  the collect loop forgets a worker's handle in the statement after it unpins the segment: got [obase = (char *) dsm_segment_address(outseg);] want [slots[i].outHandle = DSM_HANDLE_INVALID;]
TWIN FAIL: the collect loop forgets a worker's handle in the statement after it unpins the segment

I also measured the behaviour behind arm 5. I injected an ereport(ERROR) for worker 0 right after the collect loop's unpin, only in my worktree (300000 rows, 8 columns, mmap cluster):

tree + injection	error reported	segments left
this head	`ERROR: injected collect`	0
this head, no injection	none; `300000 rows, content equal to the generator: true`	0
`lateforget`	`ERROR: can't attach the same segment more than once`	3

Registries and CI

compare_to_bash.py: literal matches: 10 | template matches: 0 | missing: 0.
The suite's 10 ledger rows match the shell's RESULT names and the twin's record names (diff is empty).
Counted from the ledger: 5743c3c0 has 7957 rows, 4543 never and 3414 dated. This head has 7962 rows, 4545 never and 3417 dated. 4545 + 3417 = 7962, and the budget line is checks_never_observed_red 4545. cluster_tests goes 4221 to 4226.
The new row's date is backed by lateforget above. The two arm-4 rows are backed by nohold.
The suite has no pgc_skip line. The pyarrow import scan prints [].
CI at e90e8ade: 15 check runs, 15 success.

Not blocking

e90e8ade changes a comment in #1741's entry of check_ledger_budget.txt from "suites_not_covered stays 28" to "stays 34". The value line in the same file is still suites_not_covered 28, so the comment is now wrong.
This branch's base is main, but it sits on #1744. GitHub lists 63 commits: your two, one already on main (2c7cf896, #1688), and 60 from the open PRs below it. All of those PRs show as approved, but merging #1746 merges all of them. Merge them in order, from the bottom.
My earlier note still holds: no test covers the cleanup freeing a slot that is not DONE. I did not re-run doneonly this time.

Logs: logs/1746-rr2/ (head-clean-*, basecode-headtest-*, mutant-nohold-*, mutant-lateforget-*, inject-*).

🤖 Generated with Claude Code

jdatcmd requested changes Oct 22, 2026

View reviewed changes

jdatcmd mentioned this pull request Oct 22, 2026

The parallel flush's error-cleanup callback can raise its own error, replacing the original and leaving segments pinned #1753

Closed

OffgridwithJD force-pushed the fix/1745-pflush-double-cancel branch from c5b22d1 to 81151ed Compare October 22, 2026 23:20

This was referenced Oct 22, 2026

Missing tests: 87 confirmed behaviours no shell or pytest arm exercises (#1339 audit) #1341

Closed

alpha5: clearing the board to release — who has what #1283

Open

jdatcmd and others added 21 commits October 23, 2026 03:15

jdatcmd and others added 25 commits October 23, 2026 05:35

OffgridwithJD force-pushed the fix/1745-pflush-double-cancel branch from 81151ed to e90e8ad Compare October 23, 2026 07:59

jdatcmd approved these changes Oct 23, 2026

View reviewed changes

jdatcmd merged commit 34eb209 into main Oct 23, 2026
15 checks passed

Navigation Menu

Repository navigation

fix: a second cancel during the parallel flush's error cleanup frees what the workers pinned (#1745) - #1746

fix: a second cancel during the parallel flush's error cleanup frees what the workers pinned (#1745)#1746
jdatcmd merged 63 commits into
mainfrom
fix/1745-pflush-double-cancel

OffgridwithJD commented Oct 22, 2026

Uh oh!

jdatcmd left a comment

Uh oh!

jdatcmd left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

OffgridwithJD commented Oct 22, 2026

The defect, measured

A correction to the issue

The fix (src/columnar_write_state.c)

Tests: parallel_flush_optin arm 4 and its twin

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

Blocking: arm 4's restart does not happen as root

The fix, measured outside the suite (PG17, mmap cluster, 4 workers)

Error-handling rules

Not covered by the test (non-blocking)

Other observations

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

Runs

Blocking item from my last review: fixed

Arm 5 (#1753) is red first

Registries and CI

Not blocking

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

The fix (`src/columnar_write_state.c`)

Tests: `parallel_flush_optin` arm 4 and its twin