fix: a cancelled remote parallel export aborts its workers' uploads (#1722) by OffgridwithJD · Pull Request #1727 · commandprompt/pgcolumnar

OffgridwithJD · 2026-10-22T11:49:13Z

@jdatcmd Fixes #1722. Stacked on #1721 (db178391), whose fixture lists open uploads, so this merges right after it. The diff against #1721 is one commit, 469540e2. Main is still 0c5e924d, so this waits for the freeze on #1341 like the rest of the stack.

The bug

When a cancel arrives, pexport_cleanup terminates each parallel_export worker, and SIGTERM ends it in FATAL. FATAL calls proc_exit without unwinding, so the writers' PG_CATCH never reaches PgColumnarSinkAbort. Each worker's multipart upload then stays open on the store, billed until a lifecycle rule reaps it. The dispatcher's DELETEs only remove plain keys.

Red first, on #1721's code

The new check, cancel: the server holds no open upload under the prefix, is in both the shell suite and the twin.

major	shell	twin
15–19	FAIL `got [4] want [0]`	FAIL `got '4' want '0'`

The fix

src/columnar_sink.c: remote sinks with an upload in progress go on a list.
- A before_shmem_exit callback aborts any sink still on the list. It is registered on the first remote open, after InitPostgres's ShutdownPostgres, so it runs first, while the transaction and the module's handles are still intact.
- Finish unlists a sink only after the complete succeeds, so a complete that raised still goes through the writer's abort. Abort unlists it too, and transaction end empties the list.
- Under proc_exit, InterruptHoldoffCount is 1, so the abort's waits don't check for interrupts. Each wait is bounded by the module's own 60 s quiet limit (OS_MAX_QUIET_SLICES).
- Under proc_exit, an ERROR is promoted to FATAL, which the module's PG_TRY can't catch. A failed DELETE therefore ends this callback with one more FATAL line, and the remaining callbacks still run, because shmem_exit has already popped this one.
objstore/columnar_objstore_module.c: objstore_sink_abort now reconnects before its DELETE. The SIGTERM can land mid-request, with a part's body still on the connection.
- In one failing run, the request log shows worker 0's PUT partNumber=2 completing after the dispatcher's DELETEs, and no DELETE ?uploadId= for its upload ever arriving.
- Over the twin's cancel arm: 4 of 20 runs left one upload open without the reconnect, 0 of 15 with it.
- Because the failure is intermittent, this mutant dates nothing.

Green and mutants

Head: 36/36 on PG15–19 in both the shell suite and the twin. The build is clean under -Werror, for the extension on 17 and the module on 16.
noreg (the before_shmem_exit registration dropped): the new check fails by name in both harnesses on 15, 17 and 19 (got [4] want [0]). This mutant dates the row.
nodisc (the reconnect dropped): intermittent, as above.

Not fixed here

If the SIGTERM lands while a worker's POST ?uploads is in flight, the store creates an upload whose ID the client never learned. Nothing on the client side can abort it, short of the dispatcher listing ?uploads&prefix=, which needs new module API. It never occurred in these runs.

Census

never 3907 → 3907, dated 2293 → 2294, rows 6200 → 6201
cluster_tests 2783 → 2784
suites_not_covered unchanged at 101
One TESTS.md row added in test: objstore_sink_write gains its pytest twin, and multipart is read as S3 does it (#1341) #1721's §268
Gate: guards 476 passed, collection 2784, census agrees, ledger gate rc=0, self-test passed

🤖 Generated with Claude Code

jdatcmd

@OffgridwithJD Requesting changes on #1727 at 68d8b0c0.

The fix closes #1722 as reported. The red, the green and the noreg mutant all reproduce. One claim in the PR does not hold, though: "Each wait is bounded by the module's own 60 s quiet limit." No such bound exists. With a silent store, a worker that SIGTERM ended now hangs in its exit callback until the store answers. Before this PR it exited in about 1 s.

All runs are on PG17, against my own prefix copy, as root with the server running as postgres. "Parent code" means src/columnar_sink.c and objstore/columnar_objstore_module.c checked out from 68d8b0c0~1 (#1721's head), with this PR's tests kept.

Runs

run	result
PR shell, clean	36 passed + 0 failed + 0 unrunnable + 0 skipped = 36
PR twin, clean, serial (as CI runs it)	36 pass + 0 fail + 0 unrun = 36
parent code, shell	`FAIL cancel: the server holds no open upload under the prefix: got [4] want [0]`, 35 + 1
parent code, twin	fails exactly that name: `got '4' want '0'`, 35 + 1
`noreg` (the `before_shmem_exit` call removed), shell	`FAIL cancel: the server holds no open upload under the prefix: got [4] want [0]`
`noreg`, twin	fails exactly `cancel: the server holds no open upload under the prefix`
`parallel_export_parquet.sh` (local target, includes the local cancel arm)	54 passed + 0 failed

compare_to_bash.py: 36 literal matches. The 2 missing are the optional Garage integration: checks, and they are missing on the parent too. The ledger has 7467 rows: 4350 never, 3117 dated. That matches check_ledger_budget.txt. The pyarrow import scan prints [].

Defect: a terminated worker hangs at exit while the store is silent

Measured. I started a 4-worker export to s3://, waited for a part PUT, stopped the fixture server with SIGSTOP, and ran pg_terminate_backend on the 4 workers:

code	workers after SIGTERM
parent	`workers gone after 1 s`
PR head	all 4 still alive at +92 s; they exited only after SIGCONT
PR head + the fix below	`workers gone after 60 s`, each with `FATAL: columnar: timeout reading "s3://pgc-bucket/h1/part-000N.parquet"`

Here is a stack from one hung worker on the head (all 4 match):

#5  WaitLatchOrSocket (latch=..., wakeEvents=wakeEvents@entry=35, sock=7, timeout=-1, timeout@entry=5000, ...)
#6  os_wait (...) at columnar_objstore_module.c:248
#9  os_write_request (..., method="DELETE", rawQuery="uploadId=up%2F51%2Bx", ...)
#10 objstore_sink_abort (...) at columnar_objstore_module.c:2181
#11 sink_abort_open_at_exit (...) at src/columnar_sink.c:74
#12 shmem_exit (code=code@entry=1) at ipc.c:250

Why. os_wait passes WL_LATCH_SET | WL_EXIT_ON_PM_DEATH | sockEvents with no WL_TIMEOUT. WaitLatchOrSocket therefore turns the 5000 ms into timeout=-1 (frame #5 above, wakeEvents=35 = 1|2|32). The wait never times out, so quiet advances only when the latch wakes the process. In normal running, a cancel still gets through CHECK_FOR_INTERRUPTS. Under proc_exit, InterruptHoldoffCount is 1, as you note, so nothing breaks the wait. A network partition that drops packets without a reset behaves like this. The worker keeps its bgworker slot, and SIGTERM cannot end it.

Before this PR, the 60 s "timeout reading" error was also dead code on every read path. That part is not yours, but this PR is the first code to depend on the bound.

Suggested fix. Add WL_TIMEOUT in os_wait:

rc = WaitLatchOrSocket(MyLatch,
                       WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH | sockEvents,
                       h->fd, OS_WAIT_SLICE_MS, PG_WAIT_EXTENSION);

With it, the exit is bounded at 60 s (table above). objstore_sink_write.sh (36/36), objstore_http_read.sh (24/24) and objstore_s3_read.sh (17/17) still pass. A check would pin it: terminate the workers with the fixture stopped, and require them gone within a bound.

pg_ctl stop -m fast with a silent store failed to stop within 40 s on both the parent and the head. On the parent, the dispatcher's pexport_remove_outputs causes this. It is not a regression from this PR, but the same fix would bound it too.

Not blocking

The window you named is real, and a slow store opens it. The fixture copy I used sleeps 3 s after it creates an upload and before it answers POST ?uploads. Cancelled during that sleep, the head leaves open uploads under st1/: 4, with DELETE uploadId: 0. A slow CreateMultipartUpload is enough to cause this. #1722's option (b), where the dispatcher lists ?uploads&prefix= and aborts each, would cover it and SIGKILL as well. Please open a follow-up issue rather than leave it only in the PR body.
The other ways to end the run pass on the head. I measured each on the parent code and on the head (open uploads after it):
- terminating the dispatcher: 4 on the parent, 0 on the head;
- terminating a single-backend export_parquet to s3://: 1 on the parent, 0 on the head;
- eleven cancels 20 ms apart: 4 on the parent, 0 on the head.
The server log shows no crash, no TRAP and no "leaked to commit" warning.
Already on the parent: a second cancel turns the error into ERROR: errstart was not called. It happens on both the parent and the head, 4 times in 4 runs on the head. The likely cause is the FlushErrorState() in the module's best-effort delete (line 2672). It runs inside the dispatcher's error cleanup and empties the outer error's stack. This deserves its own issue.
#1721's twin is unsafe under pytest -n 2. Under the default load distribution, both xdist workers build the module fixture. Both call pick_port(30) and pick the same port. Three runs out of three failed (premise: sigv4 fixture server is up: got '0' want '1', or premise: the large export took the multipart protocol: got 'no' want 'yes'). A private network namespace changed nothing. With --dist loadfile the result is 36/36. CI runs it serially, so CI does not see this.
The PR body is stale. It names the commit 469540e2 and gives a census of 3907 / 2293→2294 / 6200→6201 and cluster_tests 2783→2784. The head commit is 68d8b0c0, and its files say 4350 / 3116→3117 / 7466→7467 and 3755→3756.

Logs: logs/1727/ (red-parentcode-*.log, mutant-noreg-pr-*.log, hung-store-*.log, break-head.log, break-parentcode.log, clean-*.log).

🤖 Generated with Claude Code

jdatcmd · 2026-10-22T15:39:19Z

One more data point for my review above: suites (PG 17) on #1723's head, which sits above this PR in the stack, reddened this PR's new check, cancel: the server holds no open upload under the prefix: got [1] want [0]. That fits the window the review measured: an upload whose POST ?uploads is still in flight when the cancel lands isn't registered yet, so the exit callback can't abort it. So the window shows up on a CI runner too, not only under a deliberately slow store. Closing it, with the dispatcher listing the prefix (#1722's option b), would also make this check deterministic.

OffgridwithJD · 2026-10-22T15:41:36Z

@jdatcmd Thanks, both points taken. The fix is being prepared on this PR's head, and it goes in when the stack restate reaches it:

Bounded exit: WL_TIMEOUT in os_wait, as you suggested, plus a check that SIGTERMed workers with a stopped (SIGSTOP) store exit within the 60 s bound. It's red on the current head.
The CI flake: the cancel arm waits until the fixture's __open_uploads__ shows every worker's upload registered, then cancels. A premise with its own name covers not getting there. The check then pins what this PR fixes, not the in-flight window, measured 10× for flakes.
The in-flight window is now A cancel during CreateMultipartUpload still leaves an upload open: the dispatcher should list and abort uploads under the export prefix #1743 (option b, the dispatcher lists and aborts under the prefix), with its own red-first arm.
The double cancel is A second cancel of a parallel_export_parquet to s3:// turns the dispatcher's error into "errstart was not called" #1741, which I've claimed.
test: objstore_sink_write gains its pytest twin, and multipart is read as S3 does it (#1341) #1721's -n 2 port race is fixed in the same batch, and the PR body's numbers will be restated.

🤖 Generated with Claude Code

…ew, item 4) jdatcmd's #1727 review: under `pytest -n 2` (load distribution) both xdist workers build this module's fixture, and both called pick_port(30) and pick_port(31), walking from the same start to the same free port before either bound it. The fixture server or the private cluster of one worker then failed to come up. Both ports now start from a slot offset by the worker's index (base + 100 * n, the derivation make_cluster uses), so two workers walk from ports 200 apart. PG17, `pytest -n 2 test_objstore_sink_write.py`: - before: 3 runs of 3 red, 33 pass + 2 fail each ("premise: sigv4 fixture server is up" plus a multipart check); - after: 3 runs of 3 green, 35 pass + 0 fail; serial 35/35. No check name, ledger row or count changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…old a terminated export worker (#1727 review) jdatcmd's #1727 review: os_wait passed no WL_TIMEOUT, so WaitLatchOrSocket ignored the 5 s slice (timeout=-1) and a silent store never counted a quiet slice; the 60 s quiet limit was dead code. Under proc_exit interrupts are held, so a parallel_export worker that SIGTERM ended stayed in its exit-callback upload abort until the store answered. os_wait now passes WL_TIMEOUT. objstore_sink_write gains "terminate: export workers exit within 75 s while the store is silent": SIGSTOP the fixture mid-upload, terminate the workers, time their exit. - red first on #1727's head, PG17, both twins: "got [slow (4 workers, 4 left after 91s)]" (twin: 90s), which dates the row; - with the fix: 61 s on each of PG15-19, 38/38 in both twins. pg_ctl stop -m fast with the store silent mid-export: 60 s with the fix; without it the stop did not finish within 150 s (rc=1). The coordinator's addition: jdatcmd saw "cancel: the server holds no open upload under the prefix: got [1] want [0]" flake in CI on #1723's head, an upload whose POST ?uploads was still in flight when the cancel landed (#1743's window). The cancel arm now cancels only once the fixture's __open_uploads__ shows all four workers' uploads, with its own check ("all four workers' uploads were open before the cancel"), so the open-upload check pins exactly what #1722 fixes. That check is not a premise: nomultipart (the sink's flush loop never runs, so no export opens a multipart upload; one line in columnar_objstore_module.c) reddens it by name in both twins on PG17 ("got [no (0 open)] want [yes]"; the same 11 names red in shell and twin), which dates it. PG17 at the fix: 10 runs of shell and twin, 0 flakes (38/38 each). noreg (the before_shmem_exit registration dropped) still reddens "cancel: the server holds no open upload under the prefix: got [4] want [0]" in both twins. Census: +2 rows, never 4350 -> 4350, dated 3117 -> 3119; cluster_tests 3756 -> 3758; two TESTS.md rows in objstore_sink_write's section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… from the table under test no longer vouches for it (#1341) Ports test/native_fold_deferral.sh to test/pytest/test_native_fold_deferral.py, all 13 checks by the shell's name, one test per check, on a private database. Three holes in the shell, each measured red-first on PG17 against main and fixed in both harnesses: - The heap mirror was `CREATE TABLE fd_heap AS SELECT * FROM fd_t`, read through the scan under test. With the writer storing one p1 value off by one (writecorrupt), the fold, the row path and the mirror all agreed: main passed 13/13. The mirror is now written from the generator, and writecorrupt reddens both heap-mirror checks in both twins. - The answers were read with `psql ... | tail -1` and compared with check. With the server stopped before the correctness section, all three answer checks passed on empty strings. Readings now run with ON_ERROR_STOP and come back empty on failure; the checks are check_text, and the counts go to check_num raw instead of defaulting to 0. Stopped server: only the three checks read before the stop pass. - Three premises are reddened by a one-line pgcolumnar change: the fold's ANALYZE line (foldno), the EXPLAIN group total (badtotal), the writer's stripe limit (nostripe). They lost the prefix (#1339). Eight mutations on PG17 each redden the same names in shell and twin: nodefer 3, deferall 1, nocursor 3, dropsurv 4, writecorrupt 2, foldno 1, badtotal 2, nostripe 1. Head is 13/13 in both on each of 15-19. Seeded: 13 rows, 9 dated, 4 never. Census 4192 -> 4196 never, 2952 -> 2961 dated, 7157 rows; suites_not_covered 69 -> 68; cluster_tests 3416 -> 3429; TESTS.md §301. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d through the reader no longer vouches for it (#1341) Ports test/native_lazy_slot.sh to test/pytest/test_native_lazy_slot.py, all 10 checks by the shell's name, one test per check. Two holes in the shell, each measured red-first on main's shell on PG17 and fixed in both harnesses: - The heap mirror was a copy of lz_c (INSERT INTO lz_h SELECT * FROM lz_c), read through the reader under test. With the writer storing c40 off by one (writec40), main passed 10/10. It is now written from the generator, and writec40 reddens 4 checks. - Five comparisons read `q ... | tail -1` and compared with check, so two failed reads agreed, and a timing whose statement failed read as fast. With the server stopped after setup, 6 of 10 passed on main. A new qv helper names the side in its failure, the comparisons use check_text, and a failed timing is "no reading": all 10 now fail. Mutants on PG17, the same failing checks by name in shell and twin: writec40 4, stale (the fetch no longer clears the slot) 2, eager (decode the whole row) 1, short (one column too few) 6. An earlier stale mutant that kept only PgColumnarSlotStoreDeferred's clear was inert, because the fetch clears the slot first; the faithful one drops both. Head is 10/10 in both on PG15-19. Seeded: 10 rows, 7 dated, 3 never. Census 4196 -> 4199 never, 2961 -> 2968 dated, 7167 rows; suites_not_covered 68 -> 67; cluster_tests 3429 -> 3439; TESTS.md §302. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… counted nothing no longer passes (#1341) Ports test/native_metadata_flush.sh to test/pytest/test_native_metadata_flush.py, every check by the shell's name, one test per check (#1341). Four holes, each measured on PG17 against main's shell, which passed all six checks under each: - A witness that counted nothing passed. With the flush session never counting an open (zerocount, opens=0 for every flush), "narrow and wide open the same number" (0 = 0) and "a small constant" (0 <= 6) both passed. A new check wants at least the three tables every flush writes; zerocount now reddens it. - The heap mirror was copied from the columnar table, so a writer that stored 1777 as 1778 (writecorrupt) agreed with itself. The mirror is now written from the generator, and the sum check reddens: got 40020020, want 40020000. - Nothing showed the zone rows were read. The reader rules vectors out by value after decoding, with or without a usable zone map, so Vectors Skipped alone cannot say. A new check wants a vector ruled out by its zone row alone (Skipped minus Ruled Out by Value); zone rows with no min/max (zonenull) redden it. - "premise: the flush emits the metadata-open witness" reads pgcolumnar's own behaviour: a flush that never begins its session (nosession) reddens it. It lost the prefix (#1339). Mutants on PG17, the same names red in shell and twin: noreuse (#445 reverted) 2, zerocount 1, writecorrupt 1, zonenull 1, nosession 4. Head is 8/8 in both on each of 15-19. Seeded: 8 rows, 6 dated, 2 never (the row count and the range read against the mirror). Census 4199 -> 4201 never, 2968 -> 2974 dated, 7175 rows; suites_not_covered 67 -> 66; cluster_tests 3439 -> 3447; TESTS.md §303. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d counter no longer reads as 0 (#1341) Every counter in the shell was read through `${x:-0}`, and four checks want exactly 0. With the server stopped before the feature-off arm, "with the feature off nothing is filtered early" and "a volatile qual is refused the two-pass path" passed on main (PG17). The counters are always printed, so a missing line is a failed reading: counter() now returns "no reading (...)" when the EXPLAIN fails or prints no such line, the feature-off readings run under ON_ERROR_STOP, and check_num refuses them. With the server stopped both checks now fail. Four premises a one-line pgcolumnar change reddens lost the prefix (#1339): "the query runs on the columnar custom scan" (noscan), the two "nothing was pruned" checks (noscan: their counters are not printed), and "the same shape without the volatile term does use it" (nocount, noscan, droprow). The twin test_native_late_materialization.py makes every check by the shell's name, one per test, on a private database with serial plans (a parallel plan prints per-worker averages). Every reading is a value: a failed query is a query_error(), an unprinted counter is "no reading", never 0. Measured on PG17, the same failing checks by name in shell and twin: nocount 2, noscan 7, droprow 3, skipkeep 1, nocursor 1, novolguard 1. matall (a rejected row's payload built anyway, still counted) reddens nothing: no counter reports per-row materialization, and the answers are the same by design. Head is 14/14 in both on each of 15-19. Seeded: 14 rows, 8 dated, 6 never. Census 4201 -> 4207 never, 2974 -> 2982 dated, 7189 rows; suites_not_covered 66 -> 65; cluster_tests 3447 -> 3461; TESTS.md §304. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nd a writer mutant dates four rows (#1691 review) jdatcmd's #1691 review: three premises read the columnar table n, and dropseven (pgcolumnar_tuple_insert drops the row whose id is 7, one line) reddened all three. The two fixture premises ("the fixture loaded every row", "the needle is selective") now read the heap mirror h, which pgcolumnar does not write, so no pgcolumnar change reddens them. "heap agrees on how many rows match" compares n with h and is not a premise: the prefix is dropped and the row is renamed, not removed. dropseven on PG17, shell and twin: the same 7 names red, the two fixture premises green (7 pass + 7 fail = 14 in both). It dates four rows that were never: "heap agrees on how many rows match", "the executor still reports the rows the scan filtered", "and the answers are the same with it off" and "and the volatile query still matches the heap mirror". Clean: 14/14 in both on PG17. Census for this suite: 14 rows, 12 dated + 2 never (was 8 + 6): never -4, dated +4. The header's "nothing pruned at the vector level" premise is a check since #1691. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… that drops every free row no longer passes (#1341) Every reconcile check counted only the injected stale free_space row, so a reconcile that dropped EVERY free row, leaking all reusable space, passed all 11 checks on main (overreconcile). A plain compact, which does not reuse, must now keep the legitimate free ranges the earlier compact_rewrite left (count and bytes), with a premise that one exists. And "row count intact" compared two readings with plain check: with the server stopped before it, main passed it on two empty strings. It is check_num now. The reuse-cycle loop is unrolled so its two check names are literals the grader reads. New twin test/pytest/test_native_reclaim_reconcile.py, one check per test under the shell's names, on a private database; readings are values (query_error()), row sets compared as an md5 over sorted rows. Mutants on PG17, same names red in shell and twin: noreconcile 4, overreconcile 1. 13/13 shell and twin on each of 15-19. Seeded: 13 rows, 5 dated, 8 never. Census 4203 -> 4211 never, 2986 -> 2991 dated, 7202 rows; suites_not_covered 65 -> 64; cluster_tests 3461 -> 3474; TESTS.md §305. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nd nofree dates it (#1692 review) jdatcmd's #1692 review: "premise: a legitimate free range exists before the plain compact" reads pgcolumnar's own free list, which holds ranges only because compact_rewrite recorded them. nofree (record_free_space drops its insert, one line in columnar_metadata.c) leaks every freed range and reddened only that premise, so a real leak read as "the fixture was wrong". It is renamed "compact_rewrite recorded the free ranges it left" (the row is kept, renamed) and dated by nofree. nofree on PG17: shell "FAIL compact_rewrite recorded the free ranges it left: got [no (0|0)] want [yes]", 12 + 1; the twin fails the same name, 12 pass + 1 fail. Clean: 13/13 in both. The next check still compares against a non-empty legit_before. Census for this suite: 13 rows, 6 dated + 7 never (was 5 + 8): never -1, dated +1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jdatcmd

@OffgridwithJD Approving #1727 at 4dc63aae.

My blocking finding is fixed. A terminated export worker no longer hangs while the store is silent: it exits in 61 s. The new check fails by name, in the shell and the twin, when WL_TIMEOUT is removed. I reviewed the PR's own two commits (98ca1a74, 4dc63aae), not the stack beneath.

All runs are on PG17, against my own prefix copy, as root with the server running as postgres.

The blocking finding is fixed

os_wait now passes WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH | sockEvents. To show the new check pins it, I removed WL_TIMEOUT again at the head (noWLT):

run	result
shell, clean	`store silent: 4 workers terminated, 0 left after 61s`; `38 passed + 0 failed`
shell, `noWLT`	`store silent: 4 workers terminated, 4 left after 91s` and `FAIL terminate: export workers exit within 75 s while the store is silent: got [slow (4 workers, 4 left after 91s)] want [bounded]`; `37 passed + 1 failed`
twin, clean (`-n 2`)	`38 pass + 0 fail + 0 unrun = 38`, `38 passed in 86.14s`
twin, `noWLT`	`37 pass + 1 fail + 0 unrun = 38`; it fails exactly `terminate: export workers exit within 75 s while the store is silent`

So the bound I asked for exists, and the row's date and mutant text are backed by a run.

The other checks

mutant	shell	twin
`noreg`: the `before_shmem_exit(sink_abort_open_at_exit, …)` call removed	`FAIL cancel: the server holds no open upload under the prefix: got [4] want [0]`, 37 + 1	the same one name, 37 + 1
`nomultipart`: `while (false && s->buf.len >= s->partSize)` in `objstore_sink_write`	11 red, including `FAIL all four workers' uploads were open before the cancel: got [no (0 open)] want [yes]`	the same 11 names, 27 + 11

So noreg still dates the #1722 check. nomultipart dates the new "all four uploads were open" check, which is correctly not a premise.

Registries

The suite's 38 ledger rows match the shell's 38 RESULT names (diff is empty).
Counted from the file at the head: 7474 rows, 4332 never, 3142 dated. 4332 + 3142 = 7474. The budget line is checks_never_observed_red 4332, and cluster_tests 3763.
compare_to_bash.py: literal matches: 38. The 2 missing are the optional Garage integration: checks, as before.
objstore_sink_write.sh has no pgc_skip line. The pyarrow import scan prints [].

How this composes with #1755

#1755 (mine, being reworked) also changes os_wait. It adds WL_TIMEOUT only when a handle has a deadline, and cuts the wait to the time left. The two do not merge cleanly: git merge-tree prs/1727 prs/1755 reports CONFLICT (content): Merge conflict in objstore/columnar_objstore_module.c, in the WaitLatchOrSocket call. Other conflicts are in the ledger, budget, TESTS.md, expected_tests.txt and objstore_http_server.py.

The two changes do not contradict each other. With this PR's unconditional WL_TIMEOUT, #1755's events |= WL_TIMEOUT becomes redundant, and its timeout = Min(timeout, left) and deadline checks still apply. #1755 should rebase onto this PR and drop its own WL_TIMEOUT line. I have not built or run that merged version. That part is unverified.

Not blocking

CI is not green at this head. pytest (cluster tests, PG 17) and PG 18 fail 8 tests in test_objstore_stash_recovery.py (for example the suite runs to completion with debris beside a valid module: got 1 want 0). That starts at #1717 (f2a1bf8c), beneath this PR: #1716's PG17 job passed with 0 fail, and #1717's failed these 8. suites (PG 17) and PG 18 pass here.
Under noreg, the terminate check passes in 1 s (0 left after 1s), because no abort runs at exit. That is right: the cancel check catches noreg. But it means the terminate check proves only the bound, not that the abort ran.
The bound is 75 s against a measured 61 s. That leaves 14 s for a slow CI runner.
expected_tests.txt calls the new cancel check a "premise" ("gains the all-uploads-open premise"). The check has no premise: label and should not have one. The comment should say "check".

Logs: logs/1727-rr/ (clean-*.log, mutant-noWLT-*, mutant-noreg-*, mutant-nomultipart-*, merge-1727-1755-module.c, ci-*.log).

🤖 Generated with Claude Code

…longer passes as refused (#1341) Ports test/native_page_offset_bound.sh to test/pytest/test_native_page_offset_bound.py, every check by the shell's name, one test per check. Three holes in the shell, each measured red-first on PG17 against main and fixed in both harnesses: - Every refusal arm counted any psql failure as refused. With the whole-group containment guard removed (noguardwhole) the backend segfaulted and the arms still passed; with the projected guard removed (noguardproj) a later XX001 ("row group 1 is inconsistent with its column chunks") passed the projected overrun arm. Each arm now wants XX001 with the guard's own message, "lies outside row group". - Survival was a bare SELECT 1, which answers again once crash recovery has finished. With the probe 4s late, main passed both whole-group survival arms with the backend killed. It now also reads the server log's count of backends killed by a signal since the arm began. - Nothing read an intact table, and every arm wants a refusal. A guard over-applied to refuse every chunk (overall), or a legal chunk ending exactly at its group end (offby1), passed all 10 checks on main. Two controls now read rows behind an OFFSET 0 fence on each path; an aggregate over an intact table is answered without decoding a chunk, which is why the first version of these controls stayed green. Mutants on PG17, same failing checks by name in shell and twin: noguardwhole 6, noguardproj 2, offby1 1, overall 2. Without the whole-group guard the read either segfaults or fails decoding, depending on where it lands (three runs: shell 4-6, twin 6); its three refusal checks are red every time. 12/12 shell and twin on each of 15-19. Seeded: 12 rows, 10 dated, 2 never (the projected survival arms). Census 4210 -> 4212 never, 2992 -> 3002 dated, 7214 rows; suites_not_covered 64 -> 63; cluster_tests 3474 -> 3486; TESTS.md §306. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s left open (#1743) A worker of parallel_export_parquet learns its multipart upload id only from the answer to CreateMultipartUpload (POST ?uploads), and the store creates the upload when the request arrives. A worker that ended while it waited for that answer left an upload open that nothing could abort. #1727's exit callback (in this base) aborts an upload a terminated worker holds the id of; it cannot help a worker that never got the id. The dispatcher's cleanup now lists the uploads under the export's prefix (ListMultipartUploads) and aborts each one whose key is exactly one of the export's part names, part-0000.parquet to part-(nkeys-1).parquet. Any other key under the prefix is left alone. - On a cancel or other error the aborts run inside #1744's PgColumnarCleanupAndRethrow, in the cleanup callback after pexport_cleanup. The helper has copied the handled error out and emptied the stack, so a second cancel that lands in a part DELETE cuts short only that DELETE: the aborts still run and the first cancel is re-thrown. They also run after a worker failed (for example, the workers were terminated). - Interrupts are held only around bounded waits: 5 s on a timed latch for the terminated workers to stop, then 10 s for the listing and aborts. - New module entries list_uploads and abort_upload, which take a timeout that sets a deadline on the handle; ABI 5 -> 6. - os_wait: this builds on #1727's WL_TIMEOUT (4dc63aa, in this base through #1744's stack, not on main). It keeps WL_TIMEOUT for every wait and cuts the slice to the time left when a handle has a deadline. The first version's comment "os_wait passes no WL_TIMEOUT" is gone; this hunk depends on #1727 landing. - The fixture server already wrote __open_uploads__ (#1721) and had __delete_delay__ (#1744). One copy of each is kept; the __open_uploads__ writer is the module function _write_open_uploads, which --seed-upload also uses. Review of the first version (80ee33e): a second cancel during pexport_remove_outputs' part DELETEs emptied the error stack (#1741), the first version's outer PG_CATCH then raised "errstart was not called", and the aborts never ran. That outer PG_TRY, its copy-flush-rethrow helper and pexport_spawn_and_wait are removed. Test: new suite objstore_export_open_uploads and its twin, 30 checks each, identical names. The fixture server gains ListMultipartUploads, delay hooks on POST ?uploads, part PUTs and the listing, and --seed-upload. Five arms: cancel while parts are sent; cancel while every CreateMultipartUpload is held; the workers terminated there; cancel there, then cancel again once the first part DELETE has arrived (__delete_delay__ 3 s); and cancel there while the listing is held. The double-cancel arm was red first on the rebased first version, PG17, shell and twin alike: "XX000: errstart was not called", one more errstart line in the server log, and 4 uploads left open. With the rework: 30/30 on PG15-19, shell and twin. Ledger: 30 rows, 12 dated by four mutations that redden the same names in both twins on PG17 (no listing; every listed key aborted; the part match off by one; the cleanup run with the handled error on the stack), 18 never. "parts: no upload of this export is left open" is never: on this base #1727 also closes it, so no single mutation reddens it. never 4488 -> 4506, dated 3260 -> 3272, rows 7748 -> 7778; suites_not_covered stays 34; cluster_tests 4027 -> 4057; TESTS.md section 338. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lated param no longer passes (#1341) Ports test/native_param_pushdown.sh to test/pytest/test_native_param_pushdown.py, every check by the shell's name, one test per check, and closes three holes in the shell, each measured on PG17 against main and fixed in both harnesses. - The PARAM_EXEC guard was untested. With it removed (freezeexec) main passed 5/5: Begin freezes before any outer row sets the param, so the frozen value is its initial one and "ts >= <that>" prunes nothing; the lower-bound correlated answer stayed 3002. The arm now reads the scan's Usable Skip Predicates (must be 0), asks the upper bound "ts <= thr.lo" (77000), where the frozen value prunes every group, and checks the plan rescans the columnar scan per outer row. - A value frozen once and reused by a later execution (staleparam) passed 5/5: every arm ran one execution per session. Two executions of one generic plan in one session now each count (2001) and prune (18 of 20) with their own value. - Survival was a bare SELECT 1: with a backend SIGKILLed before it, main passed. It now also requires the server log's crash count to be unchanged. Mutants on PG17, same failing checks by name in shell and twin: shiftparam 4, noparam 2, freezeexec 2, staleparam 2. The harness kill reds "backend alive" in both. 10/10 shell and twin on each of 15-19. Seeded: 10 rows, 6 dated, 4 never. Census 4212 -> 4216 never, 3002 -> 3008 dated, 7224 rows; suites_not_covered 63 -> 62; cluster_tests 3486 -> 3496; TESTS.md §307. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…wrong order no longer passes (#1341) Ports test/native_sort_by.sh to test/pytest/test_native_sort_by.py, every check by the shell's name, one test per check, and fixes two holes in the shell, each measured red-first against main's shell on PG17 (PG18 for the virtual arm): - The physical order was never read. Group skipping on host was the only evidence of a sort, so a rewrite that applied only the first key (firstonly: ts left unordered) passed 16/16. The rows are now read in scan order and every adjacent pair is compared on (host, ts) ("the scan order follows the declared key (host, ts)"), behind a premise that the inserted order is not already sorted, and the explicit-column sort must lay the rows out as the declared key does. - Every refusal counted any psql failure. A backend aborting in place of the no-key refusal (keyabort) passed its own check; a crash in place of the virtual-column refusal (virtabort) passed 18/18 on PG18; a dropped declared column given the generic message (persgeneric) passed 16/16. Each refusal now wants its guard's SQLSTATE and a phrase of its message, every reading after a refusal waits for the server, and "backend still up after the refusals" reads the server log's crash count. The empty and single-row sorts read psql's exit status. Mutants on PG17, the same checks red by name in shell and twin: firstonly 1, persgeneric 1, keyabort 2, nosort 2, wrongcol 4, desc 4; virtabort on PG18 2. Clean: shell 20/20 (+1 skip) on 15-17 and 22/22 on 18-19, twin 22/22 on each. Before PG18 the virtual arm is recorded as a skip, and its two twin tests decline (expected_unrunnable.txt). 23 rows, 9 dated, 14 never. Census 4216 -> 4230 never, 3008 -> 3017 dated, 7247 rows; suites_not_covered 62 -> 61; cluster_tests 3496 -> 3518; TESTS.md §308. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e index stale no longer passes (#1341) Ports test/native_rewrite.sh to test/pytest/test_native_rewrite.py, every check by the shell's name, one test per check, and fixes four holes in the shell, each measured on PG17 against main and fixed in both harnesses: - The threshold was untested from both sides. With every group that has a live row rewritten (overrewrite), or min_deleted_fraction ignored (nothreshold), main passed 17/17. Two new checks: "a second rewrite with nothing deleted rewrites no group" and "a group deleted below the threshold is not rewritten". - vacuum's index rebuild was untested. With pgcolumnar_compact_relation's reindex skipped (noreindex), main passed 17/17: over the whole table the stale entries land on every live row once, so a count and a hash through the index both still matched. A narrow index range read does not (id 4097 read back as 6146). New check: "an index range read after vacuum returns the rows asked for". - Four counts compared two readings with plain check, and "unique constraint still enforced" took any psql failure as enforcement. With the server stopped after the rewrite, four checks passed on main. Counts are check_num, and the refusal wants 23505 and n_id_uq's own message. Mutants on PG17, same failing names in shell and twin: droprow 6, noindex 5, overrewrite 2, nothreshold 1, noreindex 1. With the server stopped after the rewrite, both harnesses fail 15 and pass the 5 read before it. 20/20 shell and twin on each of 15-19. Seeded: 20 rows, 9 dated, 11 never. Census 4230 -> 4241 never, 3017 -> 3026 dated, 7267 rows; suites_not_covered 61 -> 60; cluster_tests 3518 -> 3538; TESTS.md §309. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ector's first value no longer passes (#1341) The suite checked each zone-map bound for presence and width only, and every column of its fixture ascended, so a vector's first value was always its minimum. A writer that never raised a vector's max past its first value (nomaxupd), never lowered its min (nominupd), or never folded a chunk max past its first vector (foldmax) stored bounds of the right width, and main's shell passed 18/18 under each, on PG17. - v is now a permutation of 1..6150 (its values, nulls and counts unchanged), with a premise that it is not ascending within a vector. - Four checks decode each bound of id, k and v (zm_le reads the stored native bytes) and compare it with the heap's own for its group and vector: every vector's min, every vector's max, every whole-chunk min, every whole-chunk max. - "value_count sums (v, nullable)", "null_count sums (v, nullable)" and "whole-chunk sum(id) matches heap" compared two readings with check: with the server stopped they passed on two empty strings. They are check_num now; a stopped server fails every check read after it. The twin test_native_zonemap.py asserts all 23 checks by the shell's names, one per test. Mutants on PG17, the same names red in shell and twin: nomaxupd 2, nominupd 2, foldmax 1, nominmax 5. Clean 23/23 in both on each of 15-19. Seeded: 23 rows, 5 dated, 18 never. Census 4241 -> 4259 never, 3026 -> 3031 dated, 7290 rows; suites_not_covered 60 -> 59; cluster_tests 3538 -> 3561; TESTS.md §310. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…orage id is unread (#1697 review) jdatcmd's #1697 review: when the storage id could not be read, the twin fell back to `WHERE storage_id = NULL`, which matches no row, and a count of bad rows over no rows is the 0 nine checks want. The shell has no fallback: its empty $SID is a syntax error, so each check fails. Each zone-map reading now goes through _zint, which returns sid's own failure instead of querying; _bounds_match and the after-drop count short-circuit the same way, so expect.num refuses each one, as under a stopped server. nullsid (pgcolumnar_relation_storageid returns NULL for every relation, one line in columnar_vacuum.c) on PG17: - before: shell 3 passed + 20 failed; twin 12 pass + 6 fail, with nine names passing in the twin that fail in the shell ("id (int4) min width = 4 bytes", "k (int8) max width = 8 bytes", "min/max present for all columns", "per-vector value+null = 1024", "per-vector value_count folds into chunk (k)", "sum null for non-int4 columns", "sum present for int4 columns", "whole-chunk value+null = group rows", "zone maps cleaned on drop"); - after: the same 20 names red in both (comm prints nothing either way). Clean: 23/23 in both. No check name or ledger row changes; nullsid is not used to date rows (every reading is unreadable under it, so it says nothing about what each measures). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…red across scans no longer passes (#1341) "nested columnar scans each hold their own handle" compared only the join's count, which a shared handle still answers correctly: with one static zone-map session shared across scans (the first implementation of #744), main's shell passed it. It is now the witness, each of the join's two scans reporting its own groups opened once, and the count is a check of its own, "nested columnar scans return the right answer". The three scans around the one that errors in a subtransaction must also each probe exactly their own groups. Two premises a one-line pgcolumnar change reddens lost the prefix (#1339): the probe count (probetwice, noprobe) and the middle scan's error (excludeall). A new check holds zs2 to more than one group, which a writer ignoring stripe_row_limit fails. Six mutants on PG17 redden the same names in both twins: shared 5, reopen 5, probetwice 4, noprobe 7, excludeall 8, nostripe 1. With the server stopped after setup, only the check read before it passes. The re-resolved index oid that columnar_metadata.c documents (idxagain) stays 13/13: no witness here counts it. 13/13 shell and twin on each of 15-19. Seeded: 13 rows, 12 dated, 1 never. Census 4259 -> 4260 never, 3031 -> 3043 dated, 7303 rows; suites_not_covered 59 -> 58; cluster_tests 3561 -> 3574; TESTS.md §311. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d optx100 dates it (#1698 review) jdatcmd's #1698 review: "premise: the fixture really has more than one chunk group" reads pgcolumnar's own row_group catalog, the writer's output, so it can redden for a writer bug; zs2's twin check had already lost the prefix for that reason. It is renamed "the fixture has more than one chunk group" (the row is kept, renamed) and dated by optx100 (columnar_write_state.c: the writer multiplies the table's stripe_row_limit by 100, one line). optx100 on PG17: shell "FAIL the fixture has more than one chunk group: got [no] want [yes]" and "FAIL zs2 has more than one chunk group", 11 + 2; the twin fails the same 2 names. Clean: 13/13 in both. Census for this suite: 13 rows, 13 dated + 0 never (was 12 + 1): never -1, dated +1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ulls as values no longer passes (#1341) Ports test/native_vecdecode.sh to test/pytest/test_native_vecdecode.py, every check by the shell's name, one test per check. Three holes in the shell, each measured on PG17 against main and fixed in both harnesses: - The fixture had no nulls. With the fold ignoring the validity bitmap (nonull), main passed 24/24. A nullable column w (NULL on every seventh row) now runs through the fold ("the fold answers count and sum of a nullable column over a skipping range": 51|2444 against heap's 44|2606 under nonull) and through a scalar parity check, behind a heap premise that the range holds nulls. - Missing counters read as 0. "decoded plus skipped is the group's vectors" passed as 0 + 0 = 0 with the plans failing, and two aggregate parities compared two empty readings: with the server stopped before the plans, main passed 3. Every counter must now be read, the parities use check_text, and the same stop fails all 28. - Five "premise:" checks read pgcolumnar's own behaviour: nofold reddens the three "reaches the fold" checks, decodeall and noskip the fold arm's decode count, and noskip "the narrow range skips vectors". All lost the prefix (#1339). "a ruled-out vector is not decoded" now names both readings when it fails, so its lossy_arms.tsv row is removed. The poison check ("the fold does not count rows from a vector it never decoded") is dated by holeread: the fold ignores the skip mask, and the mutant zero-fills the hole, so the read is deterministic on a non-assert build too. Mutants on PG17, the same failing checks by name in shell and twin: decodeall 3, holeread 1, nonull 1, nofold 4, noskip 3. Head is 28/28 in both on each of 15-19. Seeded: 28 rows, 10 dated, 18 never. Census 4259 -> 4277 never, 3050 -> 3060 dated, 7337 rows; suites_not_covered 57 -> 56; cluster_tests 3580 -> 3608; TESTS.md §313. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…can and optdiv100 date eight rows (#1700 review) jdatcmd's #1700 review: four premises read the plan pgcolumnar produces, and noscan (pgcolumnar.enable_custom_scan off by default, one line; the C initializer too, or an assert build traps) reddens all four, as it did the same check in #1691. They lose the prefix (rows renamed, not removed): "the narrow plan is a columnar custom scan", "the wide plan is a columnar custom scan", "no whole group is pruned on the narrow query" and "the wide range skips none, so the arms really differ". The review asked to apply the same test to "one row group": optdiv100 (the writer divides the table's stripe_row_limit by 100, one line in columnar_write_state.c) reddens it (got [51] want [1]), so it loses the prefix too. The ground-first order stays, so a failure still names its cause. PG17, shell = twin by name: - noscan: 15 red in both (twin: 3 recorded fails + 12 named VacuityErrors); dates the four renamed checks plus "EXPLAIN reports Columnar Vectors Decoded", "and it is not stuck at zero when a full scan decodes everything" and "control: and the same counter reads every vector when nothing is skipped", all never before. - optdiv100: 6 red in both; dates "one row group, so every difference is per-vector". - clean: 28/28 in both. Census for this suite: 28 rows, 18 dated + 10 never (was 10 + 18): never -8, dated +8. The heap-side nulls check stays a premise: it reads h. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t be its own guard's (#1341) The refusal arms read only the SQLSTATE, and four of the six refusals share 22023 (absent id, count mismatch, id-less file, duplicate requested id). With the "file carries no field ids" guard removed (noanyid), an id-less file fell to the absent-id refusal, also 22023, and main's shell passed 11/11. Each arm now wants its own guard's primary message, read with VERBOSITY verbose in the shell and diag.message_primary in the twin. Survival was a bare SELECT 1, which answers again after crash recovery: with the absent-id refusal turned into abort() (abortabsent) it passed, and the crash took five later refusal arms with it. Survival now also reads the server log's count of killed backends, and every reading first waits for the server, so a crash fails its own arm and survival only. The twin's writer runs as a subprocess, so the module imports no pyarrow, and every test declines under the shell's skip name before it asks for the fixture. Measured on PG17, each giving the same failing names in shell and twin: noanyid 1, abortabsent 2, noscalar 1, offbyone 5, bypos 6. With the server stopped before the projection arms, all 11 fail. Clean: 11/11 shell and twin on each of 15-19. Seeded: 11 rows, 9 dated, 2 never. Census 4269 -> 4271 never, 3068 -> 3077 dated, 7348 rows; suites_not_covered 56 -> 55; cluster_tests 3608 -> 3619; TESTS.md §314. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… through the reader under test no longer passes (#1341) Every codec was compared with c_none.parquet read back through read_parquet, the reader under test. A defect after decompression -- in the page decode every codec shares -- corrupted both sides alike: with one decoded byte flipped in every page over 4 KB, the same way for every codec (postcorrupt), main's shell passed 9/9 on PG17. Each file now decodes to the generator's rows, built in SQL from the same generator the crafter wrote them from; postcorrupt fails five checks in both harnesses (and the gzip sum reads 199990033, not 199990000). Each file's codec is now read from its own metadata. The crafter printed it to stderr and nothing read it: with every file written uncompressed, main passed 9/9; the three codec premises now fail. Survival read a bare SELECT 1. With the crafted-size guard aborting (hdrabort) and the probe 3 s late, main passed it; it now also reads the server log's crash count and fails as "no ([1], 1 backend crashes)". The fixtures move into test/craft_parquet_codecs.py, which the twin runs as a subprocess so its fixture never imports pyarrow; each twin test declines under the shell's skip name first. The gzip sum is now check_num against 199990000, so two failed reads can't agree, and the codec and FDW loops are unrolled into literal names. postcorrupt, nocodec, mapcodec and hdrabort each fail the same checks by name in shell and twin on PG17 (5, 4, 1, 2). Head is 14/14 in both on each of 15-19. Seeded: 14 rows, 7 dated, 7 never (the four codec premises read pyarrow's metadata, and the FDW counts hold under every mutation). Census 4271 -> 4278 never, 3077 -> 3084 dated, 7362 rows; suites_not_covered 55 -> 54; cluster_tests 3619 -> 3633; TESTS.md §315. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…le read through the reader under test no longer passes (#1341) The twin, test/pytest/test_native_parquet_units.py, makes each of the shell's 20 checks under its name, on a private database, and declines under the shell's pyarrow skip name before it asks for the fixture. Two holes in the shell, each measured red-first on PG17 against main, fixed in both harnesses: - The pushdown oracle was the two-group millis file read back through read_parquet, the reader under test. With every row after a page's first taking the next dictionary entry (dictshift), the scan and the mirror shifted alike and main passed 19/19. The oracle is now the generator's hourly series, built in SQL, and a new check, "the millis file reads back exactly as written", compares the whole file with it. Under dictshift both fail, the same names in shell and twin. - The nanos truncation checks compared each file with a second file read the same way, with plain check. With the server stopped before them, main passed both on two empty reads. They now want the toward-zero microseconds as literals, through check_text, and fail with the server stopped. The per-unit and per-tag loops are unrolled, so every name is a literal the grader reads. Mutants on PG17, each failing the same checks by name in shell and twin: msasus (MILLIS read as micros) 9, nsasus (NANOS read as micros) 4, floordiv (nanos truncated toward -inf) 1, dictshift 2. Head passes 20/20 in both harnesses on each of 15-19. Not tested, because there is nothing to mutate: the reader skips isAdjustedToUTC, and a timestamptz column decodes exactly as a timestamp one, so there is no UTC adjustment for a check to catch. Seeded: 20 rows, 13 dated, 7 never. Census 4278 -> 4285 never, 3084 -> 3097 dated, 7382 rows; suites_not_covered 54 -> 53; cluster_tests 3633 -> 3653; TESTS.md §316. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…shared with the oracle no longer passes (#1341) The oracle h was filled with read_parquet of the same file, and read_parquet shares the scan's decoders. With a text decoder dropping the last byte of a string (txtdec) or an INT64 decoder off by one above 9000 (i64dec), both sides read the same wrong values, and main's shell passed 15/15 under each. h is now built in SQL from the generator test/craft_parquet_projection.py wrote the file from; under those mutants the projections that read c, or b, fail. No table had a dropped column, so a target's attribute number always equalled its index, and pqfdw_compute_needed matching the needed set by index (posneed) passed 15/15. ftd declares a column ahead of a..d and drops it; "after a dropped column, projection [d] == oracle" and "after a dropped column, one column reads 1" fail under posneed. The five projection checks were built from a loop variable the grader cannot read; they are now literal names. Crafting moved to test/craft_parquet_projection.py, which both harnesses run. Mutants on PG17, each failing the same names in shell and twin: posneed 2, txtdec 3, i64dec 2, droplast 8, noproj 8. 17/17 shell and twin on each of 15-19. Seeded: 17 rows, 15 dated, 2 never. Census 4285 -> 4287 never, 3097 -> 3112 dated, 7399 rows; suites_not_covered 53 -> 52; cluster_tests 3653 -> 3670; TESTS.md §317. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rd no longer stands in for the chunk-count refusal (#1341) Ports test/native_parquet_hardening.sh to test/pytest/test_native_parquet_hardening.py, all 18 checks by the shell's name, one test per check. Both twins now write their files with one crafter, test/craft_parquet_hardening.py, so neither can drift from the other. Four holes, each measured on main's shell on PG17 and fixed in both harnesses: - The short-chunk verdict matched any "has a malformed row group", which five guards raise. With the chunk-count guard removed (nochunk) a later guard rejected the file and main passed 18/18. The verdict now wants this guard's own detail ("carries 1 column chunks, but the schema has 2 leaf columns"). - The "still reads" arms counted rows. A date decoded one day late, or a time or timestamp one microsecond late (dateoff, timeoff, tsoff), passed 18/18. They now compare the decoded values with the ones the crafter wrote. - "no wrapped -25536 appears" counted -25536 in the in-range file, which never holds 40000, so with the int2 guard removed (noi2) it still passed. It now reads the out-of-range file itself, which must refuse with 22003. - Survival was a bare SELECT 1. With the chunk refusal turned into abort() and the probe 5 s late, main passed it. It now also reads the server log's count of killed backends, and every reading waits for the server first, so a crash fails only its own arm. The range refusals also want their own type's message, since four share 22008. Mutants on PG17, each the same failing names in shell and twin: noi2 5, overi2 2, notime 1, nodate 1, nodateall 2, nots 1, nochunk 1, chunkabort 2, tsoff 1, dateoff 1, timeoff 1. Head passes 18/18 in both harnesses on each of PG15-19. "date offset overflow" is guarded twice (the offset-overflow branch and IS_VALID_DATE), so only nodateall, which removes both, reaches it. Seeded: 18 rows, 16 dated and 2 never (parquet_schema on the bad file, and the valid two-column read). Census 4287 -> 4289 never, 3112 -> 3128 dated, 7417 rows; suites_not_covered 52 -> 51; cluster_tests 3670 -> 3688; TESTS.md §318. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ursion guard is pinned to 54001 (#1341) The footer parser's two unbounded recursions, PgColumnarThriftSkip and walk_schema, are each guarded by check_stack_depth(). The twin crafts the two deep footers by hand (no pyarrow), pins max_stack_depth, and asserts each raises 54001 rather than SIGSEGV; a shallow valid chain still reads its one column; and read_parquet refuses the crash input the same way. Measured on PG17, same failing checks by name in shell and twin: nothrift 2 (nested-struct and read_parquet refusals), nowalk 1 (schema-chain refusal), walkstrict 1 (the valid-file control). Head 8/8 in both harnesses on PG15-19. 8 rows, 4 dated and 4 never (the survival checks, which no mutation crashes at the pinned depth). Census 4289 -> 4293 never, 3128 -> 3132 dated, 7425 rows; suites_not_covered 51 -> 50; cluster_tests 3688 -> 3696; TESTS.md §319. Waits for the merge freeze on #1341. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d each runs its own input (#1708 review) jdatcmd's #1708 review: 1. The twin's read_parquet survival test never read stack_nested.parquet; it called _alive() only, so it could not see a crash in read_parquet (skipabort: 1 passed alone). It now runs the read_parquet statement first, through the same helper the refusal test uses. 2. All four survival checks, in both harnesses, were a bare SELECT 1 straight after the trigger, which answers again once crash recovery is over (with an abort() in the Thrift guard and a 3 s delay, both survival checks after the aborts passed). Following #1701 and #1705: each arm takes a baseline of "terminated by signal" in the server log right before its trigger, every reading first waits for the server, and survival wants SELECT 1 and an unchanged crash count. A crash now fails only its own arm and its survival check. Each survival check red by name in both, under an abort() at its input's decode site (PG17): - thriftabort (PgColumnarThriftSkip: `if (stack_is_too_deep()) abort();` for its guard): shell 4 red: the nested-struct and read_parquet refusals and "backend survived the nested-struct footer" / "backend survived read_parquet on the crash input" ("got [no ([1], 1 backend crashes)]"); the twin fails the same 4 names (2 recorded, 2 VacuityErrors). - walkabort (the same in walk_schema): the schema-chain refusal and "backend survived the schema-chain footer", in both. - validabort (walk_schema aborts on entry): the schema-chain refusal and survival, "a shallow valid schema chain still reads" and "backend survived the valid file", in both. Clean: 8/8 in both. The four survival rows are dated by these mutants. Census for this suite: 8 rows, 8 dated + 0 never (was 4 + 4): never -4, dated +4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ackend no longer passes as up (#1341) Ports test/native_parquet_streaming.sh to test/pytest/test_native_parquet_streaming.py, every check by the shell's name, one test per check. Both harnesses now write their files with one shared crafter, test/craft_parquet_streaming.py, extracted from the shell's four heredocs; its files are byte-identical to theirs (md5 of all 13). Measured on main's shell on PG17: - Survival was a bare SELECT 1, which answers again once crash recovery has finished. With the second-dictionary-page refusal turned into abort() (dictabort), "backend survived the crafted page areas" passed, and the crash also failed "v2 level lengths past the page are rejected", read while the server recovered. Survival now reads the server log's count of killed backends since its section began, and every reading waits for the server, so under dictabort exactly the dictionary arm and survival fail, the same names in shell and twin. - Hardening: the two refusals that took any ERROR ("value read of the sparse file fails cleanly", "unparseable page header is rejected") and the zero-page verdict now want the page loop's own refusal, "could not decode Parquet column". No mutant here separated them (hdrerr, a header-window refusal raised as a different ERROR, is inert: the noise file is refused elsewhere), so this is stated as hardening, not a measured hole. Mutants on PG17, each the same names in shell and twin: nodict 1, nosize 1, nooffset 1, noprogress 1 (the 60 s timeout), dictabort 2. nov2 stays green, as the suite's header argues. Head is 20/20 in the shell and 21/21 in the twin on each of 15-19; the twin's extra test mirrors "crafted files were built", which the shell records only when the crafter fails. Seeded: 20 rows, 5 dated and 15 never; no row for either pgc_skip name. Census 4289 -> 4304 never, 3136 -> 3141 dated, 7445 rows; suites_not_covered 50 -> 49; cluster_tests 3696 -> 3717; TESTS.md §320. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ust be its own guard's (#1341) Ports test/native_parquet_partition.sh to test_native_parquet_partition.py, every check by the shell's name, one test per check. Three holes, each measured red-first on PG17 against main's shell and fixed in both harnesses: - pruned_for() read a failed or empty EXPLAIN as 0. With the server stopped before them, "a predicate matching everything prunes nothing", "a predicate on a file column prunes no files" and "a volatile clause prunes nothing" passed on main. "Files Pruned" prints only above 0, so a missing line now counts as 0 only beside the plan's Files line, and is "no reading" otherwise; all three now fail with the server stopped. - errs() took any ERROR. With the unknown-partition-column guard removed (nocolcheck), a later 42804 stood in; with the explicit encoded-NUL guard removed (nonul), the encoding check's 22021 did; main passed 35/35 under each. Every refusal now wants its own SQLSTATE and message. Measured: what refuses a file column declared as a partition column is the missing-component guard ("has no v= directory component"), not the column-count check the comment described; the comment now says so. - Survival was a bare SELECT 1: with the unknown-column refusal turned into abort() and the probe 5 s late, main passed it. Both survival checks now read the server log's count of killed backends, and every reading waits for the server. Mutants on PG17, the same names red in shell and twin: noprune 9, invprune 20, novolguard 1, nonul 1, nocolcheck 1, abortunk 3. With the server stopped mid-suite both harnesses pass the 10 checks read before the stop and fail the other 25. Clean 35/35 in both harnesses on each of 15-19. Seeded: 35 rows, 29 dated, 6 never. Census 4304 -> 4310 never, 3141 -> 3170 dated, 7480 rows; suites_not_covered 49 -> 48; cluster_tests 3717 -> 3752; TESTS.md §321. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s nothing no longer passes (#1341) Projection and row-group skipping were never read: every arm compared answers, which the executor's recheck and a discarding projection keep right. With every column decoded (noproj) or no row group skipped (noskip), main's shell passed 14/14. Two checks now read the scan's EXPLAIN ANALYZE counters: a one-column query must read 1 of 11 columns, and over a 200,000-row file of four row groups a selective predicate must skip three. (Only a same-type constant is used for skipping: id > 196608 against an int8 column skips nothing, id > 196608::int8 skips three. Correct either way, a missed optimization only.) The refusal arms accepted any psql failure. With the scan's missing-path guard removed (nopathguard) the backend crashed and main passed; with the short-file guard removed (nolen) the bad-magic guard refused instead and main passed. Each arm now wants its own guard's SQLSTATE and message, a survival check reads the server log's count of killed backends, and setup waits for the server so a crash fails only its own arm and survival. The oracle is a heap table written from the generator rather than the columnar table the file came from, and the self-join counts are check_num, since two failed reads agreed. Seven mutants give the same failing names in shell and twin on PG17: noproj 1, noskip 1, overskip 4, nopathguard 2, nolen 1, acceptopt 1, rgsize 2. Head is 20/20 in both on each of 15-19. 20 rows, 10 dated, 10 never. Census 4310 -> 4320 never, 3170 -> 3180 dated, 7500 rows; suites_not_covered 48 -> 47; cluster_tests 3752 -> 3772; TESTS.md §322. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ps (#1711 review) jdatcmd's #1711 review: "and the skipped scan still returns every matching row" counted `id > 196608`, an int4 constant against an int8 column, a cross-type comparison that skips no row group (#1712), so the count came from a full scan and was right whatever skipping did. skipshift (pqfdw_compute_skip moves each group's skip decision to the next group: still 3 skipped, but the only matching group among them) passed 20/20 in both twins. The count now runs the plan's own query, `id > 196608::int8`, in the shell and the twin. The plan check above it ("a selective predicate skips the three row groups it cannot match", 3 skipped) reads that same query, so it already shows the count's scan skips; it is a dated check, not a premise, since a one-line reader change reddens it, so no new premise is added. skipshift on PG17: shell "FAIL and the skipped scan still returns every matching row: got [0] want [3392]", 19 + 1; the twin fails the same name. Clean: 20/20 in both. The row is re-dated by skipshift (it was dated by a mutation that skipped every group); census unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t agreed with itself no longer passes (#1341) The round trips compared read_parquet with the source table, and the columnar scan reads that table both to write the file and as the oracle. With the columnar writer storing int4 777 as 778 (writecorrupt), the file and the "source" agreed on 778, and main passed 12/12. Every round trip now hashes the generator's own rows, built in SQL. The refusals counted any psql failure, a crash included. With the scalar physical-type guard removed (notype), main passed 12/12. Two refusals share 42804, so each now requires its SQLSTATE and its own message; every reading waits for the server first, and a survival check reads the server log's count of killed backends. read_parquet's own pg_read_server_files gate was never tested: with it removed (nogate), main passed 12/12. A role holding USAGE on the schema but not the role is now refused by the gate's own message, beside a member that reads every row. Five mutants give the same failures by name in shell and twin on PG17: writecorrupt 2, notype 1, nogate 1, nocount 1, nonull 7. 15/15 shell and twin on each of 15-19. Census 4320 -> 4324 never, 3180 -> 3191 dated, 7515 rows; suites_not_covered 47 -> 46; cluster_tests 3772 -> 3787; TESTS.md §323. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ound-trip rows name their mutant's real side (#1713 review) jdatcmd's #1713 review: 1. "reject missing column definition list" wants 42601 "a column definition list is required", which PostgreSQL's parser raises before pgcolumnar runs (the message is not in pgcolumnar's source). It reads PostgreSQL alone, so it is renamed "premise: reject missing column definition list" in the shell, the twin, TESTS.md and the ledger; it stays never. 2. The two round-trip rows said "columnar_reader.c: the columnar writer stores int4 777 as 778". The mutant that dated them is in columnar_reader.c, but it is read-side (store_att_byval returns 778 for 777), so the text now says "the columnar reader returns int4 777 as 778 (store_att_byval)", reproducible from the row alone. (The review's write-side version in columnar_write_state.c reddens both rows too.) Clean on PG17: 15/15 in both twins. Census unchanged (rows renamed/re-described only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…uard (#1341) Port test/objstore_listing.sh to test/pytest/test_objstore_listing.py, one test per shell check under its literal name, on a private cluster restarted with AWS_* and the SigV4 fixture server paging two keys at a time. Three holes in the shell, measured red-first on PG17 before the fix: - the two non-progress refusals share 08P01 and read only the SQLSTATE; the no-keys page replays its token, so with the progress check removed the token check refused it and main passed 12/12. Each now wants its guard's own message (refusal_of). - no check made a LIST fail, so a non-2xx read as an empty prefix passed 12/12. objstore_http_server.py gains a __listing_status__ hook and a check wants the HTTP status refusal (08006). - the exact-key "no LIST" count read a log nothing proved alive; a control now wants the key's own GET in it. A survival check reads the server log's count of killed backends, and every refusal waits for the server first, so an abort fails only its own check. Seven mutants on PG17, the same reds in shell and twin; 15/15 on 15-19. Ledger 15 rows (11 dated, 4 never); never 4324 -> 4328, dated 3191 -> 3202, snc 46 -> 45, cluster_tests 3787 -> 3802. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1341) test/pytest/test_native_rewrite_conc.py runs the shell's 11 checks by their literal names, one per test. Holes found by mutating main's shell on PG17: - A compact_rewrite under AccessExclusiveLock passed 5/5. The workers just waited for it, and nothing claimed non-blocking. New arm: a rewrite held mid-verb (pgcolumnar.maintenance_hold_ms) with its lock proven granted in pg_locks. A reader and a writer, each under SET LOCAL lock_timeout = '3s', must finish while it still holds the lock. Then the hold must be proven still held, and the rewrite must complete. - The overlap was assumed. A new check requires the compactor to move rows during the workers' window. Head moves 24-30 rows with 3-5 serialization retries per run. Mutants, PG17, same reds in both harnesses: - noconflict (the conflict check never raises): 3 reds, the same in 3/3 runs (whole-row, count/sum, range) - ael (AccessExclusiveLock): 2 (reader, writer) - alwaysconflict (over-apply): 5 - nohold: 2 (holds-lock, still-held) Head: 11/11 in both harnesses on PG15-19. Ledger: 11 rows, 9 dated and 2 never (row count sane, the held rewrite itself completed). Census 4328 + 3202 == 6165 -> 4330 + 3211 == 7541; suites_not_covered 45 -> 44; cluster_tests 3802 -> 3813. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1341) test_objstore_addressing.py is the twin of objstore_addressing.sh, every check under the shell's name, on a private cluster with its own SigV4 fixture server started with --virtual-host-base s3.local. Two holes in the shell, red-first on main's code (PG17): - the gs:// no-endpoint check passed whenever the message did not name AWS_ENDPOINT_URL, so a mistyped default host (gswrongdef) passed 12/12. It now requires the allow-list refusal naming storage.googleapis.com:443. - no key needed URI-encoding, so a space sent as '+' (plusspace) or '+'/'=' left bare (plusraw) passed 12/12. A key "dir sp/a+b=c.parquet" is now read under both addressings, and its request path checked in the log. The twin also checks the request path the server logged for every addressing read, and asserts each refusal by SQLSTATE plus the phrase naming the cause. CI: the pytest-cluster jobs (ci.yml, nightly.yml) gain the step that resolves s3.local and pgc-bucket.s3.local, which only the suites jobs had. Without it the twin declines UNAVAILABLE_ENDPOINT and the run ends 67. Ledger: 16 rows, 11 dated by seven mutations with the same reds in both harnesses, 5 never. Census 4330 + 3211 == 7541 -> 4335 + 3222 == 7557; suites_not_covered 44 -> 43; cluster_tests 3813 -> 3829. No row for either pgc_skip name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… restores every stash no longer passes (#1341) test/pytest/test_objstore_stash_recovery.py is the twin of test/objstore_stash_recovery.sh. It runs test/objstore_module.sh as a subprocess against the same three arrangements of pkglibdir and asserts on the exit status and what was left behind, one check per test under the shell's names. Three holes in the shell, each measured on main's suite against a mutation of objstore_module.sh's start guard on PG17: - "and it did not stop at the start guard" counted 'interrupted mid-move', the refusal's text before #393's fix. Nothing prints it any more. With the guard turned back into a refusal, 8 checks went red and this one stayed green. It is replaced by a check that the installed module is still the one that was there. - The debris was a byte-identical copy of the module, so a guard with no debris branch, restoring every stash over the installed module, passed 17/17. The debris is now a different valid module. - Both restore slices parked the module at .away, so a guard that walked .away alone passed 17/17. The only-copy slice now parks it at .probe. Each slice now starts from the pristine copy, so a presence-only guard no longer reddens the next slice's premise instead of its own checks. lib.sh's initdb now passes -U postgres, as pgc_cluster.initdb does. Every psql in lib.sh connects as postgres, and a non-root run (the pytest-cluster job, which this twin drives a shell suite from) otherwise gets a cluster with no postgres role. Ledger: 18 rows, 9 dated by four mutations (the same reds by name in both harnesses), 9 never. Census 4335 + 3222 == 7557 -> 4344 + 3231 == 7575; suites_not_covered 43 -> 42; cluster_tests 3829 -> 3848 (19 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ell suite starts (#1717 review) jdatcmd's review of #1717: in the pytest-cluster job (runner user, PGDG packages) test_objstore_stash_recovery.py failed 8 checks because the inner objstore_module.sh exited 1 in all three arrangements, and the twin printed nothing about why. The cause: a packaged postgres defaults unix_socket_directories to /var/run/postgresql, which only postgres can write. lib.sh never pinned it, because the suites job drops to runuser -u postgres; the twin is the first caller that drives lib.sh as neither root nor postgres, so the postmaster FATALed on "could not create lock file .../.s.PGSQL.N.lock: Permission denied" and every start attempt failed. pgc_cluster.py has always pinned the socket to the cluster's datadir for exactly this reason; lib.sh now does the same. Every connection to lib.sh's cluster is TCP on 127.0.0.1 (the suites that connect by socket -- audit, phase3, phase6, smoke, concurrency, extension_upgrade -- run their own clusters), so nothing reads the socket's path. _run_suite now prints the last 40 lines of the inner suite's output when it exits non-zero, so CI shows the cause. Measured on PG17 with an initdb that writes PGDG's default into the generated postgresql.conf (the packaged layout), as offgridwithjd: before: 11 pass + 8 fail -- the same 8 names as CI's job 120074097818; the new tail shows "start attempt N failed; retrying on a fresh port" after: 19 pass + 0 fail Normal source-build prefixes: 19/19 on each of PG15-19. objstore_module.sh 30/30 on PG17. Guards 804 passed, collection 3654 = cluster_tests, harness selftest 1248/1248. No check names or counts change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ports test/objstore_s3_read.sh to test/pytest/test_objstore_s3_read.py: a private cluster restarted with AWS_* in the postmaster environment, reading through the shell's own SigV4 fixture server (token required, tamper bucket), every check by the shell's name, one test per check. Holes in main's shell, each passing 12/12 on PG17: - the no-environment and 403 refusals read a bare 28000, which five guards share; with s3:// falling back to a default host the region guard refused instead. Each refusal now requires the phrase naming its cause. - region handling: dropping the AWS_DEFAULT_REGION fallback, swapping the precedence, signing us-east-1 with no region, or signing with an empty secret all passed. Four checks now pin them. - no crash survival: abort() in abort cleanup passed every SQLSTATE check. Survival now reads the server log's crash count, and every reading waits for the server first. - the https check's name interpolated the build's want; it is literal now. The optional real-S3 arm declines UNMET_PRECONDITION without PGC_S3_INTEGRATION_ENDPOINT, listed in expected_unrunnable.txt on 15-19. 17 rows, 13 dated and 4 never. Census 4344 -> 4348 never, 3231 -> 3244 dated, 7575 -> 7592 rows; suites_not_covered 42 -> 41; cluster_tests 3848 -> 3866. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…is checked as the TLS one (#1341) Ports test/objstore_tls_read.sh to test/pytest/test_objstore_tls_read.py: all 29 checks under the shell's names, one test per check, on a private cluster restarted with SSL_CERT_FILE (and, for arm E, SSL_CERT_DIR alone) and the AWS environment. The PKI, the parquet fixture and the logs live in server_dir; the fixture is written by a python3 child. Four holes, each passing on main's shell (PG17): - A refusal was any 08006 plus a reason grepped anywhere in the output. With the module speaking cleartext to the TLS port, the three "refused with 08006" checks passed. Each refusal now also wants the module's own message naming the URL, and the whole DETAIL: OpenSSL's verification reason for that certificate. - A self-signed leaf had no arm. It has one now. - SSL_CERT_DIR was never read: trusting SSL_CERT_FILE only passed 19/19. Arm E restarts with SSL_CERT_DIR alone and reads the good server. - Nothing read the server log: with abort() in place of the verification refusal, the crash reddened the next arm's premise and D, and "returned no rows" passed. Each refusal now waits for SELECT 1, and a survival check counts "terminated by signal". Census 4348 -> 4355 never, 3244 -> 3266 dated, 7592 -> 7621 rows; suites_not_covered 41 -> 40; cluster_tests 3866 -> 3895. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ard (#1341) test/pytest/test_objstore_http_read.py is the twin of test/objstore_http_read.sh: 24 checks under the shell's names, one test per check. Holes on main's shell, each passing 18/18 on PG17, now red in both harnesses: - arm C took the SQLSTATE alone, so a dropped Range-ignored guard passed on 08P01 from "unexpected HTTP status 200" (noguard200). Each refusal now needs its own guard's message as well. - no fixture sent a short 206 body, cut a body off mid-transfer, or dropped a keep-alive connection: noshort, eofok, noretry and alwaysretry passed. The fixture server gains /short/, /trunc/, /dropka/ and /hangup/ modes, inert unless a path names them. - no crash check: a survival check now counts "terminated by signal" after waiting for the server, and every reading after a possible crash waits for the server first, so one arm's crash can't redden the next. Ledger: 24 rows, 16 dated by ten mutants (SAME reds in shell and twin on PG17), 8 never. Census 4355 -> 4363 never, 3266 -> 3282 dated, 7621 -> 7645 rows; suites_not_covered 40 -> 39; cluster_tests 3895 -> 3919. In the stack, three twin test names collided with names earlier ports define (test_the_fixture_server_is_up, test_the_cancel_latency_is_bounded, test_the_fixture_is_written): they are now test_the_http_read_*, in the twin and in its TESTS.md rows. No check name changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1341) Port test/objstore_sink_write.sh to test/pytest/test_objstore_sink_write.py, one test per shell check under its literal name, on a private cluster restarted with AWS_* and the SigV4 fixture server. Holes in the shell, measured red-first on PG17 before the fix: - the fixture completed an upload from every stored part, ignoring the list, so a client dropping the last part from it passed 30/30. It now assembles the listed parts, each checked against its ETag, and a failed complete leaves the upload open, as S3 does. - no check made a part fail, so a refused part read as stored passed 30/30. objstore_http_server.py gains a __part_status__ hook; four checks want the part check's own 58030, no key and no open upload. - abort was read from the DELETE count, so an abort naming the wrong upload passed 30/30. The server keeps __open_uploads__ and two checks want none open. - the Arrow arm read only export_arrow > 0, green with the object's last byte missing; the object is imported back from the bucket. - every oracle hashed the columnar table the export scanned; they are the generators now. Each refusal wants its own guard's message, and the cancel arm lets in-flight requests land before reading the prefix. The pair is declared INCOMPLETE: the two opt-in real-S3 integration arms (PGC_S3_INTEGRATION_ENDPOINT, never set in CI) are not ported. Eight mutants on PG17, the same reds in shell and twin; 35/35 on 15-19. Ledger 35 rows (11 dated, 24 never); never 4363 -> 4387, dated 3282 -> 3293, snc 39 -> 38, cluster_tests 3919 -> 3954. In the stack, the twin test test_the_sigv4_fixture_server_is_up collided with the one #1718 defines: it is now test_the_sink_write_sigv4_fixture_server_is_up, in the twin and its TESTS.md row. No check name changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1722) On cancel, pexport_cleanup terminates each parallel_export worker, and SIGTERM ends it in FATAL. FATAL calls proc_exit without unwinding, so the writers' PG_CATCH never reached PgColumnarSinkAbort, and each worker's multipart upload stayed open on the store, billed until a lifecycle rule reaps it. The sink layer now keeps the remote sinks with an upload in progress on a list and aborts any still listed from a before_shmem_exit callback. The callback is registered after InitPostgres's ShutdownPostgres, so it runs while the transaction and the module's handles are intact. Finish (after a complete that succeeded) and abort unlist a sink, and transaction end empties the list. The SIGTERM can land mid-request, with a part's body still on the connection, so the module's abort now reconnects before its DELETE. Without that, the DELETE was read as more of the part (the request log shows the part's PUT complete and no abort for its upload), and 4 of 20 cancel runs of the twin still left one upload open. With it, 0 of 15. objstore_sink_write's cancel arm gains "cancel: the server holds no open upload under the prefix". It was red on main's code on all five majors, in the shell and the twin alike (4 uploads open under cx/). With the fix it passes 36/36 on 15-19 in both. Dropping the before_shmem_exit registration reddens it by name in both twins, which dates its row. Census: never 4387 -> 4387, dated 3293 -> 3294, rows 7680 -> 7681; cluster_tests 3954 -> 3955. Stacked on #1721, whose fixture lists open uploads. Fixes #1722 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…old a terminated export worker (#1727 review) jdatcmd's #1727 review: os_wait passed no WL_TIMEOUT, so WaitLatchOrSocket ignored the 5 s slice (timeout=-1) and a silent store never counted a quiet slice; the 60 s quiet limit was dead code. Under proc_exit interrupts are held, so a parallel_export worker that SIGTERM ended stayed in its exit-callback upload abort until the store answered. os_wait now passes WL_TIMEOUT. objstore_sink_write gains "terminate: export workers exit within 75 s while the store is silent": SIGSTOP the fixture mid-upload, terminate the workers, time their exit. - red first on #1727's head, PG17, both twins: "got [slow (4 workers, 4 left after 91s)]" (twin: 90s), which dates the row; - with the fix: 61 s on each of PG15-19, 38/38 in both twins. pg_ctl stop -m fast with the store silent mid-export: 60 s with the fix; without it the stop did not finish within 150 s (rc=1). The coordinator's addition: jdatcmd saw "cancel: the server holds no open upload under the prefix: got [1] want [0]" flake in CI on #1723's head, an upload whose POST ?uploads was still in flight when the cancel landed (#1743's window). The cancel arm now cancels only once the fixture's __open_uploads__ shows all four workers' uploads, with its own check ("all four workers' uploads were open before the cancel"), so the open-upload check pins exactly what #1722 fixes. That check is not a premise: nomultipart (the sink's flush loop never runs, so no export opens a multipart upload; one line in columnar_objstore_module.c) reddens it by name in both twins on PG17 ("got [no (0 open)] want [yes]"; the same 11 names red in shell and twin), which dates it. PG17 at the fix: 10 runs of shell and twin, 0 flakes (38/38 each). noreg (the before_shmem_exit registration dropped) still reddens "cancel: the server holds no open upload under the prefix: got [4] want [0]" in both twins. Census: +2 rows, never 4350 -> 4350, dated 3117 -> 3119; cluster_tests 3756 -> 3758; two TESTS.md rows in objstore_sink_write's section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jdatcmd

Re-approved on d37d54e, the one-pass restate onto main c7f0de4. restate_check.sh against my last approval: the PR's own change is the same in every content file; only the registries (budget, TESTS.md, expected_tests) differ, and the head's census agrees with its tsv.

…s left open (#1743) A worker of parallel_export_parquet learns its multipart upload id only from the answer to CreateMultipartUpload (POST ?uploads), and the store creates the upload when the request arrives. A worker that ended while it waited for that answer left an upload open that nothing could abort. #1727's exit callback (in this base) aborts an upload a terminated worker holds the id of; it cannot help a worker that never got the id. The dispatcher's cleanup now lists the uploads under the export's prefix (ListMultipartUploads) and aborts each one whose key is exactly one of the export's part names, part-0000.parquet to part-(nkeys-1).parquet. Any other key under the prefix is left alone. - On a cancel or other error the aborts run inside #1744's PgColumnarCleanupAndRethrow, in the cleanup callback after pexport_cleanup. The helper has copied the handled error out and emptied the stack, so a second cancel that lands in a part DELETE cuts short only that DELETE: the aborts still run and the first cancel is re-thrown. They also run after a worker failed (for example, the workers were terminated). - Interrupts are held only around bounded waits: 5 s on a timed latch for the terminated workers to stop, then 10 s for the listing and aborts. - New module entries list_uploads and abort_upload, which take a timeout that sets a deadline on the handle; ABI 5 -> 6. - os_wait: this builds on #1727's WL_TIMEOUT (4dc63aa, in this base through #1744's stack, not on main). It keeps WL_TIMEOUT for every wait and cuts the slice to the time left when a handle has a deadline. The first version's comment "os_wait passes no WL_TIMEOUT" is gone; this hunk depends on #1727 landing. - The fixture server already wrote __open_uploads__ (#1721) and had __delete_delay__ (#1744). One copy of each is kept; the __open_uploads__ writer is the module function _write_open_uploads, which --seed-upload also uses. Review of the first version (80ee33e): a second cancel during pexport_remove_outputs' part DELETEs emptied the error stack (#1741), the first version's outer PG_CATCH then raised "errstart was not called", and the aborts never ran. That outer PG_TRY, its copy-flush-rethrow helper and pexport_spawn_and_wait are removed. Test: new suite objstore_export_open_uploads and its twin, 30 checks each, identical names. The fixture server gains ListMultipartUploads, delay hooks on POST ?uploads, part PUTs and the listing, and --seed-upload. Five arms: cancel while parts are sent; cancel while every CreateMultipartUpload is held; the workers terminated there; cancel there, then cancel again once the first part DELETE has arrived (__delete_delay__ 3 s); and cancel there while the listing is held. The double-cancel arm was red first on the rebased first version, PG17, shell and twin alike: "XX000: errstart was not called", one more errstart line in the server log, and 4 uploads left open. With the rework: 30/30 on PG15-19, shell and twin. Ledger: 30 rows, 12 dated by four mutations that redden the same names in both twins on PG17 (no listing; every listed key aborted; the part match off by one; the cleanup run with the handled error on the stack), 18 never. "parts: no upload of this export is left open" is never: on this base #1727 also closes it, so no single mutation reddens it. never 4488 -> 4506, dated 3260 -> 3272, rows 7748 -> 7778; suites_not_covered stays 34; cluster_tests 4027 -> 4057; TESTS.md section 338. Restated onto main 4500efd, stacked on #1752. main's #1745 added a test named test_double_cancel_reports_the_first_cancel to test_parallel_flush_optin.py, so the corpus defined that name twice and two guards in test_docs_cover_the_corpus.py failed. This twin's test is renamed test_double_cancel_reports_the_exports_own_cancel, with its TESTS.md row; the check name it records is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

OffgridwithJD force-pushed the fix/1722-export-cancel-upload-abort branch from 469540e to 68d8b0c Compare October 22, 2026 12:47

OffgridwithJD mentioned this pull request Oct 22, 2026

Missing tests: 87 confirmed behaviours no shell or pytest arm exercises (#1339 audit) #1341

Closed

87 tasks done

jdatcmd requested changes Oct 22, 2026

View reviewed changes

jdatcmd mentioned this pull request Oct 22, 2026

A second cancel of a parallel_export_parquet to s3:// turns the dispatcher's error into "errstart was not called" #1741

Closed

OffgridwithJD mentioned this pull request Oct 22, 2026

A cancel during CreateMultipartUpload still leaves an upload open: the dispatcher should list and abort uploads under the export prefix #1743

Closed

OffgridwithJD mentioned this pull request Oct 22, 2026

fix: a second cancel during a remote export's cleanup keeps the export's own error (#1741) #1744

Merged

jdatcmd mentioned this pull request Oct 22, 2026

fix: a cancelled remote parallel export aborts the uploads its workers left open (#1743) #1755

Merged

OffgridwithJD force-pushed the fix/1722-export-cancel-upload-abort branch from 68d8b0c to 4dc63aa Compare October 22, 2026 21:20

OffgridwithJD mentioned this pull request Oct 23, 2026

alpha5: clearing the board to release — who has what #1283

Open

jdatcmd and others added 7 commits October 23, 2026 03:15

jdatcmd approved these changes Oct 23, 2026

View reviewed changes

jdatcmd and others added 7 commits October 23, 2026 03:44

jdatcmd and others added 26 commits October 23, 2026 04:11

OffgridwithJD force-pushed the fix/1722-export-cancel-upload-abort branch from 4dc63aa to d37d54e Compare October 23, 2026 05:53

jdatcmd approved these changes Oct 23, 2026

View reviewed changes

jdatcmd merged commit d37d54e into main Oct 23, 2026
15 checks passed

Navigation Menu

Repository navigation

fix: a cancelled remote parallel export aborts its workers' uploads (#1722) - #1727

fix: a cancelled remote parallel export aborts its workers' uploads (#1722)#1727
jdatcmd merged 42 commits into
mainfrom
fix/1722-export-cancel-upload-abort

OffgridwithJD commented Oct 22, 2026

Uh oh!

jdatcmd left a comment

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD commented Oct 22, 2026

Uh oh!

jdatcmd left a comment

Uh oh!

jdatcmd left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

OffgridwithJD commented Oct 22, 2026

The bug

Red first, on #1721's code

The fix

Green and mutants

Not fixed here

Census

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

Runs

Defect: a terminated worker hangs at exit while the store is silent

Not blocking

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD commented Oct 22, 2026

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

The blocking finding is fixed

The other checks

Registries

How this composes with #1755

Not blocking

Uh oh!

jdatcmd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants