[jdatcmd]

The crash half of the #1341 core-am item "maintenance functions that roll back or crash part-way". #1419 is the rollback half. Test-only: no source change.

What it adds

A new suite, test/maintenance_crash.sh, and its pytest twin, test_maintenance_crash.py. The twin uses a private cluster, because every arm takes the postmaster down. The shell kills its process group, as recovery.sh does. The twin stops it with -m immediate, which leaves the same redo for the next start.

compact_rewrite, recluster, compact and vacuum each run twice on a fresh table (row groups of 1000 rows, two in five rows deleted, one group deleted whole, an index):

crash point	after recovery
function run in an open transaction; another session runs CHECKPOINT; crash	the table is as it was before the function
function committed; crash before any checkpoint, so its work is only in WAL	the table is as the function left it, with the same rows

Each arm checks that a crash recovery was logged ("database system was not properly shut down"), then the table by content (an md5 over every row), row groups, and an index scan.

Evidence

• Green: 16/16 on each of PG15–19, and the twin on each.
• Red on PG17, both twins:

mutant	what went red
data pages written without WAL (PgColumnarWriteLogicalData)	committed compact_rewrite, recluster and vacuum: zstd decompression failed or produced the wrong length
free space reused whatever its freeing transaction	uncommitted compact_rewrite: corrupt encoded chunk (DOD element width invalid)

• Ledger: 16 rows, 4 dated. The other 12 are the eight recovery arms and four table arms neither mutant reached. One of those is compact, which writes no data pages. The census is 2774 + 16 = 2790 rows, 2095 + 12 = 2107 never, and 683 dated. The gate passes against origin/main. The harness selftest passes 1245/1245 and the guard set 433/433.
• Registration: run_all_versions.sh (in C order: maintenance_crash before maintenance_due, which the selftest caught), test_compare_to_bash.py COMPLETE, TESTS.md §115. cluster_tests goes 565 → 566.

🤖 Generated with Claude Code

https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N

[OffgridwithJD]

@jdatcmd Reviewed 36fd27e.

• What it holds: compact_rewrite, recluster, compact and vacuum are each crashed twice: once uncommitted, after another session checkpoints, and once committed, with no checkpoint since. After recovery the table is checked by md5, row groups and an index scan, and the server log must show a crash recovery.
• Red (PG17): data pages written without WAL, if (false && RelationNeedsWAL(rel)) in PgColumnarWriteLogicalData (.so b9726239 -> f9467d41), gives 13 + 3: after a crash with compact_rewrite committed, the table is as compact_rewrite left it, with the same rows: got [|10/6400||], and the same for recluster and vacuum. The twin fails with zstd decompression failed or produced the wrong length (expected 235).
  • The recovery arms discriminate: with the SIGKILL in crash_cluster replaced by pg_ctl -m fast stop, all 8 go red with no recovery logged (0 before, 0 after).
• Blocker: nothing asserts that the function did any work, so a no-op passes every arm. Measured with pgcolumnar_compact_rewrite reduced to rewritten = 0 (.so ed94c1d4): shell 16/16 and twin 1 passed, including the two compact_rewrite rows the ledger dates. A function that changes nothing is trivially crash-safe, so both crash points need the premise #1419 has:
  • committed: after's row groups differ from the fresh table's;
  • uncommitted: the open session reports its row groups before it names itself (in the shell that output currently goes to /dev/null).
• Green at head: shell 16/16 and twin 1 passed on PG17 and PG18. The ledger, 2790 / 2107 / 683, matches the tsv, the gate agrees against origin/main, all 16 names have rows, and cluster_tests 566 matches collection.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, addressed in 4fc2455, rebased onto main f2bd5f1.

• Each function must do work before the crash, at both crash points, in both twins:

  • uncommitted: $name changed the row groups in its open transaction. The open session puts its in-transaction row-group count into its application_name (mc_open <groups>), which the suite's poll reads. A SIGKILL can't lose that reading, unlike psql output sent to a file.
  • committed: $name changed the row groups before the crash.

  I didn't name these premise:, because a mutant turns them red and the gate refuses a premise that has gone red.

• Your no-op mutant (compact_rewrite with rewritten = 0): 2 red, the two new compact_rewrite arms, got [unchanged (10/10000)]. The twin fails first on the same arm. Before this change, every arm passed under it.

• Your WAL mutant (if (false && RelationNeedsWAL(rel)) in PgColumnarWriteLogicalData): 3 red, the committed-crash arms of compact_rewrite, recluster and vacuum. The twin fails with the zstd error, as you saw.

• Green: 24/24 in both twins on each of PG15 to PG19.

• Bookkeeping: rebuilt from those runs on the rebased tree. The census is 3007 + 24 = 3031 rows, 2282 + 19 = 2301 never, 725 + 5 = 730 dated, matching the tsv. The gate gives rc=0, TESTS.md is §119, cluster_tests is 577 by collection, and the guard set passes 435/435.

🤖 Generated with Claude Code

[OffgridwithJD]

Approved at 4fc2455.

• The did-work arms hold. I applied a no-op mutant to each function in turn, on PG17 and PG18: rewritten = 0, reclustered = 0 in recluster_verb, retired = 0 in compact, and the pgcolumnar_compact_relation call removed from vacuum. Each gives 22 + 2 in the shell: that function's "changed the row groups in its open transaction" and "changed the row groups before the crash", with got [unchanged (10/10000)]. The twin fails on the same uncommitted arm, and a copy of the twin with the first loop emptied fails on the committed arm for all four. (PG17 .so bb5b7477 → 71ae4cf7 / 2461aefe / fd8e74d0 / 6b0854c8; PG18 4d5a6995 → d1ef9aea / 675cc1d5 / 0f4caaa9 / 94c4e21a.)
• The application_name reading is the in-transaction count, taken before the SIGKILL. The suite read compact_rewrite 10/6400, recluster 1/5400, compact 9/9000 and vacuum 1/5400 about 15 ms before the kill. At the same moment a second session read 10/10000, and the open session was still active with an xid. With the open session made to report a fixed mc_open 10/10000, all 4 arms go red: 20 + 4.
• The WAL mutant gives 3 red on PG17 and PG18: the committed arms of compact_rewrite, recluster and vacuum, with got [|10/6400||] and [|1/5400||]. The twin fails with the zstd error. The recovery arms still discriminate: with the kill replaced by pg_ctl -m fast stop, all 8 go red with no recovery logged (0 before, 0 after).
• Green at head: shell 24/24 and twin 24 pass, on PG17 and PG18. The census, 3031 / 2301 / 730, matches the tsv and the paragraph, and cluster_tests 577 matches collection.
• The 8 split recovery_minimal_wal rows in this diff are the main-side defect that #1434 fixes. When you rebase onto it, take main's text for those rows. A branch that keeps the split text will be refused by the ledger reader there, so it can't slip through.
• Minor: TESTS.md §119 doesn't mention the new did-work arms yet.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, restated as 08f762a, in a chain on main 670a58c: #1413 → #1415 → #1419 → #1422 → #1423 → #1424, this one on top of #1422. The patch is identical to 4fc2455, the head you approved, apart from re-derived bookkeeping: the budget paragraph, re-anchored on the previous link's census and matching the tsv; cluster_tests, matching collection; and the TESTS.md section number. The gate gives rc=0 and the guard set passes 435/435. I'll merge the chain in order as CI goes green.

🤖 Generated with Claude Code