[OffgridwithJD]

#1341 item 25, its first suite: committed columnar rows survive an immediate stop and redo, and an uncommitted transaction leaves nothing. recovery.sh had no pytest twin. This PR adds one, every check by the shell's name, and seeds the suite into the ledger. The port found two premises in the shell that could not fail, and they're fixed in both twins. Item 25's other two suites, native_backend_crash.sh and corruption.sh, come in their own PRs.

Stacked on #1441, which is stacked on #1438. Merge them in that order; this branch is #1441's head plus two commits.

What changes

• test/pytest/test_recovery.py, the twin. It has three tests on a private cluster, which it crashes with pg_ctl -m immediate, as test_maintenance_crash.py does.
  • Scenarios 1 to 3 share the heap/columnar pair, as in the shell.
  • compare_to_bash.py grades it as covering all 29 shell names, and it joins COMPLETE.
• "Recovery ran" now proves that scenario's own crash. It used to count redo lines over the whole server log, so once scenario 1 had recovered, every later "recovery ran" passed whether or not its own crash happened.
  • Measured, with every crash after the first skipped: the shell stayed 26/26. With the VM clear's WAL removed as well, still 26/26. s4's critical cleared-bit arm passed because no crash happened.
  • The fix: each crash notes the count before the kill, and recovery ran when the restart added a "database system was not properly shut down" line.
  • s2 and s3 gain the same arm, since they had no proof of a crash at all.
• Scenario 2's uncommitted INSERT is now proven to have run. It used to wait sleep 5, and nothing checked the INSERT.
  • Measured: with the INSERT pointed at a table that doesn't exist, scenario 2 passed whole, with nothing to roll back.
  • The fix: the suite now polls for the session sleeping with an xid assigned, and asserts that as its premise. The twin asserts the same xid on its holder connection.
• Ledger: recovery is seeded with 29 rows from five-major runs, and 11 of the 29 rows dated by the mutants below. suites_not_covered goes 222 → 221, and the census is 3527 + 29 = 3556 rows, 2618 + 18 = 2636 never and 909 + 11 = 920 dated, matching the tsv. The gate gives rc=0 against origin/main.

Red first

Each mutation was run one at a time on PG18, through both twins, with the .so md5 printed every time.

Source mutations (each one makes a WAL record go unwritten):

mutation	recovery.sh	test_recovery.py	the arms that went red
none (the head)	29/29	29/29	none
a stripe data page, columnar_storage.c:511	19/29	2 of 3 tests fail	s1's whole-row, aggregate and range; s2's whole-row; s3's count, deletes, whole-row and aggregate; s5's two projection arms (corrupt encoded chunk)
the metapage's reserved-offset advance, :237	24/29	1 fails	s2's whole-row and s3's four (zstd decompression failed, got [714] want [0])
the visibility-map clear, columnar_visibilitymap.c:143	28/29	1 fails	s4 post-crash deleted block stays not-all-visible: got [t] want [f]
the visibility-map set, :98	29/29	29/29	none: the bits it sets are checkpointed before the crash

Harness mutations, for the two premises:

mutation	recovery.sh	test_recovery.py
every crash after the first skipped (after the count is noted, as a kill that failed would be)	s2, s3, s4 and s5 "recovery ran" red, where they were green before this change	not run: the twin's crash is one function, and the row below skips it
no crash at all	all five "recovery ran" red	the three tests' "recovery ran" red
the uncommitted INSERT fails	premise: s2's INSERT ran …: got [0] want [1], where scenario 2 was green before	the same premise, got 0 want 1

The harness reds observe the suite rather than the code, so the ledger doesn't date those six rows. They arrive never, along with the row counts, the heap counts and s4's other arms: 18 in all.

Runs

major	recovery.sh	test_recovery.py
15	29/29	29/29
16	29/29	29/29
17	29/29	29/29
18	29/29	29/29
19	29/29	29/29

Also measured: the guard half (438 tests), harness_selftest 1245/1245 and docs_style. cluster_tests goes 624 → 627 by collection, and TESTS.md gets §134.

🤖 Generated with Claude Code

[jdatcmd]

Approving c76d6d1.

At the head, PG17: recovery.sh 29/29; test_recovery.py 29 checks, 3 tests.

I checked your two test-side claims with my own mutants of the tests, applied to both twins, each confirmed to parse, and restored from git after the run:

mutant	shell	pytest
M1: every crash after the first is a clean pg_ctl stop -m fast	25/29: s2, s3, s4 and s5 "recovery ran", each got [no recovery since the crash]	s2, s4 and s5 "recovery ran", (1 before, 1 after); s3 is not reached, because test 1 stops at s2
M2: s2's uncommitted INSERT targets a table that doesn't exist	28/29: only premise: s2's INSERT ran in its open transaction before the crash, got [0] want [1]	the same premise, got 0 want 1

So each crash now proves its own recovery, and scenario 2 proves that its INSERT ran. Before this PR, both of those failures would have passed.

I didn't repeat your three WAL-record mutants; I took them from your table. The census at this head (3556 / 2636 / 920) matches the budget paragraph, and suites_not_covered is 221.

[jdatcmd]

Re-approving 676313b, a bookkeeping-only restate of c76d6d1, which I approved.

• Identical per-file patches: recovery.sh, test_recovery.py, the ledger rows, test_compare_to_bash.py and TESTS.md (masked).
• The census at this head matches the budget paragraph: checks_never_observed_red 2639, and cluster_tests is 641.

I'll merge the chain in order once each head is green.

[jdatcmd]

@OffgridwithJD, your chain has merged: #1438 (90a49ea), #1441 (6f12e37) and #1443 (061747f). Main is 061747f. I've left the #1341 boxes for 23, 24 and 25 to you, since 25 still has two suites to go. Open from me: #1445, the last #1339 High-entry fix, and my proposal on #1339 for test_join_vector_agg.py:122 (an unreachable refusal).