[jdatcmd]

Covers the rollback half of the #1341 core-am item "maintenance functions that roll back or crash part-way". The crash half needs an immediate stop part-way through a function. It is a separate harness shape, and I'll do it in its own PR. Test-only: no source change.

What it adds

A new suite, test/maintenance_rollback.sh, and its pytest twin, test_maintenance_rollback.py. The table has row groups of 1000 rows, two rows in five deleted, and one group deleted whole, so every function has work. It also has an index on id.

For each of compact_rewrite, recluster, compact, vacuum and vacuum_sorted:

1. Run the function inside BEGIN, and read the row groups there. An arm checks they changed: for example, compact_rewrite goes from 10/10000 to 10/6400.
2. ROLLBACK.
3. The table must hold the same rows by content (an md5 over every row), the same row groups, and the same rows through an index scan.

Then a savepoint rolls back two rewrites, rows written later in the transaction commit beside the old ones, and every function runs for real and keeps the same rows.

Evidence

• Green: 13/13 on each of PG15–19, and the twin on each.
• Red on PG17, both twins, with one mutant. PgColumnarAllocateFreeSpace reuses a free range whatever its freeing transaction, so compact_rewrite writes its new groups into the space of the groups it has just retired. After the rollback, those groups point at overwritten bytes: ERROR: columnar: corrupt encoded chunk (DOD element width invalid). Nine checks went red.
• Ledger: 13 rows seeded, 9 dated. The census is 2774 + 13 = 2787 rows, 2095 + 4 = 2099 never, and 688 dated. The gate passes against origin/main. The harness selftest passes 1245/1245 and the guard set 433/433.
• Registration: run_all_versions.sh, test_compare_to_bash.py COMPLETE, TESTS.md §115 (renumbered by whichever of the §115 PRs merges later). cluster_tests goes 565 → 566 by collection.

Two things I hit while writing it

• In bash, GROUPS is a builtin array, and assigning to it silently does nothing. My first draft stored the row-group query there, so every "same row groups" comparison compared empty with empty. The premise check exposed it.
• pgc_conn is a write-watching wrapper, so setting .autocommit = False on it never reaches the connection. The function committed, and rollback() did nothing. The twin now issues BEGIN and ROLLBACK as statements, and TESTS.md says why.

🤖 Generated with Claude Code

https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N

[OffgridwithJD]

@jdatcmd Reviewed 802243d.

• What it holds: each of the five maintenance functions runs inside BEGIN, its row-group change is read there, and the transaction rolls back. The rows by md5, the row groups and an index scan must then all be unchanged. A savepoint arm follows, then every function run for real. I checked that the "by index" read really is an Index Scan using mr_id.
• Red (PG17): PgColumnarAllocateFreeSpace with the freeing-xid gate removed (.so b9726239 -> 3b083b0f) gives 4 + 9 failed, starting after compact_rewrite rolls back, the table holds the same rows and row groups, by scan and by index: got [|10/10000|]. The twin fails at that arm with corrupt encoded chunk (DOD element width invalid).
• Blocker: 7 of the 9 dated rows are red only because compact_rewrite has already corrupted the shared table. All five functions run on one mr. Once compact_rewrite's rollback leaves it corrupt, every later read errors, and recluster, compact, vacuum and vacuum_sorted report got [|10/10000|] or unchanged ().
  • Measured: the same loop with a fresh mr per function, against the same mutant .so, gives 19 + 1. Only compact_rewrite after rollback fails; recluster, compact, vacuum and vacuum_sorted each pass premise, changed and after-rollback. On good code the same loop is 20/20.
  • The savepoint arm alone does go red on the mutant (got [|] want [1000/6400|5400]), so its dating stands.
  • That leaves seven rows dated by a mutant their own arm can't see: after {recluster,compact,vacuum,vacuum_sorted} rolls back, ... and {recluster,vacuum,vacuum_sorted} changed the row groups inside its transaction. They should be never unless a mutant of their own dates them. A fresh table per function would also stop one corruption from hiding the other four.
• Green at head: shell 13/13 and twin 1 passed on PG17 and PG18. The ledger, 2787 / 2099 / 688, matches the tsv, the gate agrees against origin/main, all 13 names have rows, and cluster_tests 566 matches collection.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, addressed in add338b, rebased onto main 1f7b5da.

• Each function now gets a fresh table, in both twins: build_mr / _build_mr runs before each of the five functions and before the savepoint arm.
• Re-dated in isolation with your mutant (PgColumnarAllocateFreeSpace reusing space whatever its freeing xid): 2 red, as you measured with the fresh-table loop. They are after compact_rewrite rolls back, ... (got [|10/10000|]) and the savepoint arm (got [|]). The twin fails first on the same compact_rewrite arm, with corrupt encoded chunk (DOD element width invalid). recluster, compact, vacuum and vacuum_sorted stay green, so their seven rows are now never.
• Green: 13/13 in both twins on each of PG15 to PG19.
• Bookkeeping, on 1f7b5da: the census is 2984 + 13 = 2997 rows, 2278 + 11 = 2289 never, 706 + 2 = 708 dated, matching the tsv. The gate gives rc=0, TESTS.md is §116, cluster_tests is 571 by collection, and the guard set passes 435/435.

🤖 Generated with Claude Code

[OffgridwithJD]

Approved at add338b. The cascade is gone. A few optional points below can ride along with the rebase this PR needs anyway.

• The fix: each function and the savepoint arm now get a fresh mr.
• Green: shell 13/13 and twin 1 passed (13 checks), on PG17 (.so 70896f0e) and PG18 (1bef88ea).
• Your mutant (PgColumnarAllocateFreeSpace with the freeing-xid gate removed; PG17 70896f0e → 316bb87c, PG18 1bef88ea → 24eeba16) gives exactly 2 red on each: after compact_rewrite rolls back, ...: got [|10/10000|] and after a savepoint rolls back two rewrites, ...: got [|] want [1000/6400|5400]. The twin fails at compact_rewrite with corrupt encoded chunk (DOD element width invalid). recluster, compact, vacuum and vacuum_sorted stay green, so nothing is red by cascade any more.
• Optional: 7 of the rows marked never can be dated by mutants of their own.
  • A vacuum mutant: replacing the relfilenode swap in pgcolumnar_compact_relation with table_relation_nontransactional_truncate(rel), so the storage is cleared in place and the abort can't undo it (.so 70896f0e → 5abd4424), turns after vacuum rolls back, ... and after vacuum_sorted rolls back, ... red with got [|10/10000|]. The twin gives corrupt encoded chunk (DOD element width invalid).
  • A no-op mutant: with all five functions made no-ops (de77718c), exactly the five X changed the row groups inside its transaction rows fail, with got [unchanged (10/10000)] want [changed].
  • recluster and compact: moving recluster's retire ahead of its write stays green (83127f95), because the gate covers it. So never is honest for those two as far as I tried.
  • Dating the 7 would take never 2289 → 2282 and dated 708 → 715 on this base.
• Nit: and every function then runs for real and keeps the same rows loops over compact_rewrite, recluster, compact and vacuum, but not vacuum_sorted. Either add it or name the four.
• Bookkeeping: the tsv gives 2997 / 2289 / 708, matching the paragraph. All 13 names have rows, cluster_tests 571 and guard_tests 435 match collection, and no new mutation text has a ";" piece starting with a space.
• Before merge: main is f2bd5f1, so the budget, TESTS.md and expected_tests.txt conflict. After the rebase I'd expect §119, cluster_tests 577, and 3020 / 2293 / 727 before any re-dating.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, restated as 2fdb9b4, in a chain on main 670a58c: #1413 → #1415 → #1419 → #1422 → #1423 → #1424, this one on top of #1415. The patch is identical to add338b, the head you approved, apart from re-derived bookkeeping: the budget paragraph, re-anchored on the previous link's census and matching the tsv; cluster_tests, matching collection; and the TESTS.md section number. The gate gives rc=0 and the guard set passes 435/435. I'll merge the chain in order as CI goes green.

🤖 Generated with Claude Code