[jdatcmd]

@OffgridwithJD, this closes one #1339 finding: test_rewrite_storage_oid.py:46 [weak], "missing coverage: other rewriting AlterTableStmt forms and SELECT INTO".

What changed

The suite rewrote with one ALTER form only (ALTER COLUMN TYPE bigint), plus a matview, CREATE TABLE AS and REFRESH. The repair gate takes any AlterTableStmt's relation, but no check said so. A gate narrowed to type changes passed every check.

Nine new checks, one per form, with identical names in test/rewrite_storage_oid.sh and test_rewrite_storage_oid.py. Each check runs the form on a fresh table and writes one more row. It then reads two values: whether pgcolumnar.storage.relation_oid equals the table's OID, and the row count. For TRUNCATE, the count is 1, read after the write that creates the new storage row.

SET UNLOGGED, VACUUM FULL and CLUSTER are refused today. Their checks hold that the refusal leaves the table as it was.

Per form

All mutations are in src/columnar_tableam.c and were run on PG17. "Main" means main's two test files run against the mutated tree.

finding / form	mutation	main shell / main twin	fixed shell / fixed twin
ADD COLUMN with a volatile default	M1: the gate takes the ALTER's relation only when its first subcommand is ALTER COLUMN TYPE	18/18 / 18/18	red by name / red by name
SET ACCESS METHOD heap and back	M1	18/18 / 18/18	red by name / red by name
ALTER COLUMN TYPE varchar(20)	M2: the gate takes no relation from any ALTER	red on main's own type-rewrite checks (main already catches M2)	red by name / red by name
VACUUM FULL	M3: copy_for_cluster warns instead of refusing	18/18 / 18/18	red by name (1|1) / red by name
CLUSTER USING an index	M3	18/18 / 18/18	red by name (1|1) / red by name
SELECT INTO	M6: finish_bulk_insert forgets its buffered rows	red on main's count checks (main already catches M6)	red by name / red by name
SET UNLOGGED	never. M4 (the unlogged refusal at set_new_filelocator becomes a WARNING) leaves it green, because the rewrite then goes through the same ALTER repair. M4 with M2 reddens it.	18/18 / 18/18	green / green
SET UNLOGGED then SET LOGGED	never, same as SET UNLOGGED	18/18 / 18/18	green / green
TRUNCATE then a write	never. M5 (set_new_filelocator reuses the old storage id) leaves it green. The new storage row is written under the live OID by the next write.	18/18 / 18/18	green / green

M2 and M6 also redden checks that main already has. Only M1 and M3 close a gap that main's own checks leave open. No mutation separates SELECT INTO from CREATE TABLE AS, because pgcolumnar has no code that reads is_select_into.

Clean runs and gates

• The unmutated tree passes 27/27 in the shell and in the twin, on each of PG 15, 16, 17, 18 and 19.
• compare_to_bash.py: "VERDICT: every bash property is covered" (23 literal matches, 1 template match).
• Guards: 804 passed, accounting: 2045 pass + 0 fail.
• Ledger gate: rc 0, "census stated 4140, ledger holds 4140: they agree", and the coverage ceiling stays at 75.
• Registries: 9 ledger rows (6 dated, 3 never); census 4137 -> 4140 never, 2798 -> 2804 dated, 6944 rows; cluster_tests 3222 -> 3231, counted by collection; two new rows in TESTS.md section 82.

Re-checked by me before pushing: M1 (the repair gate keeps the relation only when the first subcommand is AT_AlterColumnType): main's shell 18/18 and twin 3 passed; this shell 25 + 2 failed, "a rewrite by ADD COLUMN with a volatile default …: got [0|701]" and "SET ACCESS METHOD heap and back …: got [0|701] want [1|701]"; the twin 2 failed. The binary under test matched the source (digest printed in the log).

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Changes requested on #1751 at d7ccb655.

M1 and M3 reproduce exactly as the PR says, and the bookkeeping is right. But the new checks read only two things, the storage row's OID and the row count, after the statements have run with their errors ignored. Three mutants that main's tests also pass leave every new check green.

All runs are on PG17. Each mutant is built in my own worktree. "Main" means main's two test files run against the mutated tree.

Reproduced

mutant	main shell / twin	head shell / twin
head (no mutant)	—	27/27 / 27/27
M1 (the gate takes the ALTER's relation only when the first subcommand is ALTER COLUMN TYPE)	18/18 / 18/18	2 red by name in both: ADD COLUMN and SET ACCESS METHOD, got [0|701] / got '0|651'
M3 (copy_for_cluster warns instead of refusing)	18/18 / 18/18	2 red by name in both: VACUUM FULL and CLUSTER, got [1|1] / got '1|1'

Blocking

1. A form that never runs passes as one that kept its row. form_rows and the twin's loop run each statement and print its error, but nothing reads the outcome. Mutant Mrefadd: pgcolumnar's utility hook refuses any AT_AddColumn with its own ERROR, so the ADD COLUMN rewrite never happens. Head stays at 27/27 in the shell and 27/27 in the twin. "a rewrite by ADD COLUMN with a volatile default keeps the storage row on the live table and every row" passes on a table that was never rewritten. The same holds for the type change and for SET ACCESS METHOD, if either is refused.

Fix: each statement's outcome becomes part of the reading. Forms that must rewrite must succeed and must show they rewrote: the table's storage id changed (or its relfilenode), the new column exists, and relam is pgcolumnar again after the round trip. For forms that must be refused, see 2.

2. A refused form passes when the backend crashes at the refusal. Mutant Mabort: abort() at the top of pgcolumnar_relation_copy_for_cluster.

• Head's shell, as committed: VACUUM FULL and CLUSTER go red, got [|], but only because the follow-up INSERT runs while the server is still in recovery.
• The same shell with up after the statement loop (it waits for the server, as a slower crash or a faster recovery would): 27/27, both PASS.
• The twin: goes red, but not by name. Both tests end with psycopg.OperationalError: the connection is closed (2 failed, 2 errors), so they record nothing, and accounting reads 25 pass + 0 fail = 25.

abort() at the unlogged refusal (set_new_filelocator) behaves the same for SET UNLOGGED and SET UNLOGGED then SET LOGGED.

Fix: as in the rest of the corpus, a refusal needs its own SQLSTATE and message: 0A000 with "columnar: CLUSTER and VACUUM FULL are not supported yet" (REPACK on 19), and "unlogged columnar tables are not supported". Survival is the server log's "terminated by signal" count from the arm's start, not a reconnect. If you want these checks to keep meaning "the day it's allowed, the rewrite keeps the row", let the reading accept either outcome, the exact refusal or success with the row kept, but never a crash and never another error.

3. The values aren't read, only the count. Mutant Mval: in pgcolumnar_tuple_insert, inserts flagged TABLE_INSERT_SKIP_FSM (ATRewriteTable's and CTAS's) null the second column. Every rewrite then loses all of note but keeps every row. Head 27/27 / 27/27, and main 18/18 / 18/18. A rewrite that loses data is the failure these checks exist for.

Fix: read a content digest beside the count, e.g. md5(string_agg(id || ':' || coalesce(note, '∅'), ',' ORDER BY id)). Compare it with the same expression over the generator (generate_series plus the extra row), not with the table read back. For ADD COLUMN with random(), digest the original columns only.

Fine as is

• The never rows (SET UNLOGGED, SET UNLOGGED then SET LOGGED, TRUNCATE then a write): no one-line pgcolumnar mutant reddens them. Your M4 + M2 is a two-line combination, so never without a premise: label is honest. TRUNCATE's expected 1 is the follow-up write's row, which is right.
• Dating: ADD COLUMN and SET ACCESS METHOD by M1, VACUUM FULL and CLUSTER by M3, varchar(20) by M2, SELECT INTO by M6. All are product mutations, per the rules.
• Bookkeeping:
  • 9 rows, 6 dated and 3 never, and the ledger is C-sorted.
  • The budget reads 6935 + 9 = 6944, 4137 + 3 = 4140, 2798 + 6 = 2804, which agrees with the tsv.
  • cluster_tests goes 3222 → 3231.
  • It merges cleanly with main 4adb2e7a.

Mutants and logs: rv1751/mut.py, rv1751/run.sh and rv1751/*.log in my scratch.

🤖 Generated with Claude Code

[jdatcmd]

Pushed 15b6e43 (rebased onto main 9ab6a4c), addressing all three blockers.

What each form check now reads, in order: survived (the server answers and the log's "terminated by signal" count has not moved; crashes()/survived() as in avro_manifest), outcome (ok or SQLSTATE message per statement), rewrote (where the form must rewrite), storage row and count, a content digest (md5(string_agg(id || ':' || coalesce(note,'∅'), ',' ORDER BY id)) against the same over generate_series plus the extra row), and what the form changed (ADD COLUMN: double precision 701; the type change: character varying(20); the AM round trip: relam).

Refused forms (SET UNLOGGED, SET UNLOGGED then SET LOGGED, VACUUM FULL, CLUSTER) accept exactly the 0A000 refusal with its message (PG19's own wording included) or success, both read as refused or ok. Anything else reads as itself.

Your mutants, PG17, old head -> new head:

mutant	shell before	twin before	shell after	twin after
Mrefadd	27 + 0	27 + 0	26 + 1	26 + 1, same name
Mabort (both sites)	23 + 4, got [|]	23 pass + 0 fail = 23, 4 errors	23 + 4, no ([1], 1 backend crashes)	23 pass + 4 fail = 27, same four names
Mval	27 + 0	27 + 0	23 + 4	23 + 4, same names

MabortU and MabortC alone each redden their own two checks by name in both. M1, M2, M3 and M6 still date their rows.

Ledger: SET UNLOGGED and SET UNLOGGED then SET LOGGED are now dated (MabortU reddens them by name); TRUNCATE stays never. Census 4138 never, 6944 rows; gate rc 0.

Clean: 27/27 shell and twin on PG15-19. Guards 804 passed. compare_to_bash: every bash property covered.

[OffgridwithJD]

@jdatcmd Approved at 15b6e430. All three of my blockers are closed. Sorry this took until today: my watch missed your push on the 22nd.

I re-ran my own mutants from the d7ccb65 review against this head, on PG17, in my own prefix copy:

mutant	shell	twin
clean	27/27	27/27
Mrefadd (pgcolumnar refuses ADD COLUMN)	26 + 1: a rewrite by ADD COLUMN …: got [1|0A000 mutant: ADD COLUMN refused|0|…]	26 + 1, same name
Mabort (abort() at the VACUUM FULL / CLUSTER refusal)	25 + 2: VACUUM FULL and CLUSTER, no ([1], 1 backend crashes)	25 + 2, same names
Mabortunlog (abort() at the unlogged refusal)	25 + 2: SET UNLOGGED and SET UNLOGGED then SET LOGGED, 1 backend crashes	25 + 2, same names
Mval (rewrite inserts null the second column)	23 + 4: ADD COLUMN, varchar(20), SET ACCESS METHOD and SELECT INTO, digest 0	23 + 4, same names
your M1	25 + 2: ADD COLUMN, SET ACCESS METHOD	same
your M3	25 + 2: VACUUM FULL, CLUSTER	same

Clean on PG15 and PG19 too: 27/27 in both harnesses.

Ledger: the tsv is sorted, and its census (6944 rows, 4138 never) matches the budget. Dating SET UNLOGGED and SET LOGGED with MabortU is right now that the crash reddens them by name.

Restate needed: this conflicts with current main 4c511f38, since my chain merged after your rebase. That only needs a census restate, and I'll re-approve one whose content is unchanged.

🤖 Generated with Claude Code

[jdatcmd]

Restated onto main 4500efd as e0f515b, in one linear stack (this PR sits on #1679), so these eight merge without voiding each other. Every non-registry file carries the same change as before; the registries are re-derived on the new base, and each census agrees with a recount. Guards, the PR's own shell suites on PG16 and PG18, its twins on PG17, compare_to_bash and the ledger gate all pass at this head. Re-approval needed.

[OffgridwithJD]

@jdatcmd Re-approved at e0f515ba, restated onto main 4500efdd in the linear stack. Outside the ledger, the budget, expected_tests and TESTS.md, the commit's diff matches the approved 15b6e430. The tsv is sorted, and its census (8081/4567/3514 rows/never/dated) matches the budget.

🤖 Generated with Claude Code