test: recovery gets its pytest twin, and each crash proves its own recovery (#1341 item 25)

test/check_ledger_budget.txt

@@ -15,7 +15,7 @@
 #
 #   Adding a check to a suite that is already covered does not move it, which is
 #   what makes it safe to bound.
-suites_not_covered 224
+suites_not_covered 221
 #
 # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
 #   one.
@@ -2815,4 +2815,62 @@ suites_not_covered 224
 #     never         2602 - 1     = 2601
 #     dated          872 + 1     = 873
 #     2601 + 873 == 3474
-checks_never_observed_red 2601
+#
+# #1341, unique_conc seeded, with its twin (OffgridwithJD): 2601 -> 2607, and
+# suites_not_covered 224 -> 223. The suite had no rows: its local check() recorded
+# through pgc_record with no accounting line, and it now uses lib.sh's recorders.
+# It enters with all 33 checks from five-major runs (33/33 on each of 15-19). 27 are
+# dated by six mutations of columnar_unique.c: the per-key lock never taken (the 18
+# serialize, unique_violation and one-row arms), the lock taken with the setting off
+# (1a, 2a), every key in one bucket (3, and 4's differing key), only the first key
+# column hashed (4's differing key), the partial predicate ignored (5 outside it),
+# and a NULL key locked under NULLS DISTINCT (6). The "does not block" arms could not
+# fail before this change, so their dates are new evidence. Six arrive never: the two
+# bucket-count premises, the same-statement duplicate (7, the btree's own check), and
+# the aborted first inserter (8), which waits on the first's xid with or without the
+# key lock.
+#
+#     ledger rows   3474 + 33 = 3507
+#     never         2601 +  6 = 2607
+#     dated          873 + 27 = 900
+#     2607 + 900 == 3507
+#
+# #1341, update_conc seeded, with its twin (OffgridwithJD): 2607 -> 2621, and
+# suites_not_covered 223 -> 222. The suite had no rows: its local check() recorded
+# through pgc_record with no accounting line, and it now uses lib.sh's recorders.
+# It enters with all 28 checks from five-major runs (28/28 on each of 15-19). 14 are
+# dated by seven mutations: in columnar_row_lock.c, the flush-time serialization
+# skipped, the committed-delete conflict never raised, and the re-read taken from
+# the statement snapshot (each the same 10 duplicate-row and 40001 arms at RC and
+# RR, update-vs-delete, delete-vs-update and multirow); the row lock skipped during
+# the UPDATE of a table with a unique index (the unique/RC arm); the self-modify
+# guard removed (the self-update arm); and in pgcolumnar_subxact_callback, either
+# discard skipped (the new failed-statement savepoint arms, both). Fourteen arrive
+# never: the seven heap-oracle arms, the two SERIALIZABLE guards, which held before
+# the row lock existed, the two serial controls, the scan-order premise, and the
+# completed-UPDATE savepoint arm's two, which neither discard alone reddens.
+#
+#     ledger rows   3507 + 28 = 3535
+#     never         2607 + 14 = 2621
+#     dated          900 + 14 = 914
+#     2621 + 914 == 3535
+#
+# #1341, recovery seeded, with its twin (OffgridwithJD): 2621 -> 2639, and
+# suites_not_covered 222 -> 221. The suite had no rows. It enters with all 29 checks
+# from five-major runs (29/29 on each of 15-19). 11 are dated by three mutations,
+# each a WAL record not written: a stripe data page's (the whole-row, aggregate and
+# range arms of s1, s2's whole-row, s3's four, and s5's two projection arms), the
+# metapage's reserved-offset advance (s2's whole-row and s3's four), and the
+# visibility-map clear (s4's cleared bit). Eighteen arrive never: the row counts
+# (s1, s2, s5's base, s2's phantom rows), the two heap counts, s4's three pre-crash
+# arms, its clean block, its count and contents, the five "recovery ran" arms and
+# the s2 INSERT premise. Those last six go red only under harness mutations, a crash
+# that does not happen and an INSERT that never runs, which observe the suite rather
+# than the code, so they date nothing here. The visibility-map set's WAL alone
+# reddens nothing: the bits it sets are checkpointed before the crash.
+#
+#     ledger rows   3535 + 29 = 3564
+#     never         2621 + 18 = 2639
+#     dated          914 + 11 = 925
+#     2639 + 925 == 3564
+checks_never_observed_red 2639

test/pytest/TESTS.md

@@ -179,6 +179,9 @@ behaviour, the source of that number is named.
 - [131. test_type_matrix.py: types the differential matrix never stored answer as heap](#131-test_type_matrixpy-types-the-differential-matrix-never-stored-answer-as-heap)
 - [132. test_iceberg_data_files.py: a delete entry in a data manifest is refused](#132-test_iceberg_data_filespy-a-delete-entry-in-a-data-manifest-is-refused)
 - [133. test_native_skip.py: a type binary-coercible to its opclass's type prunes](#133-test_native_skippy-a-type-binary-coercible-to-its-opclasss-type-prunes)
+- [134. test_unique_conc.py: concurrent inserters of one unique key serialize](#134-test_unique_concpy-concurrent-inserters-of-one-unique-key-serialize)
+- [135. test_update_conc.py: two writers of one row serialize](#135-test_update_concpy-two-writers-of-one-row-serialize)
+- [136. test_recovery.py: committed writes survive a crash, uncommitted ones leave nothing](#136-test_recoverypy-committed-writes-survive-a-crash-uncommitted-ones-leave-nothing)
 
 ## 1. How to read a test in here
 
@@ -7221,3 +7224,60 @@ A partial pytest twin of `test/native_skip.sh` (#1433). It ports only the arms t
 | --- | --- |
 | `test_a_range_on_a_coercible_type_prunes` | a range on inet, cidr, a domain over cidr, text, varchar and a domain over varchar returns heap's rows and removes chunk groups |
 | `test_equality_on_a_coercible_type_prunes` | equality on cidr and on varchar, which also takes the bloom probe, returns heap's row and removes chunk groups |
+
+## 134. test_unique_conc.py: concurrent inserters of one unique key serialize
+
+The pytest twin of `test/unique_conc.sh` (#5, ported for #1341), every check by the shell's name. A columnar row's btree entry is written at once while its data waits for the stripe to flush, so a second inserter of the same key could not see the first's row and would insert a duplicate. The table AM takes a transaction-scoped advisory lock per unique key, hashed with the key type's default hash. So the second inserter waits for the first to commit and gets 23505, asserted by SQLSTATE and the index's name. T1 is held mid-INSERT, its key row indexed and unflushed, by a second row that waits in `cq_block` on a lock a holder session has. Every barrier polls `pg_stat_activity` for a lock wait.
+
+The shapes: one key, with the lock off in T2 as the control that shows the bug; keys equal under the opclass with different bytes (numeric 1.0 and 1.00, and citext by case); different keys, which must not wait; a multi-column key, equal and differing in its second column; a partial index, inside and outside its predicate; NULL keys under NULLS DISTINCT and NULLS NOT DISTINCT; a duplicate within one statement; and a first inserter that aborts. A private cluster runs `pgcolumnar.unique_lock_buckets` at 100003, which is fixed at server start, so unrelated keys do not share a bucket.
+
+### Every test
+
+| test | what it holds |
+| --- | --- |
+| `test_the_bucket_count_is_fixed_at_server_start` | `SET` of the bucket count is refused 55P02, and the cluster runs 100003 |
+| `test_one_key_serializes_and_the_lock_is_what_does_it` | with the lock off in T2 both inserts commit; with it on T2 waits, gets 23505, and one row is left |
+| `test_keys_equal_under_the_opclass_serialize` | numeric 1.0 and 1.00: the same pair, lock off and on |
+| `test_citext_keys_equal_by_case_serialize` | citext 'ABC' and 'abc' serialize and conflict; unrunnable without citext |
+| `test_different_keys_do_not_block_each_other` | T2's different key returns at once, and both rows are kept |
+| `test_a_multi_column_key` | (1,2) twice serializes and conflicts; (5,6) and (5,7) do not wait |
+| `test_a_partial_unique_index` | inside the predicate the keys serialize and conflict; outside it neither waits and both rows are kept |
+| `test_null_keys` | two NULLs never wait under NULLS DISTINCT; under NULLS NOT DISTINCT they serialize and conflict |
+| `test_a_duplicate_within_one_statement` | one INSERT of the same key twice raises 23505 and inserts nothing |
+| `test_an_aborted_first_inserter` | T2 waits behind T1's flushed, uncommitted row; T1 rolls back and T2's insert succeeds |
+
+## 135. test_update_conc.py: two writers of one row serialize
+
+The pytest twin of `test/update_conc.sh` (#5, ported for #1341), every check by the shell's name. A columnar UPDATE marks the old row deleted and appends a new one, so without a lock on the row identity two transactions updating one row each kept their own new version, and the row was duplicated. `columnar_row_lock.c` takes a transaction-scoped advisory lock per row at the delete-vector flush, re-reads the latest committed state, and gives the loser a retryable 40001. A table with a unique index takes the lock during the UPDATE, so the unique check cannot answer 23505 first (#1421).
+
+Each concurrent case runs one interleaving on a heap table, the oracle, and on a columnar one: s1 writes and holds its transaction, s2's statement waits on a lock, which a poll of `pg_stat_activity` confirms, and s1 commits. At READ COMMITTED a heap re-applies s2's update where columnar fails it, so the columnar value is compared with the heap's only at REPEATABLE READ. A private database; each session runs at a 30s `lock_timeout`, as the shell's cluster does.
+
+### Every test
+
+| test | what it holds |
+| --- | --- |
+| `test_update_vs_update_read_committed` | one row is left and the loser gets 40001; the heap composes both updates into one row |
+| `test_update_vs_update_repeatable_read` | count and values match the heap, and both losers get 40001 |
+| `test_update_vs_update_serializable` | the count matches the heap and the loser gets 40001, which held before the row lock too |
+| `test_update_vs_delete_read_committed` | the heap's DELETE removes the updated row; columnar's loser gets 40001 |
+| `test_delete_vs_update_read_committed` | the committed DELETE stands on both, and columnar's loser gets 40001 |
+| `test_a_row_updated_twice_by_one_statement` | `UPDATE ... FROM (VALUES (1), (1))` leaves one row on both |
+| `test_overlapping_multi_row_updates` | 41 overlapping rows: 200 rows remain, and the loser gets 40001 |
+| `test_serial_updates_control` | the same two updates without overlap leave one row with the last writer's value |
+| `test_a_covering_unique_index` | with a unique index the loser still gets 40001, not 23505; the heap composes |
+| `test_a_savepoint_rolled_back` | an UPDATE rolled back to a savepoint leaves the pre-savepoint value, as on the heap |
+| `test_a_statement_failing_inside_a_savepoint` | an UPDATE that fails partway inside a savepoint leaves nothing behind: its buffered mark and row are discarded |
+
+## 136. test_recovery.py: committed writes survive a crash, uncommitted ones leave nothing
+
+The pytest twin of `test/recovery.sh` (ported for #1341), every check by the shell's name. Columnar stripes are written through the buffer manager with a full-page WAL record each, and the metadata lives in WAL-logged heap catalogs. So rows committed after the last checkpoint come back from WAL, a transaction open at the crash leaves nothing visible, and a visibility-map bit cleared after the checkpoint does not come back set from the page on disk. A heap mirror is the oracle.
+
+A private cluster, stopped with `-m immediate` for each crash. Each crash is proven by the server log gaining a "database system was not properly shut down" line, counted before that crash, so one recovered crash cannot vouch for a later one. The open transaction's INSERT is proven to have run by its session holding an xid.
+
+### Every test
+
+| test | what it holds |
+| --- | --- |
+| `test_the_pair_through_three_crashes` | rows committed after a checkpoint survive; an open transaction's checkpointed INSERT leaves no row; committed deletes survive; each against the heap pair |
+| `test_a_cleared_visibility_bit_stays_cleared` | a bit set and checkpointed, then cleared by a DELETE, stays cleared after the crash; a clean block's bit stays set; the rows match heap |
+| `test_a_projection_survives_a_crash` | a projection copy committed after the checkpoint replays, and reproduces the base's rows and a scan through it |

test/pytest/expected_tests.txt

@@ -733,4 +733,10 @@ guard_tests 438
 # listed controls and three parametrized refusals), by collection over the cluster files.
 # cluster_tests 609 -> 617 for #1433's test_native_skip.py, eight tests (six ranges and
 # two equalities, parametrized), by collection over the cluster files on this tree.
-cluster_tests 617
+# cluster_tests 617 -> 627 for #1341's test_unique_conc.py, the twin of unique_conc.sh:
+# ten tests, by collection over the cluster files on this tree.
+# cluster_tests 627 -> 638 for #1341's test_update_conc.py, the twin of update_conc.sh:
+# eleven tests, by collection over the cluster files on this tree.
+# cluster_tests 638 -> 641 for #1341's test_recovery.py, the twin of recovery.sh: three
+# tests, by collection over the cluster files on this tree.
+cluster_tests 641

test/pytest/test_compare_to_bash.py

@@ -146,6 +146,7 @@
             "projection_scan_io",
             "projection_update",
             "projections",
+            "recovery",
             "recovery_minimal_wal",
             "rewrite_storage_oid",
             "rls_direct_storage",
@@ -165,6 +166,8 @@
             "timeseries_recipes",
             "ttl_expire",
             "type_matrix",
+            "unique_conc",
+            "update_conc",
             "vacuum_lock_privilege",
             "validity_elision",
             "vm_privilege",