Repository navigation
test: replication gets its pytest twin, and its vacuum and compact arms replay a write (#1341) - #1463
Conversation
3e800eb to
7e0d271
Compare
jdatcmd
left a comment
There was a problem hiding this comment.
Approving 3e800eb.
Measured on PG17: replication.sh 44/44, and test_replication.py 44 checks, on three private clusters, run as root. The tsv at this head (3839 / 2812 / 1027) matches the description and its budget.
Your vacuum claim, with the mutant run myself: pgcolumnar_vacuum returns before it rewrites anything (PG_RETURN_VOID straight after reading its argument).
| shell | result |
|---|---|
main's replication.sh |
41/41 green, as you said |
| this PR's | 43/44: vacuum rewrote r into a new relfilenode on the primary, got [no] want [yes] |
| the twin | the same arm, got 'no' want 'yes' |
I took the compact, XLOG_SMGR_TRUNCATE and WAL-less-page mutants from your table.
…es ALTER EXTENSION UPDATE (#1341 item 66) Between installing a new library and running ALTER EXTENSION pgcolumnar UPDATE, a database runs the 1.0-alpha5 library over its 1.0-alpha4 catalog. The library deletes from pgcolumnar.retired_group, which that catalog lacks, so DROP TABLE, TRUNCATE and pgcolumnar.vacuum() on a columnar table fail there with 42P01 'columnar metadata table "pgcolumnar.retired_group" does not exist', which names neither the cause nor the remedy. Measured: each refusal changes nothing, reads and writes work throughout, and after the update all three succeed on the same rows. open_columnar_table's error now carries a hint naming the update. docs/installation.md's "If you skipped step 3" shows the error and the hint. Red first, both twins, PG18: native_upgrade_converge gains the window's arms in a database at the previous release's catalog (the newest fixture), and the three refusal arms fail on main with `got [42P01|no-hint]`. test_native_upgrade_converge.py is their partial twin, declared INCOMPLETE; it stages the previous release's base script from test/fixtures as the shell does. Green on 15-19 in both twins. cluster_tests 660 -> 661 by collection; TESTS.md gets section 143. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… still be set to a row store (#1341) test_fk_referencing.py asserts every shell check by name; compare_to_bash grades it as covering all 28 (27 checks on PG17+, and the pre-17 branch's own), and it joins COMPLETE. The conversion guard's over-rejection control converted a table with no foreign key back to heap, so it could not see a guard that refused every SET ACCESS METHOD on a referenced table whatever the target: with the guard's columnar-target test removed, the suite stayed 26/26. A new arm sets the referenced table itself to heap; under that mutant both twins now fail it. cluster_tests 661 -> 666 by collection; TESTS.md gets section 144. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tations of the foreign-key guards (#1341) 28 rows from five-major runs (27/27 on 17-19, 26/26 on 15-16). Dated on PG18 by: the referenced-side refusal removed (5), the guard keyed on the referencing table (8), the conversion refusal removed (8), and the conversion refused whatever the target (1). suites_not_covered 219 -> 218; the census is 3647 rows, 2686 never, 961 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_truncate_cleanup.py asserts every shell check by name; compare_to_bash grades it as covering all 25, and it joins COMPLETE. It runs in a private database, since the snapshot counts every pgcolumnar catalog row and the shared one holds every other test's. The shell suite needed no change: each of its arms already goes red under a mutation of the path it names. cluster_tests 666 -> 672 by collection; TESTS.md gets section 145. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mutations of the truncate paths (#1341) 25 rows from five-major runs (25/25 on each of 15-19). Dated on PG18 by a rewrite that keeps the retired storage's rows (7) or its stale write state (1), an in-place truncate that skips the projections' storages (8), and a storage-tree delete that skips them (1). suites_not_covered 218 -> 217; the census is 3672 rows, 2695 never, 977 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…zone-map answer that ignores deletes (#1341) test_native_dml.py asserts every shell check by name; compare_to_bash grades it as covering all 15, and it joins COMPLETE. The min/max arm could not fail as its comment claimed. The deletes (id % 7 = 0) removed neither the smallest nor the largest id, so min and max were the same whether or not the deleted rows counted; and measured with groups that hold deletes left on the zone-map path, min/max(id) still came out right while min/max(k) came out wrong (10|50000 for 20|49990). The suite stayed 13/13. The deletes now include ids 1 and 5000, a premise asserts on heap that min and max moved, and the arm takes min/max over k as well as id; under that mutant both twins fail it. The index arms gain a premise that the lookups take the primary-key index, so they test the index fetch's delete check rather than the scan. cluster_tests 672 -> 673 by collection; TESTS.md gets section 146. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ons (#1341) 15 rows from five-major runs (15/15 on each of 15-19). Dated on PG18 by: a row group with deletes still answering min/max from zone maps (1), count(*) from metadata not subtracting deleted rows (3), and the index fetch ignoring the delete vector (5). suites_not_covered 217 -> 216; the census is 3687 rows, 2702 never, 985 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…longer pass its survival arm (#1341) test_native_dict_underfill.py asserts every shell check by name; compare_to_bash grades it as covering all 4, and it joins COMPLETE. It turns the post-codec encoding choice off for its own session, where the shell sets it for its database. The survival arm was a fresh SELECT 1, which a crashed backend passes once crash recovery has finished, and the suite computed a crash flag it never checked. It now counts "terminated by signal" in the server log. The rejection arm matched 'does not match|corrupt|invalid|out of' in any message, which "out of memory" satisfies too; it now requires XX001. A premise asserts the inflation rewrote the chunk's descriptor. cluster_tests 673 -> 674 by collection; TESTS.md gets section 147. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 rows from five-major runs (4/4 on each of 15-19). Dated on PG18 by the DICT fill check removed (1) and replaced by a crash (2, one shared). suites_not_covered 216 -> 215; the census is 3691 rows, 2704 never, 987 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers can see a double count and a deleted row (#1341) test_native_decode_gating.py asserts every shell check by name; compare_to_bash grades it as covering all 29, and it joins COMPLETE. It sets the width gate off for its session, where the shell pins it off for its cluster. Two of the filter counters' guarantees could not fail: - every row of the needle's surviving vector matched, so the producer's re-test of that vector rejected nothing, and a producer counting its rejections a second time read exactly right: with the non-counting filter disabled the suite stayed 26/26. A qual that half of vector 0 passes now asserts its rejections are counted once. - the counters were checked only on the table without deletes, so with the evaluator's deleted-row test removed the suite stayed 26/26. Rows Removed by Filter is now checked on the multi-group table with deletes, against heap's live count. The multi-group fixture's DELETE ran through the gating it was built to test (its WHERE is an unprunable qual too), so a gating mutant broke the fixture rather than the arms. It now runs with the width gate out of reach, and its premise counts the surviving rows with count(*) instead of a gated WHERE. cluster_tests 674 -> 676 by collection; TESTS.md gets section 148. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our mutations of phase 2 (#1341) 29 rows from five-major runs (29/29 on each of 15-19). Dated on PG18 by: the per-vector evaluation never run (4), a deleted row counted as filtered (1), every vector skipped (7), and a double count in the producer (2). No premise is dated: the fixture no longer depends on the gating under test. suites_not_covered 215 -> 214; the census is 3720 rows, 2722 never, 998 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d settings are proven absent by SQLSTATE (#1341) test_native_decode_gate_width.py asserts every shell check by name; compare_to_bash grades it as covering all 35, and it joins COMPLETE. It holds the threshold for its session, where the shell sets it per database because every helper opens a new connection. The two premises that the rejected probe settings were not shipped compared SHOW's output with the empty string: q returns empty for a dead connection too, and an empty expectation is the shape the pytest layer refuses. Both twins now require SHOW to raise 42704 (undefined_object). cluster_tests 676 -> 677 by collection; TESTS.md gets section 149. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y three mutations of the width gate (#1341) 35 rows from five-major runs (35/35 on each of 15-19). Dated on PG18 by gating whatever the width (2), > for >= (1) and a payload count one too high (1, shared). suites_not_covered 214 -> 213; the census is 3755 rows, 2754 never, 1001 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by arms test what they name (#1341) test_pg_dump_roundtrip.py asserts every shell check by name; compare_to_bash grades it as covering all 25 names (40 checks over the two formats), and it joins COMPLETE. It dumps a private source database and restores it into fresh ones. Two arms could not fail as claimed: - "restored index answers a point lookup" disabled seqscan and bitmapscan, but the columnar scan is neither, and with the index dropped from the restored database the lookup still answered 12345: the suite stayed 36/36. A premise now asserts the lookup's plan uses dt_id_idx. - "restored sort_by resolves and applies" checked the row count after vacuum_sorted, which is 20000 whatever it sorted on: with the sort keys replaced by column 1 the suite stayed 36/36. It now counts (host, ts) inversions in scan order, which must be 0, with a premise that the restored table has some before the call. cluster_tests 677 -> 680 by collection; TESTS.md gets section 150. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… mutations (#1341) 40 rows from five-major runs (40/40 on each of 15-19). Dated on PG18 by: options (6) and projection declarations (8) no longer registered with pg_extension_config_dump, rebuild_projections() skipping missing projections (7), drop_projection keeping its declaration (8), and vacuum_sorted on the wrong column (2). suites_not_covered 213 -> 212; the census is 3795 rows, 2778 never, 1017 dated, restated on main db56168. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dex (#1341) The shell's "the index still answers the same range" never touched the index: right after pg_upgrade there are no planner statistics, the planner took the columnar scan for the range even with seqscan off, and the arm passed with c_id dropped after the upgrade. The arm and a new premise now run with the custom scan off, and the premise asserts the plan scans c_id. test_pg_upgrade.py carries all 16 checks by the shell's name. Like the shell it needs two majors: --pg-config is the new one, PGC_UPGRADE_FROM_PG_CONFIG the old one, whose extension it builds from a copy of the tree. The matrix runs one major at a time, so there it declines, listed for 15-19 in expected_unrunnable.txt. cluster_tests 680 -> 681, TESTS.md section 151. The suite has its own check() and records nothing, so no ledger rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd names a refused step (#1341) jdatcmd, reviewing #1461: under root, initdb refused ("cannot be run as root"), _sh discarded the code, and the failure surfaced one step later as a FileNotFoundError for postgresql.conf. initdb, pg_ctl and pg_upgrade now go through _server_tool, which drops to postgres with runuser when euid is 0 (as pgc_cluster._asroot and lib.sh do), chowns the data directories, and raises naming the step and its stderr. initdb passes -U postgres, and the socket directory is the datadir, as pgc_cluster pins it. A failed make of the old major's extension now shows its stderr too. Measured as root (sudo, PG17 -> PG18): 585369d fails with that FileNotFoundError, this commit is 16/16. As a normal user: 16/16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ms replay a write (#1341) Measured: compact on r retired nothing, because vacuum had already reclaimed every deleted row, so "compact replays identically" compared a state compact never changed. The fixture now gives compact one fully deleted row group, and a new arm asserts it was retired (1|1). "vacuum rewrote r into a new relfilenode on the primary" does the same for vacuum, and a premise pins r2's second batch, without which the module-less replay arms passed on a failed INSERT. Red first: with vacuum returning before it rewrites, the old shell is 41/41 green and the new one reddens the vacuum arm. With compact retiring nothing, the new compact arm is red. test_replication.py carries all 44 checks by the shell's name, on a private primary, a streaming standby and an isolated restore. COMPLETE gains replication, TESTS.md section 152, cluster_tests 681 -> 682. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions (#1341) The suite had no rows. All 44 checks enter from five-major runs (44/44 on each of 15-19). Dated by: no SMGR truncate record, data and gap pages written without WAL, vacuum returning early, compact retiring nothing. suites_not_covered 212 -> 211; census 3839 rows / 2811 never / 1028 dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7e0d271 to
3f27e47
Compare
jdatcmd
left a comment
There was a problem hiding this comment.
Re-approving 3f27e47, the restate onto main db56168. I compared every commit's patch with the approved stack using git range-diff. The test and source commits are unchanged, and the ledger-seed commits changed only by re-anchoring on #1470. The census at this head matches its budget paragraph, and the counts step consistently. At #1467's head: the guards pass (438), 687 collected (matching the file), and harness_selftest is 1245/1245.
Stacked on #1461 (merge after it). Two commits:
0d4702b3(the twin and the shell fixes) and7e0d2714(the ledger seed). This is the last suite of #1341 item 129, after pg_dump_roundtrip (#1458) and pg_upgrade (#1461).What was wrong
r, vacuum had already reclaimed every deleted row, so compact returned 0 and removed no row group. The fixture now gives compact one fully deleted row group, from an INSERT of its own followed by a DELETE. A new arm, "compact retired the fully deleted row group on the primary", requiresretired|row groups lost=1|1.Red first (PG18, both shells, then the twin)
pgcolumnar_vacuumreturns before rewritingretired = 0)XLOG_SMGR_TRUNCATErecordstandby 753664)could not read blocks)The twin stops at its first red, so under the last two mutants it shows one arm where the shell shows several. The ledger is dated from the shell's logs.
The twin
test_replication.pycarries all 44 checks by the shell's name (compare_to_bash: 44 literal, 0 missing). It runs on three private clusters, built withpgc_cluster.Clusterandpick_portslots 20, 21 and 22:pg_basebackup -R -C -Sstreaming standby;Like the shell, the twin restarts the standby without the module and then with it. Every comparison on the standby side goes through
query_error, so a failed read can't compare equal to anything. It's one test function, because the scenario is sequential.Runs
Head is 44/44 in both twins on each of 15, 16, 17, 18 and 19. The twin is also 44/44 run as root (sudo, PG18):
pgc_cluster.Clusteralready drops to postgres for initdb and pg_ctl, and pg_basebackup is a client tool.Ledger
count(*)answers from the catalog;suites_not_covered212 → 211. The census is 3839 / 2812 / 1027.pgc_ledger.py gatereturns rc 0 against origin/main (ceiling 219 → 211).Counts and guards
cluster_tests681 → 682, and pytest collects 682.COMPLETEgainsreplication, and TESTS.md gains §152.harness_selftest.shgives 1245, anddocs_style.shpasses.🤖 Generated with Claude Code