Skip to content

test: native_reclaim gets its pytest twin, and its reuse arms can see reuse turned off (#1341 item 70) - #1468

Merged
jdatcmd merged 25 commits into
mainfrom
test/1341-native-reclaim-twin
Oct 16, 2026
Merged

jdatcmd merged 25 commits into
mainfrom
test/1341-native-reclaim-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

Stacked on #1467 (merge after it). Two commits: 4edf4a78 (the twin and the shell fixes) and 5f31e7c2 (the ledger re-dating). This is my guess at item 70's third suite, the one cut off as test/nativ…. It's a twin-less native suite whose arms postdate 2026-09-09, and it sits next to recluster.

What was wrong

With an online compaction never reserving from a freed range (reusedOffset = false && PgColumnarAllocateFreeSpace(...)), the shell stayed 36/36 green. The suite exists to prove that reuse. Measured why:

  • The repeated reclusters did nothing. The four recluster('n', 'id') calls after the first each returned 0: recluster self-gates on a table already in that key's order. They wrote nothing, so the file "plateaued" at 311296 with reuse disabled. They now alternate v and id, and a new arm, "each repeated recluster rewrote the table, so none was skipped as already ordered", requires each call to rewrite. Without reuse the file then grows 311296 → 901120 and the plateau arm goes red.
  • "free space is being reused (bounded, not growing per cycle)" asserted free_space > 0, which is the arm before it again. It now compares with the count after the first cycle and goes red as 1 → 5 rows.

Red first (PG18)

mutant old shell (36) new shell (37) twin
no reservation from a freed range green red: the plateau and free-space-bound arms red: plateau (311296 -> 901120)

Fifteen more mutations redden their own arm in all three. The old shell already saw them, but the ledger had them never:

  • compact_rewrite's guard without isnan, without < 0, without > 1, strict at 0.0, strict at 1.0;
  • maintenance_due with NULL, negative or above-1 accepted for the compact threshold, and the same three for the recluster threshold;
  • maintenance_due refusing the 0.0 or the 1.0 boundary;
  • compact_rewrite_due pinned false, and pinned true.

The twin

test_native_reclaim.py carries all 37 checks by the shell's name in three tests, one per fixture (compare_to_bash: 37 literal, 0 missing). SQLSTATEs are read from exc.sqlstate, so a failure without one can't read as "OK". maintenance_due's eight refusals are a loop table. So the loop-reader guard's list of such files gains native_reclaim: that guard exists so that change shows up as a diff.

Runs

Head is 37/37 in both twins on each of 15, 16, 17, 18 and 19.

Ledger

  • The suite had 36 rows, 2 of them dated. The new arm enters never, and 17 rows are newly dated by sixteen mutations:
    • no reuse (2 arms);
    • compact_rewrite's guard (5);
    • maintenance_due's guard and verdict (10).
  • The max_groups arm was already dated, and its mutant (cap ignored: 10|0|0) still reddens it.
  • Still never:
    • the controls;
    • the data-parity arms. A shifted reuse offset reddened the plateau and cap arms, not the data, so I didn't seed it.
    • maintenance_due's NaN arms. NaN is redundant with > 1.0 under PostgreSQL's float ordering, as the SQL's own comment says.
  • The census is 3901 / 2828 / 1073. suites_not_covered doesn't move (209), and pgc_ledger.py gate returns rc 0.

Counts and guards

  • cluster_tests 687 → 690, and pytest collects 690.
  • COMPLETE gains native_reclaim, and TESTS.md gains §155.
  • The NO_CLUSTER guards pass (438), harness_selftest.sh passes, and docs_style.sh passes.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 5f31e7c.

Measured on PG17, at 5f31e7c: native_reclaim.sh 37/37, and test_native_reclaim.py 37 checks over 3 tests. The tsv is 3901 / 2828 / 1073, as described.

Your reuse claim, run myself: reusedOffset = false && PgColumnarAllocateFreeSpace(...) in columnar_write_state.c.

suite result
main's native_reclaim.sh 36/36 green: the suite that exists to prove reuse couldn't see it off
this PR's 35/37: repeated recluster reuses space (file plateaus), and free space is being reused (bounded, not growing per cycle) got [no (1 -> 5 rows)]
the twin the plateau arm, got 'no (311296 -> 901120)'

Alternating the recluster keys is what makes it measurable: the same-key reclusters returned 0 and wrote nothing. I took the fifteen guard and verdict mutants from your table.

jdatcmd and others added 23 commits October 16, 2026 05:01
…es ALTER EXTENSION UPDATE (#1341 item 66)

Between installing a new library and running ALTER EXTENSION pgcolumnar UPDATE, a
database runs the 1.0-alpha5 library over its 1.0-alpha4 catalog. The library deletes
from pgcolumnar.retired_group, which that catalog lacks, so DROP TABLE, TRUNCATE and
pgcolumnar.vacuum() on a columnar table fail there with 42P01 'columnar metadata
table "pgcolumnar.retired_group" does not exist', which names neither the cause nor
the remedy. Measured: each refusal changes nothing, reads and writes work throughout,
and after the update all three succeed on the same rows.

open_columnar_table's error now carries a hint naming the update.
docs/installation.md's "If you skipped step 3" shows the error and the hint.

Red first, both twins, PG18: native_upgrade_converge gains the window's arms in a
database at the previous release's catalog (the newest fixture), and the three
refusal arms fail on main with `got [42P01|no-hint]`. test_native_upgrade_converge.py
is their partial twin, declared INCOMPLETE; it stages the previous release's base
script from test/fixtures as the shell does. Green on 15-19 in both twins.

cluster_tests 660 -> 661 by collection; TESTS.md gets section 143.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… still be set to a row store (#1341)

test_fk_referencing.py asserts every shell check by name; compare_to_bash grades it as
covering all 28 (27 checks on PG17+, and the pre-17 branch's own), and it joins
COMPLETE.

The conversion guard's over-rejection control converted a table with no foreign key
back to heap, so it could not see a guard that refused every SET ACCESS METHOD on a
referenced table whatever the target: with the guard's columnar-target test removed,
the suite stayed 26/26. A new arm sets the referenced table itself to heap; under that
mutant both twins now fail it.

cluster_tests 661 -> 666 by collection; TESTS.md gets section 144.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tations of the foreign-key guards (#1341)

28 rows from five-major runs (27/27 on 17-19, 26/26 on 15-16). Dated on PG18 by:
the referenced-side refusal removed (5), the guard keyed on the referencing table
(8), the conversion refusal removed (8), and the conversion refused whatever the
target (1). suites_not_covered 219 -> 218; the census is 3647 rows, 2686 never,
961 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_truncate_cleanup.py asserts every shell check by name; compare_to_bash grades it
as covering all 25, and it joins COMPLETE. It runs in a private database, since the
snapshot counts every pgcolumnar catalog row and the shared one holds every other
test's. The shell suite needed no change: each of its arms already goes red under a
mutation of the path it names.

cluster_tests 666 -> 672 by collection; TESTS.md gets section 145.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mutations of the truncate paths (#1341)

25 rows from five-major runs (25/25 on each of 15-19). Dated on PG18 by a rewrite
that keeps the retired storage's rows (7) or its stale write state (1), an in-place
truncate that skips the projections' storages (8), and a storage-tree delete that
skips them (1). suites_not_covered 218 -> 217; the census is 3672 rows, 2695 never,
977 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…zone-map answer that ignores deletes (#1341)

test_native_dml.py asserts every shell check by name; compare_to_bash grades it as
covering all 15, and it joins COMPLETE.

The min/max arm could not fail as its comment claimed. The deletes (id % 7 = 0)
removed neither the smallest nor the largest id, so min and max were the same
whether or not the deleted rows counted; and measured with groups that hold deletes
left on the zone-map path, min/max(id) still came out right while min/max(k) came
out wrong (10|50000 for 20|49990). The suite stayed 13/13. The deletes now include
ids 1 and 5000, a premise asserts on heap that min and max moved, and the arm takes
min/max over k as well as id; under that mutant both twins fail it.

The index arms gain a premise that the lookups take the primary-key index, so they
test the index fetch's delete check rather than the scan.

cluster_tests 672 -> 673 by collection; TESTS.md gets section 146.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ons (#1341)

15 rows from five-major runs (15/15 on each of 15-19). Dated on PG18 by: a row group
with deletes still answering min/max from zone maps (1), count(*) from metadata not
subtracting deleted rows (3), and the index fetch ignoring the delete vector (5).
suites_not_covered 217 -> 216; the census is 3687 rows, 2702 never, 985 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…longer pass its survival arm (#1341)

test_native_dict_underfill.py asserts every shell check by name; compare_to_bash
grades it as covering all 4, and it joins COMPLETE. It turns the post-codec encoding
choice off for its own session, where the shell sets it for its database.

The survival arm was a fresh SELECT 1, which a crashed backend passes once crash
recovery has finished, and the suite computed a crash flag it never checked. It now
counts "terminated by signal" in the server log. The rejection arm matched
'does not match|corrupt|invalid|out of' in any message, which "out of memory"
satisfies too; it now requires XX001. A premise asserts the inflation rewrote the
chunk's descriptor.

cluster_tests 673 -> 674 by collection; TESTS.md gets section 147.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 rows from five-major runs (4/4 on each of 15-19). Dated on PG18 by the DICT fill
check removed (1) and replaced by a crash (2, one shared). suites_not_covered
216 -> 215; the census is 3691 rows, 2704 never, 987 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers can see a double count and a deleted row (#1341)

test_native_decode_gating.py asserts every shell check by name; compare_to_bash
grades it as covering all 29, and it joins COMPLETE. It sets the width gate off for
its session, where the shell pins it off for its cluster.

Two of the filter counters' guarantees could not fail:
- every row of the needle's surviving vector matched, so the producer's re-test of
  that vector rejected nothing, and a producer counting its rejections a second time
  read exactly right: with the non-counting filter disabled the suite stayed 26/26.
  A qual that half of vector 0 passes now asserts its rejections are counted once.
- the counters were checked only on the table without deletes, so with the
  evaluator's deleted-row test removed the suite stayed 26/26. Rows Removed by Filter
  is now checked on the multi-group table with deletes, against heap's live count.

The multi-group fixture's DELETE ran through the gating it was built to test (its
WHERE is an unprunable qual too), so a gating mutant broke the fixture rather than
the arms. It now runs with the width gate out of reach, and its premise counts the
surviving rows with count(*) instead of a gated WHERE.

cluster_tests 674 -> 676 by collection; TESTS.md gets section 148.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our mutations of phase 2 (#1341)

29 rows from five-major runs (29/29 on each of 15-19). Dated on PG18 by: the
per-vector evaluation never run (4), a deleted row counted as filtered (1), every
vector skipped (7), and a double count in the producer (2). No premise is dated: the
fixture no longer depends on the gating under test. suites_not_covered 215 -> 214;
the census is 3720 rows, 2722 never, 998 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d settings are proven absent by SQLSTATE (#1341)

test_native_decode_gate_width.py asserts every shell check by name; compare_to_bash
grades it as covering all 35, and it joins COMPLETE. It holds the threshold for its
session, where the shell sets it per database because every helper opens a new
connection.

The two premises that the rejected probe settings were not shipped compared SHOW's
output with the empty string: q returns empty for a dead connection too, and an empty
expectation is the shape the pytest layer refuses. Both twins now require SHOW to
raise 42704 (undefined_object).

cluster_tests 676 -> 677 by collection; TESTS.md gets section 149.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y three mutations of the width gate (#1341)

35 rows from five-major runs (35/35 on each of 15-19). Dated on PG18 by gating
whatever the width (2), > for >= (1) and a payload count one too high (1, shared).
suites_not_covered 214 -> 213; the census is 3755 rows, 2754 never, 1001 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by arms test what they name (#1341)

test_pg_dump_roundtrip.py asserts every shell check by name; compare_to_bash grades it
as covering all 25 names (40 checks over the two formats), and it joins COMPLETE. It
dumps a private source database and restores it into fresh ones.

Two arms could not fail as claimed:
- "restored index answers a point lookup" disabled seqscan and bitmapscan, but the
  columnar scan is neither, and with the index dropped from the restored database the
  lookup still answered 12345: the suite stayed 36/36. A premise now asserts the
  lookup's plan uses dt_id_idx.
- "restored sort_by resolves and applies" checked the row count after vacuum_sorted,
  which is 20000 whatever it sorted on: with the sort keys replaced by column 1 the
  suite stayed 36/36. It now counts (host, ts) inversions in scan order, which must be
  0, with a premise that the restored table has some before the call.

cluster_tests 677 -> 680 by collection; TESTS.md gets section 150.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… mutations (#1341)

40 rows from five-major runs (40/40 on each of 15-19). Dated on PG18 by: options
(6) and projection declarations (8) no longer registered with
pg_extension_config_dump, rebuild_projections() skipping missing projections (7),
drop_projection keeping its declaration (8), and vacuum_sorted on the wrong column
(2). suites_not_covered 213 -> 212; the census is 3795 rows, 2778 never, 1017 dated, restated on main db56168.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dex (#1341)

The shell's "the index still answers the same range" never touched the index:
right after pg_upgrade there are no planner statistics, the planner took the
columnar scan for the range even with seqscan off, and the arm passed with
c_id dropped after the upgrade. The arm and a new premise now run with the
custom scan off, and the premise asserts the plan scans c_id.

test_pg_upgrade.py carries all 16 checks by the shell's name. Like the shell it
needs two majors: --pg-config is the new one, PGC_UPGRADE_FROM_PG_CONFIG the
old one, whose extension it builds from a copy of the tree. The matrix runs one
major at a time, so there it declines, listed for 15-19 in
expected_unrunnable.txt. cluster_tests 680 -> 681, TESTS.md section 151.

The suite has its own check() and records nothing, so no ledger rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd names a refused step (#1341)

jdatcmd, reviewing #1461: under root, initdb refused ("cannot be run as root"),
_sh discarded the code, and the failure surfaced one step later as a
FileNotFoundError for postgresql.conf. initdb, pg_ctl and pg_upgrade now go
through _server_tool, which drops to postgres with runuser when euid is 0
(as pgc_cluster._asroot and lib.sh do), chowns the data directories, and
raises naming the step and its stderr. initdb passes -U postgres, and the
socket directory is the datadir, as pgc_cluster pins it. A failed make of the
old major's extension now shows its stderr too.

Measured as root (sudo, PG17 -> PG18): 585369d fails with that
FileNotFoundError, this commit is 16/16. As a normal user: 16/16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ms replay a write (#1341)

Measured: compact on r retired nothing, because vacuum had already reclaimed
every deleted row, so "compact replays identically" compared a state compact
never changed. The fixture now gives compact one fully deleted row group, and
a new arm asserts it was retired (1|1). "vacuum rewrote r into a new
relfilenode on the primary" does the same for vacuum, and a premise pins r2's
second batch, without which the module-less replay arms passed on a failed
INSERT.

Red first: with vacuum returning before it rewrites, the old shell is 41/41
green and the new one reddens the vacuum arm. With compact retiring nothing,
the new compact arm is red.

test_replication.py carries all 44 checks by the shell's name, on a private
primary, a streaming standby and an isolated restore. COMPLETE gains
replication, TESTS.md section 152, cluster_tests 681 -> 682.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions (#1341)

The suite had no rows. All 44 checks enter from five-major runs (44/44 on
each of 15-19). Dated by: no SMGR truncate record, data and gap pages
written without WAL, vacuum returning early, compact retiring nothing.
suites_not_covered 212 -> 211; census 3839 rows / 2811 never / 1028 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e recluster's visibility-map rule (#1341 item 70)

Two measured gaps in "index scan returns each row once":
- with the index dropped after the recluster it stayed green, because with
  only seqscan and the custom scan off the count went through the disabled
  seqscan. Bitmap scans are off now too, and "premise: the count takes an
  index-only scan of n_id_uq" asserts the plan.
- with recluster's PgColumnarVMClearForRowRange removed the whole suite
  stayed green: no VM bit was ever set, so every index-only probe fetched
  from the heap. A VACUUM before the recluster sets them (an arm checks the
  pre-recluster count answers from the index), and the old entries, still in
  the index, are then counted twice: 81409 for 40960.

test_native_recluster.py carries all 14 checks by the shell's name.
COMPLETE gains native_recluster, TESTS.md section 153, cluster_tests
682 -> 683.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…utations (#1341 item 70)

The suite had no rows. All 14 checks enter from five-major runs (14/14 on
each of 15-19). Dated by: no index entry for a rewritten row, no old group
retired, the VM left set over the retired row numbers, AccessExclusiveLock,
and a sort key of no columns. suites_not_covered 211 -> 210; census
3853 rows / 2817 never / 1036 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its own reason (#1341 item 70)

Three attribution faults, measured under mutations of the drop hook:
- every cleanup arm compared with the suite's starting snapshot, so one leak
  reddened every arm after it: with the cleanup gated on the columnar AM,
  the never-columnar heap arm went red on aac_t's rows. Each now compares
  with the snapshot taken just before its own table.
- REFRESH MATERIALIZED VIEW reused the CREATE arm's matview and went red as
  'relation "rwm" does not exist'. It now refreshes a matview of its own,
  made WITH NO DATA (a premise), and reddens on the hook's own error.
- "premise: and took pgcolumnar.options with it" went red when the member
  skip was removed, because DROP EXTENSION aborted: a product change, so an
  arm (#1236).

test_alter_am_cleanup.py carries all 47 checks by the shell's name, in four
tests on databases of their own. COMPLETE gains alter_am_cleanup, TESTS.md
section 154, cluster_tests 683 -> 687.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… mutations of the drop hook (#1341 item 70)

The suite had no rows. All 47 checks enter from five-major runs (47/47 on
each of 15-19). suites_not_covered 210 -> 209; census 3900 rows / 2843
never / 1057 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the test/1341-native-reclaim-twin branch from 5f31e7c to ffa62b2 Compare October 16, 2026 05:01
@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, thanks for restating the stack. I'm checking it now, and found one regression in it.

The restate dropped two edits to the loop-reader guard's file list in test_compare_to_bash.py:

I compared each commit's patch before and after the restate, and only those lines differ. At the top of the stack (#1474, 46db379), test_compare_to_bash.py fails:

FAILED test_compare_to_bash.py::test_the_loop_reader_invents_nothing_in_this_corpus
and it is these files, …: got 'build_refusal, differential, join_runtime_filter, native_reclaim, native_repack, native_vacuum_race, sorted_pathkeys' want 'build_refusal, differential, join_runtime_filter, native_vacuum_race, sorted_pathkeys'

So from #1468 up, the guard's CI job will go red. The fix is to restore those two lines in a5764b3 and 2476bfb. Everything else I've compared so far is unchanged apart from the ledger and the counts: #1450's fix commit is patch-identical, and the other changed commits are ledger seeds. I'll finish checking the bookkeeping and re-approve #1450 → #1467 as soon as that's done, since the dropped lines are above them.

… reuse turned off (#1341 item 70)

With an online compaction never reserving from a freed range, the shell
stayed 36/36 green. Measured why:
- the four "repeated" recluster('n', 'id') calls each returned 0: recluster
  self-gates on a table already in that key's order, so they wrote nothing
  and the file plateaued with reuse disabled. They now alternate v and id,
  and a new arm requires each to rewrite. Without reuse the file then grows
  311296 -> 901120, and the plateau arm reddens.
- "free space is being reused (bounded, not growing per cycle)" asserted
  free_space > 0, the arm before it again. It now compares with the count
  after the first cycle, and reddens as 1 -> 5 rows.

test_native_reclaim.py carries all 37 checks by the shell's name in three
tests; maintenance_due's eight refusals are a loop table, so the loop-reader
guard's file list gains native_reclaim. COMPLETE gains native_reclaim,
TESTS.md section 155, cluster_tests 687 -> 690.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd the threshold guards (#1341 item 70)

One new check enters never; 17 of the suite's never rows are dated from
five-major runs (37/37 on each of 15-19). Census 3901 rows / 2828 never /
1073 dated; suites_not_covered unchanged (209).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the test/1341-native-reclaim-twin branch from ffa62b2 to 8c74c45 Compare October 16, 2026 05:22

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving 8c74c45. The upper stack's range-diff against the previous restate (base 62c80a4) shows only the two restored guard-list lines, in fd5c9cb and 7e8d310; every other commit is unchanged. test_compare_to_bash.py passes at the top (5b03158), 40/40, with the list naming native_reclaim and native_repack.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, the prefix #1450 → #1467 has merged. #1468 → #1474 are re-approved on their fixed heads, and I'm merging them in order as each goes green, unless you say hold.

@jdatcmd
jdatcmd merged commit 0053560 into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants