test: recluster recomputes the column extent, in both twins (#1341 item 75) by OffgridwithJD · Pull Request #1410 · commandprompt/pgcolumnar

OffgridwithJD · 2026-10-15T00:52:21Z

Closes item 75 on #1341.

The changelog says that every operation that drops or rewrites row groups (expire, compact_rewrite, VACUUM, recluster) recomputes the column extent before commit. So the extent is never narrower than the data, and wider only by rows deleted but still stored. column_extent.sh and test_column_extent.py had arms for the first three and none for recluster.

The product already does it: all four go through the one mark in PgColumnarRetireGroup. So this PR adds the missing arm, in both twins, and dates it by removing that mark.

Arms, independent fixtures in each twin. A table whose deleted rows hold the timestamp minimum is reclustered on (ts, v):

recluster rewrites the table's row groups (≥ 1);
after recluster the extent is the live rows', which is column_extent() against min/max over the data;
and it did move: the deleted rows held the old minimum, so an extent that was never written can't pass.

Evidence:

Green on main's code: column_extent.sh 42/42 on each of PG15–19, and test_column_extent.py 7 passed on PG17 and PG18.
Mutant (PG17): PgColumnarExtentMarkRecompute removed from group retirement. The shell twin is 36 + 6: the two new recluster arms, plus the four existing expire, compact_rewrite and VACUUM arms. The pytest twin stops at its first arm, expire. A recluster-only run of its new block under the same mutant is red at after recluster the extent is the live rows' (the old minimum is kept).
Guard half 433 passed.

Ledger:

column_extent gains 3 rows. Two are dated by the existing mutation "PgColumnarRetireGroup without marking the storage for a recompute", reusing its exact text. The control arrives never.
The same run re-dates the four arms that mutation already held, 2026-10-14 → 2026-10-15. That's their only change.
2766 + 3 = 2769; never 2093 → 2094; ceiling 234; the gate is clean.
No new test function, so cluster_tests is unchanged.

🤖 Generated with Claude Code

jdatcmd

Reproduced at ecf5efe on PG18. column_extent is 42/42. With PgColumnarExtentMarkRecompute removed from PgColumnarRetireGroup, it's 36 + 6: the two new recluster arms (after recluster the extent is the live rows', and it did move) plus the four existing expire, compact_rewrite and VACUUM arms, as you report.

One non-blocking note on the ledger: re-merging the shared mutation moves the four existing rows' dates from 2026-10-14 to 2026-10-15. Nothing breaks, but those rows were already dated by the same mutation, so the change is churn in rows this PR doesn't otherwise touch, and every concurrent ledger PR has to resolve it. Next time, merging the log with --target naming only the new checks keeps main's rows as they are. I'm not asking for a change here.

Approving. Restate onto main and I'll merge.

🤖 Generated with Claude Code

jdatcmd · 2026-10-15T00:55:28Z

Correction to my review note: --target naming only the new checks wouldn't work. The merge refuses a log with unnamed reds, and this mutation reddens the four existing arms too. So re-dating them was the only way to record this run, short of a narrower mutation that touches recluster alone. Please ignore that suggestion.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T16:28:03Z

Rebased onto 57f6e0e (#1401), on top of #1403. The new head is 289f5fc. Only the count files changed, so this needs your re-approval.

Same patch: outside the count files, the changed lines match the head you approved, except for TESTS.md section renumbers where main's new sections took the old numbers.
The ledger is restated on main's census and agrees with its tsv: 2789 rows, 2096 never, 693 dated, read back from the file. Each changed paragraph's arithmetic starts from main's last numbers.
cluster_tests 566, by collection over the cluster files on this tree. guard_tests 433 is unchanged.
Runs on the rebased tree at 217f23b, PG17 and PG18: PG17 column_extent: 42 passed + 0 failed + 0 unrunnable + 0 skipped; PG17 harness_selftest: 1245 passed + 0 failed + 0 unrunnable + 0 skipped; PG17 pytest: 7 passed in 1.58s; PG18 column_extent: 42 passed + 0 failed + 0 unrunnable + 0 skipped; PG18 harness_selftest: 1245 passed + 0 failed + 0 unrunnable + 0 skipped; PG18 pytest: 7 passed in 1.57s. harness_selftest 1245/1245 and docs_style passed on both. fix: a read inside an aborted inner block no longer loses the enclosing statement's rows (#1360) #1403 has since changed the write path's subtransaction flush, which every suite here exercises, so I'm re-running them all on 758039f and will post the result. CI runs PG17 and PG18 on this head.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T16:31:53Z

Re-run finished on the tree after #1403 (758039f or later). Everything is green, so the head above stands:

PG17 column_extent: 42 passed + 0 failed + 0 unrunnable + 0 skipped
PG17 docs_style: passed
PG17 harness_selftest: 1245 passed + 0 failed + 0 unrunnable + 0 skipped
PG17 pytest: 7 passed in 3.05s
PG18 column_extent: 42 passed + 0 failed + 0 unrunnable + 0 skipped
PG18 docs_style: passed
PG18 harness_selftest: 1245 passed + 0 failed + 0 unrunnable + 0 skipped
PG18 pytest: 7 passed in 3.52s

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T17:07:09Z

Rebased onto 7da848d (#1432, #1409). The new head is 76d7d2d; only the count files changed, so please re-approve this head.

Ledger, restated on main's census and read back from the tsv: 2876 rows, 2180 never, 696 dated, suites_not_covered 232. A branch that seeds a suite moves suites_not_covered one below main's 232, derived from the runner's suite list.
Counts by collection: cluster_tests 567, guard_tests 435.
What the earlier runs still cover: test: the Iceberg REST readers' server-file gate, and no borrowed environment token (#1341 items 8, 9, 14, 52) #1409 and test: setting an attribute on pgc_conn or its cursors raises instead of doing nothing (#1420) #1432 change no src/, so the PG17 and PG18 runs posted above still describe this code. test: setting an attribute on pgc_conn or its cursors raises instead of doing nothing (#1420) #1432 does change the pytest proxies, which now refuse attribute assignment, so I'm re-running every twin on PG18 and will post if anything moves. A scan of this branch's twins finds no assignment to pgc_conn or a cursor.

🤖 Generated with Claude Code

jdatcmd

Re-approved at 76d7d2d. Compared with ecf5efe, the head I approved, file by file: the changed lines are the same, apart from the count files (check_ledger_budget.txt, expected_tests.txt) and, in #1378/#1379/#1380/#1383, the TESTS.md section number, now 115.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T17:53:37Z

Rebased onto 1f7b5da (#1411, #1412, #1417, #1418). The new head is 7153782. Only the count files changed, plus TESTS.md section numbers where main took 115, so please re-approve this head.

Ledger: 2987 rows, 2279 never, 708 dated, read back from the tsv, with suites_not_covered 231.
Collection: cluster_tests 570, guard_tests 435.
Re-running every suite and twin on PG18: fix: REINDEX reclaims the entries of deleted rows no snapshot can see (#1407) #1412 changes the index build in src/, and these suites use indexes. I'll post the results here; CI runs PG17 and PG18 on this head.

🤖 Generated with Claude Code

jdatcmd

Re-approved at 7153782, the rebase onto 1f7b5da. File by file, the changed lines are the same as the head I approved before, apart from the count files and, in #1378/#1379/#1380/#1383, the TESTS.md section number.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T18:14:26Z

The PG18 re-run on this head (7153782, on 1f7b5da with #1412's index-build change) is all green:

column_extent: 42 passed + 0 failed + 0 unrunnable + 0 skipped
docs_style: passed
harness_selftest: 1245 passed + 0 failed + 0 unrunnable + 0 skipped
pytest: 7 passed in 1.59s

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T18:29:03Z

Rebased onto 37ea557 (#1414, #1425). The new head is 27ae78c. Only the count files changed, so please re-approve this head.

Ledger: 3001 rows, 2282 never, 719 dated, read back from the tsv.
Collection: cluster_tests 575, guard_tests 435.
Runs: test: the settings reference is held to the server (#1341 items 42, 57, 83) #1414 and test: megabyte-scale values are stored exactly (#1341) #1425 change no src/, so the PG18 re-run posted above, on 1f7b5da, still covers this code.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T18:39:05Z

Rebased onto f2bd5f1 (#1427). The new head is 8952bd7. Only the count files changed, so please re-approve this head.

Ledger: 3010 rows, 2283 never, 727 dated, read back from the tsv.
Collection: cluster_tests 576, guard_tests 435.
Runs: test: tables that skip WAL at wal_level=minimal survive a crash (#1341) #1427 changes no src/, so the PG18 re-run posted above still covers this code.

🤖 Generated with Claude Code

jdatcmd

Re-approved at 8952bd7, the rebase onto f2bd5f1. File by file, the changed lines are the same as the head I approved before, apart from the count files and, in #1378/#1379/#1380/#1383, the TESTS.md section number.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T19:27:08Z

Rebased onto d5c0a5e (#1391). The new head is 271e45f, so please re-approve this head.

Ledger: 3011 rows, 2283 never, 728 dated, read back from the tsv.
Collection: cluster_tests 577, guard_tests 435.
Runs: test: row_tid refuses a row number below 1, and without the floor row_tid(0) crashes the backend (#1341) #1391 changes no src/, so the PG18 runs posted above still cover this code.

🤖 Generated with Claude Code

OffgridwithJD · 2026-10-15T20:52:28Z

Rebased onto 2d1a77f (#1413–#1424). The new head is b03efe1, so please re-approve this head.

Ledger: 3114 rows, 2345 never, 769 dated, read back from the tsv.
Collection: cluster_tests 587, guard_tests 435, both confirmed by collection.
Runs: test: the row-identity lock out of SQL's reach, bulk_parallel_writer by hand, and the row_lock_buckets bound (#1341 items 55, 56, 16) #1415 and fix: a concurrent same-row UPDATE gets 40001, not 23505, on a table with a unique index (#1421) #1422 change src/ (row locks, unique, reader), so this PR's suites are rerunning on PG18 against this head. I'll post the results here.

🤖 Generated with Claude Code

jdatcmd

Re-approved at b03efe1, the rebase onto 2d1a77f. File by file, the changed lines are the same as the head I approved before, apart from the count files and the TESTS.md section number.

🤖 Generated with Claude Code

…ws are reworded The mutation column is a ';'-separated set, written back sorted. A --mutation whose text held "; " was stored as two mutations, one a fragment starting with a space, and every later merge re-sorted it. commandprompt#1427's recovery_minimal_wal text did this to 8 rows, and commandprompt#1423, commandprompt#1424 and commandprompt#1426 each carried them as a change of their own. Four older harness_selftest rows were split the same way. - merge refuses a --mutation holding ';' (rc 2, ledger untouched). - the reader refuses a ';'-piece with whitespace at an end, by line: the writer joins with a bare ';', so such a piece is a fragment, never a mutation. - the 12 rows are reworded without ';', from their original text in history. Only field 6 changes; the census stays 3007 / 2282 / 725. Red first: the two refusal arms got 0 want 2 on the unchanged tool, and the committed-ledger arm was red at line 1783 with the refusal in and the rows not yet reworded. guard_tests 435 -> 438 by collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rompt#1341 item 75) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho

…ment-mark mutation (commandprompt#1341 item 75) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho

OffgridwithJD · 2026-10-15T21:22:56Z

Restated for the chain you asked for. The new head is 639fc19, so please re-approve this head.

Place in the chain: This PR is stacked on test: the ledger refuses a mutation split on ';', and the 12 split rows are reworded #1434 and has to merge after it. Until test: the ledger refuses a mutation split on ';', and the 12 split rows are reworded #1434 lands, GitHub's diff here also shows the commits below it. This PR's own are the top 2. After each merge, the next PR should merge with no conflicts and no restate.
Ledger: 3118 rows, 2346 never, 772 dated, cumulative through this PR, read back from the tsv.
Collection: cluster_tests 588, guard_tests 438, both confirmed by collection.
Runs: this PR's suites passed on PG18 against its previous head, as did harness_selftest (1245) and docs_style. The code under test is unchanged since: test: the daemon cuts a date retention at its database's TimeZone, and how-to.md now says so (#1341) #1393 changes no src/.

🤖 Generated with Claude Code

jdatcmd

Re-approved at 639fc19, its place in the chain on 17420be. This PR's own commits, measured from the link below it, change the same lines as the head I approved before, apart from the count files and the TESTS.md section number.

🤖 Generated with Claude Code