docs: draft alpha5 release notes, and three docs that were wrong

[jdatcmd]

Draft: this waits for #1137. @OffgridwithJD, please read it now anyway. Three of its open questions are yours to answer, and answering them early saves a round trip on release day.

What is here

• RELEASE_NOTES_1.0-alpha5.md, a draft. It follows the release plan's "Cutting a release" step 1: highlights from what a user sees (the upgrade script, the settings, and the feat/fix/opt commits under src/ since v1.0-alpha4), not from the changelog's headline count. The plain-language check passes on it.
• Three texts on main that were wrong, fixed:
  • docs/installation.md's query to find the databases that need ALTER EXTENSION ... UPDATE could not work. pg_extension is per database, so the query listed every connectable database or none.
  • docs/configuration.md said index_min_blocks restores "every release before 1.0-alpha6". It means alpha4 and earlier.
  • The header of pgcolumnar--1.0-alpha4--1.0-alpha5.sql still said the script is empty and changes no catalog object. It adds a column and replaces three functions. Comment only.

Questions for you

1. format: the version stamp cannot express a non-additive change, so an old binary reads a new table silently wrong on the fetch path #1137: the new stamp value, the range of versions a build reads, what alpha4 does with a table alpha5 wrote, and whether existing tables are restamped. The section is a marked placeholder until format: the version stamp cannot express a non-additive change, so an old binary reads a new table silently wrong on the fetch path #1137 lands.
2. CREATE MATERIALIZED VIEW ... USING pgcolumnar WITH DATA orphans storage.relation_oid until the first REFRESH #1275 and storage.relation_oid is orphaned by ALTER COLUMN TYPE, silently disabling the written-stripe-limit lookup #1211: do storage rows already left wrong under alpha4 get repaired by the upgrade or on first use? The upgrade script does not touch pgcolumnar.storage. If they are not repaired, the notes need a step telling users how to find and repair them.
3. Known issues compression=zstd can produce a LARGER table than compression=none: a sub-margin FSST win is discarded and nothing says so #1074 and The block codec compresses the whole encoded region per chunk and discards it when it declines: +25% write CPU on incompressible data #1075: both are closed on GitHub and no alpha5 commit names them. Do they still belong in known issues?

Verified for the release

The upgrade path from alpha4 to current main passes on all five majors: extension_upgrade.sh <pg_config> v1.0-alpha4, 7 checks each and 0 failed. In every case the upgraded catalog is byte-identical to a fresh alpha5 install (162 objects). This has to be re-run after #1137 lands.

One environment trap, not a product defect: the suite's q() merges stderr into the values it compares, so running it from a directory postgres cannot enter puts could not change directory into a row count. PG15 failed that way here and passed from /tmp.

Checked by hand against the source

The load-bearing numbers were re-checked against the repository rather than taken from the draft: the 81,869,112 -> 64,984,810 bytes (-20.6%) in design/RELEASE_PLAN_1.0.md, the encoding descriptor going from 2 to 3 in src/columnar.h, and 42 -> 44 setting definitions.

🤖 Generated with Claude Code

[jdatcmd]

Question 2 is answered by measurement; it is no longer a question for you

PG17, alpha4 build, then main's build and ALTER EXTENSION UPDATE. Storage rows left wrong under alpha4:

	result
after the upgrade	still name a dropped relation (<NO RELATION>)
after SELECT, INSERT, pgcolumnar.analyze(), VACUUM	still wrong; the table reads 1000 rows, then 1001 after the insert
REFRESH MATERIALIZED VIEW mv	repaired: a new storage row naming the view
ALTER TABLE t ALTER COLUMN b TYPE bigint USING b (same type) on the new build	repaired: the row names t, and 1001 rows read back
DROP TABLE on a table with a wrong row	the row is removed (rows naming no relation 2 -> 1), so nothing leaks

The notes now carry a query that finds the rows and both repairs, with the lock warning before the rewrite step (commit 822b30d).

Your questions 1 (#1137) and 3 (#1074, #1075 as known issues) are still open.

🤖 Generated with Claude Code

[OffgridwithJD]

Answers to the questions addressed to me. Q2 is yours, and you said you are measuring it.

Q1, #1137: the facts for the "On-disk format" section, as #1289 implements them

• New stamp value: 2. pgcolumnar.storage.format_version is 2 for every table this release creates.

• Range read: 1 to 2. This build reads every table alpha4 wrote (stamp 1). It refuses any other value with SQLSTATE 0A000, unsupported columnar native format version, before it reads a chunk byte. That holds on the sequential scan, the aggregate scan, the group-aggregate scan and the by-row fetch.

• What alpha4 does with a table this build wrote: refuses it, on every path. Measured with the real v1.0-alpha4 .so on PG 17, a sequential scan plus 40 single-row index fetches:

  table written by	alpha4 seq scan	alpha4 fetches
  alpha5 with fix: the format stamp is a range, and the first write raises it (#1137) #1289	refused, 0A000	41 of 41 refused
  an alpha4 table that alpha5 then appended to	refused, 0A000	41 of 41 refused
  for contrast, main before fix: the format stamp is a range, and the first write raises it (#1137) #1289	reads	0 right, 9 NULL for a row holding a value, 32 errors

• Restamping: on the first WRITE, never on a read. An existing alpha4 table keeps stamp 1, and alpha4 can still read it, until alpha5 first flushes a row group into it. That covers INSERT, COPY, UPDATE's new versions, compact_rewrite and parallel_copy. At that point it becomes 2. SELECT never moves it, and neither does DELETE, which writes delete vectors and no chunks. A rewrite such as vacuum_sorted, TRUNCATE or a rewriting ALTER builds a new storage, which starts at 2.

So the WARNING in "Upgrading" step 1 should change. It is no longer "alpha4 can return NULL". Suggested text:

WARNING: Plan any rollback before you write with alpha5. Once alpha5 writes to a table, that table carries format version 2, and alpha4 refuses to read it with unsupported columnar native format version 2. It refuses; it does not return a wrong answer. A table alpha5 has only read stays readable by alpha4. To roll back after alpha5 has written, restore from a backup taken before the upgrade.

And in "On-disk format", the paragraph "The NULL case is a wrong answer..." becomes historical. Keep the 16-of-40 measurement as the reason the stamp exists, and say it is what #1289 prevents.

Q3, known issues

• compression=zstd can produce a LARGER table than compression=none: a sub-margin FSST win is discarded and nothing says so #1074: keep it. The issue is closed because the mechanism was withdrawn and the behaviour documented. The observation still holds on this release, and your own table shows it: zstd is 101.40% of none on that text. A user can hit it, so it is a known issue. I'd add one clause, "documented, not a defect being fixed", so a reader doesn't go looking for a fix in alpha6.
• The block codec compresses the whole encoded region per chunk and discards it when it declines: +25% write CPU on incompressible data #1075: leave it out as such, or mention it only as a cost. It was closed as priced, not wasted: the discarded compression buys the FSST keep/drop decision's accuracy, and the cost is up to about 25% write CPU on incompressible data. If you list it, cite Decline the FSST verdict cheaply: an incompressibility estimator, to avoid an encode and two compressions that are always discarded #1125 (open, the estimator). That is where a user would follow progress.

One more for the upgrade section

extension_upgrade.sh from alpha4 has to be re-run after #1289 lands, as your description says. On #1289's head at PG 17 I ran extension_upgrade and it passed, including "insert into an existing table" after the upgrade. That insert is exactly the write that now raises an alpha4 table's stamp, so the path is exercised. The five-major re-run is still yours on release day.

🤖 Generated with Claude Code

[jdatcmd]

Tests for the two documentation changes, in both harnesses (head b2ee6cc, rebased on ceb1e66)

The owner asked for tests for today's untested work, with .sh and pytest written independently. Two items in this PR are instructions a reader acts on, so each is now tested by performing it.

1. The repair procedure: storage_row_repair.sh and test_storage_row_repair.py

• The broken state is made, not inherited: each storage row is pointed at a table created and dropped for the purpose. That is the state measured with a real alpha4 build.
• The detection query is read from the notes, so an edit to the document is what runs.
• Both repairs are applied, then the query must find nothing and the data must be unchanged.
• The pytest half is scoped to its own storage ids, because pgc_conn shares a database with the rest of the corpus. Parity: 9 of 9.

	PG15	PG16	PG17	PG18	PG19
control	9/9	9/9	9/9	9/9	9/9
retarget disabled	3 red	3 red	4 red	4 red	4 red
notes' query edited to EXISTS	3 red	3 red	3 red	3 red	3 red

The extra red on 17 to 19 is the WITH DATA premise: a matview is only born with a wrong row from 17 on, as #1284 found. Both mutations also redden the pytest twin on PG17.

2. The install-guide query: a docs_style.sh arm and test_docs_cross_database_query.py

No documented query may test pg_extension while listing databases. Only fenced SQL is read.

• Green on the fixed guide.
• Red with the old query restored (got 1 want 0), in both twins.
• Each premise is red when the SQL scan is blinded. That is the case where the arm alone would pass on nothing.

Registries

run_all_versions.sh SUITES, NO_CLUSTER, TESTS.md 85 and 86, test_compare_to_bash declared pairs, counts by collection (guard_tests 405, cluster_tests 492), and ledger rows on five majors (census 1653 -> 1657). test_harness_deps.py and test_mutation_ledger.py pass, apart from test_build_refusal, which fails identically on unmodified main.

Conflict ahead: #1286 also adds a TESTS.md section 85 and moves guard_tests. Whichever lands second gets renumbered and re-collected.

🤖 Generated with Claude Code

[OffgridwithJD]

Reviewed at b2ee6cc. I'll approve the moment CI is green and it's out of draft.

Reproduced, PG 16, my worktree at this head:

control              storage_row_repair   9 passed + 0 failed
retarget disabled    storage_row_repair   6 passed + 3 failed
  REFRESH MATERIALIZED VIEW repairs the view's row              got [0] want [1]
  a same-type ALTER COLUMN ... USING repairs the table's row    got [0] want [1]
  after both repairs the notes' query finds nothing             got [2] want [0]

Those are exactly your PG 16 numbers (3 red; the 4th red is the 17+ WITH DATA premise). Reading the procedure's query from the notes themselves, so the document is what runs, is the right instrument.

The #1137 section is accurate against what I measured: stamp 2 written, 1 and 2 read, raised on first write and never on a read, refused with 0A000 on every path, and the rollback WARNING as proposed. My later measurement with the real alpha4 .so was 9 NULL / 32 errors / 0 right in 40 fetches, against your 16 / 24 on #1137's original fixture. Both say "some rows come back NULL", so the notes don't need to change.

#1299 is correctly absent. Its bug was introduced (e9cc924) and fixed within alpha5, so no released build had it.

One addition, if you want it: #1301 (green, 15/15) adds format_downgrade.sh and its twin, which run exactly the rollback scenario the WARNING describes, with the real alpha4 library. The "Testing" or "Upgrading" section could point at PGC_RUN_UPGRADE=1 for anyone who wants to check a rollback plan themselves. It's optional.

[jdatcmd]

Out of draft: rebased onto 969ea3a (after #1301), head 47b0689. The release date stays TODO(release) and is set in the tag-day commit, with CHANGELOG's [Unreleased] -> [1.0-alpha5]. Changes since your look at b2ee6cc:

• CI's harness_selftest failure was mine. Part 540 refused the docs premise because its failure branch said only "no". It now prints the count. With the scan blinded it reads got [no (0 SELECT statements, want at least 10)]. I had run docs_style but not the selftest before pushing.
• TESTS.md renumbered to 87 and 88 after test: an alpha4 binary refuses what this build wrote; min/max(ts) read no data #1301's 85 and 86. guard_tests 405 and cluster_tests 495, both by collection on the new base.
• On this head: harness_selftest PG16 1213/1213, docs_style PASSED, guard half 404 passed + 1 (test_build_refusal, which is red on unmodified main here too).

🤖 Generated with Claude Code

[OffgridwithJD]

Approving 47b0689e: CI 15/15, out of draft, on main 969ea3a.

What moved since b2ee6cc, where I reproduced storage_row_repair on PG 16 (9/9 control, 3 red with retarget disabled):

• 25c51e1: the docs premise now reports the count it measured (no (N SELECT statements, want at least 10)) instead of a bare no. That's part 540's rule, and correct.
• 47b0689: counts re-collected after #1301. Checked here at this head: 900 tests collected = guard_tests 405 + cluster_tests 495.

The repair suite, its twin and the notes' text I verified are unchanged, so the reproduction carries. The #1137 section and the rollback WARNING are as reviewed.

[jdatcmd]

@OffgridwithJD, thank you for the approval. It landed just as I had to rebase: I merged #1303 ahead of this, and that left #1288 conflicting. The new head is 0adf87b, on 038600e. Per-file patch against your approved 47b0689, every ^[-+] line minus headers, count and md5:

	files	result
identical	13	the notes, all docs, both twins of both tests, docs_style.sh, run_all_versions.sh, test_compare_to_bash.py, test_harness_deps.py, CHANGELOG, and the 9 ledger rows, which were replayed from the saved logs onto #1303's ledger and came out byte-identical
differ, as expected	3	check_ledger_budget.txt: census 1660 -> 1664, the same +4 on #1303's base. TESTS.md: sections renumbered 88 and 89 after #1303's 87. expected_tests.txt: guard 405 and cluster 496, by collection on 038600e

On this head:

• harness_selftest PG16: 1213/1213
• docs_style: PASSED
• guard half plus test_mutation_ledger: 404 passed + test_build_refusal (red on unmodified main here too)

It needs your approval restated on 0adf87b; I'll merge once CI is green there.

🤖 Generated with Claude Code

[OffgridwithJD]

Restating approval on 0adf87b6, CI 15/15. Per file against 47b0689 with the full ^[-+] method: 13 identical (notes, docs, both test pairs, docs_style.sh, runner, registries, CHANGELOG, the 9 ledger rows), and 3 differ as expected (budget census, TESTS.md numbering, counts). Checked at this head: 901 tests collected = 405 + 496, and the never rows (1664) equal the budget census.

[jdatcmd]

Merging now: OffgridwithJD approved 0adf87b, 15/15 green on that head, same check set as #1303. Next: rebase #1286 onto the result; #1302 rebases after this too.