From 00235caf03d850d49decdac3a1ea8285b3191997 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 18:33:59 +0000 Subject: [PATCH 01/11] docs: draft alpha5 release notes, and three docs that were wrong RELEASE_NOTES_1.0-alpha5.md is a draft. Its highlights come from what a user sees, as the release plan asks: the upgrade script, the settings (42 -> 44 definitions, two added, none removed, no default changed) and the feat/fix/opt commits under src/ since v1.0-alpha4. It holds marked TODO(release) lines for #1137, which has not landed, and for questions only the author of a change can answer. Three repository texts were wrong: - docs/installation.md: the query meant to list databases that need the upgrade tested pg_extension, which is per database, so it listed every connectable database or none. It now lists them and checks each. - docs/configuration.md: index_min_blocks restores what alpha4 and earlier did, not 'every release before 1.0-alpha6'. - pgcolumnar--1.0-alpha4--1.0-alpha5.sql: the header still said the script is empty and changes no catalog object. It adds a column and replaces three functions. Comment only. plain_language_check.py: 16 files ok, the notes included. docs_style.sh on PG17: 55 checks, PASSED. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 + RELEASE_NOTES_1.0-alpha5.md | 333 +++++++++++++++++++++++++ docs/configuration.md | 2 +- docs/installation.md | 13 +- pgcolumnar--1.0-alpha4--1.0-alpha5.sql | 18 +- 5 files changed, 356 insertions(+), 16 deletions(-) create mode 100644 RELEASE_NOTES_1.0-alpha5.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 62517d4d8..0bf09dab2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -134,6 +134,12 @@ true until the next version shipped. against 1000 with one, identically across twelve readings with and without concurrent DDL. With the guard it tests removed, it read 2000 against 1000 and failed in both twins. The cause of the buffer noise is still unknown. +- `docs/installation.md` gave a query to find the databases that need + `ALTER EXTENSION pgcolumnar UPDATE`. It could not work: `pg_extension` lists + only the current database's extensions, so the query listed every connectable + database or none. The guide now lists the databases and checks each one. + `docs/configuration.md` said `index_min_blocks` restores "every release before + 1.0-alpha6"; it means alpha4 and earlier. - `test/pytest/README.md` now names all four registries a new pytest test has to be entered in, and which of them a local run can catch. Adding one test reddens diff --git a/RELEASE_NOTES_1.0-alpha5.md b/RELEASE_NOTES_1.0-alpha5.md new file mode 100644 index 000000000..362e2ce31 --- /dev/null +++ b/RELEASE_NOTES_1.0-alpha5.md @@ -0,0 +1,333 @@ +# pgColumnar 1.0-alpha5 release notes + +Release date: TODO(release) +Previous release: 1.0-alpha4 (2026-09-17) + +pgColumnar is a columnar table access method for PostgreSQL. This is the fifth +alpha. Its theme is encoding, as measured. Tables that this release writes are +smaller: on the ClickBench sample, the same data takes 20.6% fewer bytes than +before this release's two encoding changes. Range columns can now skip chunk +groups on overlap and containment queries. Catalog reads cost less in a database +with many columnar tables. + +This release changes how column chunks are stored. An alpha4 build cannot read +safely what this release writes. Read "On-disk format" before you upgrade. + +This release requires one upgrade command. See "Upgrading" at the end. + +## Highlights + +- **TODO(release): #1137, the on-disk format version check.** See "On-disk + format". This is likely the most important item for anyone who upgrades. +- **Smaller tables.** On ClickBench `hits_0.parquet` (1,000,000 rows, 105 + columns), stored chunk bytes fell from 81,869,112 to 64,984,810. Two changes + produce this. An encoding is kept only when it is smaller after compression. + A chunk that holds no null stores no validity bitmap. +- **Range columns skip on `&&` and `@>`.** A zone map now records the greatest + upper bound of a range column. On a table clustered on the range, an overlap + query read 1 chunk group of 20 instead of all 20. +- **Less catalog work in a large database.** The pgcolumnar catalogs are shared + by every columnar table in a database. Many lookups used to read a whole + catalog. They now use its index once the catalog is large enough. A new + setting, `pgcolumnar.index_min_blocks`, sets that size. +- **Two crashes fixed.** Both involve a partitioned table that names + `pgcolumnar` as its access method. See "Fixed". +- **Columnar materialized views work in more places.** A columnar materialized + view can now hold per-table options. Its catalog entry is correct from the + moment it is created. + +## On-disk format: TODO(release): #1137 + +TODO(release): #1137. Replace this section once #1137 lands. Issue #1137 adds a +format version stamp that this release writes, and a range check on read. Its +purpose is to make an older build refuse a newer table early and loudly on +every read path. State here: + +- the new stamp value +- the range of versions this build reads +- what an alpha4 build does with a table this build wrote +- whether existing tables are restamped + +What is true without #1137: + +- The native data format stays PGCN v1. The metapage version is unchanged. +- Each column chunk carries an encoding descriptor (a small header that records + how the chunk's values are encoded). Its version moves from 2 to 3. This + release reads versions 2 and 3, so it reads every table that alpha4 wrote. No + conversion is needed. +- An alpha4 build cannot read a version 3 chunk safely. A sequential scan fails + with `unrecognized native encoding descriptor`. An index fetch can fail, or it + can return NULL for a row that holds a value. Measured on an alpha4 build: 16 + of 40 single-row fetches returned NULL, and the other 24 raised the error. + +The NULL case is a wrong answer, not an error. That is why downgrading is not +supported. See the warning in "Upgrading". + +## Smaller tables + +### An encoding is kept only when it is smaller after compression (#1132) + +The writer chooses an encoding, such as bit-packing, by comparing uncompressed +sizes. The block codec (`zstd` or `lz4`) then compresses the result. An +encoding can shrink the bytes and still make the compressed chunk larger, +because it hides patterns the codec would have found. + +The writer now compresses both the encoded chunk and the raw chunk and keeps the +smaller. Measured on the ClickBench sample: + +| measure | before | after | +| --- | ---: | ---: | +| stored bytes | 81,869,112 | 78,109,810 (-4.59%) | +| `ClientEventTime` column | 3,530,536 | 1,777,051 (-49.7%) | + +The price is write CPU: 4.40% more backend instructions on the same load. Query +results are unchanged. + +`pgcolumnar.enable_post_codec_encoding_choice` controls this. It is on by +default. Set it to `off` to restore the earlier choice. + +### A chunk with no nulls stores no validity bitmap (#1130) + +A validity bitmap is one bit per row that records which rows are NULL. The +writer stored it for every chunk, uncompressed, even when no row was NULL. It +now leaves the bitmap out of any chunk that holds no null. The decision depends +on the rows written, not on a `NOT NULL` constraint. A nullable column with no +nulls in a chunk also saves the space. + +Measured on the ClickBench sample, which has no nulls: + +| measure | before | after | +| --- | ---: | ---: | +| stored bytes | 78,109,810 | 64,984,810 (-16.80%) | +| validity bitmap bytes | 13,125,000 | 0 | +| chunks without a bitmap | 0 of 735 | 735 of 735 | + +Both changes apply to data this release writes. Data written before the upgrade +keeps its old layout until it is rewritten. TODO(release): verify. + +## Range pruning (#1144) + +A query with `&&` (overlap) or `@>` (contains) on a range column now skips +chunk groups that cannot match. Before this release, both operators read every +chunk group. + +The zone map (the per-group summary the scan uses to skip data) gains the +greatest upper bound of the range column. The minimum and maximum it already +held cannot answer overlap, because a range type sorts by lower bound first. + +Measured on 200,000 rows of `tstzrange` in 20 chunk groups: + +| table | groups read, before | groups read, after | +| --- | ---: | ---: | +| clustered on the range column | 20 | 1 | +| rows in random order | 20 | 20 | + +Clustering is a precondition, not an option. On a table whose ranges are +scattered, every group holds a bound near the maximum, so nothing is skipped. + +Three further points: + +- A range with no upper bound is never skipped from above. +- The scan compares under the collation the range type was declared with. +- Data written before the upgrade has no summary. It is read in full, and never + skipped wrongly, until it is rewritten. + +See [docs/limitations.md](docs/limitations.md), section "Which predicates +prune". + +## Catalog work in a large database + +pgColumnar keeps its metadata in catalogs that every columnar table in a +database shares. Many lookups read a whole catalog to find one table's rows. In +a database with many columnar tables, each lookup paid for every other table. +This release routes those lookups through each catalog's existing index. No +index is added and no catalog changes. + +An index probe costs more than reading a catalog of one or two pages. So a +lookup uses the index only when the catalog has at least +`pgcolumnar.index_min_blocks` pages. The default is 3. It was chosen from +measurements at 24 catalog sizes on PostgreSQL 15, 17 and 19. + +Measured on PostgreSQL 17 with 400 other columnar tables that have projections, +in catalog buffers read: + +| operation | catalog | before | after | +| --- | --- | ---: | ---: | +| one-row `DELETE` | `row_group` | 45 | 19 | +| `INSERT` | `storage` | 19 | 13 | +| `pgcolumnar.vacuum_sorted` | `storage` | 128 | 93 | +| `pgcolumnar.truncate` | `free_space` | 26 | 17 | + +Other paths that changed: + +- **Planning.** Planning a query read `pgcolumnar.storage` in full for the + newest table. It now reads the table's own row through its metapage. Measured + on a 37-page catalog: 37 blocks per planned query before, 4 after. +- **Retiring a row group.** Compaction made 7 sequential catalog scans per + retired row group, plus 3. It now uses the catalogs' indexes instead, once + they reach `pgcolumnar.index_min_blocks` pages. +- **`DELETE`.** The session that ran a `DELETE` scanned `delete_vector` in full + once per row group it touched. It now uses the index. +- **Empty catalogs.** `pgcolumnar.options` and `pgcolumnar.projection` are empty + until you set an option or add a projection. Planning no longer probes their + indexes while they are empty or small. + +## Covering projections and the planner + +A covering projection is a sorted copy of some of a table's columns, which a +query can scan instead of the table. These changes affect which plan the planner +chooses. They do not change query results. + +- **Parallel scan of a covering projection.** The planner can now scan a covering + projection in parallel. Before, a plan had to give up either the projection or + the parallel workers. +- **A projection scan is priced from its own pages.** The planner used the size + of the whole relation file. That file holds the base table and every + projection. +- **A base scan is no longer priced from its projections' pages**, for the same + reason. +- **A projection is priced from what its sort order can skip.** The cost used to + assume half the table for every query. It now follows the selectivity of + conditions on the sort key. A condition that mentions the sort key but cannot + use it to skip, such as `sk BETWEEN 1 AND 2000 OR kind = 'odd'`, earns no + discount. + +## Retention + +`pgcolumnar.expire` now accepts a `date` retention column. Before, it accepted +only `timestamp` and `timestamptz`. + +For a `date` column, the cutoff is rounded down to the start of its day. As a +result, `expire` can keep a row up to one day older than the retention window. +It never removes a row that is inside the window. + +## Settings + +Two settings are added. None is removed, and no default changes. + +| setting | type | default | purpose | +| --- | --- | --- | --- | +| `pgcolumnar.enable_post_codec_encoding_choice` | boolean | `on` | Keep an encoding only when it is smaller after block compression. `off` restores the earlier choice. | +| `pgcolumnar.index_min_blocks` | integer | `3` | Page count at or above which a catalog lookup uses the catalog's index. `0` always uses the index. A very large value never does. | + +Both can be set per session. See [docs/configuration.md](docs/configuration.md). + +## Fixed + +- **A partitioned table that names `pgcolumnar` could abort the backend** + (#1259). From PostgreSQL 17, a partitioned table can carry + `USING pgcolumnar` as the default for its partitions. It has no storage of its + own. Functions such as `pgcolumnar.stats` and `pgcolumnar.sort_status` treated + it as a columnar table. An assert-enabled build aborted. A production build + returned a wrong answer instead. These functions now treat the partitioned + parent as not columnar. +- **`pgcolumnar.import_parquet` crashed the server on a partitioned parent** + (#1263). It now refuses a target without storage. The crash was a + segmentation fault on production builds too. Importing into a heap table still + works. +- **`CREATE MATERIALIZED VIEW ... USING pgcolumnar ... WITH DATA` left a wrong + catalog entry** (#1275). The view's storage row named a relation that no + longer existed until the first `REFRESH`. It now names the view. +- **`ALTER TABLE ... ALTER COLUMN TYPE` left a wrong catalog entry** (#1211). + After the rewrite, the table's storage row named the dropped temporary + relation. The rows stayed readable. The storage row now names the table. +- **`pgcolumnar.analyze()` could skip columns without saying so** (#1276). On a + table with a covering projection, it could resolve the projection's storage + instead of the table's. Measured on a five-column table with a one-column + projection: statistics for 2 columns instead of 5, and a successful return. +- **A columnar materialized view can hold per-table options** (#1265). + `pgcolumnar.set_options` and `pgcolumnar.reset_options` now accept one, and + `DROP MATERIALIZED VIEW` removes its options. +- **`pgcolumnar.reset_options` reported success on a non-columnar relation** + (#1265). It accepted a heap table, a partitioned table and a view, while + `set_options` refused them. It now refuses the same relations, with SQLSTATE + `42809`. + +## Known issues + +- TODO(release): verify whether the alpha4 known issues still apply. Both are + closed on GitHub, and no alpha5 commit under `src/` names either: + - #1074: a codec setting can make a high-entropy text table larger. + - #1075: the block codec compresses a region it then discards. + +## Upgrading + +### Before you start + +- You need: + - the alpha5 build + - the same `PG_CONFIG` you installed alpha4 with + - permission to restart the server + - the right to run `ALTER EXTENSION` in each database that has the extension +- List the databases you can connect to. You will visit each one: + + ```sql + SELECT datname FROM pg_database WHERE datallowconn; + ``` + +### Steps + +1. Run `make install` with the same `PG_CONFIG`. + + **WARNING:** You cannot safely go back to alpha4 after the next step. Once + the server loads the alpha5 library, the chunks it writes use encoding + descriptor version 3. An alpha4 build can return NULL for a row that holds a + value in such a chunk. If you may need to roll back, take a backup first. + TODO(release): confirm after #1137 lands. + +2. Restart the server, so that it loads the new library. +3. Connect to one database from the list. +4. Check whether the extension is installed there: + + ```sql + SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar'; + ``` + + If the query returns no row, skip to step 7. + +5. Run: + + ```sql + ALTER EXTENSION pgcolumnar UPDATE; + ``` + +6. Run the query from step 4 again. + + Expected result: `1.0-alpha5`. + +7. Repeat steps 3 to 6 for every other database in the list. + +### What the upgrade changes + +- It adds the column `max_upper` (type `bytea`) to `pgcolumnar.zone_map`. + Existing rows get NULL, which means "no summary". +- It replaces `pgcolumnar.set_options`. The new version also accepts a columnar + materialized view. +- It replaces `pgcolumnar.reset_options`. The new version refuses any relation + that `set_options` refuses. A script that called it on a heap table, a + partitioned table or a view used to succeed. It now raises an error. +- It replaces `pgcolumnar."analyze"`. The new version finds a table's storage + through the table's metapage. +- No table data is converted. TODO(release): confirm after #1137 lands, and + state whether existing tables keep their current format stamp. +- TODO(release): verify whether a storage row left wrong by `ALTER COLUMN TYPE` + (#1211) or by `CREATE MATERIALIZED VIEW ... WITH DATA` (#1275) under alpha4 is + repaired by the upgrade. The upgrade script does not touch + `pgcolumnar.storage`. + +See [docs/installation.md](docs/installation.md) for the full upgrade procedure. + +## Scope and limitations + +- This is an alpha. Interfaces may change before 1.0. +- Cascading encodings (one encoding applied on top of another) are not in this + release. Issue #1139 tracks them until a measurement shows a size win. +- `pgcolumnar.vacuum_full` still skips columnar materialized views. +- A GIN or BRIN index builds on a columnar table, but the planner can never use + it. Both need a bitmap scan, which the columnar access method does not + provide. See [docs/limitations.md](docs/limitations.md). +- On PGXN this release is `1.0.0-alpha.5`, while `CREATE EXTENSION` reports + `1.0-alpha5`. PGXN requires a semantic version, which needs three integer + components. The two names refer to the same release. + +The complete, itemized list of changes is in `CHANGELOG.md`. diff --git a/docs/configuration.md b/docs/configuration.md index 5c1e9abc1..278542818 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -83,7 +83,7 @@ disk. It never changes the values that a table returns. | Setting | Type | Default | Description | | --- | --- | --- | --- | -| `pgcolumnar.index_min_blocks` | integer | `3` | Heap pages at or above which a keyed read of a `pgcolumnar` metadata catalog probes that catalog's index instead of reading it sequentially. These catalogs are shared by every columnar table in the database, so the same catalog is one page in a small database and hundreds in a large one, and below a few pages a sequential read is cheaper than a probe. The default is measured rather than chosen, and it is a default rather than a truth: the best value differs per catalog, because the hot catalogs are read more often per retired row group. Change it only to undo a regression you have measured. A very large value restores the behaviour of every release before 1.0-alpha6, which always read sequentially; `0` always probes. Range 0 to INT_MAX. | +| `pgcolumnar.index_min_blocks` | integer | `3` | Heap pages at or above which a keyed read of a `pgcolumnar` metadata catalog probes that catalog's index instead of reading it sequentially. These catalogs are shared by every columnar table in the database, so the same catalog is one page in a small database and hundreds in a large one, and below a few pages a sequential read is cheaper than a probe. The default is measured rather than chosen, and it is a default rather than a truth: the best value differs per catalog, because the hot catalogs are read more often per retired row group. Change it only to undo a regression you have measured. A very large value makes every converted lookup read its catalog sequentially, as 1.0-alpha4 and earlier did for most of them; `0` always probes. Range 0 to INT_MAX. | | `pgcolumnar.reclaim_coalesce` | boolean | `on` | During online compaction, split an oversized freed range on reuse and coalesce adjacent freed ranges, so space is reclaimed under fragmentation. Off reverts to whole-range reuse. | | `pgcolumnar.enable_end_truncation` | boolean | `off` | Allow `pgcolumnar.truncate()` to return trailing reclaimed blocks to the operating system. Off makes `pgcolumnar.truncate()` a no-op. Requires superuser to set. | | `pgcolumnar.autovacuum` | boolean | `off` | Run the maintenance daemon. When on, it runs `compact_rewrite` and `recluster` on columnar tables that cross a threshold. It uses only `ShareUpdateExclusiveLock` and yields to any stronger lock. It never blocks a reader or a writer. See the [administration guide](administration.md#the-maintenance-daemon-pgcolumnarautovacuum). Reloadable, not a per-session setting. | diff --git a/docs/installation.md b/docs/installation.md index e25b69889..2618c47bf 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -123,11 +123,18 @@ ALTER EXTENSION pgcolumnar UPDATE; SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar'; ``` -To find the databases that need it: +To find the databases that need it, list the databases you can connect to: ```sql -SELECT datname FROM pg_database WHERE datallowconn - AND EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcolumnar'); +SELECT datname FROM pg_database WHERE datallowconn; +``` + +Then connect to each one and check whether it has the extension. A query cannot +do this from one database, because `pg_extension` lists only the extensions of +the database you are connected to: + +```sql +SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar'; ``` ### If you skipped step 3 diff --git a/pgcolumnar--1.0-alpha4--1.0-alpha5.sql b/pgcolumnar--1.0-alpha4--1.0-alpha5.sql index 3a815e849..88cf8e1cb 100644 --- a/pgcolumnar--1.0-alpha4--1.0-alpha5.sql +++ b/pgcolumnar--1.0-alpha4--1.0-alpha5.sql @@ -9,18 +9,12 @@ * alpha4 install would never run it and a fresh install would silently get a * different catalog from an upgraded one. * - * EMPTY ON PURPOSE, AND THAT IS WHY IT EXISTS. This change opens the cycle and - * nothing else. The file is here so that the work already pointed at alpha5 -- - * the zone-map summary that overlap and containment pruning needs (#1144), and - * cascading step 2 (#1139) -- appends to a script that already exists rather - * than racing to create it, and so that the version bump is reviewable on its - * own rather than inside a feature diff. - * - * PostgreSQL requires the file to exist and to be readable; an upgrade script - * that adds nothing is a valid one. `ALTER EXTENSION pgcolumnar UPDATE` from - * alpha4 reaches alpha5 through this file and changes no catalog object, which - * is what test/extension_upgrade.sh asserts by comparing the upgraded catalog - * against a fresh install of the new version. + * IT WAS CREATED EMPTY to open the cycle, so that the work already pointed at + * alpha5 appended to a script that existed rather than racing to create it. It + * is no longer empty. It now adds `pgcolumnar.zone_map.max_upper` for range + * pruning (#1144), and replaces `reset_options`, `set_options` and `"analyze"` + * (#1265, #1276). test/extension_upgrade.sh asserts that the upgraded catalog is + * byte-identical to a fresh install of the new version. */ -- The greatest upper bound of a RANGE column's values in a zone map unit From e007c84a44dc821a62aa76cfa7931a0830e2fe81 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 18:38:27 +0000 Subject: [PATCH 02/11] docs: alpha5 notes say how to repair the catalog rows alpha4 left wrong Measured on PostgreSQL 17, alpha4 build upgraded to main: the storage rows #1211 and #1275 left naming a dropped relation survive ALTER EXTENSION UPDATE, reads, an insert, pgcolumnar.analyze() and VACUUM. The table stays readable (1000 rows, then 1001 after an insert), and DROP TABLE still removes its row (rows naming no relation 2 -> 1). REFRESH MATERIALIZED VIEW writes a correct row for the view. For a table, a rewrite on the new build fixes it: ALTER COLUMN ... TYPE to the same type with USING. Both measured. The notes now give a query to find the rows and the two repairs, with the lock warning before its step. Co-Authored-By: Claude Opus 5.5 --- RELEASE_NOTES_1.0-alpha5.md | 48 +++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-) diff --git a/RELEASE_NOTES_1.0-alpha5.md b/RELEASE_NOTES_1.0-alpha5.md index 362e2ce31..7f27b78d9 100644 --- a/RELEASE_NOTES_1.0-alpha5.md +++ b/RELEASE_NOTES_1.0-alpha5.md @@ -310,10 +310,50 @@ Both can be set per session. See [docs/configuration.md](docs/configuration.md). through the table's metapage. - No table data is converted. TODO(release): confirm after #1137 lands, and state whether existing tables keep their current format stamp. -- TODO(release): verify whether a storage row left wrong by `ALTER COLUMN TYPE` - (#1211) or by `CREATE MATERIALIZED VIEW ... WITH DATA` (#1275) under alpha4 is - repaired by the upgrade. The upgrade script does not touch - `pgcolumnar.storage`. +- It does not repair a catalog entry that alpha4 left wrong. See "After the + upgrade: repair entries alpha4 left wrong". + +### After the upgrade: repair entries alpha4 left wrong + +Two alpha4 defects, fixed in this release (#1211, #1275), left a table's storage +row naming a relation that no longer exists. The upgrade does not repair rows +that already exist. The table still reads and writes correctly, and `DROP TABLE` +still removes the row. Repair them anyway, so the catalog describes your tables. + +This was measured on PostgreSQL 17, on tables created under alpha4. After the +upgrade, the rows stayed wrong through reads, inserts, `pgcolumnar.analyze()` +and `VACUUM`. Each repair below made them correct. + +1. In each database, find the rows that name no relation: + + ```sql + SELECT s.storage_id, s.relation_oid + FROM pgcolumnar.storage s + WHERE NOT EXISTS (SELECT 1 FROM pg_class c WHERE c.oid = s.relation_oid); + ``` + + If the query returns no row, this database needs nothing. + +2. For each columnar materialized view, run: + + ```sql + REFRESH MATERIALIZED VIEW view_name; + ``` + + **CAUTION:** the next step takes an `ACCESS EXCLUSIVE` lock and rewrites the + whole table. Run it when the table can be locked for that long. + +3. For each columnar table that ran `ALTER TABLE ... ALTER COLUMN ... TYPE` + under alpha4, rewrite it once. Converting a column to its own type with + `USING` forces the rewrite and changes no data: + + ```sql + ALTER TABLE table_name ALTER COLUMN column_name TYPE column_type USING column_name; + ``` + +4. Run the query from step 1 again. + + Expected result: no row. See [docs/installation.md](docs/installation.md) for the full upgrade procedure. From b6170007eb871670c037a02f37c9702a47808526 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 18:39:39 +0000 Subject: [PATCH 03/11] docs: alpha5 known issues, measured on this release #1074 still holds on alpha5: 200,000 rows of random 64-character hex text are 7,036,928 bytes under none and lz4, and 7,135,232 under zstd, 101.40%. Kept as a known issue with that measurement. #1075 is removed. It was closed as priced, not wasted: the discarded compression buys the accuracy of the FSST keep decision, at a measured 9-12% of write CPU, and the cheaper estimator is #1125. Co-Authored-By: Claude Opus 5.5 --- RELEASE_NOTES_1.0-alpha5.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/RELEASE_NOTES_1.0-alpha5.md b/RELEASE_NOTES_1.0-alpha5.md index 7f27b78d9..e4d6463fb 100644 --- a/RELEASE_NOTES_1.0-alpha5.md +++ b/RELEASE_NOTES_1.0-alpha5.md @@ -245,10 +245,18 @@ Both can be set per session. See [docs/configuration.md](docs/configuration.md). ## Known issues -- TODO(release): verify whether the alpha4 known issues still apply. Both are - closed on GitHub, and no alpha5 commit under `src/` names either: - - #1074: a codec setting can make a high-entropy text table larger. - - #1075: the block codec compresses a region it then discards. +- **Setting a codec can make a table larger, on high-entropy text** (#1074). + This is unchanged from alpha4. Measured on this release, 200,000 rows of + random 64-character hex text: + + | `pgcolumnar.compression` | relation bytes | against `none` | + | --- | ---: | ---: | + | `none` | 7,036,928 | 100% | + | `lz4` | 7,036,928 | 100% | + | `zstd` | 7,135,232 | 101.40% | + + If a table stores long high-entropy text, measure both settings. See + [docs/features.md](docs/features.md). ## Upgrading From d3f444e05b0ff9aa30a6087c7431cefcb34fd370 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 18:40:30 +0000 Subject: [PATCH 04/11] docs: alpha5 notes, old chunks keep their layout, measured PG17: a row group alpha4 wrote kept encoding descriptor version 2 after ALTER EXTENSION UPDATE; the next insert on the new build wrote version 3; the table read back 4000 rows with the expected sum (8002000). The TODO on that sentence is replaced by the measurement. Co-Authored-By: Claude Opus 5.5 --- RELEASE_NOTES_1.0-alpha5.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/RELEASE_NOTES_1.0-alpha5.md b/RELEASE_NOTES_1.0-alpha5.md index e4d6463fb..78084c7fe 100644 --- a/RELEASE_NOTES_1.0-alpha5.md +++ b/RELEASE_NOTES_1.0-alpha5.md @@ -103,7 +103,9 @@ Measured on the ClickBench sample, which has no nulls: | chunks without a bitmap | 0 of 735 | 735 of 735 | Both changes apply to data this release writes. Data written before the upgrade -keeps its old layout until it is rewritten. TODO(release): verify. +keeps its old layout until it is rewritten. Measured on PostgreSQL 17: a row +group alpha4 wrote kept descriptor version 2 after the upgrade, the next insert +wrote version 3, and the table read back every row correctly. ## Range pruning (#1144) From 4215507ae1f44486d02d452eb00224006871aeaf Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 18:40:39 +0000 Subject: [PATCH 05/11] docs: split a 30-word sentence the language check refused Co-Authored-By: Claude Opus 5.5 --- RELEASE_NOTES_1.0-alpha5.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/RELEASE_NOTES_1.0-alpha5.md b/RELEASE_NOTES_1.0-alpha5.md index 78084c7fe..90ec8da02 100644 --- a/RELEASE_NOTES_1.0-alpha5.md +++ b/RELEASE_NOTES_1.0-alpha5.md @@ -103,9 +103,9 @@ Measured on the ClickBench sample, which has no nulls: | chunks without a bitmap | 0 of 735 | 735 of 735 | Both changes apply to data this release writes. Data written before the upgrade -keeps its old layout until it is rewritten. Measured on PostgreSQL 17: a row -group alpha4 wrote kept descriptor version 2 after the upgrade, the next insert -wrote version 3, and the table read back every row correctly. +keeps its old layout until it is rewritten. This was measured on PostgreSQL +17. A row group that alpha4 wrote kept descriptor version 2 after the upgrade. +The next insert wrote version 3. The table read back every row correctly. ## Range pruning (#1144) From 20cbd6b7e18bf44eebaa25b92e36b47406051c61 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 19:05:10 +0000 Subject: [PATCH 06/11] docs: alpha5 notes, the on-disk format section now that #1137 landed Filled from #1289 and OffgridwithJD's answers on #1288, and from a run of #1289's build against a real alpha4 build on PG17: - this release writes format version 2 and reads 1 and 2; anything else is refused with 0A000 before a byte is read; - the first write raises a table from 1 to 2, a read never does (measured: stamps stayed 1 through ALTER EXTENSION UPDATE and reads, and became 2 after an INSERT, including a covering projection's own storage row, and after an online recluster); - alpha4 refuses a version 2 table on every path rather than misread it. The upgrade WARNING is OffgridwithJD's text: plan a rollback before the first write, because alpha4 refuses what alpha5 has written to. Only the release date is left as TODO(release). Co-Authored-By: Claude Opus 5.5 --- RELEASE_NOTES_1.0-alpha5.md | 79 ++++++++++++++++++++----------------- 1 file changed, 42 insertions(+), 37 deletions(-) diff --git a/RELEASE_NOTES_1.0-alpha5.md b/RELEASE_NOTES_1.0-alpha5.md index 90ec8da02..28dc8fd03 100644 --- a/RELEASE_NOTES_1.0-alpha5.md +++ b/RELEASE_NOTES_1.0-alpha5.md @@ -10,15 +10,17 @@ before this release's two encoding changes. Range columns can now skip chunk groups on overlap and containment queries. Catalog reads cost less in a database with many columnar tables. -This release changes how column chunks are stored. An alpha4 build cannot read -safely what this release writes. Read "On-disk format" before you upgrade. +This release changes how column chunks are stored. Once it writes to a table, +an alpha4 build refuses to read that table. Read "On-disk format" before you +upgrade. This release requires one upgrade command. See "Upgrading" at the end. ## Highlights -- **TODO(release): #1137, the on-disk format version check.** See "On-disk - format". This is likely the most important item for anyone who upgrades. +- **An older build refuses a table it cannot read** (#1137). A table this + release has written to carries format version 2. An alpha4 build refuses it on + every read path, instead of returning a wrong answer. See "On-disk format". - **Smaller tables.** On ClickBench `hits_0.parquet` (1,000,000 rows, 105 columns), stored chunk bytes fell from 81,869,112 to 64,984,810. Two changes produce this. An encoding is kept only when it is smaller after compression. @@ -36,32 +38,36 @@ This release requires one upgrade command. See "Upgrading" at the end. view can now hold per-table options. Its catalog entry is correct from the moment it is created. -## On-disk format: TODO(release): #1137 - -TODO(release): #1137. Replace this section once #1137 lands. Issue #1137 adds a -format version stamp that this release writes, and a range check on read. Its -purpose is to make an older build refuse a newer table early and loudly on -every read path. State here: - -- the new stamp value -- the range of versions this build reads -- what an alpha4 build does with a table this build wrote -- whether existing tables are restamped - -What is true without #1137: - -- The native data format stays PGCN v1. The metapage version is unchanged. -- Each column chunk carries an encoding descriptor (a small header that records - how the chunk's values are encoded). Its version moves from 2 to 3. This - release reads versions 2 and 3, so it reads every table that alpha4 wrote. No - conversion is needed. -- An alpha4 build cannot read a version 3 chunk safely. A sequential scan fails - with `unrecognized native encoding descriptor`. An index fetch can fail, or it - can return NULL for a row that holds a value. Measured on an alpha4 build: 16 - of 40 single-row fetches returned NULL, and the other 24 raised the error. - -The NULL case is a wrong answer, not an error. That is why downgrading is not -supported. See the warning in "Upgrading". +## On-disk format (#1137) + +Each columnar table records a format version, `pgcolumnar.storage.format_version`. +This release changes it as follows: + +- **This release writes version 2.** Every table it creates starts at 2. +- **It reads versions 1 and 2.** It reads every table alpha4 wrote. It refuses + any other version, with SQLSTATE `0A000` (`unsupported columnar native format + version`), before it reads any data. +- **A write raises a table from 1 to 2. A read never does.** An alpha4 table + stays at 1 until this release first writes a row group into it. Until then, + alpha4 can still read it. That covers `INSERT`, `COPY`, the new row versions of + `UPDATE`, `compact_rewrite`, `recluster` and `parallel_copy`. `SELECT` never + moves it, and neither does `DELETE`. A rewrite builds new storage, which + starts at 2. +- **alpha4 refuses a version 2 table on every read path.** It does not return a + wrong answer. Measured with an alpha4 build on PostgreSQL 17: a sequential scan + and 40 single-row fetches were all refused. That held for a table this release + created and for an alpha4 table it then appended to. A query on a table with a + covering projection was refused too. + +Why the version moved: each column chunk carries an encoding descriptor (a small +header that records how the chunk's values are encoded). Its version moves from 2 +to 3, because a chunk with no nulls no longer stores a validity bitmap (#1130). +An alpha4 build that read such a chunk returned NULL for some rows that hold a +value. Measured before this change: 16 of 40 single-row fetches returned NULL, +and the other 24 raised an error. The format version is what turns that wrong +answer into a refusal. + +The native data format stays PGCN v1. The metapage version is unchanged. ## Smaller tables @@ -279,11 +285,10 @@ Both can be set per session. See [docs/configuration.md](docs/configuration.md). 1. Run `make install` with the same `PG_CONFIG`. - **WARNING:** You cannot safely go back to alpha4 after the next step. Once - the server loads the alpha5 library, the chunks it writes use encoding - descriptor version 3. An alpha4 build can return NULL for a row that holds a - value in such a chunk. If you may need to roll back, take a backup first. - TODO(release): confirm after #1137 lands. + **WARNING:** Plan any rollback before you write with alpha5. Once alpha5 + writes to a table, that table carries format version 2, and alpha4 refuses to + read it. A table alpha5 has only read stays readable by alpha4. To roll back + after alpha5 has written, restore from a backup taken before the upgrade. 2. Restart the server, so that it loads the new library. 3. Connect to one database from the list. @@ -318,8 +323,8 @@ Both can be set per session. See [docs/configuration.md](docs/configuration.md). partitioned table or a view used to succeed. It now raises an error. - It replaces `pgcolumnar."analyze"`. The new version finds a table's storage through the table's metapage. -- No table data is converted. TODO(release): confirm after #1137 lands, and - state whether existing tables keep their current format stamp. +- No table data is converted, and no table's format version changes. A table + moves from version 1 to 2 the first time alpha5 writes to it. - It does not repair a catalog entry that alpha4 left wrong. See "After the upgrade: repair entries alpha4 left wrong". From 2fca31a070591d7134f7bef17d4e5ec5e4114ae7 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 20:06:57 +0000 Subject: [PATCH 07/11] test: the release notes' repair procedure and the install guide's query, both twins Two of today's changes were documentation that a reader acts on, and neither had a test. storage_row_repair.sh and test_storage_row_repair.py perform the alpha5 notes' 'repair entries alpha4 left wrong' procedure: they make the broken state (a storage row naming a dropped relation, as measured with a real alpha4 build), run the query READ FROM THE NOTES, apply both repairs, and check the query then finds nothing and the data is intact. PG17: shell 9 passed, pytest 9 checks passed, parity 9 of 9. Proven able to fail, both twins: M1 PgColumnarRetargetStorageRelation returns at once: shell 4 red (both repairs, the final query, and the WITH DATA premise); pytest red at the premise M2 the notes' query edited to EXISTS: shell 3 red, pytest red, so the test reads the document rather than a copy of it docs_style.sh and test_docs_cross_database_query.py: no documented query tests pg_extension while listing databases (pg_extension is per database). Green on the fixed guide; red with the old query restored (got 1 want 0, both twins); each premise red when the SQL scan is blinded. Co-Authored-By: Claude Opus 5.5 --- test/docs_style.sh | 14 +++ test/pytest/test_docs_cross_database_query.py | 36 +++++++ test/pytest/test_storage_row_repair.py | 94 +++++++++++++++++++ test/run_all_versions.sh | 1 + test/storage_row_repair.sh | 77 +++++++++++++++ 5 files changed, 222 insertions(+) create mode 100644 test/pytest/test_docs_cross_database_query.py create mode 100644 test/pytest/test_storage_row_repair.py create mode 100755 test/storage_row_repair.sh diff --git a/test/docs_style.sh b/test/docs_style.sh index 8e4ffe46a..681aff61d 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -795,6 +795,20 @@ check "and says WHY rebasing works where merging does not" \ "$(grep -c 'merging \*\*into\*\*' "$SRCDIR/CONTEXT.md")" "1" +# A QUERY CANNOT ASK ANOTHER DATABASE WHAT EXTENSIONS IT HAS (#1288). pg_extension is a +# per-database catalog, so a statement that reads pg_database and tests pg_extension in the +# same breath answers for the CURRENT database only: it lists every connectable database or +# none. docs/installation.md shipped exactly that as "the databases that need the upgrade". +# Read per statement inside ```sql blocks, so prose that explains the rule cannot trip it. +_xdb_stmts="$(for f in $docs; do + awk '/^[[:space:]]*```sql/ {b=1; next} b && /^[[:space:]]*```/ {b=0; print ";"; next} b {printf "%s ", $0}' "$f" +done | tr ';' '\n')" +check "premise: the scan read SQL statements from the documents" \ + "$([ "$(printf '%s\n' "$_xdb_stmts" | grep -c 'SELECT')" -ge 10 ] && echo yes || echo no)" "yes" +check "no documented query tests pg_extension while listing databases" \ + "$(printf '%s\n' "$_xdb_stmts" | grep 'pg_database' | grep -c 'pg_extension')" "0" + + echo "checks run: $checks" if [ "$fail" = 0 ]; then echo "docs_style.sh: PASSED" diff --git a/test/pytest/test_docs_cross_database_query.py b/test/pytest/test_docs_cross_database_query.py new file mode 100644 index 000000000..b794b4f7f --- /dev/null +++ b/test/pytest/test_docs_cross_database_query.py @@ -0,0 +1,36 @@ +"""No documented query asks another database what extensions it has (#1288). + +pg_extension is a per-database catalog. A statement that reads pg_database and tests +pg_extension in the same breath answers for the CURRENT database only, so it lists every +connectable database or none. docs/installation.md shipped exactly that query as the way +to find the databases that need `ALTER EXTENSION pgcolumnar UPDATE`. + +The pytest half of the docs_style.sh arm of the same name, written independently: its +own parser for the ```sql blocks, split into statements on ';'. Prose that explains the +rule cannot trip it, because only fenced SQL is read. + +Needs the repository only, not a cluster or the driver. +""" +import pathlib +import re + +REPO = pathlib.Path(__file__).resolve().parents[2] +DOCS = sorted((REPO / "docs").glob("*.md")) + [REPO / "README.md"] +FENCE = re.compile(r"^[ \t]*```sql[ \t]*\n(.*?)^[ \t]*```", re.S | re.M) + + +def _statements(): + out = [] + for path in DOCS: + for block in FENCE.findall(path.read_text()): + out.extend(s.strip() for s in block.split(";") if s.strip()) + return out + + +def test_no_documented_query_asks_another_database_for_its_extensions(expect): + stmts = _statements() + expect.at_least(sum(1 for s in stmts if re.search(r"\bSELECT\b", s)), 10, + "premise: the scan read SQL statements from the documents") + bad = [s for s in stmts if "pg_database" in s and "pg_extension" in s] + expect.num(len(bad), 0, + "no documented query tests pg_extension while listing databases") diff --git a/test/pytest/test_storage_row_repair.py b/test/pytest/test_storage_row_repair.py new file mode 100644 index 000000000..ec3f908d1 --- /dev/null +++ b/test/pytest/test_storage_row_repair.py @@ -0,0 +1,94 @@ +"""The alpha5 release notes' repair procedure works as written. + +Two alpha4 defects (#1211, #1275) left a table's pgcolumnar.storage row naming a +relation that no longer exists, and the upgrade does not repair existing rows. +RELEASE_NOTES_1.0-alpha5.md, "After the upgrade: repair entries alpha4 left wrong", +gives a query that finds them and two repairs. This performs them. + +Independent of test/storage_row_repair.sh: own tables, own row count, own way of +making the damage. The notes' query is read from the notes, so an edit to the +document is what this runs. + +SCOPED TO ITS OWN STORAGE IDS. pgc_conn gives a private schema, not a private +database, and the pgcolumnar catalogs are per database, so the notes' query sees +every other test's storage rows too. The count asserted is of rows the query +returns for the two storage ids this test damaged, which only this test can move. +""" +import pathlib +import re + +ROWS = 1500 +NOTES = pathlib.Path(__file__).resolve().parents[2] / "RELEASE_NOTES_1.0-alpha5.md" + + +def _notes_query(): + text = NOTES.read_text() + after = text.split("find the rows that name no relation", 1) + if len(after) != 2: + return None + m = re.search(r"```sql\n(.*?)```", after[1], re.S) + return m.group(1).strip().rstrip(";") if m else None + + +def _one(cur, sql): + cur.execute(sql) + row = cur.fetchone() + return row[0] if row else None + + +def test_the_release_notes_repair_procedure_works(pgc_conn, expect): + find = _notes_query() + expect.text("found" if find and "pgcolumnar.storage" in find else "missing", "found", + "premise: the release notes give the query that finds wrong rows") + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE rep_t (k int, w int) USING pgcolumnar") + cur.execute(f"INSERT INTO rep_t SELECT g, g * 2 FROM generate_series(1,{ROWS}) g") + cur.execute("CREATE MATERIALIZED VIEW rep_mv USING pgcolumnar AS " + "SELECT k FROM rep_t WITH DATA") + t_sid = _one(cur, "SELECT pgcolumnar.get_storage_id('rep_t')") + mv_sid = _one(cur, "SELECT pgcolumnar.get_storage_id('rep_mv')") + mine = f"({t_sid}, {mv_sid})" + + def found_mine(): + return _one(cur, f"SELECT count(*) FROM ({find}) z WHERE z.storage_id IN {mine}") + + expect.num(found_mine(), 0, "premise: no row names a missing relation before the damage") + + # The damage: each row names a relation created and dropped for the purpose. + cur.execute("CREATE TABLE rep_gone_1 (x int)") + cur.execute("CREATE TABLE rep_gone_2 (x int)") + g1 = _one(cur, "SELECT 'rep_gone_1'::regclass::oid") + g2 = _one(cur, "SELECT 'rep_gone_2'::regclass::oid") + cur.execute("DROP TABLE rep_gone_1, rep_gone_2") + cur.execute(f"UPDATE pgcolumnar.storage SET relation_oid = {g1} WHERE storage_id = {t_sid}") + cur.execute(f"UPDATE pgcolumnar.storage SET relation_oid = {g2} WHERE storage_id = {mv_sid}") + + expect.num(found_mine(), 2, "the notes' query finds both rows alpha4 left wrong") + expect.text(_one(cur, "SELECT count(*) || '/' || sum(w) FROM rep_t"), + f"{ROWS}/{ROWS * (ROWS + 1)}", + "premise: the damaged table still reads every row") + + cur.execute("REFRESH MATERIALIZED VIEW rep_mv") + expect.num(_one(cur, "SELECT (relation_oid = 'rep_mv'::regclass)::int " + "FROM pgcolumnar.storage WHERE storage_id = " + "pgcolumnar.get_storage_id('rep_mv')"), 1, + "REFRESH MATERIALIZED VIEW repairs the view's row") + cur.execute("ALTER TABLE rep_t ALTER COLUMN w TYPE int USING w") + expect.num(_one(cur, "SELECT (relation_oid = 'rep_t'::regclass)::int " + "FROM pgcolumnar.storage WHERE storage_id = " + "pgcolumnar.get_storage_id('rep_t')"), 1, + "a same-type ALTER COLUMN ... USING repairs the table's row") + + # A REFRESH gives the view a NEW storage id, so the final count covers the + # original ids AND the current ones: a damaged row left behind under its old + # id must still be counted. + t_now = _one(cur, "SELECT pgcolumnar.get_storage_id('rep_t')") + mv_now = _one(cur, "SELECT pgcolumnar.get_storage_id('rep_mv')") + mine = f"({t_sid}, {mv_sid}, {t_now}, {mv_now})" + expect.num(found_mine(), 0, "after both repairs the notes' query finds nothing") + expect.text(_one(cur, "SELECT count(*) || '/' || sum(w) FROM rep_t"), + f"{ROWS}/{ROWS * (ROWS + 1)}", + "and the repaired table holds the same rows") + expect.text(_one(cur, "SELECT count(*) || '/' || sum(k) FROM rep_mv"), + f"{ROWS}/{ROWS * (ROWS + 1) // 2}", + "and so does the refreshed view") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 5c01d7657..499e36501 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -289,6 +289,7 @@ SUITES=( sorted_pathkeys sorted_projection stats_privilege + storage_row_repair tablesample temporal truncate_cleanup diff --git a/test/storage_row_repair.sh b/test/storage_row_repair.sh new file mode 100755 index 000000000..48607f5cc --- /dev/null +++ b/test/storage_row_repair.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# The alpha5 release notes' repair procedure works as written. +# +# Two alpha4 defects (#1211, #1275) left a table's pgcolumnar.storage row naming a +# relation that no longer exists, and the upgrade does not repair rows that already +# exist. RELEASE_NOTES_1.0-alpha5.md, "After the upgrade: repair entries alpha4 left +# wrong", gives a query that finds them and two repairs. Instructions are validated +# by performing them, so this suite performs them. +# +# THE BROKEN STATE IS MADE, NOT INHERITED. This build no longer produces it, so the +# suite writes it directly: each storage row is pointed at the OID of a table created +# and dropped for the purpose. That is the state measured on PG17 with a real alpha4 +# build (#1288): relation_oid naming a dropped relation, everything else intact. +# +# THE QUERY IS READ FROM THE NOTES, not copied here. If the notes' query is edited, +# this suite runs the edited one, so the document and the check cannot drift apart. +# +# Usage: test/storage_row_repair.sh [PG_CONFIG] + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/lib/postgresql/18/bin/pg_config}" + +q "CREATE EXTENSION IF NOT EXISTS pgcolumnar;" >/dev/null + +NOTES="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/RELEASE_NOTES_1.0-alpha5.md" +# The first ```sql block after the step that introduces the query. +FIND_SQL="$(awk '/find the rows that name no relation/ {f=1; next} + f && /```sql/ {g=1; next} g && /```/ {exit} g {print}' "$NOTES" 2>/dev/null)" +# A statement in the notes ends with ';', which cannot sit inside a subquery. +FIND_SQL="${FIND_SQL%;}" +check "premise: the release notes give the query that finds wrong rows" \ + "$(printf '%s\n' "$FIND_SQL" | grep -c 'pgcolumnar.storage')" "1" + +q "CREATE TABLE srr_t (id int, b int) USING pgcolumnar; + INSERT INTO srr_t SELECT g, g FROM generate_series(1,1000) g; + CREATE MATERIALIZED VIEW srr_mv USING pgcolumnar AS SELECT id FROM srr_t WITH DATA; + CREATE TABLE srr_gone_a (x int); CREATE TABLE srr_gone_b (x int);" >/dev/null +gone_a="$(q "SELECT 'srr_gone_a'::regclass::oid;")" +gone_b="$(q "SELECT 'srr_gone_b'::regclass::oid;")" +q "DROP TABLE srr_gone_a, srr_gone_b;" >/dev/null + +# Count of rows the notes' query reports. +found() { q "SELECT count(*) FROM ($FIND_SQL) z;"; } +names_itself() { # names_itself RELNAME -> 1 when its storage row names it + q "SELECT (relation_oid = '$1'::regclass)::int FROM pgcolumnar.storage + WHERE storage_id = pgcolumnar.get_storage_id('$1');" +} + +check_num "premise: no row names a missing relation before the damage" "$(found)" "0" + +q "UPDATE pgcolumnar.storage SET relation_oid = $gone_a + WHERE storage_id = pgcolumnar.get_storage_id('srr_t'); + UPDATE pgcolumnar.storage SET relation_oid = $gone_b + WHERE storage_id = pgcolumnar.get_storage_id('srr_mv');" >/dev/null + +check_num "the notes' query finds both rows alpha4 left wrong" "$(found)" "2" +check_text "premise: the damaged table still reads every row" \ + "$(q "SELECT count(*) || '/' || sum(b) FROM srr_t;")" "1000/500500" + +# Step 2 of the notes: refresh each columnar materialized view. +q "REFRESH MATERIALIZED VIEW srr_mv;" >/dev/null +check_num "REFRESH MATERIALIZED VIEW repairs the view's row" "$(names_itself srr_mv)" "1" + +# Step 3: convert a column to its own type with USING, which forces the rewrite. +q "ALTER TABLE srr_t ALTER COLUMN b TYPE int USING b;" >/dev/null +check_num "a same-type ALTER COLUMN ... USING repairs the table's row" \ + "$(names_itself srr_t)" "1" + +# Step 4: the notes' query again, and the data unchanged. +check_num "after both repairs the notes' query finds nothing" "$(found)" "0" +check_text "and the repaired table holds the same rows" \ + "$(q "SELECT count(*) || '/' || sum(b) FROM srr_t;")" "1000/500500" +check_text "and so does the refreshed view" \ + "$(q "SELECT count(*) || '/' || sum(id) FROM srr_mv;")" "1000/500500" + +pgc_summary From 2b4dd1e27d14dfd1859cac04fcb1b499cf14c436 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 20:08:26 +0000 Subject: [PATCH 08/11] test: register the two new pytest files in every registry (#1288) NO_CLUSTER (test_docs_cross_database_query.py), TESTS.md 85 and 86 with contents entries, test_compare_to_bash's declared pairs (storage_row_repair), and both counts by collection: guard_tests 404 -> 405, cluster_tests 491 -> 492. Co-Authored-By: Claude Opus 5.5 --- test/pytest/TESTS.md | 30 +++++++++++++++++++++++++++++ test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_harness_deps.py | 2 ++ 3 files changed, 33 insertions(+) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index b3b165873..b58705f14 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -133,6 +133,8 @@ behaviour, the source of that number is named. - [85. test_native_agg.py: min/max over timestamptz read no data](#85-test_native_aggpy-minmax-over-timestamptz-read-no-data) - [86. test_format_downgrade.py: an alpha4 binary refuses what this build wrote](#86-test_format_downgradepy-an-alpha4-binary-refuses-what-this-build-wrote) - [87. test_latest_reading.py: the latest-reading recipe does what the document says](#87-test_latest_readingpy-the-latest-reading-recipe-does-what-the-document-says) +- [88. test_storage_row_repair.py: the alpha5 release notes' repair procedure works as written](#88-test_storage_row_repairpy-the-alpha5-release-notes-repair-procedure-works-as-written) +- [89. test_docs_cross_database_query.py: no documented query asks another database for its extensions](#89-test_docs_cross_database_querypy-no-documented-query-asks-another-database-for-its-extensions) ## 1. How to read a test in here @@ -6404,3 +6406,31 @@ Removal proof: the document's `LATERAL` edited to `ORDER BY ts` (ascending) retu | test | what it holds | | --- | --- | | `test_the_latest_reading_recipe_does_what_the_document_says` | the document's three blocks are found; the table holds its rows; without the btree the `LATERAL` form reads the table about once per device; the `GROUP BY` form and, with the document's index, the `LATERAL` form return the rows `DISTINCT ON` returns; the indexed `LATERAL` reads at least one block and under a tenth of the table | + +## 88. test_storage_row_repair.py: the alpha5 release notes' repair procedure works as written + +Two alpha4 defects (#1211, #1275) left a table's `pgcolumnar.storage` row naming a relation that no longer exists, and the upgrade does not repair existing rows. `RELEASE_NOTES_1.0-alpha5.md` gives a query that finds them and two repairs: `REFRESH MATERIALIZED VIEW` for a view, and a same-type `ALTER COLUMN ... USING` for a table. Instructions are validated by performing them, so this performs them. + +THE BROKEN STATE IS MADE, NOT INHERITED: each storage row is pointed at a table created and dropped for the purpose. That matches what a real alpha4 build left behind, measured on PG17. THE QUERY IS READ FROM THE NOTES, so an edit to the document is what runs. + +SCOPED TO ITS OWN STORAGE IDS. `pgc_conn` gives a private schema in a shared database, and the pgcolumnar catalogs are per database. So the count asserted is of the rows the notes' query returns for the ids this test damaged, the original and the post-repair ones both. + +Removal proofs on PG17: disabling `PgColumnarRetargetStorageRelation` reddens it at the premise. Editing the notes' query from `NOT EXISTS` to `EXISTS` reddens it too. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_the_release_notes_repair_procedure_works` | the notes give the query; it finds neither row before the damage and both after; the damaged table still reads every row; each repair repoints its row; the query then finds nothing; the table and the view hold the same rows | + +## 89. test_docs_cross_database_query.py: no documented query asks another database for its extensions + +`pg_extension` is a per-database catalog. A statement that reads `pg_database` and tests `pg_extension` answers for the current database only, so it lists every connectable database or none. `docs/installation.md` shipped exactly that as the way to find the databases that need the upgrade (#1288). Only fenced SQL is read, split into statements, so prose that explains the rule cannot trip it. It needs no cluster, so it is in `NO_CLUSTER`. + +Red with the old query restored (got 1, want 0). Its premise is red when the fence pattern is broken and nothing is scanned (got 0, wanted at least 10). That is the case where the arm alone would pass on nothing. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_no_documented_query_asks_another_database_for_its_extensions` | at least ten SQL statements were read; none reads `pg_database` and tests `pg_extension` | diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 63206b037..14976e0a5 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -128,6 +128,7 @@ "scan_decode_cost", "sorted_pathkeys", "stats_privilege", + "storage_row_repair", "temporal", "ttl_expire", "validity_elision", diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index 66d6c7e2a..f9441a811 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -71,6 +71,8 @@ # Landed on main in #922 after this list was written, and the arm above caught # it: the property says it needs no database, so the declaration must say so too. "test_suite_accounting.py", + # #1288: reads docs/*.md and README.md only. + "test_docs_cross_database_query.py", # The classifier's own controls, split out of this file so the job whose file # list IS this list actually runs them. This file cannot be in the list: it # hands cluster-bound file names to pytest, so it needs what they need. From 65e8c716478e59a7238604a5adb6c34105c93bfd Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 20:25:33 +0000 Subject: [PATCH 09/11] test: ledger rows for storage_row_repair, all on five majors (#1288) Nine checks. Five dated by two mutations run on all five majors, each log merged with its own failed set as targets: retarget disabled 3 red on 15/16, 4 on 17-19 notes' query to EXISTS 3 red on every major Control 9 passed + 0 failed on every major. Census 1653 -> 1657. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 ++++ test/check_ledger.tsv | 9 +++++++++ test/check_ledger_budget.txt | 17 ++++++++++++++++- 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bf09dab2..521f328af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -140,6 +140,10 @@ true until the next version shipped. database or none. The guide now lists the databases and checks each one. `docs/configuration.md` said `index_min_blocks` restores "every release before 1.0-alpha6"; it means alpha4 and earlier. + Both are now tested, in both harnesses: `storage_row_repair` performs the + release notes' repair procedure, reading its query from the notes, and a docs + check refuses any documented query that tests `pg_extension` while listing + databases. - `test/pytest/README.md` now names all four registries a new pytest test has to be entered in, and which of them a local run can catch. Adding one test reddens diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 905afced6..1903f5f7b 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1847,6 +1847,15 @@ rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partitio rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp ATTACH PARTITION lp_3 FOR VALUES FROM ('2026-03-01') TO ('2026-04-01') 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 CONCURRENTLY 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds) +storage_row_repair storage_row_repair REFRESH MATERIALIZED VIEW repairs the view's row 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row +storage_row_repair storage_row_repair a same-type ALTER COLUMN ... USING repairs the table's row 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row +storage_row_repair storage_row_repair after both repairs the notes' query finds nothing 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row;the release notes' query is edited from NOT EXISTS to EXISTS +storage_row_repair storage_row_repair and so does the refreshed view 15;16;17;18;19 never - +storage_row_repair storage_row_repair and the repaired table holds the same rows 15;16;17;18;19 never - +storage_row_repair storage_row_repair premise: no row names a missing relation before the damage 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row;the release notes' query is edited from NOT EXISTS to EXISTS +storage_row_repair storage_row_repair premise: the damaged table still reads every row 15;16;17;18;19 never - +storage_row_repair storage_row_repair premise: the release notes give the query that finds wrong rows 15;16;17;18;19 never - +storage_row_repair storage_row_repair the notes' query finds both rows alpha4 left wrong 15;16;17;18;19 2026-10-12 the release notes' query is edited from NOT EXISTS to EXISTS validity_elision validity_elision a chunk claiming no bitmap while it holds fewer values than rows is refused 15;16;17;18;19 never - validity_elision validity_elision a column with no nulls stores no validity bitmap 15;16;17;18;19 2026-09-18 writer: flush_one_column's presentCount == rowCount forced false, so the bitmap is written even when the chunk holds no null validity_elision validity_elision a null-free column elides its bitmap beside a null-bearing one in the same row group 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 0a7b69f2a..96f8a404f 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -1243,4 +1243,19 @@ suites_not_covered 247 # never 1653 + 7 = 1660 # dated 213 + 1 = 214 # 1660 + 214 == 1874 -checks_never_observed_red 1660 +# +# #1288: 1660 -> 1664. storage_row_repair is a new suite, nine checks, all on five +# majors. Five are dated by two mutations, each run on all five majors and merged one +# log at a time with its own failed set as the targets: +# +# PgColumnarRetargetStorageRelation returns at once 3 red on 15/16, 4 on 17+ +# (the WITH DATA premise is +# born wrong only from 17) +# the notes' query edited from NOT EXISTS to EXISTS 3 red on every major +# +# Four stay never: two premises, and the two arms that the data is unchanged after +# the repair, which no repair mutation here reaches. Replayed from the saved logs +# onto #1303's ledger, not merged by hand. +# +# never 1660 + 4 = 1664 +checks_never_observed_red 1664 From effc30e05e8c882266db51ee1a2a2287295a2d72 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 20:40:26 +0000 Subject: [PATCH 10/11] fix: the docs premise reports the count it measured (#1288) CI on b2ee6cc: harness_selftest part 540 refused the new docs_style premise, 'a new arm that cannot say what it measured is refused by name', because its failure branch answered only 'no'. It now prints the number of SELECT statements it read. My mistake: I ran docs_style but not harness_selftest before pushing, and part 540 sweeps every suite. PG17: harness_selftest 1213 passed + 0 failed; docs_style 57 checks PASSED; with the scan blinded the premise is red and says 'no (0 SELECT statements, want at least 10)'. Co-Authored-By: Claude Opus 5.5 --- test/docs_style.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/test/docs_style.sh b/test/docs_style.sh index 681aff61d..64ad27787 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -803,8 +803,11 @@ check "and says WHY rebasing works where merging does not" \ _xdb_stmts="$(for f in $docs; do awk '/^[[:space:]]*```sql/ {b=1; next} b && /^[[:space:]]*```/ {b=0; print ";"; next} b {printf "%s ", $0}' "$f" done | tr ';' '\n')" +# The failure carries the count it measured: selftest part 540 refuses an arm that +# answers only "no". +_xdb_n="$(printf '%s\n' "$_xdb_stmts" | grep -c 'SELECT')" check "premise: the scan read SQL statements from the documents" \ - "$([ "$(printf '%s\n' "$_xdb_stmts" | grep -c 'SELECT')" -ge 10 ] && echo yes || echo no)" "yes" + "$([ "$_xdb_n" -ge 10 ] && echo yes || echo "no ($_xdb_n SELECT statements, want at least 10)")" "yes" check "no documented query tests pg_extension while listing databases" \ "$(printf '%s\n' "$_xdb_stmts" | grep 'pg_database' | grep -c 'pg_extension')" "0" From 0adf87b6a59cb2de9c5e63e73fbb26606a8a67e3 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 12 Oct 2026 21:00:32 +0000 Subject: [PATCH 11/11] test: re-collect both counts after #1303 landed (#1288) guard_tests 405 and cluster_tests 496, by collection on 038600e. TESTS.md sections renumbered 88 and 89 after #1303's 87; ledger rows replayed from the saved logs onto #1303's ledger (census 1660 -> 1664). Co-Authored-By: Claude Opus 5.5 --- test/pytest/expected_tests.txt | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index ade202328..d19254285 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -261,7 +261,9 @@ # adding one: `403 tests collected` over the 24 NO_CLUSTER files. `cluster_tests` # was re-derived in the same run and did NOT move -- 467 -- which is the expected # answer for a change that touches only the layer and its own guard. -guard_tests 404 +# 404 -> 405 for #1288: test_docs_cross_database_query.py, cluster-free, in NO_CLUSTER. +# Re-derived by collection on 038600e (after #1303): `405 tests collected`. +guard_tests 405 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and @@ -543,4 +545,6 @@ guard_tests 404 # own test is not in it. Git conflicted again -- loudly, because the key moved # on both sides -- and the answer is neither branch's number. Collected on the # rebased tree, not added up. -cluster_tests 495 +# 495 -> 496 for #1288: test_storage_row_repair.py. Re-derived by collection on +# 038600e (after #1303): `496 tests collected`. +cluster_tests 496