Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,16 @@ true until the next version shipped.
against 1000 with one, identically across twelve readings with and without
concurrent DDL. With the guard it tests removed, it read 2000 against 1000 and
failed in both twins. The cause of the buffer noise is still unknown.
- `docs/installation.md` gave a query to find the databases that need
`ALTER EXTENSION pgcolumnar UPDATE`. It could not work: `pg_extension` lists
only the current database's extensions, so the query listed every connectable
database or none. The guide now lists the databases and checks each one.
`docs/configuration.md` said `index_min_blocks` restores "every release before
1.0-alpha6"; it means alpha4 and earlier.
Both are now tested, in both harnesses: `storage_row_repair` performs the
release notes' repair procedure, reading its query from the notes, and a docs
check refuses any documented query that tests `pg_extension` while listing
databases.

- `test/pytest/README.md` now names all four registries a new pytest test has to
be entered in, and which of them a local run can catch. Adding one test reddens
Expand Down
388 changes: 388 additions & 0 deletions RELEASE_NOTES_1.0-alpha5.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ disk. It never changes the values that a table returns.

| Setting | Type | Default | Description |
| --- | --- | --- | --- |
| `pgcolumnar.index_min_blocks` | integer | `3` | Heap pages at or above which a keyed read of a `pgcolumnar` metadata catalog probes that catalog's index instead of reading it sequentially. These catalogs are shared by every columnar table in the database, so the same catalog is one page in a small database and hundreds in a large one, and below a few pages a sequential read is cheaper than a probe. The default is measured rather than chosen, and it is a default rather than a truth: the best value differs per catalog, because the hot catalogs are read more often per retired row group. Change it only to undo a regression you have measured. A very large value restores the behaviour of every release before 1.0-alpha6, which always read sequentially; `0` always probes. Range 0 to INT_MAX. |
| `pgcolumnar.index_min_blocks` | integer | `3` | Heap pages at or above which a keyed read of a `pgcolumnar` metadata catalog probes that catalog's index instead of reading it sequentially. These catalogs are shared by every columnar table in the database, so the same catalog is one page in a small database and hundreds in a large one, and below a few pages a sequential read is cheaper than a probe. The default is measured rather than chosen, and it is a default rather than a truth: the best value differs per catalog, because the hot catalogs are read more often per retired row group. Change it only to undo a regression you have measured. A very large value makes every converted lookup read its catalog sequentially, as 1.0-alpha4 and earlier did for most of them; `0` always probes. Range 0 to INT_MAX. |
| `pgcolumnar.reclaim_coalesce` | boolean | `on` | During online compaction, split an oversized freed range on reuse and coalesce adjacent freed ranges, so space is reclaimed under fragmentation. Off reverts to whole-range reuse. |
| `pgcolumnar.enable_end_truncation` | boolean | `off` | Allow `pgcolumnar.truncate()` to return trailing reclaimed blocks to the operating system. Off makes `pgcolumnar.truncate()` a no-op. Requires superuser to set. |
| `pgcolumnar.autovacuum` | boolean | `off` | Run the maintenance daemon. When on, it runs `compact_rewrite` and `recluster` on columnar tables that cross a threshold. It uses only `ShareUpdateExclusiveLock` and yields to any stronger lock. It never blocks a reader or a writer. See the [administration guide](administration.md#the-maintenance-daemon-pgcolumnarautovacuum). Reloadable, not a per-session setting. |
Expand Down
13 changes: 10 additions & 3 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,18 @@ ALTER EXTENSION pgcolumnar UPDATE;
SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar';
```

To find the databases that need it:
To find the databases that need it, list the databases you can connect to:

```sql
SELECT datname FROM pg_database WHERE datallowconn
AND EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcolumnar');
SELECT datname FROM pg_database WHERE datallowconn;
```

Then connect to each one and check whether it has the extension. A query cannot
do this from one database, because `pg_extension` lists only the extensions of
the database you are connected to:

```sql
SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar';
```

### If you skipped step 3
Expand Down
18 changes: 6 additions & 12 deletions pgcolumnar--1.0-alpha4--1.0-alpha5.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,12 @@
* alpha4 install would never run it and a fresh install would silently get a
* different catalog from an upgraded one.
*
* EMPTY ON PURPOSE, AND THAT IS WHY IT EXISTS. This change opens the cycle and
* nothing else. The file is here so that the work already pointed at alpha5 --
* the zone-map summary that overlap and containment pruning needs (#1144), and
* cascading step 2 (#1139) -- appends to a script that already exists rather
* than racing to create it, and so that the version bump is reviewable on its
* own rather than inside a feature diff.
*
* PostgreSQL requires the file to exist and to be readable; an upgrade script
* that adds nothing is a valid one. `ALTER EXTENSION pgcolumnar UPDATE` from
* alpha4 reaches alpha5 through this file and changes no catalog object, which
* is what test/extension_upgrade.sh asserts by comparing the upgraded catalog
* against a fresh install of the new version.
* IT WAS CREATED EMPTY to open the cycle, so that the work already pointed at
* alpha5 appended to a script that existed rather than racing to create it. It
* is no longer empty. It now adds `pgcolumnar.zone_map.max_upper` for range
* pruning (#1144), and replaces `reset_options`, `set_options` and `"analyze"`
* (#1265, #1276). test/extension_upgrade.sh asserts that the upgraded catalog is
* byte-identical to a fresh install of the new version.
*/

-- The greatest upper bound of a RANGE column's values in a zone map unit
Expand Down
9 changes: 9 additions & 0 deletions test/check_ledger.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -1847,6 +1847,15 @@ rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partitio
rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp ATTACH PARTITION lp_3 FOR VALUES FROM ('2026-03-01') TO ('2026-04-01') 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds)
rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds)
rewrite_storage_oid rewrite_storage_oid the repair gate does not read a partition the ALTER did not touch: lp DETACH PARTITION lp_2 CONCURRENTLY 15;16;17;18;19 2026-10-12 the repair gate opens every inheritor with NoLock again, so it reads the metapage of a partition the ALTER never locked (an Assert on assert builds)
storage_row_repair storage_row_repair REFRESH MATERIALIZED VIEW repairs the view's row 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row
storage_row_repair storage_row_repair a same-type ALTER COLUMN ... USING repairs the table's row 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row
storage_row_repair storage_row_repair after both repairs the notes' query finds nothing 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row;the release notes' query is edited from NOT EXISTS to EXISTS
storage_row_repair storage_row_repair and so does the refreshed view 15;16;17;18;19 never -
storage_row_repair storage_row_repair and the repaired table holds the same rows 15;16;17;18;19 never -
storage_row_repair storage_row_repair premise: no row names a missing relation before the damage 15;16;17;18;19 2026-10-12 PgColumnarRetargetStorageRelation returns at once, so no repair repoints a storage row;the release notes' query is edited from NOT EXISTS to EXISTS
storage_row_repair storage_row_repair premise: the damaged table still reads every row 15;16;17;18;19 never -
storage_row_repair storage_row_repair premise: the release notes give the query that finds wrong rows 15;16;17;18;19 never -
storage_row_repair storage_row_repair the notes' query finds both rows alpha4 left wrong 15;16;17;18;19 2026-10-12 the release notes' query is edited from NOT EXISTS to EXISTS
validity_elision validity_elision a chunk claiming no bitmap while it holds fewer values than rows is refused 15;16;17;18;19 never -
validity_elision validity_elision a column with no nulls stores no validity bitmap 15;16;17;18;19 2026-09-18 writer: flush_one_column's presentCount == rowCount forced false, so the bitmap is written even when the chunk holds no null
validity_elision validity_elision a null-free column elides its bitmap beside a null-bearing one in the same row group 15;16;17;18;19 never -
Expand Down
17 changes: 16 additions & 1 deletion test/check_ledger_budget.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1243,4 +1243,19 @@ suites_not_covered 247
# never 1653 + 7 = 1660
# dated 213 + 1 = 214
# 1660 + 214 == 1874
checks_never_observed_red 1660
#
# #1288: 1660 -> 1664. storage_row_repair is a new suite, nine checks, all on five
# majors. Five are dated by two mutations, each run on all five majors and merged one
# log at a time with its own failed set as the targets:
#
# PgColumnarRetargetStorageRelation returns at once 3 red on 15/16, 4 on 17+
# (the WITH DATA premise is
# born wrong only from 17)
# the notes' query edited from NOT EXISTS to EXISTS 3 red on every major
#
# Four stay never: two premises, and the two arms that the data is unchanged after
# the repair, which no repair mutation here reaches. Replayed from the saved logs
# onto #1303's ledger, not merged by hand.
#
# never 1660 + 4 = 1664
checks_never_observed_red 1664
17 changes: 17 additions & 0 deletions test/docs_style.sh
Original file line number Diff line number Diff line change
Expand Up @@ -795,6 +795,23 @@ check "and says WHY rebasing works where merging does not" \
"$(grep -c 'merging \*\*into\*\*' "$SRCDIR/CONTEXT.md")" "1"


# A QUERY CANNOT ASK ANOTHER DATABASE WHAT EXTENSIONS IT HAS (#1288). pg_extension is a
# per-database catalog, so a statement that reads pg_database and tests pg_extension in the
# same breath answers for the CURRENT database only: it lists every connectable database or
# none. docs/installation.md shipped exactly that as "the databases that need the upgrade".
# Read per statement inside ```sql blocks, so prose that explains the rule cannot trip it.
_xdb_stmts="$(for f in $docs; do
awk '/^[[:space:]]*```sql/ {b=1; next} b && /^[[:space:]]*```/ {b=0; print ";"; next} b {printf "%s ", $0}' "$f"
done | tr ';' '\n')"
# The failure carries the count it measured: selftest part 540 refuses an arm that
# answers only "no".
_xdb_n="$(printf '%s\n' "$_xdb_stmts" | grep -c 'SELECT')"
check "premise: the scan read SQL statements from the documents" \
"$([ "$_xdb_n" -ge 10 ] && echo yes || echo "no ($_xdb_n SELECT statements, want at least 10)")" "yes"
check "no documented query tests pg_extension while listing databases" \
"$(printf '%s\n' "$_xdb_stmts" | grep 'pg_database' | grep -c 'pg_extension')" "0"


echo "checks run: $checks"
if [ "$fail" = 0 ]; then
echo "docs_style.sh: PASSED"
Expand Down
30 changes: 30 additions & 0 deletions test/pytest/TESTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,8 @@ behaviour, the source of that number is named.
- [85. test_native_agg.py: min/max over timestamptz read no data](#85-test_native_aggpy-minmax-over-timestamptz-read-no-data)
- [86. test_format_downgrade.py: an alpha4 binary refuses what this build wrote](#86-test_format_downgradepy-an-alpha4-binary-refuses-what-this-build-wrote)
- [87. test_latest_reading.py: the latest-reading recipe does what the document says](#87-test_latest_readingpy-the-latest-reading-recipe-does-what-the-document-says)
- [88. test_storage_row_repair.py: the alpha5 release notes' repair procedure works as written](#88-test_storage_row_repairpy-the-alpha5-release-notes-repair-procedure-works-as-written)
- [89. test_docs_cross_database_query.py: no documented query asks another database for its extensions](#89-test_docs_cross_database_querypy-no-documented-query-asks-another-database-for-its-extensions)

## 1. How to read a test in here

Expand Down Expand Up @@ -6404,3 +6406,31 @@ Removal proof: the document's `LATERAL` edited to `ORDER BY ts` (ascending) retu
| test | what it holds |
| --- | --- |
| `test_the_latest_reading_recipe_does_what_the_document_says` | the document's three blocks are found; the table holds its rows; without the btree the `LATERAL` form reads the table about once per device; the `GROUP BY` form and, with the document's index, the `LATERAL` form return the rows `DISTINCT ON` returns; the indexed `LATERAL` reads at least one block and under a tenth of the table |

## 88. test_storage_row_repair.py: the alpha5 release notes' repair procedure works as written

Two alpha4 defects (#1211, #1275) left a table's `pgcolumnar.storage` row naming a relation that no longer exists, and the upgrade does not repair existing rows. `RELEASE_NOTES_1.0-alpha5.md` gives a query that finds them and two repairs: `REFRESH MATERIALIZED VIEW` for a view, and a same-type `ALTER COLUMN ... USING` for a table. Instructions are validated by performing them, so this performs them.

THE BROKEN STATE IS MADE, NOT INHERITED: each storage row is pointed at a table created and dropped for the purpose. That matches what a real alpha4 build left behind, measured on PG17. THE QUERY IS READ FROM THE NOTES, so an edit to the document is what runs.

SCOPED TO ITS OWN STORAGE IDS. `pgc_conn` gives a private schema in a shared database, and the pgcolumnar catalogs are per database. So the count asserted is of the rows the notes' query returns for the ids this test damaged, the original and the post-repair ones both.

Removal proofs on PG17: disabling `PgColumnarRetargetStorageRelation` reddens it at the premise. Editing the notes' query from `NOT EXISTS` to `EXISTS` reddens it too.

### Every test

| test | what it holds |
| --- | --- |
| `test_the_release_notes_repair_procedure_works` | the notes give the query; it finds neither row before the damage and both after; the damaged table still reads every row; each repair repoints its row; the query then finds nothing; the table and the view hold the same rows |

## 89. test_docs_cross_database_query.py: no documented query asks another database for its extensions

`pg_extension` is a per-database catalog. A statement that reads `pg_database` and tests `pg_extension` answers for the current database only, so it lists every connectable database or none. `docs/installation.md` shipped exactly that as the way to find the databases that need the upgrade (#1288). Only fenced SQL is read, split into statements, so prose that explains the rule cannot trip it. It needs no cluster, so it is in `NO_CLUSTER`.

Red with the old query restored (got 1, want 0). Its premise is red when the fence pattern is broken and nothing is scanned (got 0, wanted at least 10). That is the case where the arm alone would pass on nothing.

### Every test

| test | what it holds |
| --- | --- |
| `test_no_documented_query_asks_another_database_for_its_extensions` | at least ten SQL statements were read; none reads `pg_database` and tests `pg_extension` |
8 changes: 6 additions & 2 deletions test/pytest/expected_tests.txt
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,9 @@
# adding one: `403 tests collected` over the 24 NO_CLUSTER files. `cluster_tests`
# was re-derived in the same run and did NOT move -- 467 -- which is the expected
# answer for a change that touches only the layer and its own guard.
guard_tests 404
# 404 -> 405 for #1288: test_docs_cross_database_query.py, cluster-free, in NO_CLUSTER.
# Re-derived by collection on 038600e (after #1303): `405 tests collected`.
guard_tests 405

# The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran
# in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and
Expand Down Expand Up @@ -543,4 +545,6 @@ guard_tests 404
# own test is not in it. Git conflicted again -- loudly, because the key moved
# on both sides -- and the answer is neither branch's number. Collected on the
# rebased tree, not added up.
cluster_tests 495
# 495 -> 496 for #1288: test_storage_row_repair.py. Re-derived by collection on
# 038600e (after #1303): `496 tests collected`.
cluster_tests 496
1 change: 1 addition & 0 deletions test/pytest/test_compare_to_bash.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@
"scan_decode_cost",
"sorted_pathkeys",
"stats_privilege",
"storage_row_repair",
"temporal",
"ttl_expire",
"validity_elision",
Expand Down
36 changes: 36 additions & 0 deletions test/pytest/test_docs_cross_database_query.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
"""No documented query asks another database what extensions it has (#1288).

pg_extension is a per-database catalog. A statement that reads pg_database and tests
pg_extension in the same breath answers for the CURRENT database only, so it lists every
connectable database or none. docs/installation.md shipped exactly that query as the way
to find the databases that need `ALTER EXTENSION pgcolumnar UPDATE`.

The pytest half of the docs_style.sh arm of the same name, written independently: its
own parser for the ```sql blocks, split into statements on ';'. Prose that explains the
rule cannot trip it, because only fenced SQL is read.

Needs the repository only, not a cluster or the driver.
"""
import pathlib
import re

REPO = pathlib.Path(__file__).resolve().parents[2]
DOCS = sorted((REPO / "docs").glob("*.md")) + [REPO / "README.md"]
FENCE = re.compile(r"^[ \t]*```sql[ \t]*\n(.*?)^[ \t]*```", re.S | re.M)


def _statements():
out = []
for path in DOCS:
for block in FENCE.findall(path.read_text()):
out.extend(s.strip() for s in block.split(";") if s.strip())
return out


def test_no_documented_query_asks_another_database_for_its_extensions(expect):
stmts = _statements()
expect.at_least(sum(1 for s in stmts if re.search(r"\bSELECT\b", s)), 10,
"premise: the scan read SQL statements from the documents")
bad = [s for s in stmts if "pg_database" in s and "pg_extension" in s]
expect.num(len(bad), 0,
"no documented query tests pg_extension while listing databases")
2 changes: 2 additions & 0 deletions test/pytest/test_harness_deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@
# Landed on main in #922 after this list was written, and the arm above caught
# it: the property says it needs no database, so the declaration must say so too.
"test_suite_accounting.py",
# #1288: reads docs/*.md and README.md only.
"test_docs_cross_database_query.py",
# The classifier's own controls, split out of this file so the job whose file
# list IS this list actually runs them. This file cannot be in the list: it
# hands cluster-bound file names to pytest, so it needs what they need.
Expand Down
Loading
Loading