Skip to content

fix: a collation change without a rewrite forgets the column's stored bounds (#1327) - #1329

Merged
jdatcmd merged 6 commits into
commandprompt:mainfrom
OffgridwithJD:fix/1327-collation-zonemaps
Oct 13, 2026
Merged

jdatcmd merged 6 commits into
commandprompt:mainfrom
OffgridwithJD:fix/1327-collation-zonemaps

Conversation

@OffgridwithJD

@OffgridwithJD OffgridwithJD commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1327, which you filed while reviewing #1325.

The bug

ALTER COLUMN ... TYPE text COLLATE <other> keeps the relfilenode, so no zone map was rewritten. The stored minima and maxima were the old collation's, and the reader compared them under the new one. Red first, on PG17 at 11f954e, against heap:

query after C → en_US.utf8 columnar heap
min(s) || '|' || max(s) B0|m6 a0|m6
count(*) WHERE s < 'B' 0 750
count(*) WHERE s = 'a0' 0 107
a columnar partition, ALTER through its partitioned parent wrong
a projection sorted on the column, range wrong

column_extent() also still printed the C bound.

The fix: forget the bounds rather than recompute them

The ProcessUtility hook handles any ALTER TABLE that changes a column type.

  • Before the statement, it records each columnar relation's relfilenode and column collations from the syscache. That covers the named relation and every inheritor, which is how an ALTER on a partitioned parent reaches its partitions. It does not open the relations, so it takes no lock that the ALTER would then have to upgrade.
  • After the statement, for each relation whose relfilenode is unchanged (not rewritten), every column whose collation changed gets PgColumnarForgetColumnOrdering:
    • The zone-map minima and maxima are nulled, whole-chunk and per-vector, in the base storage and in every projection storage. A projection stores [rownumber, cols…], so its column index is the list position + 1.
    • The column's extent rows are deleted.
    • Counts and sums stay, because no ordering decides them.

A zone map with values and no bounds is already the state a type with no ordering writes, and the readers treat it as "nothing known":

  • no pruning;
  • expire keeps the group;
  • the extent recompute refuses, so the extent stays absent until a rewrite.

One reader was wrong about that state. The ungrouped min/max fold skipped such a group as if it were all NULL, and returned NULL where heap returned a value. It now sends the group to be read, as it already does for an added column.

Recomputing instead would decode the column in every group, inside the ALTER and under its ACCESS EXCLUSIVE lock. Not rewriting was supposed to spare the user exactly that cost. A later rewrite builds correct zone maps, either vacuum_sorted/pgcolumnar.vacuum or a type change that does rewrite.

The sorted-pathkeys claim was already safe. It refuses every collatable column for exactly this reason. vacuum_sorted's ORDER BY arm is green on main and is that refusal's guard, dated by a mutant that removes it.

Found on the way

PgColumnarReadExtentRows was first given an unregistered GetLatestSnapshot(). PG18 asserts a heap scan's snapshot is registered, so every collation change aborted on the 18 and 19 assert builds. Caught by the five-major run and fixed in its own commit.

Evidence

  • collation_change.sh (11 checks) and the independent test_collation_change.py pass on PG15–19, with no TRAPs. The twins grade 12/12, the 12th being the shell's pgc_skip name, mirrored by the twin's cannot_run.
  • Seven mutations, each red on its own arms:
    • nothing forgotten (7 arms);
    • projections skipped;
    • the projection index off by one;
    • the named relation only;
    • the min/max fold guard removed;
    • the extent kept;
    • the sorted claim's collation refusal removed.
  • Ledger: 11 rows, 8 dated, 3 never (all premises). Rebased onto 4c72242 (feat: a no-op expire answers from the column's extent, not a zone-map probe per group (#1310) #1326): 1796 → 1799 over 2270 rows. The gate gives rc=0, and suites_not_covered stays 246.
  • PG17 neighbours: native_agg 35/35, native_agg_addcolumn 17/17, native_agg_deletes 8/8, native_zonemap 18/18, sorted_pathkeys 113/113, projections 73/73, projection_rewrite 84/84, column_extent 36/36, alter_column_type 21/21, docs_style 58/58.
  • Guard half 419/419. cluster_tests 525 → 526. TESTS.md section 103. CHANGELOG Fixed.

A server with no en_US collation fails the shell suite. pgc_skip passes it only when PGC_ALLOW_MISSING_EN_US_COLLATION=1 waives it, and the twin's cannot_run makes the run INCOMPLETE. Neither can pass vacuously. CI's Ubuntu images have en_US.utf8, but I haven't confirmed that inside the cluster jobs; this PR's CI will.

🤖 Generated with Claude Code

jdatcmd and others added 6 commits October 13, 2026 13:47
…ing as heap does, red (commandprompt#1327)

Seen red on PG17 at 11f954e: min|max B0|m6 against heap's a0|m6, WHERE s < 'B'
empty against 750 rows, an equality on 'a0' 0 against 107, the extent still
the C bound, and the same through a partitioned parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…andprompt#1327)

The projection arm was seen red on PG17 at 11f954e, and red again with only the
base storage's ordering forgotten. The sorted-table arm is green on main: the
sorted-pathkeys claim already refuses collatable columns, and this arm is what
keeps that refusal. Two arms that nothing could redden are removed (a
projection never claims an order, and a rolled-back ALTER rolls its catalog
back by construction).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… bounds (commandprompt#1327)

ALTER COLUMN ... TYPE text COLLATE <other> keeps the relfilenode, so the zone
maps kept minima and maxima under the old collation and the reader pruned by
them under the new one. The ProcessUtility hook records each columnar
relation's relfilenode and column collations before an ALTER that changes a
type (the named relation and its inheritors), and after it, for every relation
not rewritten, forgets the ordering of each column whose collation changed:
zone-map minima and maxima nulled (base and projection storages), extent rows
deleted. Counts and sums stay. A zone map with values and no bounds is the
state a type with no ordering already writes; the ungrouped min/max fold now
reads such a group instead of skipping it as all-NULL. collation_change.sh
11/11 on PG17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ELOG (commandprompt#1327)

The twin was seen red on PG17 against main (min|max Q0|x8 against heap's
c0|x8) and passes 11/11 with the fix. Its own values, geometry, two list
partitions, and a projection that lists the column second. A server with no
en_US collation is cannot_run under the shell's pgc_skip name, so the pair
grades 12/12. cluster_tests 524 -> 525 by collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ollation change (commandprompt#1327)

PG18 asserts that a heap scan's snapshot is registered or active; on an assert
build every collation change aborted on 18 and 19. Seen in the five-major run;
collation_change.sh 11/11 and its twin on 15-19 after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… mutations (commandprompt#1327)

collation_change.sh 11/11 and test_collation_change.py on each of PG15-19.
Ledger 2203 / 1748, gate rc=0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the fix/1327-collation-zonemaps branch from c5023d1 to e21d3f3 Compare October 13, 2026 13:48

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at e21d3f3. Merging when CI is green on this head.

The risk I most wanted to rule out is closed. Could a projection sorted on the column still supply ORDER BY in the old collation's order? That would be wrong even with the bounds nulled. It can't happen: the sorted-scan path refuses any column with a collation (columnar_customscan.c, OidIsValid(att->attcollation)), and the comment there records this very case. Probed on PG17: ORDER BY s LIMIT, DISTINCT ... ORDER BY and GROUP BY ... ORDER BY all plan a Sort (s COLLATE "en_US.utf8"). Range and equality reads through the projection match heap.

Aggregates and awkward ALTER shapes, all matching heap on PG17:

  • min/max under the default plan, the ungrouped fold (with and without a filter), and the grouped path;
  • a collation change plus a rewriting type change (rewritten, fine);
  • a collation change plus ADD COLUMN (6/6 bounds nulled);
  • a rolled-back ALTER, a savepoint rollback, ALTER then a read in the same transaction, and a DO block;
  • a partitioned parent with a heap and a columnar partition.

A domain with the same C collation nulls nothing, so no bounds are lost.

Runs at e21d3f3:

  • collation_change.sh: 11/11 on PG17, PG18 and PG19. PG19 is not in CI's build matrix.
  • test_collation_change.py on PG17: green.
  • native_projection.sh on PG17: 8/8.

The ledger is right. Main's rows are unchanged. 2270 rows, of which 1799 are never red, equal to the budget. The three never-red collation rows are premises.

Not examined: dump/restore and pg_upgrade of nulled bounds.

@jdatcmd
jdatcmd merged commit c90976d into commandprompt:main Oct 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zone maps keep the old collation after ALTER COLUMN ... COLLATE, so queries return wrong results

2 participants