Skip to content

fix: the storage-row repair opened partitions an ALTER never locked - #1299

Merged
jdatcmd merged 1 commit into
commandprompt:mainfrom
OffgridwithJD:fix/retarget-locks-partition
Oct 12, 2026
Merged

jdatcmd merged 1 commit into
commandprompt:mainfrom
OffgridwithJD:fix/retarget-locks-partition

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd, this is the alpha5 regression I asked to hold the tag for (#1283). It lands only on your approval.

The defect

The #1211 repair gate (e9cc924, new in alpha5) calls PgColumnarRetargetStorageRelation for every inheritor of an ALTER's target, and that function opened each with NoLock. An ALTER that doesn't recurse locks only the parent and the partition it names. With two or more columnar partitions:

statement assert build release build
ALTER TABLE m DETACH PARTITION m_2 CONCURRENTLY backend aborts; partition left half-detached unlocked open of m_1
ALTER TABLE m DETACH PARTITION m_2 backend aborts unlocked open
ALTER TABLE m ATTACH PARTITION m_3 ... backend aborts unlocked open
ALTER TABLE ONLY m ALTER COLUMN v SET DEFAULT 0 backend aborts unlocked open
ALTER TABLE m ALTER COLUMN v SET DEFAULT 0 (recurses) fine fine

The backtrace from gdb attached to the backend:

ExceptionalCondition  "lockmode != NoLock || IsBootstrapProcessingMode() || CheckRelationLockedByMe(r, AccessShareLock, true)"  relation.c:67
relation_open (relationId=16520, lockmode=0)
PgColumnarRetargetStorageRelation (relid=16520)   src/columnar_metadata.c:2297
pgcolumnar_process_utility                        src/columnar_tableam.c:3015

With one partition there is no other partition to open, which is why nothing has caught it. The sibling PgColumnarRerecordProjectionsAfterRewrite takes AccessShareLock, as it did in alpha4.

The fix, and the alternative I rejected

if (!CheckRelationOidLockedByMe(relid, AccessShareLock, true))
    return;

A relation the statement rewrote is locked by it until commit, so a relation this transaction holds no lock on can't need its storage row repointed. CheckRelationOidLockedByMe(Oid, LOCKMODE, bool) has the same signature on 15 through 19 (checked in each prefix's lmgr.h).

Rejected: taking AccessShareLock instead. It's correct, but it costs one lock per partition on every non-recursing ALTER. A time-series table with thousands of partitions could exhaust the shared lock table on an ALTER TABLE ONLY that needs one lock. Its sibling avoids the same cost by returning before it opens anything when a partition has no projection declarations.

Arms that work on every build

An "it did not crash" arm is green on unfixed code on a release build, because the Assert is compiled out. So each arm reads the blocks of the partition the ALTER did not name, from pg_statio, flushed by the ALTER's own session. That also works for DETACH CONCURRENTLY, which can't run in a transaction block. The unfixed gate reads that partition's metapage; the fix reads nothing. There's a control showing a real read of it is visible.

                     PG15 PG16 PG17 PG18 PG19 (assert)     PG18 release
fix                  18   18   18   18   18  passed        18 passed
unfixed              4 red each: "session aborted"         4 red: got [1] want [0]
skip-every-relation  3/3/4/4/4 red: the existing repair arms plus
                     "a rewrite through the parent still repoints every partition's storage row"

The skip-every-relation mutation shows the new early return doesn't skip what it has to repair. It's 3 on 15/16 because the matview WITH DATA arm is green there by design, as #1284 recorded.

Two of my own vacuous drafts, both caught by running the unfixed code. The first DETACH CONCURRENTLY arm checked row counts, which are right either way, because the abort comes after the detach commits. And my first lock-based design read debug_assertions after the crashing ALTERs: SHOW hit a restarting server and the arm turned into a skip. Neither shape is left.

The rest

  • Pytest twin test_the_repair_gate_leaves_untouched_partitions_alone: 3 passed. Unfixed, it refuses the aborted session's None on an assert build and measures got 1 on a release build. Parity is 0 missing. cluster_tests is 491 by collection.
  • Ledger 1859 → 1866 rows; never 1651 → 1653; dated 208 → 213. Merged one major at a time with named targets.
  • harness_selftest, docs_style and the pytest guard half (404) pass on the rebased head (9f2cbc6 on d5eb40b).

🤖 Generated with Claude Code

The repair gate added for commandprompt#1211 runs PgColumnarRetargetStorageRelation for
every inheritor of an ALTER's target, and opened each with NoLock. An ALTER
that does not recurse -- DETACH, DETACH CONCURRENTLY, ATTACH, ALTER TABLE
ONLY -- locks only the parent and the partition it names, so with two or more
columnar partitions the gate opened a relation nothing held:

    TRAP: failed Assert("lockmode != NoLock || ... CheckRelationLockedByMe ...")
    relation_open(16520, NoLock) <- PgColumnarRetargetStorageRelation
                                 <- pgcolumnar_process_utility

Every assert-enabled build aborts the backend; a release build opens the
partition unlocked, racing a concurrent DROP. Measured on PG 15 and 17. New in
alpha5 (e9cc924); absent from v1.0-alpha4. Found probing DETACH CONCURRENTLY
for commandprompt#1297, which the abort also left half-detached.

The gate now skips a relation the statement holds no lock on: a relation the
statement rewrote stays locked by it until commit. AccessShareLock on every
partition would also be correct, at one lock per partition on every such
ALTER, which a table of thousands of partitions can exhaust.

The arms read the blocks of the partition the ALTER did not name, flushed by
the ALTER's own session, so they hold on every build:

    fix          18 passed                    PG 15..19 assert; PG 18 release
    unfixed      4 red: session aborted       assert builds, every major
                 4 red: got [1] want [0]      PG 18 release build
    skip-all     the existing repair arms and "a rewrite through the parent
                 still repoints every partition's storage row" go red

Pytest twin: 3 passed; unfixed it refuses the aborted session's None on an
assert build and measures got 1 on a release build. Parity 0 missing.

Ledger 1859 -> 1866 rows, never 1651 -> 1653, dated 208 -> 213.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 9f2cbc6. Merging now: CI is 15/15 green on this head. This was the tag blocker.

Reproduced on a release build, where "it didn't crash" proves nothing

PG17 non-assert (/usr/local/pg17_nc), my own worktrees from this head. The second run has the new early return deleted:

fix        rewrite_storage_oid   18 passed + 0 failed
unfixed    rewrite_storage_oid   14 passed + 4 failed
  the repair gate does not read a partition the ALTER did not touch: ONLY lp ALTER COLUMN v SET DEFAULT 0   got [1] want [0]
  ... ATTACH PARTITION lp_3 ...                                                                            got [1] want [0]
  ... DETACH PARTITION lp_2                                                                                got [1] want [0]
  ... DETACH PARTITION lp_2 CONCURRENTLY                                                                   got [1] want [0]

Those are exactly your four arms, and they're red on the build type where the Assert is compiled out. That is the property the arms needed.

The reasoning holds

  • The skip is safe. A relation the statement rewrote is locked by it until commit, so CheckRelationOidLockedByMe(relid, AccessShareLock, true) is true for every relation that can need repointing. Your skip-every-relation mutation is the evidence that the early return doesn't drop one of those: the existing repair arms go red under it.
  • A lock taken by an earlier statement in the same transaction also passes the check. That is correct: the open is then genuinely covered by a held lock, which is the Assert's own condition.
  • Rejecting AccessShareLock per partition is the right call. One lock per partition on an ALTER TABLE ONLY could exhaust the shared lock table on a table with thousands of partitions, which is exactly the #1297 workload.

Ledger: 1866 rows, 1653 never-red, budget 1653, all on five majors. That reproduces your stated arithmetic.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants