Skip to content

fix: dump a rollup's definition; refresh refuses a restored source by its OID (#1377) - #1405

Merged
jdatcmd merged 7 commits into
commandprompt:mainfrom
OffgridwithJD:fix/1377-rollup-dump
Oct 14, 2026
Merged

jdatcmd merged 7 commits into
commandprompt:mainfrom
OffgridwithJD:fix/1377-rollup-dump

Conversation

@OffgridwithJD

@OffgridwithJD OffgridwithJD commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Update, restated onto fbbd16c (after #1406), head 48ea997. The figures below this section are from the first head (ac873b4). The current ones:


Fixes #1377. This is the design @jdatcmd agreed on #1377, using his plain-OID detection.

Change (in pgcolumnar--1.0-alpha5.sql and identically in the alpha4→alpha5 upgrade script):

  • pgcolumnar.rollup is registered with pg_extension_config_dump, so a dump carries the definition. rollup_group stays undumped, because it names row groups of storage that a restore replaces.
  • A new column source_oid oid, set by create_rollup and by rebase_rollup. A dump writes source as a name and source_oid as a number. After any restore the two disagree, so refresh_rollup refuses with 55000. The DETAIL says the rollup was restored, and the HINT names rebase_rollup. The check needs no privilege. pg_upgrade keeps relation OIDs and the storage files, so a rollup stays valid across an upgrade without a rebase.

Why the storage id isn't enough (measured): storageid_seq restarts with the extension, so in both a new database and the same database after DROP EXTENSION the restored source gets the recorded storage id (10000000000).

Arms (test/rollup.sh and test_rollup.py::test_rollup_dump_and_restore, independent fixtures). Each runs in its own databases with a non-superuser owner. The partials are heap, so the source is the only columnar table and the storage id collides by construction:

  • the definition survives the restore;
  • a restore into another database, and into the same database after DROP EXTENSION, gets the owner's refresh refused with 55000 and the rollup untouched;
  • the refusal names rebase_rollup;
  • after the rebase a refresh folds 0, then exactly the rows appended since, and the rollup equals a full recompute.

Evidence:

  • Red on PG17 against the tree without the fix: 7 arms, 42704 "not a rollup" and no definition. The pytest twin is red at the definition arm.
  • Mutant, the fix without the source_oid check: 4 arms red on PG17. The refresh succeeds and folds the restored rows a second time (177|808d… vs the recompute's 177|de33…), so the OID check is what catches the collision.
  • Green: rollup.sh 95/95 on PG15, 16, 17, 18 and 19, and test_rollup.py 2/2 on PG17 and PG18.
  • Guard half 433 passed. cluster_tests 551 → 552: one new test function, and the bare collect delta against main is +1.
  • Ledger: 13 new rows from the five-major logs. Seven are dated by the two reds above; the six fixture dump/restore arms arrive never. Budget 2041 → 2047, and the census agrees.

Merge order: after #1376. Before it, a rollup row whose source was dropped still exists. config_dump writes a dangling regclass as a bare number (see alter_am_cleanup.sh), so the dump would carry a row pointing at whatever later gets that OID. Once #1376 removes those rows, nothing dangling is dumped. rebase_rollup should then also re-record #1376's dependency; I'll add that when rebasing onto it.

Note: the pgcolumnar schema outlives DROP EXTENSION because it isn't an extension member, as alter_am_cleanup.sh already records. So the same-database fixture drops it before restoring, as a user would have to.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced at ac873b4 on PG18. rollup is 95/95. Without the source_oid check (IF false AND r.source::oid <> r.source_oid), it's 91 + 4: both restore refusals got [ok|151|cf00…] want [55000|151|4c34…], the hint, and and the restored rollup equals a full recompute: got [177|808d…] want [177|de33…]. That's the double count, caught only by the OID check, as you measured.

On the merge order and rebase_rollup: #1406 (#1376) records the dependencies from an AFTER INSERT OR UPDATE OF rollup, partials, source trigger on pgcolumnar.rollup, not from create_rollup. A restore writes the config-dumped rows back with COPY, which fires the trigger, so the restored view gets its dependency on the restored source with no change to rebase_rollup. Once you rebase onto #1406, one arm would pin it: dump → restore → DROP TABLE src refused (2BP01), in both twins. Your restored-database fixture already has everything that arm needs.

Approving. I'll merge it after #1406, as you propose, once it's restated and green.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, #1406 is in (fbbd16c), so #1405 can rebase onto it. As noted in my approval, the rollup trigger fires when a restore re-inserts the rows, which re-records the dependency. One dump → restore → DROP TABLE src refused arm would pin that.

🤖 Generated with Claude Code

jdatcmd and others added 6 commits October 14, 2026 23:28
… its OID (commandprompt#1377)

pgcolumnar.rollup is registered with pg_extension_config_dump, so a dump
carries the definition. rollup_group is not dumped: it names row groups of
storage a restore replaces. The rollup records its source's OID beside the
storage id. A dump writes source as a name and source_oid as a number, so
after a restore into another database, or into the same one after DROP
EXTENSION, the two disagree. Refresh then refuses and names rebase_rollup,
which records the OID again. The storage id alone cannot tell: storageid_seq
restarts with the extension, so a restored source gets the recorded id, and
refresh would fold every restored row into the restored partials again.
Design by jdatcmd on commandprompt#1377.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho
…the NOT NULL column commandprompt#1377 adds

Without it their INSERTs fail on 23502 before the dependency trigger runs, and
the four arms that expect the trigger's refusal see the wrong error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjTxfQJj5pnii9wzmPP9Ho
@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd I'm restating onto fbbd16c now. The 2BP01 arm (dump → restore → DROP TABLE src refused) is in both twins. It's red without #1406 (got [ok|]) and green with it on PG17. Merging the two also needs one change on my side: #1406's hand-written pgcolumnar.rollup rows in both twins now give source_oid. Without it they fail with 23502 (my NOT NULL column) before your trigger runs, and four of your arms saw the wrong SQLSTATE. Five-major runs are going now. I'll post the head once the ledger row is in.

🤖 Generated with Claude Code

@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd #1405 is restated onto fbbd16c, now 48ea997. The update section at the top of the description has the figures: 104/104 on PG15–19, the 2BP01 arm dated by the tree without #1406, and ledger 2766 / 2093, ceiling 234. Ready once CI is green.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified 48ea997 against my approval of ac873b4. It sits on current main (fbbd16c). Outside the count files, the patch adds the restore → DROP TABLE refused (2BP01) arm in both twins, which pins the trigger re-recording the dependency on restore, and fills source_oid in #1406's direct-insert fixtures for the new NOT NULL column. No main row changes (2766 / 2093, ceiling 234). I'll merge once CI is green.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rollups do not survive pg_dump/pg_restore: the view and partials come back, the rollup catalog does not, and refresh says 'not a rollup'

2 participants