feat: incremental rollups, phase 1 (append-only) (#1294)

[OffgridwithJD]

Incremental rollups, phase 1: append-only (#1294)

This follows the design settled on #1294, points 1–12, with the owner's decisions. The SQL goes into the alpha5 scripts.

pgcolumnar.create_rollup(rollup name, source regclass, ts_column name, width interval,
                         origin timestamptz, group_by name[], agg_functions text[],
                         agg_columns name[], access_method name DEFAULT 'pgcolumnar') RETURNS regclass
pgcolumnar.refresh_rollup(rollup regclass) RETURNS bigint   -- rows folded
pgcolumnar.rebase_rollup(rollup regclass) RETURNS bigint    -- row groups recorded

The model. A rollup is a view over a <rollup>__partials table, keyed by (date_bin bucket, group_by...). The table holds count, sum, min and max, plus a sum and a count for each avg. The view turns those into the aggregates.

The refresh. It folds the rows of the row groups appended since the last refresh, in one statement (point 1). The group list, the folded rows and the recorded groups therefore share one snapshot. Each new group is read through its own TID range, which #1306 made possible.

Settled points, each an arm

• One snapshot (point 1): rows committed while a refresh runs (held open by the developer setting pgcolumnar.rollup_pause) are folded by the next refresh, exactly once.
• Buckets from date_bin (point 4): refreshes from sessions in different time zones build the same rollup. A width with months is refused (22023).
• Exactness (point 5): count, integer sum, min and max are exact, and so is the avg of an integer column. The avg of a float8 column is within a relative 1e-12. The two twins cover one case each.
• Concurrency (point 7): two refreshes started together fold each row once, because SHARE UPDATE EXCLUSIVE on the partials table conflicts with itself. REPEATABLE READ is refused (25000).
• Source identity (point 9): a rewritten source (a new storage id) is refused with 55000. rebase_rollup then records its groups without touching the rollup.
• Point 10: the refresh and rebase functions take regclass.
• Point 11: aggregates are structured arrays checked against the five that combine. Each bad input is refused with 22023, with an arm per refusal.
• Point 12: the access_method option is in, and a heap rollup behaves the same. Your columnar-vs-heap measurement is still to do.
• Phase 1 scope: any delete or update in the source is refused (0A000), and the rollup is left unchanged. Dirty buckets are phase 2.

What the arms found while it was written

• A lateral per-group subquery was pulled up into a join, and PostgreSQL builds a TID range scan only from a relation's own restrictions. So the refresh read every row group: 114 blocks of a 103-block table, for one new group. OFFSET 0 keeps it a scan, at 15 blocks. The cost arm requires a refresh after a small append to read under a fifth of the source, and at least one block.
• pgcolumnar.vacuum gives a table a new storage id and may keep its group numbers, so the identity check uses the storage id.
• Superuser runs bypass row-level security. Every arm run as postgres went past the policies on the rollup catalogs. So an owner and a stranger are tested as ordinary roles: the owner creates, refreshes and reads its rollup, and the stranger sees none of its catalog rows and can't refresh it. PUBLIC has no USAGE on schema pgcolumnar, by design (docs/administration.md), so the arms grant it as an administrator would. The docs say so.
• The first draft's source load overflowed int, leaving the table empty, and three arms passed on emptiness. Each now has a premise that requires a real measure.

Tests

• test/rollup.sh and the independent test/pytest/test_rollup.py: 45/45 on PG15–19, each.
• compare_to_bash covers every shell property.

21 mutants, each red on its arm:

• no OFFSET 0;
• no isolation check;
• no lock;
• no identity check;
• no changed-group check;
• the folded groups recorded from a fresh read;
• min combined as max;
• no row-level security;
• a rebase that clears the rollup;
• the five creation checks, each removed;
• only the first new group folded;
• the snapshot ignored;
• date_trunc bucketing;
• access_method ignored;
• no SECURITY DEFINER on the group lister;
• no catalog grants;
• a rebase that returns 0.

Three arms didn't hold their own guard on the first try: a shorter columns array was caught by the column check, a missing column under sum by the cannot-be-folded check, and a refused create that succeeds leaves a table. Their inputs were sharpened so each now discriminates.

Registries

• Ledger: 40 rows from a five-major run and the 21 mutations, on main's ledger (1984/1705). 31 are dated, and no premise is among them; three checks that measure the code were renamed from premises first (test: 87 arms have survived every mutation aimed at their neighbours: vacuous, or named for something they do not measure #1236). 9 are never: 7 premises, plus 2 arms the budget comment names. Census 1705 → 1714, and the gate agrees.
• TESTS.md section 96, the same number as feat: pgcolumnar.tier_partition moves a cold partition to Parquet, exactly once (#1295) #1317 and docs, test: gap-fill, LOCF and interpolation recipes, held to a reference (#1296 phase 1) #1311; whichever merges later renumbers. cluster_tests 504 by collection.
• Guard half: 415. docs_style passes. native_upgrade_converge PG17: 21/21, so the rollup catalogs, policies and grants converge. harness_selftest PG17: 1223.

Next, as agreed on #1314

Once #1314 lands, the refresh will honour pgcolumnar.retired_group. A recorded group that's missing because retention retired it is dropped from the snapshot, and its buckets are left alone; that's the owner's keep-history default. It needs no dirty buckets, so it comes as a follow-up PR with its own arm.

🤖 Generated with Claude Code

[jdatcmd]

Review of 62d5fb9. This is the design we settled on #1294, implemented carefully. The single-statement refresh, IS NOT DISTINCT FROM key matching, the SUEL lock, the identity check, and the OFFSET 0 TID-range finding (114 → 15 blocks) are all good. One thing blocks: a hidden coupling that gives silently wrong rows on non-default builds.

Blocking: the row-number → TID mapping hard-codes 291

refresh_rollup builds each new group's TID range as first_row_number / 291 and % 291 + 1. In C, the mapping is COLUMNAR_VALID_ITEMPOINTER_OFFSETS, which is (uint64) MaxHeapTuplesPerPage (columnar.h:168). That's 291 only at the default BLCKSZ of 8192. On a PostgreSQL built with --with-blocksize=16 it's 584, and at 32 it's 1169.

Each new group's ctid range would then cover the wrong rows: some rows folded twice, some never, and nothing errors. A refresh is supposed to equal a full recompute, and this is a silent way for it not to.

Fix: don't restate the constant in SQL. Either expose it from C (pgcolumnar.row_tid(row_number bigint) RETURNS tid calling PgColumnarRowNumberToItemPointer is the most direct), or derive it from current_setting('block_size') with the same formula as MaxHeapTuplesPerPage. Arm: the SQL-side mapping equals the C-side ctid for the first and last row of several groups. That's red if the SQL constant ever disagrees, which a mutation to 290 or 292 shows.

Measured, no change needed

On PG18 assert, built from this head, I ran a 4500-row source in 1000-row groups. Every 7th device id is NULL and every 11th ts is NULL. There were two refreshes, with an append between them, and the rollup was '{count,sum,min,max,avg}' over '{*,v,v,v,v}', grouped by dev, in 1-hour buckets.

full recompute minus rollup   0 rows
rollup minus full recompute   0 rows
groups: full 457, rollup 457; NULL-dev groups 76; NULL-bucket groups 6

Both directions were compared as text, with avg rounded to 9 places.

So NULL group keys and NULL timestamps fold correctly, because IS NOT DISTINCT FROM is used in both the UPDATE and the anti-join. If neither twin has an arm with NULL keys and NULL ts, it's worth one: a future = in place of IS NOT DISTINCT FROM would duplicate the NULL partials, and only such a fixture would notice.

Noting, for the follow-up

With #1314 merged, the daemon's expire (#1307, once merged) retires source groups. In phase 1, every refresh after that is refused with 0A000 until rebase_rollup. That's correct for phase 1, but users who turn on both will hit it. Your keep-history follow-up resolves it; until it lands, a line in the docs would help.

[OffgridwithJD]

All taken, and the retention follow-up is folded in, since #1314 has landed. Committed locally on a rebase onto 953c192. The five-major and mutation runs are going; I'll push when they're in.

• The 291 (blocking). pgcolumnar.row_tid(bigint) RETURNS tid, in C, calls PgColumnarRowNumberToItemPointer, and the refresh uses it. The SQL holds no block-size constant now. Arm: each row group's TID range holds exactly that group's rows, checked against reality: for every group, the count of rows in [row_tid(first), row_tid(last)] equals row_count - deleted_count. It was red before the function existed. A mutant computing row_tid with 290 per block is in the run.
• NULL keys and NULL ts. There's an arm in each twin with NULL group keys and NULL timestamps, refreshed twice around an append. Writing it exposed a weakness in my own digests: both twins built them with ||, which turns a row with any NULL into NULL, and string_agg then silently dropped those rows. They use concat_ws over coalesce now. A mutant with = in place of IS NOT DISTINCT FROM is in the run.
• Retention. A recorded group that expire retired, read from retired_group through a STABLE SECURITY DEFINER helper, now leaves the snapshot, and its buckets keep their rows. That's the keep-history default, so users with the daemon's expire on aren't refused. Any other missing group is still 0A000. Arms: after expire, a refresh folds nothing, refuses nothing and keeps every bucket; an append still folds; recent buckets equal a recompute while the expired ones keep theirs. The docs say so.

So far both twins are 55/55 on PG15 and 16, and every major builds after a clean. That check caught PG_RETURN_ITEMPOINTER being 16+ only.

🤖 Generated with Claude Code

[OffgridwithJD]

Pushed. Head is c6f097d, rebased onto b6656ac (after #1307). Every point of your review is in it, as listed in my comment above.

The gate found something before this push. One check name, and the rollup equals a full recompute, was used by six checks, and gate refused it ("one ledger key covers 6 checks"): under one key, a red in one check would be recorded against the others. Each one now says which step it follows (and after the rebase the rollup equals a full recompute and so on), in both twins. Because the ledger keys on the name, I threw away every earlier log and ran everything again.

Evidence

• Clean, on this exact tree (after feat: the maintenance daemon runs expire when autovacuum_expire is on (#1290) #1307, whose expire changes the retention arms depend on): rollup.sh and test_rollup.py are 55/55 on each of PG15–19.
• 24 mutations, all red on PG16. These ran on the same rollup code before the rebase onto feat: the maintenance daemon runs expire when autovacuum_expire is on (#1290) #1307; my SQL and C are byte-identical across it, and the alpha5 SQL differs from that tree by exactly feat: the maintenance daemon runs expire when autovacuum_expire is on (#1290) #1307's patch. Failed checks per mutant: no_offset0 1, no_iso_check 1, no_lock 1, no_identity_check 1, no_changed_check 1, record_fresh 6, min_as_max 7, no_rls 2, rebase_clears 2, months_check 2, func_allowlist 1, length_check 2, column_check 1, star_check 2, first_group_only 23, ignore_snapshot 20, zone_bucket 3, ignore_am 1, no_definer 2, no_grants 5, rebase_zero 1, eq_keys 1, no_retired_drop 3, map_wrong 19.
• Ledger: 55 rows, merged one log at a time with --mutation and --target. 43 are dated, and no premise is among them. 12 are never: 5 premises, plus 7 arms the budget comment names. The census goes 1686 → 1698 on 2058 rows, and gate passes (rc=0).
• Registries: TESTS.md section 97 (after feat: the maintenance daemon runs expire when autovacuum_expire is on (#1290) #1307's 96). cluster_tests 510 → 511 by collection.
• Guards: the guard half passes 415/415, and docs_style passes. native_upgrade_converge PG17 is 21/21, and harness_selftest PG17 is 1223/1223.

Separately, found while checking that run: native_upgrade_converge.sh leaves its PostgreSQL server running after every run. It calls pgc_setup, which arms trap pgc_on_exit EXIT, and then replaces that trap with its own trap cleanup EXIT. So pgc_teardown never runs: the server keeps going, the workdir stays in /tmp, and a run that dies loses the TERMINATED marker from #1233. Each of my converge runs today left exactly one PG17 server, and their start times match the runs' log times to the second. I'll file it with a guard and a fix as its own PR, not in this one.

[jdatcmd]

Approved at c6f097d. I verified the fixes on this head's build (PG18 assert), without relying on the description.

The block-size coupling is gone: refresh_rollup contains no 291; pgcolumnar.row_tid() appears four times.

Retention, the keep-history default, measured. The source: 3000 rows 30 days old in three 1000-row groups, plus 1000 current rows with every 11th ts NULL and every 7th device NULL, under a 7-day TTL. The rollup: {count,sum,min,max,avg} by dev, in 1-hour buckets.

refresh                                   4000 folded
expire                                    3 groups retired, 3 rows in retired_group
refresh after expire                      0 folded, no refusal, rollup identical to before (EXCEPT both ways: 0)
append 1000, refresh                      1000 folded
buckets the source still backs            full recompute vs rollup: 0 / 0 differences, 210 groups (NULL ts and NULL dev included)
buckets whose raw rows expired            305 kept, still counting their 3000 rows

My first try at this retired nothing: my fixture put NULL timestamps in the old groups, and expire rightly keeps those. So the run above puts the NULLs only in current rows.

The duplicate check name that your gate caught (and the rollup equals a full recompute ×6) is a good find, and so is throwing away every earlier log because of it. I'll merge once CI is green.

[jdatcmd]

@OffgridwithJD, CI at c6f097d fails suites on both PG17 and PG18, on one check:

entry_point_privilege  FAIL  coverage: no entry point is unbucketed: got [ create_rollup row_tid] want [none]

Every SQL entry point has to be placed in one of entry_point_privilege.sh's privilege buckets. refresh_rollup and rebase_rollup evidently are, but create_rollup and row_tid aren't.

row_tid deserves a thought about which bucket it belongs in. It maps a row number to a TID with no table argument and reads nothing, so "no privilege needed" may be right. If it ever takes a relation, it would need the caller-privilege check the others have.

My approval stands on the content. I'll merge once the bucketing is in and CI is green, after comparing the new head's other files against c6f097d.

[OffgridwithJD]

Fixed at 3282b28, one commit on top of c6f097d. The other files' patches are unchanged.

• row_tid → EXEMPT. It maps a row number to a TID by arithmetic, takes no relation and reads nothing. The comment beside it says that a version taking a relation would need the SELECT check, as you suggested. EXEMPT's own guard ("no exempt name actually takes a relation") still holds: its argument is bigint.
• create_rollup → RELATION_BY_OTHER_MEANS. Its relation is the second argument, which the proargtypes[0] rule cannot see. The class goes from two to three, so its arm is renamed the relation-by-other-means class is the three that are known. That suite has no ledger rows, so the rename changes no key.
• Does create_rollup refuse a caller that can't read the source? It does, though not by a check of its own. The arm I wrote expecting red was green on the unfixed code. It reads the source's storage id through get_storage_id, one of your GUARDED_BEHAVIOURAL functions, which raises 42501. So there's no code change. Both twins now assert it: a role that cannot read the source cannot create a rollup of it → 42501. It is red only under a mutation removing get_storage_id's pg_class_aclcheck, and that mutation also reddens entry_point_privilege's get_storage_id refuses an unprivileged caller with 42501.

Evidence on this tree: rollup.sh and test_rollup.py are 56/56 on each of PG15–19, and entry_point_privilege is 31/31 on each. The ledger is 2059 rows and the census stays at 1698, since the new arm is dated. gate rc=0, the guard half passes 415/415, and docs_style passes. TESTS.md section 97 has a paragraph on this.

My miss: I ran my own suites and the guards, but not entry_point_privilege, which is the suite a new SQL function has to satisfy.

[jdatcmd]

Re-approved at 3282b28. The one commit on top of c6f097d touches only tests and the ledger (entry_point_privilege.sh, both rollup twins, TESTS.md, ledger and budget); git diff --stat c6f097d 3282b28 shows no source file. Run here on PG17 from this head: entry_point_privilege is 31/31, including coverage: no entry point is unbucketed and the relation-by-other-means class is the three that are known.

Both buckets are right. row_tid is EXEMPT because it takes a bigint and reads nothing. create_rollup is relation-by-other-means, because its relation is the second argument. Asserting create_rollup's refusal is also right: it's held by get_storage_id's check, and your arm now proves that, where before it was only assumed. Merging when CI is green.

[jdatcmd]

Merging now: approved at 3282b28, 15/15 green.