feat: incremental rollups, phase 1 (append-only) (#1294)

CHANGELOG.md

@@ -142,6 +142,25 @@ true until the next version shipped.
   long, and refuses, before creating anything, a call that would create more
   than 1000 partitions (`22023`) and a parent with a partition bounded by
   `MAXVALUE` (`55000`).
+- Incremental rollups, phase 1 (#1294). `pgcolumnar.create_rollup` defines an
+  aggregate per `date_bin` time bucket and `group_by` key (count, sum, min, max,
+  avg), kept as a view over a partials table; `pgcolumnar.refresh_rollup` folds
+  only the row groups appended since the last refresh, in one statement, reading
+  each new group through its own TID range. Late rows are folded into their
+  buckets, rows committed during a refresh are folded by the next one, and two
+  concurrent refreshes fold each row once. The rollup equals a full recompute:
+  exactly for counts, integer sums and averages, min and max, and within 1e-12
+  for `float8` averages. Rows that `expire` retires keep their buckets in the
+  rollup, so the rollup's history outlives the raw data. This phase refuses any
+  other delete or update in the source (`0A000`) and a rewritten source
+  (`55000`, until `pgcolumnar.rebase_rollup`). Each group's TID range comes from
+  the access method's own mapping (`pgcolumnar.row_tid`), not a block size
+  restated in SQL.
+  The rollup catalogs carry row-level security, so each role sees only the
+  rollups it owns. Found while it was written: a lateral per-group subquery
+  pulled up into a join read every row group (114 blocks of a 103-block table
+  for one new group), because PostgreSQL builds a TID range scan only from a
+  relation's own restrictions; `OFFSET 0` keeps it a scan (15 blocks).
 
 ### Changed
 

docs/how-to.md

@@ -457,6 +457,56 @@ When a partition is dropped by retention, its pgcolumnar catalog rows go with it
 
 `test/partman_columnar.sh` checks each of these statements.
 
+## Keep a rollup up to date
+
+A rollup keeps an aggregate per time bucket, such as hourly averages, and folds
+in only the rows added since the last refresh. Queries read the rollup instead
+of the raw table.
+
+```sql
+SELECT pgcolumnar.create_rollup(
+    'metrics_hourly', 'metrics', 'ts', '1 hour', '2026-01-01 00:00:00+00',
+    group_by      => '{device}',
+    agg_functions => '{count,sum,min,max,avg}',
+    agg_columns   => '{*,value,value,value,value}');
+
+SELECT pgcolumnar.refresh_rollup('metrics_hourly');   -- returns rows folded
+SELECT * FROM metrics_hourly WHERE bucket >= now() - interval '1 day';
+```
+
+The rollup is a view with a `bucket` column, the `group_by` columns, and one
+column for each aggregate. The aggregate columns are named `count`, or the
+function and the column, as in `sum_value` and `avg_value`. The buckets come
+from `date_bin(width, ts, origin)`, so every session gets the same buckets,
+whatever its time zone. The width cannot include months.
+
+A refresh reads only the row groups added since the last refresh. Rows that
+arrive late, with old timestamps, are still folded into their buckets. Rows
+that commit while a refresh runs are folded by the next refresh. Two refreshes
+that run at the same time fold each row once. A refresh after a small append
+reads a small part of the table.
+
+The rollup equals what the same `GROUP BY` over the whole table returns. Count,
+sum, min and max are exact, and so is the average of an integer column. The
+average of a `float8` column can differ from a full recompute in the last digits.
+
+**Retention keeps the rollup's history.** When `pgcolumnar.expire`, or the
+maintenance daemon, drops old rows of the source, the rollup keeps its rows for
+those buckets. So, for example, hourly averages can outlive the raw data.
+
+**This version folds appended rows only.** If rows of the source are deleted or
+updated in any other way, `refresh_rollup` refuses (`0A000`) and changes
+nothing. If the source
+is rewritten, for example by `pgcolumnar.vacuum`, `refresh_rollup` refuses with
+`55000`. If the rollup was current before the rewrite, run
+`pgcolumnar.rebase_rollup` to record the rewritten table and continue.
+
+Run `refresh_rollup` at READ COMMITTED. A role that is not a superuser needs
+`USAGE` on schema `pgcolumnar`, which a superuser grants. Each role then sees
+and refreshes only the rollups that it owns.
+
+`test/rollup.sh` checks each of these statements.
+
 ## Keep tables optimized automatically
 
 A background daemon can compact and recluster tables on a schedule. The threshold

docs/sql-reference.md

@@ -591,6 +591,37 @@ refused before it creates any (`22023`). The function takes
 `SHARE UPDATE EXCLUSIVE` on the parent, never `ACCESS EXCLUSIVE`. See
 [Create time partitions ahead](how-to.md#create-time-partitions-ahead).
 
+### pgcolumnar.create_rollup(rollup name, source regclass, ts_column name, width interval, origin timestamptz, group_by name[], agg_functions text[], agg_columns name[], access_method name DEFAULT 'pgcolumnar') returns regclass
+
+Creates a rollup of `source` in the source's schema. A rollup is a view of
+`date_bin(width, ts_column, origin)` buckets per `group_by` key, with one column
+per aggregate. Each aggregate is a function from `agg_functions` (`count`,
+`sum`, `min`, `max` or `avg`) applied to the column at the same position in
+`agg_columns`. `*` goes only with `count`. `ts_column` must be `timestamptz`.
+The data is kept in a table named `<rollup>__partials`, with access method
+`access_method`. The function refuses a width with months, an unknown function,
+arrays of different lengths, and a column the source does not have (`22023`).
+See [Keep a rollup up to date](how-to.md#keep-a-rollup-up-to-date).
+
+### pgcolumnar.refresh_rollup(rollup regclass) returns bigint
+
+Folds the rows of the row groups added to the source since the last refresh,
+and returns the number of rows folded. Runs at READ COMMITTED only (`25000`).
+Row groups that `pgcolumnar.expire` retired are dropped from the refresh's
+record, and the rollup keeps their buckets. Refuses after any other delete or
+update in the source (`0A000`), and after a rewrite of the source (`55000`).
+
+### pgcolumnar.row_tid(row_number bigint) returns tid
+
+Returns the item pointer that a columnar table gives the row with this row
+number. Row numbers start at 1 (`22003` otherwise).
+
+### pgcolumnar.rebase_rollup(rollup regclass) returns bigint
+
+Records the source's current row groups as folded, without changing the rollup,
+and returns the number of row groups recorded. Use it after a rewrite of the
+source, when the rollup was current before the rewrite.
+
 ### pgcolumnar.import_arrow(rel regclass, path text) returns bigint
 
 Inserts the rows of an Arrow IPC stream file at `path` into the existing table