feat: rollups phase 2, dirty buckets: deletes and rewrites are recomputed, history is never (#1294)

CHANGELOG.md

@@ -184,16 +184,33 @@ true until the next version shipped.
   concurrent refreshes fold each row once. The rollup equals a full recompute:
   exactly for counts, integer sums and averages, min and max, and within 1e-12
   for `float8` averages. Rows that `expire` retires keep their buckets in the
-  rollup, so the rollup's history outlives the raw data. This phase refuses any
-  other delete or update in the source (`0A000`) and a rewritten source
-  (`55000`, until `pgcolumnar.rebase_rollup`). Each group's TID range comes from
+  rollup, so the rollup's history outlives the raw data. A rewritten source is
+  refused (`55000`, until `pgcolumnar.rebase_rollup`). Each group's TID range comes from
   the access method's own mapping (`pgcolumnar.row_tid`), not a block size
   restated in SQL.
   The rollup catalogs carry row-level security, so each role sees only the
   rollups it owns. Found while it was written: a lateral per-group subquery
   pulled up into a join read every row group (114 blocks of a 103-block table
   for one new group), because PostgreSQL builds a TID range scan only from a
   relation's own restrictions; `OFFSET 0` keeps it a scan (15 blocks).
+- Incremental rollups, phase 2: dirty buckets (#1294). `refresh_rollup` no longer
+  refuses deletes, updates and `compact_rewrite` in the source. In the same one
+  statement, it recomputes from the source every bucket a changed row group's
+  rows fall in, and folds new groups into the other buckets only, so a row is
+  never counted twice. Each recorded group now carries the time range of its
+  rows, since a group that is gone has no zone maps left. A recompute never
+  touches history: when a bucket to recompute holds rows retention dropped, the
+  refresh is refused (`0A000`) and writes nothing. The retired groups' ranges
+  are kept as a `tstzmultirange` on `pgcolumnar.rollup`. `rebase_rollup` now reads
+  each group's range from its rows. Found while it was written: retirement read
+  in a statement before the fold would let an `expire` that commits in between
+  look like a delete and lose history, so it is read in the fold's own snapshot;
+  and a bucket's end is its start plus the width in plain seconds, because a
+  `timestamptz` plus `'1 day'` is 23 or 25 hours across a DST change.
+  From @OffgridwithJD's review: rows deleted from a group that `expire` then
+  retires, before the next refresh, cannot be recomputed and would be kept as
+  history, so such a refresh is refused (`0A000`) too; `expire` now records each
+  retired group's deleted count in `pgcolumnar.retired_group` for this.
 - `pgcolumnar.tier_partition(partition, server, path, lock_wait)` moves a cold
   columnar partition to a Parquet file and attaches the file in its place as a
   foreign partition with the same name, bounds, grants and comment (#1295). It

docs/how-to.md

@@ -568,9 +568,10 @@ When a partition is dropped by retention, its pgcolumnar catalog rows go with it
 
 ## Keep a rollup up to date
 
-A rollup keeps an aggregate per time bucket, such as hourly averages, and folds
-in only the rows added since the last refresh. Queries read the rollup instead
-of the raw table.
+A rollup keeps an aggregate per time bucket, such as hourly averages. A refresh
+folds in the rows added since the last refresh, and recomputes only the buckets
+whose rows were deleted or updated. Queries read the rollup instead of the raw
+table.
 
 ```sql
 SELECT pgcolumnar.create_rollup(
@@ -579,7 +580,7 @@ SELECT pgcolumnar.create_rollup(
     agg_functions => '{count,sum,min,max,avg}',
     agg_columns   => '{*,value,value,value,value}');
 
-SELECT pgcolumnar.refresh_rollup('metrics_hourly');   -- returns rows folded
+SELECT pgcolumnar.refresh_rollup('metrics_hourly');   -- returns rows read
 SELECT * FROM metrics_hourly WHERE bucket >= now() - interval '1 day';
 ```
 
@@ -603,11 +604,20 @@ average of a `float8` column can differ from a full recompute in the last digits
 maintenance daemon, drops old rows of the source, the rollup keeps its rows for
 those buckets. So, for example, hourly averages can outlive the raw data.
 
-**This version folds appended rows only.** If rows of the source are deleted or
-updated in any other way, `refresh_rollup` refuses (`0A000`) and changes
-nothing. If the source
-is rewritten, for example by `pgcolumnar.vacuum`, `refresh_rollup` refuses with
-`55000`. If the rollup was current before the rewrite, run
+**Deletes and updates are recomputed.** Rows can be deleted or updated, or
+`pgcolumnar.compact_rewrite` can rewrite a row group. The next refresh then
+recomputes, from the source, each bucket that the changed row groups covered.
+It leaves the other buckets as they are.
+
+**A refresh never recomputes history.** A bucket can hold rows that retention
+dropped. If such a bucket needs a recompute, `refresh_rollup` refuses (`0A000`)
+and changes nothing, because a recompute would lose those rows. A delete that reaches a row group
+spanning such a bucket is refused too, even when the deleted rows are recent.
+If the rollup is right as it stands, run `pgcolumnar.rebase_rollup` to accept
+the source as it is now.
+
+If the source is rewritten, for example by `pgcolumnar.vacuum`, `refresh_rollup`
+refuses with `55000`. If the rollup was current before the rewrite, run
 `pgcolumnar.rebase_rollup` to record the rewritten table and continue.
 
 Run `refresh_rollup` at READ COMMITTED. A role that is not a superuser needs

docs/sql-reference.md

@@ -636,11 +636,16 @@ See [Keep a rollup up to date](how-to.md#keep-a-rollup-up-to-date).
 
 ### pgcolumnar.refresh_rollup(rollup regclass) returns bigint
 
-Folds the rows of the row groups added to the source since the last refresh,
-and returns the number of rows folded. Runs at READ COMMITTED only (`25000`).
+Brings the rollup up to date in one statement, and returns the number of
+source rows it read. It folds the rows of the row groups added to the source
+since the last refresh. For each row group deleted from or rewritten since, it
+recomputes from the source the buckets that the group's rows fall in, and
+nothing else. Runs at READ COMMITTED only (`25000`).
+
 Row groups that `pgcolumnar.expire` retired are dropped from the refresh's
-record, and the rollup keeps their buckets. Refuses after any other delete or
-update in the source (`0A000`), and after a rewrite of the source (`55000`).
+record, and the rollup keeps their buckets as history. Refuses, changing
+nothing, when a bucket to recompute holds history (`0A000`), and after a rewrite
+of the source (`55000`).
 
 ### pgcolumnar.row_tid(row_number bigint) returns tid
 
@@ -650,8 +655,10 @@ number. Row numbers start at 1 (`22003` otherwise).
 ### pgcolumnar.rebase_rollup(rollup regclass) returns bigint
 
 Records the source's current row groups as folded, without changing the rollup,
-and returns the number of row groups recorded. Use it after a rewrite of the
-source, when the rollup was current before the rewrite.
+and returns the number of row groups recorded. It reads every row group to
+record the time range of its rows, and keeps the rollup's history. Use it after
+a rewrite of the source, or after a refusal to recompute history, when the
+rollup is right as it stands.
 
 ### pgcolumnar.tier_partition(partition regclass, server name, path text, lock_wait interval DEFAULT '10 seconds') returns bigint