[jdatcmd]

Phase 2 of #1294: dirty buckets. With this, the seven time-series features that gate alpha5 lose one more open item. The plan, and the four changes made to it while building, are on #1294 (plan, changes).

What changes for a user

• refresh_rollup no longer refuses deletes, updates or compact_rewrite in the source. It recomputes from the source every bucket that a changed row group covered, and leaves every other bucket's row alone.
• A recompute never touches history. If a bucket to recompute holds rows that retention dropped, the refresh is refused with 0A000 and writes nothing. It is also refused when rows were deleted from a group that expire then retired before the refresh, because those rows can't be recomputed. rebase_rollup is the way out.
• pgcolumnar.retired_group gains deleted_count, which expire records (found in @OffgridwithJD's review).
• refresh_rollup now returns the rows it read: folded plus recomputed.
• rebase_rollup now reads the whole source, to record each group's time range.

How it works

The whole refresh is still one statement, so everything below shares one snapshot:

1. Changed groups. A recorded group that is gone, or whose deleted count moved, is changed, unless expire retired it. Retirement is read in this same snapshot. Read earlier, an expire committing in between would make a retired group look deleted.
2. Dirty buckets. Each changed group's recorded [ts_min, ts_max] becomes a span of buckets, merged with range_agg, plus the NULL bucket when the group had rows without a ts. A bucket's end is its start plus the width in plain seconds: a timestamptz plus '1 day' is 23 or 25 hours across a DST change.
3. Recompute and fold. The dirty buckets' partials are deleted and recomputed, one ts range predicate per span so zone maps prune. New groups fold into clean buckets only, because the recompute already counts their rows in the dirty ones.
4. History. Retired groups' ranges collect in pgcolumnar.rollup.history (a tstzmultirange) and history_has_null. A dirty bucket that meets them sets a flag, and the function raises after the statement, so everything it wrote rolls back.
5. The record. Changed groups are recounted or forgotten; new groups are recorded with the range of their rows.

Catalog: rollup_group gains ts_min, ts_max and ts_has_null. rollup gains history and history_has_null. The change is in both the alpha5 install script and the alpha4→alpha5 upgrade, with byte-identical patches.

Tests: both twins, the same check names

arm	shell	pytest
a delete; and no partial outside the changed groups' range is rewritten	✓	✓
an update that moves rows to another bucket	✓	✓
a delete plus an append straddling its bucket: counted once	✓	✓
compact_rewrite	✓	✓
a deleted row whose ts is NULL	✓	✓
a delete next to retired history folds; one reaching it is refused	✓	✓
history retired by the very refresh that would recompute it is refused	✓	✓
rows deleted from a group that expire then retires: refused (found in review)	✓	✓
a day bucket refreshed in a DST zone is recomputed whole	✓ (New York)	✓ (Berlin)
recount, forget, retired groups dropped from the record	✓	✓

Red first: on main's refresh, the shell twin had 10 checks red on each of 15–19, with identical failure lists. The pytest twin stopped at the first, got '0A000' want 'folded'.

Green (phase 2 at 3dd2ebd; the review fix re-run at 9683118, below):

• rollup.sh: 72/72 on each of 15–19 with the review fix, and 72/72 on PG17 at 9683118.
• test_rollup.py: green on each of 15–19, at this head.
• native_upgrade_converge: 21/21 on PG17.
• extension_upgrade: PASS on PG17, upgraded catalog byte-identical to a fresh install (210 objects).
• docs_style: 58/58.

Mutations, each on 15–19 for the shell twin (with the same red set on every major) and on PG17 for the pytest twin. Each is red at the named arm in both:

mutation	red at
new rows folded into dirty buckets too	straddle (+ compact_rewrite in shell)
every bucket recomputed once anything changed	8 checks, including "no partial outside the range"
no history refusal	both refusal arms
blocking on stored history only	history retired by the same refresh
rebase_rollup recording no ranges	the delete arms after the rebase
the NULL bucket never dirty	the NULL-ts arm
the width added in the session's zone	the DST arm
new groups recorded without ranges	5, including both history arms
no recount / no forget	"the next refresh recomputes nothing", each
retired groups kept in the record	"no longer records the groups retention retired"
retired ranges never stored	the second refusal arm
changed means "row count moved" too	nothing, so that condition was removed (57ff07e)

Found by a mutation: the pytest DST arm was green under the zone mutation at first. That file's ORIGIN is 06:00 UTC, so the bucket I aimed at started after Berlin's switch. The rows were moved into the bucket that contains it (aaaf10d).

Not covered by an arm: an expire committing between two statements of one refresh. Nothing can force that deterministically. The fix is structural: there is no second statement.

Ledger

checks_never_observed_red goes from 1799 to 1801 against main at c90976d. The arithmetic is in the budget file. Main's other rows are byte-identical. Three phase-1 rows are removed with the behaviour they tested.

🤖 Generated with Claude Code

[OffgridwithJD]

Review at 3dd2ebd. I read the whole refresh_rollup and the catalog change, ran the twins on PG17 (rollup.sh 70/70, test_rollup.py green), and probed one case adversarially. The dirty-bucket logic holds up on every path I traced:

• Delete and update: the deleted count moves.
• compact_rewrite and VACUUM: the group is gone, its new groups fold only into clean buckets, and the recompute counts their rows once.
• The NULL bucket.
• DST: the wabs seconds width.
• The history overlap: dirty spans are bucket-aligned, so any history row inside a dirty bucket overlaps it.
• One snapshot for all of it.
• The data-modifying CTEs on disjoint rows: upd only touches clean buckets, and cleared only dirty ones.

One silent wrong answer: a delete, then expire, before the next refresh

If rows are deleted from a group and expire then retires that group, the next refresh treats it as retiring and records its range as history. Its buckets keep counting the deleted rows. Nothing is refused or reported, and the rollup is wrong for good.

Reproduced on PG17 at 3dd2ebd:

CREATE TABLE s (ts timestamptz, v int) USING pgcolumnar;
SELECT pgcolumnar.set_options('s', stripe_row_limit => 1000, chunk_group_row_limit => 1000);
INSERT INTO s SELECT now() - interval '60 days' + g * interval '1 s', g FROM generate_series(1, 1000) g;
INSERT INTO s SELECT now() - interval '1 day'   + g * interval '1 s', g FROM generate_series(1, 1000) g;
SELECT pgcolumnar.create_rollup('r', 's', 'ts', '1 day', '2000-01-01', '{}', '{count}', '{*}');
SELECT pgcolumnar.refresh_rollup('r');                                   -- 2000
DELETE FROM s WHERE ts < now() - interval '30 days' AND v <= 400;        -- 400 old rows
SELECT pgcolumnar.set_options('s', ttl_column => 'ts', ttl_interval => '30 days');
SELECT pgcolumnar.expire('s');                                           -- 1
SELECT pgcolumnar.refresh_rollup('r');                                   -- 0, no error
SELECT sum(count) FROM r WHERE bucket < now() - interval '30 days';      -- 1000, the source had 600

The control is the same delete with no expire in between: the refresh recomputes, and the count is 600. The docs now say "Deletes and updates are recomputed", so this is the case where they silently aren't.

Why it happens: changed excludes every retired group (rg.group_number NOT IN (SELECT ... FROM ret)) before the deleted count is compared. The group is gone, so there's nothing to compare against anyway: pgcolumnar.retired_group records only the group number.

Two ways I can see to close it. You know the design better, so tell me if there's a third.

1. Refuse, as for history. expire also writes the group's deleted count at retirement into retired_group (#1314's catalog; that's one column). A retiring group whose count differs from the recorded rollup_group.deleted_count sets blocked, and the refresh refuses with 0A000 and the same rebase_rollup hint. The buckets can't be recomputed, because the rows are gone, but the user is told instead of silently miscounted.
2. Order the daemon. Have it refresh a source's rollups before it expires that source. That narrows the window but doesn't close it, because a manual expire still reaches it. So I'd want (1) in any case.

The arm I'd add, in both twins: the repro above, requiring either the 0A000 refusal or a count that equals the source plus history. Without the fix it reads 1000 against 600.

Smaller

• The ledger statement is right on main at 4c72242: 1796 → 1797. I didn't recount the rows.
• Not reviewed: the pytest DST arm's choice of Berlin over New York, beyond reading it; and rebase_rollup's whole-source read cost on a large table. The docs say it reads everything, and that's fine for a recovery operation.

[jdatcmd]

@OffgridwithJD, thank you: that was a real silent wrong answer. It's fixed at 9683118, as your option 1. The tree is rebased onto c90976d (#1329).

The fix. expire now records each retired group's deleted count in pgcolumnar.retired_group, as a new deleted_count column. It reads the count before the group's delete vector is removed with it, and it reads the same stored column the refresh compares. When a refresh retires a recorded group whose count moved since the rollup last folded it, the refresh refuses with 0A000, writes nothing, and gives the same rebase_rollup hint. The DETAIL now says the rows to recompute from may be gone. Option 2, ordering the daemon, isn't done: as you said, a manual expire still reaches the case.

The arm, in both twins: your repro, requiring 0A000 and an unchanged rollup.

• Red first on PG17: the shell twin read ok and 1000 counted.
• Green:
  • rollup.sh 72/72 on each of 15–19, and again 72/72 on PG17 at this rebased head;
  • test_rollup.py green on 15–19;
  • ttl_expire.sh, whose catalog row gained a column, 67/67 on 15–19;
  • collation_change.sh 11/11 on PG17 at this head;
  • native_upgrade_converge 21/21 and extension_upgrade PASS on PG17.
• Two mutations, each red at this arm alone, on every major and in the pytest twin on PG17: the refusal removed, and expire recording the count as 0.

Ledger: 1799 → 1801 against main at c90976d. Phase 2's rows are restated from main's numbers; then the new arm is dated, and its premise is never red. Main's other rows are unchanged.

[OffgridwithJD]

Approved at 9683118.

The fix. expire records the group's deleted count before the delete vector goes with it. delete_vector_pkey is unique per (storage_id, group_number), so expire's sum over the list and rollup_source_groups' join read the same single count. blocked compares it with the count the rollup last folded or recounted. That is exactly the delete-then-expire window, and nothing else.

Run on PG17 at this head:

case	result
my repro	the second refresh refuses with 0A000, and the old bucket still reads 1000, unchanged
delete, refresh, then expire	no refusal, history 600 (the count was recounted before retirement, so they match)
delete and expire before the group was ever folded	the first refresh folds only the live group, 1000; the retired group was never recorded, so it is neither history nor refused
rollup.sh	72/72
ttl_expire.sh	67/67
test_rollup.py + test_ttl_expire.py	10 passed

The third row is a choice rather than a bug. Rows retired before a rollup ever saw them never enter it. That reads right to me for retention.

[jdatcmd]

Merging: approved by @OffgridwithJD at 9683118, and CI is green on that head.