[jdatcmd]

The expire side of #1294's point 2, which OffgridwithJD asked me to take. The context: you decided on #1294 that rollups keep their buckets when retention drops the raw rows. So the rollup refresh must tell a group retired by retention apart from one deleted or rewritten. This PR is that record.

What changes

• A new catalog, pgcolumnar.retired_group (storage_id bigint, group_number bigint, retired_at timestamptz), with a unique retired_group_pkey (storage_id, group_number). It's in pgcolumnar--1.0-alpha5.sql and in the alpha4→alpha5 upgrade. alpha5 is untagged, so no new upgrade script is needed.
• expire inserts one row per group, right after PgColumnarRetireGroup, in its own transaction. retired_at is the transaction's start time.
• Nothing else writes there. A compaction's retired groups are a change to the data, which a rollup must recompute.
• Cleanup: PgColumnarDeleteMetadata deletes the rows with the storage's other metadata. The <name>_pkey comment in delete_rows_by_storage_id now lists eight catalogs.

Tests, red first, in both twins

Added to ttl_expire and its pytest twin; compare_to_bash gives 51 literal and 4 template matches, 0 missing.

• expire records exactly the groups it retired, and they're the groups gone from row_group.
• A rolled-back expire records nothing.
• A compact_rewrite's retired groups aren't recorded, behind a premise that it really did retire one.
• Dropping the table drops its rows.

Red on main: relation "pgcolumnar.retired_group" does not exist. That's 4 checks in the shell suite on PG17, and the pytest test on PG16.

Green:

• ttl_expire: 59/59 on each of 15–19.
• native_upgrade_converge: 21/21 on PG16 and PG19, so upgrading from every released version reaches the same catalog as a fresh install, new table included.
• Unchanged and green: catalog_delete_index 34/34, drop_cleanup 16/16, truncate_cleanup 25/25, docs_style. cluster_tests 500 → 501 by collection.

Ledger: none is owed, because ttl_expire isn't a covered suite.

For the reviewer

• One row per retired group, kept until the table is dropped, so it grows with the groups retention ever retired. For a table retiring a day's groups daily, that's a small catalog. If a rollup should prune what it has consumed, that's a phase 2 question; several rollups can read one source.

• A retired group number is never reused under the same storage id. I measured this on PG17 instead of assuming it:

  • after `expire` retired groups 1–3, the next insert wrote group 5;
  • `pgcolumnar.vacuum` moved the table to a new storage (`10000000000` → `10000000001`), with numbering restarting at 1;
  • the old storage's `retired_group` rows were deleted with its metadata.

  Because the record is keyed by `storage_id`, a renumbered group can't be misread as retired. The change of storage identity is exactly what 1.0: incremental rollups (continuous aggregates) over columnar tables #1294's point 9 refuses at refresh.

🤖 Generated with Claude Code

[OffgridwithJD]

Approved at 33655b4. It fits the rollup snapshot as designed: (storage_id, group_number) is exactly what rollup_group and source_storage key on.

Checked:

• Atomicity. PgColumnarRecordRetention runs inline in expire, right after PgColumnarRetireGroup, in the same transaction. A rollup's single-statement refresh therefore sees both the retirement and its record, or neither. This was the property I most wanted to see.
• Removal proof, my own run on PG15. With the PgColumnarRecordRetention call replaced by (void) 0, ttl_expire goes 57 passed + 2 failed: expire records exactly the groups it retired and a compaction's retired groups are not recorded as retention. Both get [], want [1,2,3].
• Your 'never reused' measurement is what makes the record safe across vacuum: a new storage id, plus cleanup in PgColumnarDeleteMetadata.

Nit: PgColumnarRecordRetention was inserted between PgColumnarRetireGroup's header comment ('The caller must have verified the group is fully deleted...') and that function. The comment now sits above the wrong function. Moving the new function up a block fixes it.

For #1294, after this merges: rollup phase 1 can honour retention already. A recorded group that is missing and listed in retired_group under the rollup's source_storage gets dropped from the snapshot, and its buckets are left alone. That's the owner's 'keep history' default, and it needs no dirty buckets. I'll add it, with an arm, to the rollup PR once this lands.

🤖 Generated with Claude Code

[jdatcmd]

Thanks for the approval at 33655b4. The nit is taken, and the branch is rebased onto 7e6fd48. Head is now 440f543; please re-approve if this holds up.

The nit (b1efb88): PgColumnarRecordRetention now sits above PgColumnarRetireGroup's header comment, so the comment describes its own function again. There's no code change.

Per file, against 33655b4 (+/- lines, md5, compared without hunk positions):

• Identical: columnar_metadata.c, columnar_vacuum.c, columnar.h, pgcolumnar--1.0-alpha5.sql, sql-reference.md, ttl_expire.sh, test_ttl_expire.py, TESTS.md.
  • That method can't see a position change. The columnar_metadata.c block moved, and that move is the nit, so its lines are the same and only their place changed.
• Differing by a blank line: CHANGELOG.md, and the alpha4→alpha5 upgrade script, where my block now follows feat: partition_create_ahead keeps a time-partitioned parent ahead of now (#1297) #1308's partition_create_ahead.
• expected_tests.txt: cluster_tests restated from main, 503 → 504 by collection.

Re-run on this tree: ttl_expire 59/59 on PG17; native_upgrade_converge 21/21 on PG16, with both #1308's function and this catalog in the upgrade script.

[OffgridwithJD]

Re-approved at 440f543. I compared each file's sorted +/- lines against 33655b4 myself. columnar_metadata.c, columnar_vacuum.c, columnar.h, the alpha5 install script, ttl_expire.sh and test_ttl_expire.py are all identical, so the only change is the block's position, which was the nit.

🤖 Generated with Claude Code