feat: a no-op expire answers from the column's extent, not a zone-map probe per group (#1310)

CHANGELOG.md

@@ -18,6 +18,14 @@ true until the next version shipped.
 
 ### Added
 
+- A `pgcolumnar.expire` that has nothing to retire now answers from the
+  retention column's extent in one index lookup, instead of one zone-map probe
+  per row group (#1310). Most passes of the maintenance daemon are such passes. When
+  the extent's minimum is not below the cutoff, no group can qualify; otherwise
+  `expire` decides group by group as before. Measured as zone-map scans for a
+  no-op `expire` over 50 groups: 50 before, 0 after. `test/ttl_expire.sh` and
+  `test_ttl_expire.py`; `ttl_expire` now enters the mutation ledger.
+
 - The planner's estimate of how many time buckets a `GROUP BY date_trunc(...)`
   or `date_bin(...)` produces now reads a per-column extent
   (`pgcolumnar.column_extent`) in one index lookup, instead of every row group's

src/columnar_vacuum.c

@@ -2491,6 +2491,36 @@ pgcolumnar_expire(PG_FUNCTION_ARGS)
 	}
 
 	storageId = PgColumnarStorageId(rel);
+
+	/*
+	 * THE NO-OP PASS ENDS HERE (#1310). Most of the daemon's passes retire
+	 * nothing, and the loop below proves that with one zone-map probe per group.
+	 * The column's extent (#1304 option 5) bounds every value a group visible to
+	 * this snapshot can hold, this transaction's own flushed groups included, so
+	 * when its minimum is not below the cutoff no group's maximum is either, and
+	 * nothing can qualify. A column with no value at all has nothing to retire
+	 * either: its groups have no minimum and maximum, which the loop keeps.
+	 *
+	 * An extent can be wider than the data, never narrower, and a wider one only
+	 * means no early exit. With no complete extent the loop decides as before.
+	 */
+	{
+		/* read only when the column has a value; initialised for gcc's LTO pass */
+		Datum		emin = (Datum) 0,
+					emax = (Datum) 0;
+		bool		hasValues = false;
+
+		if (PgColumnarExtentBound(rel, attno - 1, GetActiveSnapshot(),
+								  &emin, &emax, &hasValues) &&
+			(!hasValues ||
+			 DatumGetInt32(FunctionCall2Coll(&tce->cmp_proc_finfo, att->attcollation,
+											 emin, cutoff)) >= 0))
+		{
+			table_close(rel, NoLock);
+			PG_RETURN_INT64(0);
+		}
+	}
+
 	groups = PgColumnarReadRowGroupList(storageId, GetActiveSnapshot());
 
 	/*

test/check_ledger.tsv

@@ -2165,6 +2165,73 @@ timeseries_recipes	timeseries_recipes	the how-to gives the LOCF recipe	15;16;17;
 timeseries_recipes	timeseries_recipes	the how-to gives the gap-fill recipe	15;16;17;18;19	2026-10-13	the how-to's 'Fill empty buckets' heading renamed, so the recipe is no longer found
 timeseries_recipes	timeseries_recipes	the how-to gives the interpolation recipe	15;16;17;18;19	2026-10-13	the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found
 timeseries_recipes	timeseries_recipes	the interpolation recipe fills a gap linearly, and leaves both ends NULL	15;16;17;18;19	2026-10-13	the how-to's 'Interpolate between known values' heading renamed, so the recipe is no longer found;the how-to's interpolation recipe uses prev_v in place of next_v, so every gap is filled flat;the how-to's recipes join the series to the aggregate with JOIN instead of LEFT JOIN
+ttl_expire	ttl_expire	NO row still inside the retention was dropped (#403 item 5a)	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	NO row still inside the retention was dropped, on a date column	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	a compaction's retired groups are not recorded as retention	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	a date retention column is accepted, not refused (#1135)	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	a group whose only NULLs have been deleted is retired	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	a later expire on a column of only NULLs reads no zone map	15;16;17;18;19	2026-10-13	expire's early exit removed (the tree before #1310);the early exit not taken for a column with no value
+ttl_expire	ttl_expire	a negative ttl_interval is refused with 22023	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	a no-op expire reads no zone map when the extent's minimum is past the cutoff	15;16;17;18;19	2026-10-13	expire's early exit removed (the tree before #1310)
+ttl_expire	ttl_expire	a rolled-back expire records nothing	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	a row dated exactly on the cutoff is kept, so truncation errs toward keeping	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	a table with no declared retention is an error, not a silent success	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	and a zero ttl_interval is refused too	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	and its rows are gone	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	and no row older than Pacific/Midway's own cutoff survives	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	and no row older than UTC's own cutoff survives	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	and the date table really is smaller than it was	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	and the expired timestamps sharing the group were kept with them	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	and the table lost exactly the groups it retired	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	and the table really is smaller than it was	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	and they are the groups gone from the row-group catalog	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	and those NULL rows are still there	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	control: a group whose NULLs are still live is NOT retired	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	control: and a positive one is still accepted	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	control: and the rows are all still there	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	control: and those NULL rows survive	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	dropping the table drops its retention record	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	every remaining date row reads back its own value	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	every remaining row reads back its own value	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	expire does not retire a group that still holds NULL retention rows	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	expire in Pacific/Midway keeps exactly the rows at or after that session's cutoff	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	expire in UTC keeps exactly the rows at or after that session's cutoff	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	expire in the transaction that wrote aged rows retires their groups	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum;the extent bound leaves out the transaction's own pending fold (#1304 option 5's fix)
+ttl_expire	ttl_expire	expire on a date column retires at least one group	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	expire records exactly the groups it retired	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	expire retired the three aged groups	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	expire retires at least one group (#403 item 5a)	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	expire retires the all-visible expired group	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	index-only scan does not return rows expire already retired	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: an index-only scan is chosen on the all-visible table	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: and VACUUM really did set visibility-map bits to clear	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: and every live row is past the retention	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: and rows dated exactly ON the cutoff, which decide the rounding	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: and the SAME row group holds expired rows and NULL ones	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: but the zone map still records the deleted NULLs	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: compact_rewrite retired a group	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: expire retires the aged groups and keeps those of NULLs	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: in Pacific/Midway the fixture straddles that session's cutoff	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: in UTC the fixture straddles that session's cutoff	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: no LIVE row holds a NULL retention value any more	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: one date per row group in Pacific/Midway, so no group straddles the cutoff	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: one date per row group in UTC, so no group straddles the cutoff	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: rows exist on both sides of the cutoff within one day of it	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: some rows are past the retention and some are not	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the date fixture has rows on both sides of the cutoff	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the date fixture is laid out in several row groups	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the fixture is ONE row group, so the NULLs share it	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the fixture is laid out in several row groups	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the no-op fixture has many row groups	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the record fixture has three aged-out row groups ahead of the current ones	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: the table holds NULL retention rows at all	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	premise: without the extent, expire retires the aged groups	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	running it again retires nothing, because nothing new expired	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	seqscan agrees the expired rows are gone	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	the index-only scan is still the plan after expire	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	the retention is recorded where the other options live	15;16;17;18;19	never	-
+ttl_expire	ttl_expire	with the extent, expire retires the same groups	15;16;17;18;19	2026-10-13	the early exit compares the extent's maximum to the cutoff instead of its minimum
+ttl_expire	ttl_expire	without the extent the same expire probes every group's zone map	15;16;17;18;19	never	-
 validity_elision	validity_elision	a chunk claiming no bitmap while it holds fewer values than rows is refused	15;16;17;18;19	never	-
 validity_elision	validity_elision	a column with no nulls stores no validity bitmap	15;16;17;18;19	2026-09-18	writer: flush_one_column's presentCount == rowCount forced false, so the bitmap is written even when the chunk holds no null
 validity_elision	validity_elision	a null-free column elides its bitmap beside a null-bearing one in the same row group	15;16;17;18;19	never	-

test/check_ledger_budget.txt

@@ -15,7 +15,7 @@
 #
 #   Adding a check to a suite that is already covered does not move it, which is
 #   what makes it safe to bound.
-suites_not_covered 247
+suites_not_covered 246
 #
 # checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
 #   one.
@@ -1540,4 +1540,23 @@ suites_not_covered 247
 #     never         1746 - 1     = 1745
 #     dated          446 + 1     =  447
 #     1745 + 447 == 2192
-checks_never_observed_red 1745
+#
+# #1310, a no-op expire answers from the extent (OffgridwithJD): 1745 -> 1796,
+# and suites_not_covered 247 -> 246. ttl_expire enters the ledger with all 67
+# checks from a five-major run (ttl_expire.sh 67/67 on each of 15-19), at
+# jdatcmd's request on #1304: it is the suite for the one function that deletes
+# rows. Sixteen are dated by four mutations, each merged one log at a time with
+# --mutation and --target: the early exit removed (the tree before #1310), the
+# extent's maximum compared instead of its minimum (fourteen checks, among them
+# twelve of the suite's existing arms), the exit not taken for a column with no
+# value, and the extent bound without option 5's own-transaction fold. No
+# premise is among them: 'expire retired the three aged groups' was a premise
+# that the second mutation reddened, and was renamed to an arm first (#1236).
+# The other fifty-one arrive never -- the suite's older arms, which no mutation
+# of this change reaches, and its premises.
+#
+#     ledger rows   2192 + 67 = 2259
+#     never         1745 + 51 = 1796
+#     dated          447 + 16 =  463
+#     1796 + 463 == 2259
+checks_never_observed_red 1796

test/pytest/TESTS.md

@@ -6195,8 +6195,9 @@ restored byte-identical.
 | `test_a_date_retention_column_truncates_toward_keeping` | the rounding, which only this arm notices |
 | `test_the_cutoff_is_the_sessions_own_date` | the cutoff is the session's, asserted as an identity rather than as a difference |
 | `test_expire_records_what_it_retires_and_nothing_else` | `expire` writes exactly the groups it retires to `pgcolumnar.retired_group` (#1294); a rolled-back `expire` and a `compact_rewrite` record nothing; dropping the table drops its rows |
+| `test_a_no_op_expire_reads_the_extent_not_every_zone_map` | #1310: with the retention column's extent (#1304 option 5) past the cutoff, a no-op `expire` makes 0 zone-map scans where it made one per group; the same groups retire with or without the extent; aged rows written in `expire`'s own transaction retire; a column left with only NULLs needs no reads. Counted in a fresh connection, with the test schema's `search_path` carried over. Seen red first: 40 scans |
 
-The shell twin is `test/ttl_expire.sh`, 51 checks. The two halves share no code:
+The shell twin is `test/ttl_expire.sh`, 67 checks since #1310, all in the mutation ledger. The two halves share no code:
 different tables, different row counts, different retention windows, and the
 refusals asserted by SQLSTATE here (`55000` for no declared retention, `22023` for
 a non-positive interval) where the shell greps its message.

test/pytest/expected_tests.txt

@@ -591,4 +591,7 @@ guard_tests 419
 #
 # cluster_tests 523 -> 524 for option 5's test_a_transaction_reads_its_own_flushed_groups,
 # by collection on the same tree.
-cluster_tests 524
+#
+# cluster_tests 524 -> 525 for #1310's test_a_no_op_expire_reads_the_extent_not_every_zone_map,
+# by collection on this branch (stacked on #1304 option 5).
+cluster_tests 525

test/pytest/test_ttl_expire.py

@@ -649,7 +649,7 @@ def rec(sid):
     expect.text(f"{rec(sid)}|{rgs(sid)}", "|1,2,3,4,5", "a rolled-back expire records nothing")
 
     n = _one(pgc_conn, "SELECT pgcolumnar.expire('ttl_rr')")
-    expect.num(n, 3, "premise: expire retired the three aged groups")
+    expect.num(n, 3, "expire retired the three aged groups")
     expect.text(rec(sid), "1,2,3", "expire records exactly the groups it retired")
     expect.text(rgs(sid), "4,5", "and they are the groups gone from the row-group catalog")
 
@@ -663,3 +663,71 @@ def rec(sid):
     pgc_conn.execute("DROP TABLE ttl_rr")
     expect.num(_one(pgc_conn, "SELECT count(*) FROM pgcolumnar.retired_group WHERE storage_id = %s",
                     (sid,)), 0, "dropping the table drops its retention record")
+
+
+def test_a_no_op_expire_reads_the_extent_not_every_zone_map(pgc_conn, pgc_cluster, expect):
+    """Most of the daemon's passes retire nothing (#1310). When the retention
+    column's extent (#1304 option 5) puts its minimum past the cutoff, no group can
+    qualify, and expire answers 0 without a zone-map probe per group.
+
+    Counted as native_groupagg counts: the transaction's own scans of zone_map and
+    its index, in a fresh connection, because the counters carry a backend's earlier
+    reads for about a second.
+    """
+    dsn = pgc_cluster.dsn(dbname=_one(pgc_conn, "SELECT current_database()"))
+    path = _one(pgc_conn, "SHOW search_path")
+
+    def run(table, first=None):
+        """'n=<retired> scans=<zone-map scans>', in one transaction rolled back."""
+        with psycopg.connect(dsn) as c:
+            with c.cursor() as cur:
+                cur.execute(f"SET search_path TO {path}")
+                if first:
+                    cur.execute(first)
+                cur.execute(f"SELECT pgcolumnar.expire('{table}'),"
+                            " pg_stat_get_xact_numscans('pgcolumnar.zone_map'::regclass)"
+                            " + pg_stat_get_xact_numscans('pgcolumnar.zone_map_pkey'::regclass)")
+                n, scans = cur.fetchone()
+            c.rollback()
+        return f"n={n} scans={scans}"
+
+    def table(name, interval, *inserts):
+        with pgc_conn.cursor() as cur:
+            cur.execute(f"CREATE TABLE {name} (ts timestamptz, v int) USING pgcolumnar")
+            cur.execute(f"SELECT pgcolumnar.set_options('{name}', stripe_row_limit => {GROUP},"
+                        f" ttl_column => 'ts', ttl_interval => '{interval}')")
+            for sql in inserts:
+                cur.execute(sql)
+        return _one(pgc_conn, f"SELECT pgcolumnar.get_storage_id('{name}')")
+
+    sid = table("ttl_nx", "4000 days",
+                f"INSERT INTO ttl_nx SELECT now() - interval '20 hours' + g * interval '1 s', g"
+                f" FROM generate_series(1, {40 * GROUP}) g")
+    expect.num(_one(pgc_conn, "SELECT count(*) FROM pgcolumnar.row_group WHERE storage_id = %s", (sid,)), 40,
+               "premise: the no-op fixture has many row groups")
+    expect.text(run("ttl_nx"), "n=0 scans=0",
+                "a no-op expire reads no zone map when the extent's minimum is past the cutoff")
+    expect.text(run("ttl_nx", f"DELETE FROM pgcolumnar.column_extent WHERE storage_id = {sid}"), "n=0 scans=40",
+                "without the extent the same expire probes every group's zone map")
+
+    sid = table("ttl_ny", "3 days",
+                f"INSERT INTO ttl_ny SELECT now() - interval '12 days' + g * interval '1 s', g"
+                f" FROM generate_series(1, {4 * GROUP}) g",
+                f"INSERT INTO ttl_ny SELECT now() + g * interval '1 s', g FROM generate_series(1, {GROUP}) g")
+    without = run("ttl_ny", f"DELETE FROM pgcolumnar.column_extent WHERE storage_id = {sid}").split(" ")[0]
+    expect.text(without, "n=4", "premise: without the extent, expire retires the aged groups")
+    expect.text(run("ttl_ny").split(" ")[0], without, "with the extent, expire retires the same groups")
+
+    table("ttl_nz", "3 days",
+          f"INSERT INTO ttl_nz SELECT now() + g * interval '1 s', g FROM generate_series(1, {GROUP}) g")
+    expect.text(run("ttl_nz", f"INSERT INTO ttl_nz SELECT now() - interval '12 days' + g * interval '1 s', g"
+                              f" FROM generate_series(1, {4 * GROUP}) g").split(" ")[0], "n=4",
+                "expire in the transaction that wrote aged rows retires their groups")
+
+    table("ttl_nn", "3 days",
+          f"INSERT INTO ttl_nn SELECT now() - interval '12 days' + g * interval '1 s', g"
+          f" FROM generate_series(1, {3 * GROUP}) g",
+          f"INSERT INTO ttl_nn SELECT NULL, g FROM generate_series(1, {2 * GROUP}) g")
+    expect.num(_one(pgc_conn, "SELECT pgcolumnar.expire('ttl_nn')"), 3,
+               "premise: expire retires the aged groups and keeps those of NULLs")
+    expect.text(run("ttl_nn"), "n=0 scans=0", "a later expire on a column of only NULLs reads no zone map")