feat: the maintenance daemon runs expire when autovacuum_expire is on (#1290)

CHANGELOG.md

@@ -39,6 +39,22 @@ true until the next version shipped.
   and the alpha4 to alpha5 upgrade; `native_upgrade_converge` 21/21. ttl_expire and
   its pytest twin hold it: exactly the retired groups are recorded, a rolled-back
   expire and a `compact_rewrite` record nothing, and a drop clears the rows.
+- The maintenance daemon can apply retention (#1290). With
+  `pgcolumnar.autovacuum_expire` on, as well as `pgcolumnar.autovacuum`, each pass
+  runs `pgcolumnar.expire` on every table whose options declare both
+  `ttl_column` and `ttl_interval`. The new setting is off by default, by the
+  owner's decision on #1290: `expire` deletes rows, so turning the daemon on does
+  not by itself apply a retention. `expire` runs first, in its own transaction,
+  so one that fails (its retention column was dropped) is logged and skipped
+  without undoing the compaction that follows. `expire` now calls the maintenance
+  hold after taking its lock, as `compact_rewrite` does, so the daemon's yield is
+  testable there too. A no-op `expire` costs one zone-map scan per row group:
+  0.05 ms at 10 groups, 2.7 ms at 1000, measured on PG17. `test/autovacuum_expire.sh`
+  and an independent `test_autovacuum_expire.py` each hold 16 checks: the switch
+  is off by default; with it off, the daemon compacts but retires nothing; with it
+  on, the aged rows go, the fresh rows stay, and a table with no retention is never
+  touched or named; a failing `expire` does not undo a compaction; and the daemon
+  yields `expire`'s lock to an `ACCESS EXCLUSIVE` request within 8 s of a 30 s hold.
 
 - `test/format_downgrade.sh` and `test_format_downgrade.py` test #1137's promise
   with both real libraries. They build `v1.0-alpha4`, write with it, upgrade to the

docs/ARCHITECTURE.md

@@ -254,8 +254,10 @@ The maintenance daemon (`pgcolumnar.autovacuum`). A background-worker launcher
 starts one short-lived worker per database on a schedule. Each worker consults
 `pgcolumnar.maintenance_due` for the tables that have crossed the compact-rewrite
 or recluster thresholds, then runs `pgcolumnar.compact_rewrite` and
-`pgcolumnar.recluster` on them through SPI, each in its own transaction. It calls
-only those two `ShareUpdateExclusiveLock` verbs, and it registers as an
+`pgcolumnar.recluster` on them through SPI, each in its own transaction. With
+`pgcolumnar.autovacuum_expire` on, it first runs `pgcolumnar.expire` on each table
+that declares a retention, in a transaction of its own. It calls only these
+`ShareUpdateExclusiveLock` verbs, and it registers as an
 autovacuum-kind process so it yields to a stronger lock a session requests.
 
 ### columnar_unique.c

docs/administration.md

@@ -214,6 +214,12 @@ It is off by default. When it is on, a launcher wakes every
 database. Each worker asks `pgcolumnar.maintenance_due()` which columnar tables
 have crossed a threshold, then runs the verb it recommends.
 
+With `pgcolumnar.autovacuum_expire` also on (default off), each worker first
+runs `pgcolumnar.expire` on every table that declares a retention. Each `expire`
+runs in its own transaction. One that fails is logged and skipped, and the
+compaction that follows still runs. The switch is separate because `expire` deletes rows:
+turning the daemon on does not by itself apply a retention.
+
 The daemon calls only the online `ShareUpdateExclusiveLock` verbs. It never
 calls `vacuum`, `vacuum_sorted`, or `cluster`. So it does not block readers or
 writers. It also yields the way autovacuum does: it cancels its own maintenance

docs/configuration.md

@@ -86,7 +86,8 @@ disk. It never changes the values that a table returns.
 | `pgcolumnar.index_min_blocks` | integer | `3` | Heap pages at or above which a keyed read of a `pgcolumnar` metadata catalog probes that catalog's index instead of reading it sequentially. These catalogs are shared by every columnar table in the database, so the same catalog is one page in a small database and hundreds in a large one, and below a few pages a sequential read is cheaper than a probe. The default is measured rather than chosen, and it is a default rather than a truth: the best value differs per catalog, because the hot catalogs are read more often per retired row group. Change it only to undo a regression you have measured. A very large value makes every converted lookup read its catalog sequentially, as 1.0-alpha4 and earlier did for most of them; `0` always probes. Range 0 to INT_MAX. |
 | `pgcolumnar.reclaim_coalesce` | boolean | `on` | During online compaction, split an oversized freed range on reuse and coalesce adjacent freed ranges, so space is reclaimed under fragmentation. Off reverts to whole-range reuse. |
 | `pgcolumnar.enable_end_truncation` | boolean | `off` | Allow `pgcolumnar.truncate()` to return trailing reclaimed blocks to the operating system. Off makes `pgcolumnar.truncate()` a no-op. Requires superuser to set. |
-| `pgcolumnar.autovacuum` | boolean | `off` | Run the maintenance daemon. When on, it runs `compact_rewrite` and `recluster` on columnar tables that cross a threshold. It uses only `ShareUpdateExclusiveLock` and yields to any stronger lock. It never blocks a reader or a writer. See the [administration guide](administration.md#the-maintenance-daemon-pgcolumnarautovacuum). Reloadable, not a per-session setting. |
+| `pgcolumnar.autovacuum` | boolean | `off` | Run the maintenance daemon. When on, it runs `compact_rewrite` and `recluster` on columnar tables that cross a threshold, and `expire` when `pgcolumnar.autovacuum_expire` is also on. It uses only `ShareUpdateExclusiveLock` and yields to any stronger lock. It never blocks a reader or a writer. See the [administration guide](administration.md#the-maintenance-daemon-pgcolumnarautovacuum). Reloadable, not a per-session setting. |
+| `pgcolumnar.autovacuum_expire` | boolean | `off` | Let the daemon run `pgcolumnar.expire` on every table that declares a retention, once per sweep, before compaction. Off by default because `expire` deletes rows: turning the daemon on does not by itself apply a retention. No effect while `pgcolumnar.autovacuum` is off. Reloadable. |
 | `pgcolumnar.autovacuum_naptime` | integer | `60` | Seconds between daemon sweeps. Each sweep starts one worker per database. Range 1 to 86400. Reloadable. |
 | `pgcolumnar.autovacuum_compact_threshold` | float | `0.2` | Deleted fraction at which the daemon rewrites a table with `compact_rewrite`. Range 0.0 to 1.0. Reloadable. |
 | `pgcolumnar.autovacuum_recluster_threshold` | float | `0.05` | Appended fraction at which the daemon reclusters a table that has a recorded clustering key. Range 0.0 to 1.0. Reloadable. |
@@ -164,7 +165,7 @@ SELECT pgcolumnar.set_options(
 | `compression_level` | integer | Level for the `zstd` codec, 1 to 22. |
 | `encode_effort` | name | `full` (default) or `fast`. How much work the writer spends choosing an encoding. See below. |
 | `sort_by` | name[] | Declared physical sort key (#288), applied by `pgcolumnar.vacuum_sorted(t)` with no columns. Column names, so it survives `pg_dump`/restore. Not auto-maintained; re-run after inserts. Cannot name a virtual generated column. Clear with `reset_options(t, sort_by => true)`. |
-| `ttl_column` | name | The `date`, `timestamp` or `timestamptz` column a retention is measured on. On a `date` column the cutoff is truncated to the day, so a row dated on the cutoff is kept. Set it with `ttl_interval`; neither works alone. Nothing is deleted until you call `pgcolumnar.expire(t)` by name. |
+| `ttl_column` | name | The `date`, `timestamp` or `timestamptz` column a retention is measured on. On a `date` column the cutoff is truncated to the day, so a row dated on the cutoff is kept. Set it with `ttl_interval`; neither works alone. Nothing is deleted until `pgcolumnar.expire(t)` runs: by name, or from the daemon with `pgcolumnar.autovacuum_expire` on. |
 | `ttl_interval` | interval | How long a row is kept, measured from `ttl_column`. `pgcolumnar.expire(t)` then drops row groups whose rows are all older than this. A group with one live row is kept whole. |
 
 **A `stripe_row_limit` below 1024 disables FSST on text columns.** A vector is a

docs/features.md

@@ -231,8 +231,9 @@ and writes continue.
   curve.
 - `pgcolumnar.expire(table)` drops row groups whose rows have all passed the
   retention declared with `set_options(ttl_column, ttl_interval)`. A group with
-  one live row is kept whole, so nothing inside the retention is dropped. It
-  never runs on its own; you call it by name.
+  one live row is kept whole, so nothing inside the retention is dropped. You
+  call it by name, or the maintenance daemon calls it when
+  `pgcolumnar.autovacuum_expire` is on.
 - `pgcolumnar.truncate(table)` returns the reclaimed end blocks to the operating
   system, taking a brief `AccessExclusiveLock` only when it is free.
 
@@ -244,9 +245,10 @@ and writes continue.
   whether a table has crossed the compact and recluster thresholds.
 - `pgcolumnar.autovacuum` is a background maintenance daemon. When on, a per-database
   worker runs `compact_rewrite` and `recluster` on a schedule for tables that
-  `maintenance_due` reports as due. It runs only those two online operations, so it
-  never takes an exclusive lock, and it yields to a stronger lock a session
-  requests.
+  `maintenance_due` reports as due. With `pgcolumnar.autovacuum_expire` also on,
+  it runs `expire` on every table that declares a retention. It runs only these
+  online operations, so it never takes an exclusive lock, and it yields to a
+  stronger lock a session requests.
 
 ## Parallel bulk ingest
 

docs/how-to.md

@@ -292,8 +292,8 @@ to bound each call and keep it incremental.
 
 Declare how long rows are kept, then drop what has passed it.
 
-Before you start: the column must be `timestamp` or `timestamptz`, and you must
-own the table.
+Before you start: the column must be `date`, `timestamp` or `timestamptz`, and
+you must own the table.
 
 1. Declare the retention. Both halves are required.
 
@@ -316,9 +316,28 @@ has passed the retention, so a group holding one live row is kept whole. Some
 rows older than the interval therefore survive until the rest of their group
 ages out. Nothing inside the retention is ever dropped.
 
-It takes `ShareUpdateExclusiveLock`, so reads and writes continue. It never runs
-on its own: call it by name, or schedule it. Calling it on a table with no
-declared retention raises an error rather than doing nothing.
+It takes `ShareUpdateExclusiveLock`, so reads and writes continue. Calling it on
+a table with no declared retention raises an error rather than doing nothing.
+
+3. To expire on a schedule instead, let the maintenance daemon run `expire`.
+   Both settings are needed; the second is off by default, so turning on the
+   daemon alone never deletes rows.
+
+```sql
+ALTER SYSTEM SET pgcolumnar.autovacuum = on;
+ALTER SYSTEM SET pgcolumnar.autovacuum_expire = on;
+SELECT pg_reload_conf();
+```
+
+The daemon then runs `expire` on every table that declares a retention, once
+per `pgcolumnar.autovacuum_naptime` (default 60 seconds), and logs
+`pgcolumnar autovacuum: expire <table> retired <n> row groups` when it retires
+any. A pass that retires nothing logs at `DEBUG1` only.
+
+For a `date` or `timestamp` retention column, the cutoff depends on the time
+zone. The daemon uses the server's `TimeZone` setting, which may differ from
+the time zone your sessions use. For a cutoff in your session's time zone, run
+`pgcolumnar.expire` by hand from that session.
 
 `pgcolumnar.reset_options` has no argument for the retention. To stop expiring,
 set `ttl_interval` to a span no row will reach.

docs/sql-reference.md

@@ -34,9 +34,11 @@ PostgreSQL `CLUSTER`. Column names must exist and cannot be virtual generated
 columns.
 
 `ttl_column name` and `ttl_interval interval` declare a retention. They are read
-only by [`pgcolumnar.expire`](#pgcolumnarexpiretablename-regclass-returns-bigint),
-which you run yourself. Declaring a retention does not delete anything on its
-own. Both are needed: either one alone means no retention.
+only by [`pgcolumnar.expire`](#pgcolumnarexpiretablename-regclass-returns-bigint).
+Declaring a retention does not delete anything on its own. `expire` runs when
+you call it, or when the maintenance daemon calls it with
+`pgcolumnar.autovacuum_expire` on. Both are needed: either one alone means no
+retention.
 
 `ttl_interval` must be a positive interval. Zero and negative intervals raise
 `22023`. A negative interval would put the cutoff in the future, so `expire`
@@ -46,7 +48,7 @@ would drop rows that are still inside their retention.
 SELECT pgcolumnar.set_options('events', sort_by => ARRAY['customer_id','ts']);
 SELECT pgcolumnar.reset_options('events', sort_by => true);   -- clear it
 
--- declare a retention; nothing is deleted until you call expire
+-- declare a retention; nothing is deleted until expire runs
 SELECT pgcolumnar.set_options('events', ttl_column => 'ts',
                               ttl_interval => '90 days');
 ```
@@ -208,8 +210,10 @@ Drops row groups whose rows are all older than the retention declared by
 `set_options`. Returns the number of groups dropped. Holds only
 `ShareUpdateExclusiveLock`, so it runs against a live table.
 
-**This deletes rows.** It runs only when you call it. No other operation applies
-a retention, and `VACUUM` never does.
+**This deletes rows.** It runs when you call it, or when the maintenance daemon
+calls it with both `pgcolumnar.autovacuum` and `pgcolumnar.autovacuum_expire`
+on. The second is off by default. No other operation applies a retention, and
+`VACUUM` never does.
 
 Each group it drops is recorded in `pgcolumnar.retired_group`, in the same
 transaction. An incremental rollup reads that record to keep the summary of rows

pgcolumnar--1.0-alpha5.sql

@@ -57,7 +57,8 @@ CREATE TABLE pgcolumnar.options (
 	encode_effort name,
 	sort_by name[],                    -- declared physical sort key (#288)
 	-- Declared retention (#403 item 5a), read only by pgcolumnar.expire.
-	-- Nothing drops rows on its own: expire is called by name.
+	-- Nothing drops rows on its own: expire is called by name, or by the
+	-- daemon when pgcolumnar.autovacuum_expire is on (off by default, #1290).
 	ttl_column name,
 	ttl_interval interval
 );

src/columnar.h

@@ -262,6 +262,7 @@ extern bool pgcolumnar_autovacuum;
 extern int	pgcolumnar_autovacuum_naptime;
 extern double pgcolumnar_autovacuum_compact_threshold;
 extern double pgcolumnar_autovacuum_recluster_threshold;
+extern bool pgcolumnar_autovacuum_expire;
 extern void PgColumnarAutovacuumRegister(void);
 extern int	pgcolumnar_maintenance_hold_ms;	/* dev/test: hold SUEL this long in a maintenance verb */
 

src/columnar_autovacuum.c

@@ -12,8 +12,10 @@
  *
  * TWO INVARIANTS make this safe to run unattended:
  *
- * 1. Only the ShareUpdateExclusiveLock verbs -- pgcolumnar.compact_rewrite and
- *    pgcolumnar.recluster (now self-gating, #415 part A) -- are ever called.
+ * 1. Only the ShareUpdateExclusiveLock verbs -- pgcolumnar.compact_rewrite,
+ *    pgcolumnar.recluster (now self-gating, #415 part A) and, when
+ *    pgcolumnar.autovacuum_expire is on, pgcolumnar.expire (#1290) -- are ever
+ *    called.
  *    Never vacuum()/vacuum_sorted()/cluster(), which take AccessExclusiveLock.
  *    SUEL does not block readers or ordinary writers, so the daemon cannot block
  *    production by construction.
@@ -218,6 +220,74 @@ av_mark_as_autovacuum(void)
 #endif
 }
 
+/*
+ * Run pgcolumnar.expire on one table that declares a retention (#1290), in its
+ * own transaction inside PG_TRY.
+ *
+ * ITS OWN TRANSACTION, NOT av_maintain_one's: expire deletes rows and can fail
+ * on its own account -- the declared retention column was dropped, or its type
+ * changed -- and a failure here must not roll back a compact_rewrite that was
+ * due on the same table. It runs FIRST, so a compaction does not rewrite row
+ * groups that are about to be retired.
+ *
+ * Only reached when pgcolumnar.autovacuum_expire is on. That switch is off by
+ * default (the owner's decision on #1290): turning the daemon on must not
+ * start deleting rows by itself.
+ */
+static void
+av_expire_one(const char *qualname)
+{
+	StartTransactionCommand();
+	PushActiveSnapshot(GetTransactionSnapshot());
+	SetCurrentStatementStartTimestamp();
+
+	PG_TRY();
+	{
+		char		q[512];
+		int64		retired = 0;
+
+		if (SPI_connect() != SPI_OK_CONNECT)
+			elog(ERROR, "pgcolumnar autovacuum: SPI_connect failed");
+
+		snprintf(q, sizeof(q), "SELECT pgcolumnar.expire(%s)",
+				 quote_literal_cstr(qualname));
+		if (SPI_execute(q, false, 1) == SPI_OK_SELECT && SPI_processed == 1)
+		{
+			bool		isnull;
+			Datum		d = SPI_getbinval(SPI_tuptable->vals[0],
+										 SPI_tuptable->tupdesc, 1, &isnull);
+
+			if (!isnull)
+				retired = DatumGetInt64(d);
+		}
+		/*
+		 * LOG only when something was retired: a line per retention table per
+		 * pass, forever, is noise an administrator learns to ignore
+		 * (OffgridwithJD on #1307). av_maintain_one logs only a verb that ran.
+		 */
+		elog(retired > 0 ? LOG : DEBUG1,
+			 "pgcolumnar autovacuum: expire %s retired " INT64_FORMAT " row groups",
+			 qualname, retired);
+
+		SPI_finish();
+		PopActiveSnapshot();
+		CommitTransactionCommand();
+	}
+	PG_CATCH();
+	{
+		MemoryContext ecxt = MemoryContextSwitchTo(TopMemoryContext);
+		ErrorData  *ed = CopyErrorData();
+
+		elog(LOG, "pgcolumnar autovacuum: skipped expire on %s: %s", qualname, ed->message);
+		FreeErrorData(ed);
+		MemoryContextSwitchTo(ecxt);
+
+		FlushErrorState();
+		AbortCurrentTransaction();
+	}
+	PG_END_TRY();
+}
+
 /*
  * Run one maintenance verb on one table, each in its own transaction inside
  * PG_TRY: a query-cancel (the lock yield) or any error aborts just this op and
@@ -358,7 +428,9 @@ pgcolumnar_av_worker_main(Datum arg)
 	Oid			dbid;
 	uint32		conn_flags = 0;
 	List	   *tables = NIL;
+	List	   *hasTtl = NIL;
 	ListCell   *lc;
+	ListCell   *lt;
 
 	memcpy(&dbid, MyBgworkerEntry->bgw_extra, sizeof(Oid));
 
@@ -373,17 +445,25 @@ pgcolumnar_av_worker_main(Datum arg)
 
 	av_mark_as_autovacuum();
 
-	/* Snapshot the columnar-table list once, in its own transaction. */
+	/*
+	 * Snapshot the columnar-table list once, in its own transaction, with
+	 * whether each declares a retention. Both halves must be set, as
+	 * PgColumnarReadTtl reads it: a half declaration is no retention, and
+	 * calling expire on it would only log an error every pass.
+	 */
 	StartTransactionCommand();
 	PushActiveSnapshot(GetTransactionSnapshot());
 	if (SPI_connect() == SPI_OK_CONNECT)
 	{
 		int			ret = SPI_execute(
-			"SELECT quote_ident(n.nspname) || '.' || quote_ident(c.relname) "
+			"SELECT quote_ident(n.nspname) || '.' || quote_ident(c.relname), "
+			"       COALESCE(o.ttl_column IS NOT NULL AND o.ttl_interval IS NOT NULL, false) "
 			"FROM pg_class c "
 			"JOIN pg_am a ON a.oid = c.relam "
 			"JOIN pg_namespace n ON n.oid = c.relnamespace "
-			"WHERE a.amname = 'pgcolumnar' AND c.relkind = 'r'",
+			"LEFT JOIN pgcolumnar.options o ON o.regclass = c.oid "
+			"WHERE a.amname = 'pgcolumnar' AND c.relkind = 'r' "
+			"ORDER BY c.oid",
 			true, 0);
 		if (ret == SPI_OK_SELECT)
 		{
@@ -397,9 +477,13 @@ pgcolumnar_av_worker_main(Datum arg)
 
 				if (!isnull)
 				{
+					bool		ttlnull;
+					Datum		t = SPI_getbinval(SPI_tuptable->vals[i],
+												 SPI_tuptable->tupdesc, 2, &ttlnull);
 					MemoryContext old = MemoryContextSwitchTo(TopMemoryContext);
 
 					tables = lappend(tables, pstrdup(TextDatumGetCString(d)));
+					hasTtl = lappend_int(hasTtl, !ttlnull && DatumGetBool(t));
 					MemoryContextSwitchTo(old);
 				}
 			}
@@ -409,9 +493,12 @@ pgcolumnar_av_worker_main(Datum arg)
 	PopActiveSnapshot();
 	CommitTransactionCommand();
 
-	foreach(lc, tables)
+	forboth(lc, tables, lt, hasTtl)
 	{
 		CHECK_FOR_INTERRUPTS();
+		if (pgcolumnar_autovacuum_expire && lfirst_int(lt))
+			av_expire_one((const char *) lfirst(lc));
+		CHECK_FOR_INTERRUPTS();
 		av_maintain_one((const char *) lfirst(lc));
 	}
 	/* worker exits; the launcher starts a fresh one next naptime */