feat: pgcolumnar.partition_drop_plan returns the statements that retire old time partitions (#1297)

CHANGELOG.md

@@ -18,6 +18,16 @@ true until the next version shipped.
 
 ### Added
 
+- `pgcolumnar.partition_drop_plan(parent, cutoff)` returns the statements that
+  retire every time partition whose whole range is older than `cutoff`: a
+  `DETACH PARTITION ... CONCURRENTLY` and a `DROP TABLE` each, in range order,
+  or `DETACH ... FINALIZE` for a partition an interrupted detach left pending
+  (#1297). Run them with `\gexec`. The parent is never locked exclusively. The
+  cutoff has the key's own type, so a plan does not depend on the session's
+  time zone. A partition moved to Parquet by `tier_partition` is dropped as a
+  foreign table, and its file stays.
+  `test/partition_lifecycle.sh` and `test_partition_lifecycle.py`.
+
 - A `pgcolumnar.expire` that has nothing to retire now answers from the
   retention column's extent in one index lookup, instead of one zone-map probe
   per row group (#1310). Most passes of the maintenance daemon are such passes. When

docs/how-to.md

@@ -482,6 +482,49 @@ partitions, before it creates any. Use a wider width or a shorter lead.
 `test/partition_lifecycle.sh` checks each of these statements, in sessions
 west and east of UTC.
 
+## Drop old time partitions
+
+`pgcolumnar.partition_drop_plan` returns the statements that retire every
+partition whose whole range is older than a cutoff. It does not run them, because
+`DETACH PARTITION ... CONCURRENTLY` cannot run inside a function. Run each one as
+its own statement, for example with psql's `\gexec`:
+
+```sql
+\set ON_ERROR_STOP on
+SELECT * FROM pgcolumnar.partition_drop_plan('metrics', now() - interval '90 days') \gexec
+```
+
+For each old partition, in range order, the plan detaches it concurrently and
+then drops it. The concurrent detach takes no exclusive lock on the parent, so
+queries and inserts on the table continue. The drop locks only the detached
+partition, which no query reaches through the parent any more. A partition whose
+upper bound equals the cutoff is old. A partition that straddles the cutoff is
+kept.
+
+If a concurrent detach was interrupted, the partition stays pending. The next
+plan finishes that detach with `DETACH PARTITION ... FINALIZE` instead of starting
+it again.
+
+The drop refuses while its partition is still attached (`55000`). So if a detach
+fails, for example on a lock timeout, the drop after it does not lock the parent
+or remove the partition. A later plan retries it. `ON_ERROR_STOP` stops the run
+at the first failure, which also keeps the plan's other statements from running.
+
+The cutoff must have the key's type. A `timestamptz` key takes a `timestamptz`
+cutoff, as in the example. A `timestamp` key takes a `timestamp` cutoff, such
+as `localtimestamp - interval '90 days'`. A cutoff of the other type is refused
+(`42804`). Comparing across the two types depends on the session's time zone,
+so the same call could plan differently from two sessions.
+
+A partition that is itself partitioned is planned as one, and the `DROP` takes
+its children with it. A partition that `tier_partition` moved to Parquet is
+planned with `DROP FOREIGN TABLE`. That removes the foreign table, and the
+Parquet file stays where it is: delete it yourself when you no longer need it.
+
+The function refuses the same tables as `partition_create_ahead`, because a
+DEFAULT partition rules out a concurrent detach. A run of the plan drops the
+partitions' rows: nothing keeps them.
+
 ## Count distinct values and percentiles approximately
 
 For large tables, approximate distinct counts and percentiles cost far less than

docs/sql-reference.md

@@ -622,6 +622,25 @@ refused before it creates any (`22023`). The function takes
 `SHARE UPDATE EXCLUSIVE` on the parent, never `ACCESS EXCLUSIVE`. See
 [Create time partitions ahead](how-to.md#create-time-partitions-ahead).
 
+### pgcolumnar.partition_drop_plan(parent regclass, cutoff timestamptz | timestamp) returns setof text
+
+Returns the statements that retire every partition of `parent` whose upper bound
+is at or before `cutoff`, in range order. Each partition gets
+`ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY`, then a `DO` block that runs
+`DROP TABLE`, or `DROP FOREIGN TABLE` for a partition `tier_partition` moved to
+Parquet (the file stays). The block refuses (`55000`) while the partition is
+still attached, so a failed detach is never followed by a drop that locks the
+parent. A partition that an interrupted concurrent detach
+left pending gets `DETACH PARTITION ... FINALIZE` instead. A partition that is
+itself partitioned is listed whole. The names are schema-qualified. The
+function runs nothing: run each statement at top level, for example with
+`\gexec`, because a concurrent detach cannot run inside a function. `cutoff`
+must have the key's type; the other type is refused (`42804`), and an untyped
+literal is a `timestamptz`. It refuses a parent with the same codes as
+`partition_create_ahead` (`55000`), and a NULL cutoff (`22023`).
+`pgcolumnar.partition_drop_candidates(parent, cutoff_type)` holds the checks
+the two overloads share. See [Drop old time partitions](how-to.md#drop-old-time-partitions).
+
 ### pgcolumnar.create_rollup(rollup name, source regclass, ts_column name, width interval, origin timestamptz, group_by name[], agg_functions text[], agg_columns name[], access_method name DEFAULT 'pgcolumnar') returns regclass
 
 Creates a rollup of `source` in the source's schema. A rollup is a view of

pgcolumnar--1.0-alpha4--1.0-alpha5.sql

@@ -963,6 +963,142 @@ BEGIN
 	RETURN made;
 END;
 $create_ahead$;
+
+/*
+ * pgcolumnar.partition_drop_plan (#1297)
+ *
+ * The statements that retire every partition of a RANGE-partitioned parent
+ * whose whole range is older than cutoff (its upper bound at or before it), in
+ * range order: ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY, then DROP
+ * TABLE (DROP FOREIGN TABLE for one tier_partition moved to Parquet, whose file
+ * stays). A partition a concurrent detach left pending gets DETACH ... FINALIZE
+ * instead. A partition reaching MAXVALUE is never old; one from MINVALUE is, once
+ * its upper bound is past. A partition that is itself partitioned is listed whole.
+ *
+ * RETURNED, NOT RUN. DETACH ... CONCURRENTLY is refused inside any function, DO
+ * block or procedure, on every major (#1297 note 1), and a plain DETACH or a
+ * DROP of an attached partition takes ACCESS EXCLUSIVE on the parent, which
+ * blocks every query on the table. The caller runs each statement at top level:
+ *
+ *     SELECT * FROM pgcolumnar.partition_drop_plan('metrics', now() - interval '90 days') \gexec
+ *
+ * THE CUTOFF'S TYPE IS THE KEY'S (jdatcmd, #1297). A timestamp bound compared
+ * with a timestamptz cutoff converts in the session's zone, so one call could
+ * plan differently from two sessions. Two overloads, each refusing the other
+ * type of key (42804); an untyped literal resolves to timestamptz.
+ *
+ * THE DROP REFUSES WHILE THE PARTITION IS ATTACHED (jdatcmd, #1331 review). Any
+ * DETACH can fail -- a lock_timeout, a cancelled concurrent wait, a permission
+ * error -- and \gexec carries on unless ON_ERROR_STOP is set. A plain DROP of an
+ * attached partition would then take ACCESS EXCLUSIVE on the parent. So each
+ * drop is a DO block that raises 55000 while the partition is still in
+ * pg_inherits, a pending one included; the next plan's FINALIZE picks that up.
+ * A DROP may run inside DO; only DETACH ... CONCURRENTLY must be top-level.
+ *
+ * The same refusals as partition_create_ahead, with the same codes: a DEFAULT
+ * partition rules out the concurrent detach, and is refused rather than dropped
+ * with the exclusive one. No ownership check: this reads catalogs the caller
+ * can see, and DETACH checks ownership when the statements run.
+ */
+CREATE FUNCTION pgcolumnar.partition_drop_candidates(parent regclass, cutoff_type regtype)
+	RETURNS TABLE (part regclass, pending boolean, foreign_table boolean, upper_bound text)
+	LANGUAGE plpgsql STABLE
+	SET search_path = pg_catalog, pg_temp
+	AS $drop_candidates$
+DECLARE
+	strat   "char";
+	natts   int;
+	defid   oid;
+	keytype regtype;
+BEGIN
+	SELECT pt.partstrat, pt.partnatts, pt.partdefid, a.atttypid::regtype
+	  INTO strat, natts, defid, keytype
+	  FROM pg_partitioned_table pt
+	  JOIN pg_attribute a ON a.attrelid = pt.partrelid AND a.attnum = pt.partattrs[0]
+	 WHERE pt.partrelid = parent;
+	IF NOT FOUND OR strat <> 'r' OR natts <> 1 THEN
+		RAISE EXCEPTION '% is not RANGE-partitioned on one column', parent
+			USING ERRCODE = '55000';
+	END IF;
+	IF keytype NOT IN ('timestamptz'::regtype, 'timestamp'::regtype) THEN
+		RAISE EXCEPTION '% is partitioned on %, not on a timestamp', parent, keytype
+			USING ERRCODE = '55000';
+	END IF;
+	IF defid <> 0 THEN
+		RAISE EXCEPTION '% has a DEFAULT partition', parent
+			USING ERRCODE = '55000',
+			      HINT = 'A DEFAULT partition rules out DETACH CONCURRENTLY, so the partition lifecycle refuses to manage this table.';
+	END IF;
+	IF keytype <> cutoff_type THEN
+		RAISE EXCEPTION '% is partitioned on %, so its cutoff must be a %, not a %',
+			parent, keytype, keytype, cutoff_type
+			USING ERRCODE = '42804',
+			      HINT = CASE WHEN keytype = 'timestamp'::regtype
+			                  THEN 'Pass a timestamp, for example localtimestamp - interval ''90 days''.'
+			                  ELSE 'Pass a timestamptz, for example now() - interval ''90 days''.' END;
+	END IF;
+	-- MAXVALUE above has no upper bound to read, and is never old
+	RETURN QUERY
+		SELECT b.part, b.pending, b.foreign_table, b.upper_bound
+		  FROM (SELECT c.oid::regclass AS part, i.inhdetachpending AS pending,
+		               c.relkind = 'f' AS foreign_table,
+		               (regexp_match(pg_get_expr(c.relpartbound, c.oid),
+		                             'TO \(''([^'']+)''\)'))[1] AS upper_bound
+		          FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid
+		         WHERE i.inhparent = parent) b
+		 WHERE b.upper_bound IS NOT NULL;
+END
+$drop_candidates$;
+
+CREATE FUNCTION pgcolumnar.partition_drop_plan(parent regclass, cutoff timestamptz)
+	RETURNS SETOF text
+	LANGUAGE plpgsql STABLE
+	SET search_path = pg_catalog, pg_temp
+	AS $drop_plan_tz$
+DECLARE
+	p record;
+BEGIN
+	IF cutoff IS NULL THEN
+		RAISE EXCEPTION 'cutoff must not be NULL' USING ERRCODE = '22023';
+	END IF;
+	FOR p IN
+		SELECT d.* FROM pgcolumnar.partition_drop_candidates(parent, 'timestamptz') d
+		 WHERE d.upper_bound::timestamptz <= cutoff
+		 ORDER BY d.upper_bound::timestamptz
+	LOOP
+		RETURN NEXT format('ALTER TABLE %s DETACH PARTITION %s %s', parent, p.part,
+		                   CASE WHEN p.pending THEN 'FINALIZE' ELSE 'CONCURRENTLY' END);
+		RETURN NEXT format('DO $pgcolumnar_drop$ BEGIN IF EXISTS (SELECT 1 FROM pg_catalog.pg_inherits WHERE inhrelid = %L::regclass) THEN RAISE EXCEPTION ''%% is still attached to its parent; its DETACH did not finish'', %L USING ERRCODE = ''55000''; END IF; DROP %s %s; END $pgcolumnar_drop$',
+		                   p.part::text, p.part::text,
+		                   CASE WHEN p.foreign_table THEN 'FOREIGN TABLE' ELSE 'TABLE' END, p.part);
+	END LOOP;
+END
+$drop_plan_tz$;
+
+CREATE FUNCTION pgcolumnar.partition_drop_plan(parent regclass, cutoff timestamp)
+	RETURNS SETOF text
+	LANGUAGE plpgsql STABLE
+	SET search_path = pg_catalog, pg_temp
+	AS $drop_plan_ts$
+DECLARE
+	p record;
+BEGIN
+	IF cutoff IS NULL THEN
+		RAISE EXCEPTION 'cutoff must not be NULL' USING ERRCODE = '22023';
+	END IF;
+	FOR p IN
+		SELECT d.* FROM pgcolumnar.partition_drop_candidates(parent, 'timestamp') d
+		 WHERE d.upper_bound::timestamp <= cutoff
+		 ORDER BY d.upper_bound::timestamp
+	LOOP
+		RETURN NEXT format('ALTER TABLE %s DETACH PARTITION %s %s', parent, p.part,
+		                   CASE WHEN p.pending THEN 'FINALIZE' ELSE 'CONCURRENTLY' END);
+		RETURN NEXT format('DO $pgcolumnar_drop$ BEGIN IF EXISTS (SELECT 1 FROM pg_catalog.pg_inherits WHERE inhrelid = %L::regclass) THEN RAISE EXCEPTION ''%% is still attached to its parent; its DETACH did not finish'', %L USING ERRCODE = ''55000''; END IF; DROP %s %s; END $pgcolumnar_drop$',
+		                   p.part::text, p.part::text,
+		                   CASE WHEN p.foreign_table THEN 'FOREIGN TABLE' ELSE 'TABLE' END, p.part);
+	END LOOP;
+END
+$drop_plan_ts$;
 /* ---------------------------------------------------------------------------
  * pgcolumnar.retired_group (#1294)
  *

pgcolumnar--1.0-alpha5.sql

@@ -2208,6 +2208,142 @@ BEGIN
 END;
 $create_ahead$;
 
+/*
+ * pgcolumnar.partition_drop_plan (#1297)
+ *
+ * The statements that retire every partition of a RANGE-partitioned parent
+ * whose whole range is older than cutoff (its upper bound at or before it), in
+ * range order: ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY, then DROP
+ * TABLE (DROP FOREIGN TABLE for one tier_partition moved to Parquet, whose file
+ * stays). A partition a concurrent detach left pending gets DETACH ... FINALIZE
+ * instead. A partition reaching MAXVALUE is never old; one from MINVALUE is, once
+ * its upper bound is past. A partition that is itself partitioned is listed whole.
+ *
+ * RETURNED, NOT RUN. DETACH ... CONCURRENTLY is refused inside any function, DO
+ * block or procedure, on every major (#1297 note 1), and a plain DETACH or a
+ * DROP of an attached partition takes ACCESS EXCLUSIVE on the parent, which
+ * blocks every query on the table. The caller runs each statement at top level:
+ *
+ *     SELECT * FROM pgcolumnar.partition_drop_plan('metrics', now() - interval '90 days') \gexec
+ *
+ * THE CUTOFF'S TYPE IS THE KEY'S (jdatcmd, #1297). A timestamp bound compared
+ * with a timestamptz cutoff converts in the session's zone, so one call could
+ * plan differently from two sessions. Two overloads, each refusing the other
+ * type of key (42804); an untyped literal resolves to timestamptz.
+ *
+ * THE DROP REFUSES WHILE THE PARTITION IS ATTACHED (jdatcmd, #1331 review). Any
+ * DETACH can fail -- a lock_timeout, a cancelled concurrent wait, a permission
+ * error -- and \gexec carries on unless ON_ERROR_STOP is set. A plain DROP of an
+ * attached partition would then take ACCESS EXCLUSIVE on the parent. So each
+ * drop is a DO block that raises 55000 while the partition is still in
+ * pg_inherits, a pending one included; the next plan's FINALIZE picks that up.
+ * A DROP may run inside DO; only DETACH ... CONCURRENTLY must be top-level.
+ *
+ * The same refusals as partition_create_ahead, with the same codes: a DEFAULT
+ * partition rules out the concurrent detach, and is refused rather than dropped
+ * with the exclusive one. No ownership check: this reads catalogs the caller
+ * can see, and DETACH checks ownership when the statements run.
+ */
+CREATE FUNCTION pgcolumnar.partition_drop_candidates(parent regclass, cutoff_type regtype)
+	RETURNS TABLE (part regclass, pending boolean, foreign_table boolean, upper_bound text)
+	LANGUAGE plpgsql STABLE
+	SET search_path = pg_catalog, pg_temp
+	AS $drop_candidates$
+DECLARE
+	strat   "char";
+	natts   int;
+	defid   oid;
+	keytype regtype;
+BEGIN
+	SELECT pt.partstrat, pt.partnatts, pt.partdefid, a.atttypid::regtype
+	  INTO strat, natts, defid, keytype
+	  FROM pg_partitioned_table pt
+	  JOIN pg_attribute a ON a.attrelid = pt.partrelid AND a.attnum = pt.partattrs[0]
+	 WHERE pt.partrelid = parent;
+	IF NOT FOUND OR strat <> 'r' OR natts <> 1 THEN
+		RAISE EXCEPTION '% is not RANGE-partitioned on one column', parent
+			USING ERRCODE = '55000';
+	END IF;
+	IF keytype NOT IN ('timestamptz'::regtype, 'timestamp'::regtype) THEN
+		RAISE EXCEPTION '% is partitioned on %, not on a timestamp', parent, keytype
+			USING ERRCODE = '55000';
+	END IF;
+	IF defid <> 0 THEN
+		RAISE EXCEPTION '% has a DEFAULT partition', parent
+			USING ERRCODE = '55000',
+			      HINT = 'A DEFAULT partition rules out DETACH CONCURRENTLY, so the partition lifecycle refuses to manage this table.';
+	END IF;
+	IF keytype <> cutoff_type THEN
+		RAISE EXCEPTION '% is partitioned on %, so its cutoff must be a %, not a %',
+			parent, keytype, keytype, cutoff_type
+			USING ERRCODE = '42804',
+			      HINT = CASE WHEN keytype = 'timestamp'::regtype
+			                  THEN 'Pass a timestamp, for example localtimestamp - interval ''90 days''.'
+			                  ELSE 'Pass a timestamptz, for example now() - interval ''90 days''.' END;
+	END IF;
+	-- MAXVALUE above has no upper bound to read, and is never old
+	RETURN QUERY
+		SELECT b.part, b.pending, b.foreign_table, b.upper_bound
+		  FROM (SELECT c.oid::regclass AS part, i.inhdetachpending AS pending,
+		               c.relkind = 'f' AS foreign_table,
+		               (regexp_match(pg_get_expr(c.relpartbound, c.oid),
+		                             'TO \(''([^'']+)''\)'))[1] AS upper_bound
+		          FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid
+		         WHERE i.inhparent = parent) b
+		 WHERE b.upper_bound IS NOT NULL;
+END
+$drop_candidates$;
+
+CREATE FUNCTION pgcolumnar.partition_drop_plan(parent regclass, cutoff timestamptz)
+	RETURNS SETOF text
+	LANGUAGE plpgsql STABLE
+	SET search_path = pg_catalog, pg_temp
+	AS $drop_plan_tz$
+DECLARE
+	p record;
+BEGIN
+	IF cutoff IS NULL THEN
+		RAISE EXCEPTION 'cutoff must not be NULL' USING ERRCODE = '22023';
+	END IF;
+	FOR p IN
+		SELECT d.* FROM pgcolumnar.partition_drop_candidates(parent, 'timestamptz') d
+		 WHERE d.upper_bound::timestamptz <= cutoff
+		 ORDER BY d.upper_bound::timestamptz
+	LOOP
+		RETURN NEXT format('ALTER TABLE %s DETACH PARTITION %s %s', parent, p.part,
+		                   CASE WHEN p.pending THEN 'FINALIZE' ELSE 'CONCURRENTLY' END);
+		RETURN NEXT format('DO $pgcolumnar_drop$ BEGIN IF EXISTS (SELECT 1 FROM pg_catalog.pg_inherits WHERE inhrelid = %L::regclass) THEN RAISE EXCEPTION ''%% is still attached to its parent; its DETACH did not finish'', %L USING ERRCODE = ''55000''; END IF; DROP %s %s; END $pgcolumnar_drop$',
+		                   p.part::text, p.part::text,
+		                   CASE WHEN p.foreign_table THEN 'FOREIGN TABLE' ELSE 'TABLE' END, p.part);
+	END LOOP;
+END
+$drop_plan_tz$;
+
+CREATE FUNCTION pgcolumnar.partition_drop_plan(parent regclass, cutoff timestamp)
+	RETURNS SETOF text
+	LANGUAGE plpgsql STABLE
+	SET search_path = pg_catalog, pg_temp
+	AS $drop_plan_ts$
+DECLARE
+	p record;
+BEGIN
+	IF cutoff IS NULL THEN
+		RAISE EXCEPTION 'cutoff must not be NULL' USING ERRCODE = '22023';
+	END IF;
+	FOR p IN
+		SELECT d.* FROM pgcolumnar.partition_drop_candidates(parent, 'timestamp') d
+		 WHERE d.upper_bound::timestamp <= cutoff
+		 ORDER BY d.upper_bound::timestamp
+	LOOP
+		RETURN NEXT format('ALTER TABLE %s DETACH PARTITION %s %s', parent, p.part,
+		                   CASE WHEN p.pending THEN 'FINALIZE' ELSE 'CONCURRENTLY' END);
+		RETURN NEXT format('DO $pgcolumnar_drop$ BEGIN IF EXISTS (SELECT 1 FROM pg_catalog.pg_inherits WHERE inhrelid = %L::regclass) THEN RAISE EXCEPTION ''%% is still attached to its parent; its DETACH did not finish'', %L USING ERRCODE = ''55000''; END IF; DROP %s %s; END $pgcolumnar_drop$',
+		                   p.part::text, p.part::text,
+		                   CASE WHEN p.foreign_table THEN 'FOREIGN TABLE' ELSE 'TABLE' END, p.part);
+	END LOOP;
+END
+$drop_plan_ts$;
+
 /* ---------------------------------------------------------------------------
  * Incremental rollups (#1294).
  *