Repository navigation
feat: pgcolumnar.partition_drop_plan returns the statements that retire old time partitions (#1297) - #1331
Conversation
jdatcmd
left a comment
There was a problem hiding this comment.
Review at b4a2386. The function is right as far as I read it: tiered partitions are detached first too, names come out schema-qualified through regclass under the function's search_path, the upper_bound::timestamptz cast reads an offset-bearing bound so no zone enters, and every one of my six points has an arm. One change I'd want before merge, and one small label.
A DROP after a failed DETACH takes the parent's exclusive lock
Your point 4 found it: when the DETACH failed, the following DROP removed the still-attached partition under ACCESS EXCLUSIVE on the parent. The stronger quoting arm closes that cause, but not the failure mode. Any DETACH can fail: a lock_timeout, a cancelled CONCURRENTLY wait (which also leaves the partition pending), or a permission error. \gexec carries on to the next statement unless ON_ERROR_STOP is set, and the how-to doesn't set it, nor would most schedulers. The DROP then blocks every query on the table and drops a partition the plan said it would detach first. The docs promise the opposite ("takes no exclusive lock on the parent").
What I'd do: make the drop statement refuse when the partition is still attached, so the plan is safe whatever runs it. For example, emit
DO $d$ BEGIN
IF EXISTS (SELECT 1 FROM pg_inherits WHERE inhrelid = '<part>'::regclass) THEN
RAISE EXCEPTION '<part> is still attached; its DETACH did not finish' USING ERRCODE = '55000';
END IF;
DROP TABLE <part>;
END $d$A DROP inside DO is fine; only the DETACH ... CONCURRENTLY must be top-level. A pending partition is still in pg_inherits, so the DROP refuses that too, and the next plan's FINALIZE picks it up. Also put \set ON_ERROR_STOP on in the how-to example, as a second line of defence.
Arm, in both twins: make a DETACH fail (lock_timeout while another session holds a lock on the parent), run the rest of the plan, and require the partition to still exist and still be attached, and the DROP to fail with 55000.
Small
"premise: a second plan for the same cutoff is empty" is behaviour, not a fixture fact. A plan that kept listing dropped or pending partitions would redden it, so under #1236 it is an arm. It probably needs no new mutation: one that drops the upper_bound <= cutoff filter, or keeps detached partitions in the candidates, should redden it.
Checked
- Ledger: main's 2286 rows are byte-identical outside
partition_lifecycle. 2308 rows in all, of which 1805 are never red, equal to the budget. All ten never-redpartition_lifecyclerows are labelled premises; the one above is the only one I'd move. - CI: still running at the time of writing.
…red (commandprompt#1297) Seen red on PG17 at 4c72242: the function is absent (42883), so no plan, no drop, no refusal. Its premises pass: four partitions holding rows, and a DETACH ... CONCURRENTLY cancelled in its second phase leaves the partition pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re old partitions (commandprompt#1297) DETACH ... CONCURRENTLY is refused inside any function, so the drop half returns its statements for the caller to run at top level (\gexec): DETACH CONCURRENTLY then DROP for every partition whose upper bound is at or before the cutoff, in range order, schema-qualified, and DETACH ... FINALIZE for one a concurrent detach left pending. The parent is never locked exclusively. The same refusals and codes as partition_create_ahead. Identical patches in the alpha5 script and the alpha4 upgrade. The arms now expect schema-qualified names, and the timestamp-key arm adds a cutoff that only a wall-clock reading gets right (a UTC reading passed the first version). partition_lifecycle.sh 41/41 on PG17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erence (commandprompt#1297) The twin was seen red on PG17 against main (42883) and passes with the function. Both fixtures now create their partitions out of range order, so a plan in catalog order would fail the order arm. Its own fixture: weekly partitions in 2019, Asia/Tokyo, a private schema. compare_to_bash grades the pair with nothing missing. cluster_tests 525 -> 526 by collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…andprompt#1297) No mutation of partition_drop_plan reddens it: dropped tables are gone, so a second plan is empty by construction. Renamed in both twins (commandprompt#1236). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed (commandprompt#1297) Red on PG17: a cutoff of the other timestamp type is planned rather than refused (42804), and a tiered partition is planned with DROP TABLE, which fails on a foreign table. Green already, as guards: names that need quoting, MINVALUE and MAXVALUE bounds, a partition that is itself partitioned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tiered partitions as foreign tables (commandprompt#1297) From jdatcmd's review: a timestamp bound compared with a timestamptz cutoff converts in the session's zone, so the same call could plan differently from two sessions. Two overloads now, timestamptz and timestamp, and a key of the other type is refused (42804, with a hint naming the right expression). A partition tier_partition moved to Parquet is planned with DROP FOREIGN TABLE; its file stays. The checks the overloads share live in partition_drop_candidates(parent, cutoff_type). partition_lifecycle.sh 49/49 on PG17; identical patches in both scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ype rule, tiered and sub-partitioned partitions documented (commandprompt#1297) test_partition_lifecycle.py matches the shell suite's 49 checks by name (compare_to_bash: nothing missing), with its own fixture: Tokyo and Denver, a quoted schema of its own, half-year sub-partitions, a tiered January in 2019. Both twins green on PG17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…how-to sentences shortened (commandprompt#1297) A plan with unquoted names failed its DETACH, and the DROP still removed the attached partition (under ACCESS EXCLUSIVE on the parent), so the tables left could not show the bug: the unquoted-names mutation survived. The arm now also requires that no planned statement failed, and is red under that mutation. docs_style's sentence-length rule caught two sentences in the how-to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check is an arm (commandprompt#1331 review) Red on PG17: a DETACH that times out on the parent's lock, then the plan's DROP succeeds and removes the still-attached partition, under ACCESS EXCLUSIVE on the parent. The expectations now spell the guarded drop (gdrop). 'a second plan for the same cutoff is empty' also requires the kept partitions still attached, which makes it behaviour a mutation reaches (commandprompt#1236). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…attached (commandprompt#1331 review) Each drop is now a DO block that raises 55000 while the partition is in pg_inherits, so a DETACH that failed (a lock_timeout, a cancelled concurrent wait, a permission error) cannot be followed by a DROP that takes ACCESS EXCLUSIVE on the parent: \gexec carries on after an error. A pending partition is refused too; the next plan finalises it. partition_lifecycle.sh 50/50 on PG17; identical patches in both scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TOP and the drop's refusal documented (commandprompt#1331 review) test_partition_lifecycle.py matches the shell suite's 50 checks by name. The how-to's example sets ON_ERROR_STOP and says what the drop's refusal guards. Both twins green on PG17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@OffgridwithJD, the owner has decided that alpha5 waits for this PR. All seven time-series features are merged (#1292 last, via #1330), so this is now the last item before the tag. The release-notes PR is #1333. When you push the fix for my review (the DROP refusing while the partition is still attached, and the premise that should be an arm), I'll review it straight away. I'll also add a line about |
…nteen mutations (commandprompt#1297) partition_lifecycle.sh 50/50 and test_partition_lifecycle.py on each of PG15-19, main at b55cb23. Every never row of the suite is a premise. Ledger 2309 / 1804, gate rc=0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b4a2386 to
d7effdc
Compare
|
@jdatcmd Thanks: that was the real failure mode behind my quoting finding. Both points are addressed at d7effdc, rebased onto b55cb23 (#1330). The guarded drop. Each partition's drop is now: DO $pgcolumnar_drop$ BEGIN IF EXISTS (SELECT 1 FROM pg_catalog.pg_inherits WHERE inhrelid = '<part>'::regclass)
THEN RAISE EXCEPTION '% is still attached to its parent; its DETACH did not finish', '<part>' USING ERRCODE = '55000';
END IF; DROP TABLE <part>; END $pgcolumnar_drop$It is one line, with the names through The arm, in both twins. Another session holds
My first attempt kept the holder's lock through the DROP, so the DROP also timed out and the arm was red for the wrong reason. The drop now runs after the lock is released. "a second plan … is empty" is an arm again: "a second plan for the same cutoff is empty, and the partitions it kept are still attached". The second half is what a mutation reaches. Without the cutoff filter, the first run drops everything, and the second plan was empty anyway. Evidence:
How it was rebased. I rebuilt the branch by cherry-picking onto main without my superseded ledger commit, then regenerated the ledger from main's rows. The per-file patches match the pre-rebuild branch, except |
jdatcmd
left a comment
There was a problem hiding this comment.
Approved at d7effdc. Merging when CI is green on this head; alpha5 waits for this PR.
What I checked:
- The guarded drop: the plan text,
%Landregclassfor every name,DROP FOREIGN TABLEfor a tiered partition, and a pending partition refused by the samepg_inheritstest. - The second-plan arm: it reaches a mutation now, through "the partitions it kept are still attached".
- Ledger: main's 2286 rows are byte-identical outside
partition_lifecycle. 2309 rows, of which 1804 are never red, equal to the budget, and every never-red row in the suite is a premise. - PG19, which CI doesn't build:
partition_lifecycle.sh50/50 at this head.
A nit, not blocking. A partition name can contain $pgcolumnar_drop$, which ends the DO block's dollar quote early. I probed it on PG17 with two crafted names, and it can't be exploited. The text before the name always leaves the DO body inside an unfinished string literal, because %L doubles any quote the name carries. So the DO fails to compile, and that aborts the rest of the query string. The injected CREATE TABLE never ran, and the 63-byte name limit leaves little room besides. Still, format('DO %L', body) would make the body a proper literal and remove the question. Worth a follow-up whenever convenient.
|
@jdatcmd Thanks. I'll take the dollar-quote nit as a follow-up PR after this merges, so alpha5 isn't held for it. It will emit |
|
Merging: approved at d7effdc, and CI is green on that head. |
…rop as planned, red (#1331 review) Red on PG17 at 614eb02: the DO block's body ends at the tag inside the quoted partition name, so the statement fails to parse and the partition stays. Not exploitable (jdatcmd's probe), but the plan does not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…not dollar-quoted (#1331 review) A partition name holding $pgcolumnar_drop$ ended the DO block's dollar quote and the statement failed to parse. The body is now format('DO %L', ...), so no name can end it. partition_lifecycle.sh 52/52 on PG17; identical patches in both scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ar-quote tag (#1331 review) Red against main's dollar-quoted drop (a scratch copy of this file with main's plan text, so only the new arm can fail); green with the fix. compare_to_bash: nothing missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…review) Ledger 2311 / 1805, gate rc=0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix: partition_drop_plan's drop block carries its body as a literal, not dollar-quoted (#1331 review)
The drop half of #1297, shape b from my design note, which you confirmed on #1297. Your six review points are covered below.
What it is
pgcolumnar.partition_drop_plan(parent, cutoff)returns the statements that retire every partition whose whole range is older than the cutoff, meaning its upper bound is at or before it. It returns them in range order and schema-qualified:ALTER TABLE … DETACH PARTITION … CONCURRENTLY, thenDROP TABLE.DETACH PARTITION … FINALIZE, then the drop.DROP FOREIGN TABLE. The Parquet file stays, and the docs say so.It runs nothing, because
DETACH … CONCURRENTLYrefuses to run inside any function. You run the result with\gexec:The parent is never locked exclusively. There is no ownership check, as you said:
DETACHchecks when the statements run.Your six points
timestamptzandtimestamp. A key of the other type is refused with42804, and the hint names the right expression. An untyped literal resolves totimestamptz. Arms:timestampkey with atimestampcutoff gives the same plan from Los Angeles and Kolkata (Tokyo and Denver in the twin);DROP FOREIGN TABLE. Arms: it is planned that way, and running it leaves the Parquet file."Drop Zone"and partitions"Old Part"/"New Part"("Py Drop","Week One"in the twin). The arm runs the plan and requires every statement to succeed. My first version only checked which tables were left, and the unquoted-names mutation survived it: the DETACH failed, but the DROP then removed the still-attached partition, under ACCESS EXCLUSIVE on the parent. With the stronger arm, that mutation is red.pgcolumnar_delete_storage_tree.FINALIZE on all five majors: the pending-detach arm is green on 15–19 in both twins.
Evidence
partition_lifecycle.shgives 49/49 andtest_partition_lifecycle.py2/2 on each of PG15–19, then again on PG17 on this tree (main at 80bce53).compare_to_bashfinds nothing missing.DROP TABLE. The twin was also red against main.timestampbound totimestamptzreads it in the session's zone, so that cast is the wall-clock reading itself.suites_not_coveredstays 246. One check, "a second plan for the same cutoff is empty", held under every mutation, so it is now a premise in both twins.native_upgrade_converge,entry_point_privilege,server_file_privilege,tier_partitionanddocs_styleall pass;docs_stylecaught two long sentences in the how-to, which are fixed.cluster_testsgoes 526 → 527. The alpha5 install script and the alpha4 upgrade script carry identical patches.Not done: the daemon running this (shape a). That still waits on the question from my design note.
🤖 Generated with Claude Code