[OffgridwithJD]

pgcolumnar.tier_partition: move a cold partition to Parquet, exactly once (#1295)

pgcolumnar.tier_partition(partition regclass, server name, path text,
                          lock_wait interval DEFAULT '10 seconds') RETURNS bigint

This follows the owner's decision on #1295: the move is atomic, with a brief ACCESS EXCLUSIVE on the parent. It goes into pgcolumnar--1.0-alpha5.sql and the alpha4→alpha5 upgrade script, per the gate change on #1283. In one transaction, at READ COMMITTED, it:

1. takes a SHARE lock on the partition, so writes wait and reads go on;
2. exports with export_parquet, then checks the row count against a count(*) taken under the lock;
3. detaches and drops the partition, holding ACCESS EXCLUSIVE on the parent until commit, which covers catalog work only;
4. creates a foreign table with the partition's name, columns, owner, grants and comment;
5. attaches it with the partition's own bounds.

Every lock waits at most lock_wait (55P03), and the caller's lock_timeout is restored afterwards. pgcolumnar.tier_fault_after, a superuser-only developer GUC, raises an error after a named step, or pauses, so exactly-once and the race are tested rather than argued.

@jdatcmd's review of the proposal, point by point (each an arm, red against the first version)

1. Isolation.
   • A row committed between the caller's snapshot and the lock is the test.
   • At REPEATABLE READ and SERIALIZABLE the first version lost it: 16992|0 where 16993|1 was owed. Those levels are now refused (25000).
   • At READ COMMITTED each plpgsql statement takes a fresh snapshot, so the export after the lock sees the row; the arm proves it.
2. A UNIQUE index or primary key on the parent is refused before the export (0A000). The first version failed at ATTACH and left an orphan file.
3. Dependents of the partition are checked against a pg_depend allowlist and refused (2BP01): a view, a row-level policy, a trigger of its own, or a foreign key of its own. The first version moved a partition with a policy and dropped the policy silently. Grants and comment now carry over.
4. The racing insert. Your deadlock hypothesis is right. With the move paused after the export, an insert through the parent deadlocks with it. One side gets 40P01, and every row is exactly once.
5. Paths. Your narrower rule: a path that a foreign table in this database reads is refused (58P02), and a retry over a failed attempt's orphan file works.
6. ANALYZE: not possible. pgcolumnar_parquet has no AnalyzeForeignTable.

Findings from my own arms

• pg_stat_file(...) IS NOT NULL is false for a file that exists, because creation is NULL on Linux. So the first existing-path check never fired, and a second move overwrote a file another partition was reading.
• Without the SHARE lock, a write in flight when the move started was dropped with the partition.
• A foreign key referencing a columnar table is refused by pgcolumnar itself, so that case can't arise. My first arm for it passed on a leftover trigger: a multi-statement -c is one transaction, and the failed CREATE rolled back the previous case's DROP. Each dependent is now made in statements of its own, with a premise that it exists. A control arm shows the same partition moves once its dependents are gone.

Tests

• test/tier_partition.sh and the independent test/pytest/test_tier_partition.py, which uses threads for its concurrent sessions: 65/65 on PG15–19, each.
• compare_to_bash: 42 literal and 6 template matches, 0 missing.

Removal proofs: sixteen mutants, each red on its arm. They are:

• the original nine: no SHARE lock, no path check, a suffixed name, a shifted bound, no DEFAULT check, no wrapper check, no partition check, plus no_drop (rename aside instead of drop: the catalog-cleanup arms) and return_zero;
• the seven new: any isolation level, no UNIQUE refusal, no dependents check, no grant copy, no lock_wait, the caller's timeout not restored, and the old blanket path refusal.

The seven new ones are red in the twin too.

Not reachable by any arm, and stated plainly:

• The exported-count check can't fire at READ COMMITTED with the SHARE lock held. It's a defensive guard that turns a future gap into an error, as you suggested.
• The five fault arms and the rollback arm hold because plpgsql is transactional. Only an implementation that committed partway (dblink, a procedure with COMMIT) could redden them, so they stay never in the ledger. The ledger lists which arms are never.

Registries

• Ledger: 65 rows from a five-major run and sixteen mutations on this tree, replayed onto feat: partition_create_ahead keeps a time-partitioned parent ahead of now (#1297) #1308's ledger. 32 are dated, and no premise is among them: four checks a mutation reddened were renamed from premises first (test: 87 arms have survived every mutation aimed at their neighbours: vacuous, or named for something they do not measure #1236). 33 are never: 15 premises, and 18 arms the budget comment names (transactional, core behaviour, or export_parquet's own refusal). Census 1700 → 1733.
• TESTS.md section 94; cluster_tests 503 by collection. Guard half: 413. docs_style passes. native_upgrade_converge PG17: 21/21, now that both Native partition lifecycle for time-partitioned columnar tables: create ahead, drop past retention #1297's and this function are in the alpha5 scripts. harness_selftest PG17 is in the follow-up comment.

🤖 Generated with Claude Code

[OffgridwithJD]

harness_selftest on PG17 for this head (5eb86f2): 1223 passed, 0 failed. With tier_partition.sh at 65/65 and native_upgrade_converge at 21/21 on the same tree after the rebase onto #1308, that completes the evidence listed in the description.

One harness note, not this PR's: a PG17 test postmaster outlived its suite. It started at the end of the convergence run and received no statements. I found it because it had inherited my flock descriptor and was holding the lock. It's stopped now. If native_upgrade_converge.sh turns out to leave a server running, I'll file it separately once I've reproduced it.

🤖 Generated with Claude Code

[jdatcmd]

Review of 5eb86f2. The design is right, and the exactly-once property held under everything I threw at it. READ COMMITTED is enforced, the SHARE lock comes before the export, the count check is there, lock_timeout is transaction-local and restored, and the dependents allowlist is sound. The two bugs your own arms found were good finds.

Four inputs make the move fail, though, every one after the export. Each rolls back cleanly, so no data is lost. I checked: every partition still holds its 100 rows and is still columnar. But each leaves an orphan file, and the first two make the function unusable for ordinary tables.

How I measured

I built this head into /usr/local/pg18_assert and used one pgcolumnar_parquet server. Each partition holds 100 rows over January 2026.

case	result
partition in schema "Tier"	ERROR: schema ""Tier"" does not exist from CREATE FOREIGN TABLE """Tier""".m_old
partition owned by role "Owner X"	ERROR: role ""Owner X"" does not exist from ALTER FOREIGN TABLE public.o_old OWNER TO """Owner X"""
parent column t text COLLATE "C"	ERROR: child table "c_old" has different collation for column "t", at ATTACH
parent column w int GENERATED ALWAYS AS (v * 2) STORED	ERROR: column "w" in child table must be a generated column, at ATTACH
control: plain table, public schema, default collation	moved, 100 rows read back through the parent

All five .parquet files were left behind, four of them orphans.

Blocking 1: double quoting (schema, owner, grantees)

format('%I', rel.relnamespace::regnamespace) quotes the text of regnamespace, which is already quoted when it needs to be. The same applies to %I on rel.relowner::regrole, and to quote_ident(acl.grantee::regrole::text) in the GRANT loop. Any schema, owner or grantee whose name needs quoting breaks the move: mixed case, a space, a reserved word.

Fix: use %s for the reg* casts, which quote themselves, or %I on nspname/rolname read from the catalogs. Arms: one with a quoted schema, one with a quoted owner, one with a quoted grantee, each red on 5eb86f2.

Blocking 2: collation and generated columns

The cols list carries type and NOT NULL only, and ATTACH checks both of these.

• Collation: add COLLATE %I from attcollation when it differs from the type's default.
• Generated columns: a foreign table can carry GENERATED ALWAYS AS (expr) STORED, built from pg_get_expr(adbin, adrelid). The Parquet file already holds the stored values, so reads are right as they are.
• Or refuse these parents up front, as you do for UNIQUE. Either way, the error must come before the export.
• PG18's virtual generated columns (attgenerated = 'v') need a decision too: support them, or refuse them up front.
• Arms: a COLLATE "C" column and a stored generated column, each red on 5eb86f2.

Not blocking

• The path check is string equality ('path=' || path = ANY(ftoptions)). /x/./a.parquet, //x/a.parquet or a symlink reach the same file under another spelling. Normalising both sides, or documenting it, would do.
• Every failure after the export leaves an orphan file, as your design note says. With these four, the common cause was a late refusal. Moving every refusal before the export, as the UNIQUE check already is, makes orphans rare. Could the docs say "a failed move may leave the file at path; a retry overwrites it"?

I haven't re-run your sixteen mutants; I'll rely on your report and CI. The probes above are mine. I'll re-review the four arms red-then-green.

[OffgridwithJD]

All four are fixed locally, each with an arm I've seen red on 5eb86f2 and green after. On PG18, the shell suite fails 6 before the fix and is 71/71 after; the twin is also 71/71:

arm	on 5eb86f2
a partition in a schema whose name needs quoting ("Tier Q")	3F000
a partition owned by "Owner X", keeping that owner	42704
a grant to "Reader Y", carried to the foreign partition	42704
a COLLATE "C" column	42P21 at ATTACH
a stored generated column	42804 at ATTACH
a virtual generated column (18+; named SKIP / UNRUN below)	42804 at ATTACH

The fixes:

• %s for every regnamespace and regrole value, including GRANT's grantee.
• The column list now carries COLLATE (from attcollation::regcollation, when it isn't the type's default) and GENERATED ALWAYS AS (expr) STORED|VIRTUAL (from pg_get_expr).

I measured on PG17 and 18 that ATTACH accepts a foreign table with a stored or a virtual generated column, and that a parent identity column needs nothing. So virtual columns are supported, not refused.

The two non-blocking points: the how-to now says the path check compares the text of the path, so a symlink or another spelling isn't recognised. It already said a retry over the same path works.

Still running: the five-major runs and 19 mutants on the rebased tree (now on d208bce), including three new ones (quoting twice, no collation, no generation clause). I'll push when they're in, with the ledger replayed.

🤖 Generated with Claude Code

[OffgridwithJD]

Pushed at 663bf84, rebased onto 7e6fd48 (after #1318).

On this tree:

• tier_partition.sh and its twin: 71/71 on PG18 and 19, and 70 + 1 named SKIP / UNRUN (the virtual column) on 15–17. pytest exits 0 everywhere.
• 19 mutants on PG17, each red on its arm. That's the sixteen from before plus quote_twice (the schema arm), no_collation (the collation arm) and no_generation (the stored-column arm). The three new ones are red in the twin too. no_generation also ran on PG18, where it reddens the virtual-column arm.
• Ledger replayed onto main's (1984/1705): 71 rows, 38 dated, no dated premise, census 1738. The gate agrees.
• cluster_tests 503 → 506 by collection, since the virtual-column test is a third function. Guard half: 415. TESTS.md section 96.

🤖 Generated with Claude Code

[jdatcmd]

Approved at 663bf84. I re-ran my probe against this head's build (PG18 assert, one pgcolumnar_parquet server, 100 rows per partition over January 2026), plus two cases I hadn't tried before:

case	5eb86f2	663bf84
partition in schema "Tier"	schema ""Tier"" does not exist	moved (100); now relkind f; 100 rows through the parent
text column COLLATE "C"	collation conflict at ATTACH	moved; 100 rows read back
stored generated w = v * 2	must be a generated column	moved; sum(w) = 10100 = 2 × 5050
owner "Owner X", plus a grant to "Reader Y" (new)	role ""Owner X"" does not exist	moved; relowner is "Owner X", and relacl carries both the owner's and "Reader Y"'s entries
PG18 virtual generated w = v * 3 (new)		moved; sum(w) = 15150 = 3 × 5050

Exactly-once held throughout, here and in the first round. I'll merge once CI is green.

[jdatcmd]

@OffgridwithJD, heads-up: my approval at 663bf84 stands on the content, but #1314 and #1313 have merged since, so this now conflicts with main, most likely in the ledger budget and the cluster count. Once you rebase, I'll re-approve by comparing each file's +/- lines against 663bf84 and checking the restated census, then merge when CI is green.

[OffgridwithJD]

Rebased onto 024e737 (after #1319) and fixed the red CI. Head is now 1346113, which replaces 663bf84.

The CI failure. suites on PG17 and PG18 failed one check: server-file function is covered by the boundary test: tier_partition. server_file_privilege.sh requires every function that takes a path to have a boundary entry, and I hadn't run that suite. tier_partition writes its file through export_parquet, so its gate is pg_write_server_files.

• The new entry's fixture: a columnar partition both test roles can SELECT and UPDATE (UPDATE for the SHARE lock tier_partition takes before exporting). That leaves the role gate as the only thing between them. Past the gate, the privileged role writes the file and is then stopped by DETACH, which needs the parent's owner.
• Red first on PG17: 45 pass + 1 fail, the failure being the coverage arm.
• Each gate arm is red under its own mutation of tier_partition:
  • SECURITY DEFINER → no role is refused by the pg_write_server_files gate: tier_partition (got no);
  • a refusal naming the role on entry → the pg_write_server_files role gets past the gate: tier_partition (got yes).

The rebase. #1319 added code in the same places as this PR: the SQL scripts, the docs, and a GUC in columnar_tableam.c. I kept both sides, and every file's +/- lines against main are identical to 663bf84's against its base. The exceptions are the restatements below. One thing went wrong in resolving: git put the shared /* above the conflict, so my GUC comment came out without its opener. The per-file comparison caught it before a build did.

• Ledger: main's 2059 rows plus the 71 tier_partition rows from 663bf84, byte-identical, in the file's sort order.
• Budget restated: 1698 → 1731; rows 2059 + 71 = 2130; dated 361 + 38 = 399. gate rc=0.
• Registries: TESTS.md section 98 (docs, test: gap-fill, LOCF and interpolation recipes, held to a reference (#1296 phase 1) #1311 and fix: native_upgrade_converge.sh stops its cluster on exit (#1321) #1322 also take 98; whichever merges later renumbers). cluster_tests 511 → 514 by collection.

Re-run on this exact tree, PG15–19:

PG	tier_partition.sh	test_tier_partition.py	server_file_privilege	entry_point_privilege
15–17	70 + 1 skipped of 71	70 + 1 unrunnable of 71	48/48	31/31
18–19	71/71	71/71	48/48	31/31

The skip below 18 is the virtual-generated-column arm, named in expected_unrunnable.txt. The guard half passes 415/415, and docs_style passes.

[OffgridwithJD]

Correction to my last comment: I pushed 1346113 to feat/1295-tier-partition-v2, but this PR tracks feat/1295-tier-partition, so until a minute ago the PR still showed 663bf84, and the red CI you'd see is still the old head's. The PR head is now 1346113, and CI is running on it. Everything in that comment applies to 1346113.

[jdatcmd]

Approved at 1346113.

• Per file against 663bf84 (my own check, base 7e6fd48 then versus 024e737 now): the code, SQL, docs and both tier twins are identical. Four files differ: the budget, TESTS.md and expected_tests are restatements, and server_file_privilege.sh is the CI fix.
• The CI fix, run here: server_file_privilege is 48/48 on PG17 from this head, including server-file function is covered by the boundary test: tier_partition and both gate arms.

It conflicts with current main again, after #1311 and #1322, so one more rebase is needed. I'll re-approve by the same per-file comparison and merge when CI is green.

[OffgridwithJD]

@jdatcmd I rebased onto 2e34f90 (the #1322 merge). The head is now 99680fb.

Per-file patch identity against the head you approved (1346113 on 024e737). All 16 files are listed. 13 are byte-identical, including every src/ file and the SQL. The other three are restatements:

• check_ledger.tsv: main's file plus the same 71 tier_partition rows, unchanged.
• check_ledger_budget.txt: the entry is restated on top of docs, test: gap-fill, LOCF and interpolation recipes, held to a reference (#1296 phase 1) #1311 and fix: native_upgrade_converge.sh stops its cluster on exit (#1321) #1322: 1701 → 1734, rows 2074 + 71 = 2145, dated 373 + 38 = 411. The merged ledger's census agrees: 2145 rows, 1734 never.
• TESTS.md: test_tier_partition.py is now section 100, after 98 (timeseries_recipes) and 99 (suite_exit_trap). In run_all_versions.sh and COMPLETE, tier_partition sorts before timeseries_recipes.
• expected_tests.txt: cluster_tests 513 → 516, by collection on the rebased tree.

On the rebased tree:

• PG17: tier_partition.sh gives 70 passed + 1 skipped (the virtual-column arm, which needs 18+). test_tier_partition.py gives 3/3.
• Gate: rc=0 with that log plus the earlier clean logs from 15, 16, 18 and 19. 280 registered, 247 not covered.
• Guard half: 419/419.

[jdatcmd]

Re-approved at 99680fb. My own per-file check against 1346113: everything is identical except the budget, TESTS.md and expected_tests, which are restatements. The budget's 1734 equals the never rows in this head's ledger. Merging when CI is green.

[jdatcmd]

Merging now: approved at 99680fb, CI green.