[OffgridwithJD]

Stacked on #1458 (merge after it). Two commits: 585369d6 (the twin), and 70c186ee, which runs its server tools as postgres under root (jdatcmd's review). It is 16/16 as root and as a normal user.

What was wrong

pg_upgrade.sh's "the index still answers the same range" did not test the index. Right after pg_upgrade there are no planner statistics, and with enable_seqscan=off the planner still took the columnar custom scan for the range. Measured: with DROP INDEX c_id run after the upgrade, the old shell stayed 16/16 green.

The arm and a new "premise: the range lookup takes the upgraded index" now run with pgcolumnar.enable_custom_scan=off. The premise greps the printed plan for an index scan of c_id; a bitmap scan counts.

Red first (harness mutant, PG17 → 18)

mutant	old shell	new shell	twin
none (head)	15/15	16/16	16/16
DROP INDEX c_id after the upgrade	green	red on the premise	red on the premise

This is a harness mutant, not a source one: pg_upgrade copies the extension catalogs as files, so I found no source mutant that a single arm separates. It dates nothing either way. The suite has its own check(), records nothing, and is not registered, so there are no ledger rows and no budget change.

The twin

test_pg_upgrade.py carries all 16 checks by the shell's name (compare_to_bash: 16 literal, 0 missing). Like the shell it needs two majors:

• --pg-config names the new major.
• PGC_UPGRADE_FROM_PG_CONFIG names the old one. The twin builds and installs the extension for it from a copy of the tree.

The matrix runs one major at a time, so there the twin declines with UNMET_PRECONDITION. expected_unrunnable.txt lists it for 15–19, and a run that sets the variable is an upgrade run whose row that file would refuse.

Runs

pair	shell	twin
15 → 16	16/16	16/16
16 → 17	16/16	16/16
17 → 18	16/16	16/16
18 → 19 (19rc1)	16/16	16/16

The 18 → 19 run caught a twin bug: it read the major of 19rc1 as 191, because it kept every digit. The fix takes the leading digits only.

Bookkeeping

• COMPLETE gains pg_upgrade.
• TESTS.md gains §151.
• cluster_tests goes 680 → 681.

The NO_CLUSTER guards pass (438), harness_selftest.sh gives 1245, and docs_style.sh passes.

🤖 Generated with Claude Code

[jdatcmd]

Requesting changes on 585369d, for one blocker in the twin. The shell side holds up.

The shell side, measured on PG17 → PG18 (link mode):

• head: 16/16;
• with DROP INDEX c_id run after the upgrade: red on premise: the range lookup takes the upgraded index, got [0] want [1].

So the new premise is what makes the index arm about the index, as you measured.

The blocker: the twin cannot run as root, and this host runs pytest as root. Run in upgrade mode (PGC_PG_CONFIG=/usr/local/pgsql/bin/pg_config PGC_UPGRADE_FROM_PG_CONFIG=/usr/local/pg17/bin/pg_config, under sudo, as the harness requires here), it fails at the head:

test_pg_upgrade.py:61: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pgc-upgrade-…/old/postgresql.conf'

The cause is the line before that: initdb: error: cannot be run as root. _cluster calls initdb through _sh, which neither drops to the postgres user nor checks the return code. So the refusal is lost, and the failure shows up one step later as a missing file.

This is the host-shaped assumption the harness already handles. pgc_cluster._asroot runs initdb, pg_ctl and postgres as postgres when euid is 0, and lib.sh uses runuser for the same reason. pg_upgrade refuses root too, so all four binaries need it, and the data and work directories must be owned by postgres.

Asks:

1. Run initdb, pg_ctl and pg_upgrade through _asroot (or the same runuser drop), and chown the work tree to postgres when root.
2. Check each step's return code and fail with its stderr, so a refused step names itself rather than surfacing as FileNotFoundError.

The decline path (UNMET_PRECONDITION when the variable is unset, which is what the matrix and CI exercise) is fine as it is, and is why CI can't see this.

[OffgridwithJD]

@jdatcmd You were right, and thanks: I've reproduced it and fixed it in 70c186ee.

Reproduced as root (sudo, PG17 → PG18, on staged copies of the prefixes and the tree so postgres can traverse them). At 585369d6:

E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pgc-upgrade-…/old/postgresql.conf'

The fix, taking your two asks in order:

1. initdb, pg_ctl (start, stop and the teardown's immediate stop) and pg_upgrade now go through a _server_tool helper. It works like pgc_cluster._asroot: under euid 0 it chowns the data directories to postgres and runs the binary under runuser -u postgres --. pg_upgrade keeps cwd=work, a 0777 directory, for its logs and socket.
2. Every step checks its return code and raises "<tool> failed rc=N: <stderr>". The one exception is pg_upgrade, whose exit code stays the check "pg_upgrade exited cleanly", and which prints its output when that fails. A failed make or make install of the old major's extension now shows its stderr too.

While I was there:

• initdb passes -U postgres explicitly, as Cluster.initdb does. Without it, a host with no wrapper names the superuser after the OS user.
• unix_socket_directories is pinned to the datadir, as pgc_cluster does. A packaged build's default is a directory only postgres can write.

After the fix:

• 16/16 as root;
• 16/16 as a normal user (17 → 18).

I also ran #1463's replication twin as root before you got to it: 44/44. It uses pgc_cluster.Cluster for all three clusters, so it already drops privileges. #1463 is rebased onto 70c186ee.

🤖 Generated with Claude Code

[jdatcmd]

Approving 70c186e. Both asks are done.

Re-measured on PG17 → PG18, with pytest run as root (sudo), as on this host:

• head: test_pg_upgrade.py 16 checks, passed;
• the same harness mutant, DROP INDEX c_id after the upgrade: red on premise: the range lookup takes the upgraded index, got 0 want 1.

The twin now matches the shell side I measured on 585369d.

_server_tool runs initdb, pg_ctl and pg_upgrade through runuser -u postgres when euid is 0, with the work tree chowned first. A refused step raises with its own name and stderr, so the FileNotFoundError-one-step-later shape is gone. The make for the old major also asserts its return code now.

[jdatcmd]

Re-approving d70e5d6, the restate onto main db56168. I compared every commit's patch with the approved stack using git range-diff. The test and source commits are unchanged, and the ledger-seed commits changed only by re-anchoring on #1470. The census at this head matches its budget paragraph, and the counts step consistently. At #1467's head: the guards pass (438), 687 collected (matching the file), and harness_selftest is 1245/1245.