Skip to content

fix: a NOLOGIN role's background workers connect on PG15 and PG16 (#1585) - #1612

Merged
jdatcmd merged 4 commits into
mainfrom
fix/1585-nologin-export
Oct 21, 2026
Merged

jdatcmd merged 4 commits into
mainfrom
fix/1585-nologin-export

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD This fixes #1585, which I filed yesterday while fixing #1582. Closes #1585.

What was wrong. On PG15 and PG16, after SET ROLE to a NOLOGIN role, three features failed because their background workers connected as that role, and those majors refuse it:

  • parallel_export_parquet failed worker 0 failed: worker exited without reporting a result;
  • parallel_copy failed coordinator exited without reporting a result (the issue said this was unverified; it is now measured);
  • the parallel flush (pgcolumnar.parallel_flush = on) silently encoded every column serially: 0 of 3 columns by 3 worker(s), 3 serial.

The fix. A worker connects as the session's authenticated user, then takes the caller's user id and security context before its first transaction. This is what core's ParallelWorkerMain does. All four worker entry points share columnar_worker_connect_as() in columnar.h. On 17+ the existing BGWORKER_BYPASS_ROLELOGINCHECK stays.

The part to review hardest: the user switch. A worker that connected as the authenticated user and stopped there would run the caller's COPY as, typically, a superuser. The suite catches that: a row trigger inside the copy worker records current_user, and under that mutant it reads postgres.

New suite nologin_workers and its twin (10 checks):

run result
this branch, shell and twin, each of PG15–19 10/10, 10/10
main, PG16, shell and twin 5 failed in each: export, copy, the copy's writer, the flush split, the refused-login count (got [6] want [1])
mutant noswitch (no SetUserIdAndSecContext), PG17, shell and twin 1 failed in each: parallel_copy's workers wrote every row as the NOLOGIN role: got [postgres] want [t_nl]

The log check has a positive control: a premise makes the role try to log in directly, and the log must then show exactly one refusal.

Gates over 9e812a1c:

  • ledger gate rc=0; census 3497 never + 1720 dated = 5217;
  • suites_not_covered: 150 → 150 (the suite is registered and covered in the same change);
  • cluster_tests: 1782 → 1792, by collection;
  • guards: 464, all passing;
  • compare_to_bash: every bash property covered;
  • TESTS.md: §217.

The build is clean, with no warnings in the touched files, on 15, 16, 17 and 19.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd The diagnosis is right and the PG15/16 fix works. I reproduced 10/10 green on head on PG16 and PG17. Requesting changes for two things I measured, plus a nit. Probe: your suite plus a few steps, in ~/pgc-tools/rv1612/ on my side (I can push the probe as a patch if that helps).

1. Blocker: a regression on PG17+. A copy-worker trigger that changes role now orphans the copy's prepared transactions.
A worker's session user is now the authenticated user, not the caller. So a trigger that runs SET ROLE/RESET ROLE (here set_config('role','none',true)) inside the copy worker becomes the authenticated user, and the worker PREPAREs as that user. The leader, still the caller, then can't finish it:

PG17, SET ROLE t_nl, parallel_copy into a table whose BEFORE trigger resets the role result pg_prepared_xacts after
main 9e812a1c 2000 rows, all who = t_nl 0
head 69841ce0 ERROR: pgcolumnar.parallel_copy failed: permission denied to finish prepared transaction 2, owner postgres, holding their locks: the next TRUNCATE nw_esc waited forever until I ran ROLLBACK PREPARED

PG16 head gives the same error and the same two orphans. On main it failed earlier, at the NOLOGIN connect.

Core avoids this because a parallel worker can't change role at all: SET is refused in parallel mode. Two possible fixes, and I'd take both:

  • (a) Before its PREPARE, the worker refuses unless GetUserId() and the sec context still equal hdr->roleid/hdr->secContext. That errors normally, so nothing is left prepared.
  • (b) On 17+, keep connecting as the caller with BGWORKER_BYPASS_ROLELOGINCHECK, as main does, and use authid only below 17, where it's needed. This also keeps session_user (and is_superuser, RESET ROLE's target) as main had them on 17+.

Please add a check for this. The probe's step fits: the trigger resets the role, then parallel_copy must load every row as the caller, and pg_prepared_xacts must be empty afterwards.

2. The sec_context half is load-bearing, and no check covers it.
Mutant nosec: SetUserIdAndSecContext(userid, 0) in columnar_worker_connect_as.

  • Suite: all 10 checks of nologin_workers stay green, in shell and twin alike.
  • What it lets through: parallel_copy called inside a SECURITY DEFINER function owned by t_nl, by a superuser session, into a table whose trigger resets the role. On head the reset is refused, as core refuses it in the same function (control: a plain INSERT there keeps who = t_nl), and all 2000 rows read t_nl with 0 orphans. Under nosec the reset succeeds and the trigger runs as postgres inside the worker. Here the copy then failed on finishing and left 2 orphans, but whatever the trigger did non-transactionally before the PREPARE ran as superuser.

This is the guard against "a superuser runs someone's definer function or a security-restricted operation", so it needs a check that reddens under nosec. The probe's definer step does: on PG16 head it's green, and under nosec it's red as got [] want [t_nl].

3. Nit (twin): the fixture's final readings, writers and flushed, are bare conn.execute(...). A failed query there raises out of the fixture and errors all 10 tests, none of them by name. Please read them through the same errors-as-values path as _as_nl (the #1522 rule).

Not blocking: session_user in a worker is now the authenticated user, not the leader's session user, so a trigger reading it differs from a serial load. Fix (b) settles it on 17+.

I'll re-measure both probes and the nosec mutant on 16 and 17 against your next head.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd force-pushed the fix/1585-nologin-export branch from 69841ce to fe77bec Compare October 21, 2026 18:18
@jdatcmd

jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD Thank you, both findings were real. Fixed at fe77bec6, restated onto f6846642, and measured on PG16 and PG17 in shell and twin.

1. The role-reset regression: your fix (b), with a change below 17, plus (a).

  • On 17+ the worker connects as the caller with BGWORKER_BYPASS_ROLELOGINCHECK, exactly as main did.
  • Below 17 it connects as the authenticated user and then calls SetSessionAuthorization(caller). So on every major the worker's session user is the caller, and a trigger's RESET ROLE lands on the caller. This also settles your session_user note: it now matches main on 17+, and is the same on 15/16.
  • (a): a copy loader refuses to PREPARE unless GetUserIdAndSecContext still equals the caller's. A trigger that switches to another role the caller belongs to is now refused before PREPARE, instead of orphaning a transaction.

2. nosec: the security context, and a crash it exposed. Your definer probe also crashed the coordinator on an assert-enabled build: TRAP: failed Assert("s->prevSecContext == 0") in StartTransaction. I'd set the context before the worker's first transaction. Core requires a top-level transaction to start with none, which is why ParallelWorkerMain applies it inside the transaction. The context is now taken inside the transaction, by the workers that run user code: the copy loader and the export worker.

Five new checks, in both twins:

check head 69841ce0 fix
a copy trigger that resets the role still writes every row as the NOLOGIN role got [0:] 2000:t_nl
and that copy leaves no prepared transaction behind got [2] 0
a copy trigger that switches to another role is refused before the worker prepares — refused
and the refused copy leaves no prepared transaction behind — 0
a copy run by a SECURITY DEFINER function keeps the definer's restriction in its workers got [] (the crash) 2000:t_nl

Mutants, on PG16 and PG17, with the same names in shell and twin:

mutant reds
nosec (yours: the context dropped) 1: the definer check
headconnect (the old head: connect as the authenticated user, no session user) 1: the role-reset check
noguard (the PREPARE guard removed) 2: both role-switch checks

"and that copy leaves no prepared transaction behind" stays never. Under headconnect the guard still refuses before PREPARE, so nothing is orphaned, which is the guard working.

3. The nit: the fixture's final readings now go through _one(), which returns query_error, so a failed read reds its check by name.

Runs and gates: 15/15 in shell and twin on each of PG15–19. The ledger gate rc=0, census 3633 never, cluster_tests 2105 → 2120 by collection, and guards 467.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approving fe77bec6. Both findings are fixed and I re-measured them. Thanks for catching the assert as well.

run (shell) result
nologin_workers at head, PG16 rc=0
nosec (the context dropped, now in columnar_worker_take_context), PG16 and PG17 1 red each: "a copy run by a SECURITY DEFINER function keeps the definer's restriction in its workers"
new probe: a non-superuser login role t_lg, a member of t_nl, after SET ROLE t_nl, PG16 and PG17 export 2000; copy 2000, every row who = t_nl; flush 3 of 3 columns; pg_prepared_xacts 0

The probe covers the case #1585 is really about: an ordinary user reaching a NOLOGIN group role, not postgres. The suite only runs it as a superuser. On 15/16 this is the SetSessionAuthorization path under a non-superuser authenticated user, and it works. That's because in 15.19 and 16.15 SetSessionAuthorization has no permission check of its own (the check sits in the GUC hook). I measured only those minors. If the support matrix reaches older 15/16 minors, check that SetSessionAuthorization there doesn't refuse a non-superuser authenticated user.

Non-blocking: a version of this probe as a check (the copy and the export as a non-superuser member) would guard that path against a future change. Mine is ~/pgc-tools/rv1612/nw_nonsuper.sh if you want it.

🤖 Generated with Claude Code

jdatcmd and others added 4 commits October 21, 2026 19:44
…nger vouches for the row set (#1341)

test/pytest/test_recluster_gate.py ports all 13 checks of
test/recluster_gate.sh by name. Two shell checks could pass with nothing
measured; both are fixed in both twins.

- "no-op recluster preserved the exact row set (columnar == heap mirror)"
  compared rg_t with a heap copied from rg_t, so it agreed with whatever
  the reclusters did: with a row corrupted before the copy, it passed. The
  mirror is now built from the generator (g = 1..20000, then 1..5000
  again from the append).
- "and it leaves the physical layout byte-identical" agreed on two failed
  layout reads; it now needs both.

Measured, PG17, shell and twin:
- a row corrupted before the mirror is built: the row-set check red in
  both; the old shell passed 13 of 13.
- nogate (the #415 gate removed): 4 red. noextent (appended groups
  ignored) 2, nokey (the key ignored) 2. Each red set is identical in both
  twins.
- clean: 13 of 13 on PG15 to PG19 in both twins.

Ledger: 13 rows, 8 dated by the three mutations and 5 never.
suites_not_covered 125 -> 124, cluster_tests 2169 -> 2182, TESTS.md
section 243. Census 3650 -> 3651 never, 1930 -> 1942 dated, 5593 rows. The ledger
gate and the guards pass over 9bd1850.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nothing no longer passes as bounded (#1341)

test/pytest/test_recluster_extent.py ports all 17 checks of
test/recluster_extent.sh by name, with the races run in threads as the
shell runs them in background jobs. Three shell fixes, in both twins:

- "the run does not claim rows written during the rewrite" read
  sorted_rows <= base, which a run that recorded nothing meets: with no
  extent recorded at all (norun), it passed. It now needs at least one
  sorted row, and carries its counts.
- The race check compared two client clocks, but the rewrite ends 3-4 ms
  after the insert commits (measured twice), so the order flipped between
  the twins. It now asks the server, just before the insert, whether the
  recluster is running, and is renamed "the insert ran while the rewrite
  was still running" (the suite had no ledger rows). Both checks leave
  lossy_arms.tsv.
- The projection loop's names were built from $arm; they are literal.

Measured, PG17, shell and twin (mutations of record_online_sorted_extent):
- norun (no extent recorded): 9 red; the old shell reddened 8 and passed
  the claim check.
- noconsec (the run spans ids another writer took) 1, runall (the run
  claims every group) 4, noproj (a projection's stripe ids not merged) 2.
  Each red set is identical in both twins.
- clean: 17 of 17 on PG15 to PG19 in both twins, and twice more on PG17.

Ledger: 17 rows, 10 dated by the four mutations and 7 never.
suites_not_covered 124 -> 123, cluster_tests 2182 -> 2200, TESTS.md
section 244. Census 3651 -> 3658 never, 1942 -> 1953 dated, 5611 rows. The ledger
gate and the guards pass over 1f03f62.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ors no longer vouch for the reader (#1341)

test/pytest/test_reader_buffer_reuse.py ports all 22 checks of
test/reader_buffer_reuse.sh by name. Three shell fixes, in both twins:

- Both heap mirrors (rb_h, rbd_h) were copied from the tables under test,
  so every differential check agreed with whatever the reader returned:
  with a row corrupted before the copy, the shell passed 22 of 22. Both
  are now built from the generators, and the three full-scan comparisons
  refuse an empty side (check_text), as does the all-groups premise.
- "premise: the differential fixture spans several row groups" and the
  fault-ratio check carry their numbers, so their rows leave
  lossy_arms.tsv.
- The projection loop's eight names were built from $proj; they are
  literal.

The twin pins max_parallel_workers_per_gather = 0, as the shell's
cluster does: left at its default, a Gather split the groups and the
leader's count read 2 of 4.

Measured, PG17, shell and twin:
- a row corrupted before the mirror is built: the full-scan and
  short-group checks red in both; the old shell passed 22 of 22.
- noreuse (a fresh group buffer per row group): 1 red, the fault ratio.
- staleproj (a projected read skips column 1's bytes and decodes the
  stale buffer): 8 red in the shell and the same names in the twin, but
  it also reddens three premises, so it dates nothing.
- clean: 22 of 22 on PG15 to PG19 in both twins.

Ledger: 22 rows, 1 dated and 21 never.
suites_not_covered 123 -> 122, cluster_tests 2200 -> 2226, TESTS.md
section 245. Census 3658 -> 3666 never, 1953 -> 1971 dated, 5637 rows. The ledger
gate and the guards pass over 1ccaf4e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

parallel_export_parquet, parallel_copy and the parallel flush each start
background workers that connected as the calling role. PostgreSQL 15 and
16 refuse a background worker connection for a role without LOGIN; only
17 added BGWORKER_BYPASS_ROLELOGINCHECK. So after SET ROLE to a NOLOGIN
role, the export failed "worker 0 failed: worker exited without reporting
a result", parallel_copy failed "coordinator exited without reporting a
result", and the parallel flush encoded every column serially.

The worker's session user is now the caller on every major. On 17 and
later it connects as the caller with the login check bypassed, as before.
Below 17 it connects as the session's authenticated user, which logged
in, and then makes the caller its session user. The caller's security
context is taken inside the worker's transaction, as core's
ParallelWorkerMain does, by the workers that run user code (the copy
loader and the export worker). And a copy loader refuses to PREPARE as
any role but the caller's.

From the review (OffgridwithJD): connecting as the authenticated user and
stopping there made it the session user, so a copy trigger's RESET ROLE
became that user, the loader PREPAREd as it, and the coordinator could
not finish it, leaving prepared transactions holding locks; and without
the security context a SECURITY DEFINER copy's trigger could switch to
the session user. Setting the context before the first transaction
tripped StartTransaction's assert on an assert-enabled build.

nologin_workers and its twin, 15 checks: the export, the copy and the
flush as a NOLOGIN role; a copy trigger that resets the role still writes
as the caller and leaves nothing prepared; one that switches to another
role is refused before PREPARE and leaves nothing prepared; and a copy
run by a SECURITY DEFINER function keeps its restriction in the workers.

15/15 shell and twin on each of PG15-19. 9 checks are dated by five
mutations, with the same red sets in both harnesses: the pre-fix connect;
no caller user id; no session user (the old head); no security context;
and no PREPARE guard. Census 3492 -> 3498 never, 1715 -> 1724 dated, 5222
rows; suites_not_covered 150 -> 150; cluster_tests 1782 -> 1797; TESTS.md
section 217. The ledger gate, guards (464), name grading and the
self-test pass over 9e812a1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the fix/1585-nologin-export branch from fe77bec to 5370cf9 Compare October 21, 2026 19:52

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Re-approving the stacked head 5370cf91 (I approved fe77bec6).

  • Content unchanged: compared file by file against fe77bec6, the PR's own +/- lines match. That covers every file except the budget, expected_tests and TESTS.md, which only restate counts; check_ledger.tsv and lossy_arms.tsv are identical.
  • Gates on the stack: checked at 1f03f624, 82917995, 44d31130 and the tip 5ae35396. Guards 467 passed, collection equals cluster_tests (2182 / 2226 / 2311 / 2431), and the stated census equals the tsv's never count (3651 / 3666 / 3713 / 3787).

🤖 Generated with Claude Code

@jdatcmd
jdatcmd merged commit d39532c into main Oct 21, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parallel_export_parquet fails opaquely on PG15/16 when the current role is NOLOGIN

2 participants