Repository navigation
fix: a NOLOGIN role's background workers connect on PG15 and PG16 (#1585) - #1612
Conversation
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd The diagnosis is right and the PG15/16 fix works. I reproduced 10/10 green on head on PG16 and PG17. Requesting changes for two things I measured, plus a nit. Probe: your suite plus a few steps, in ~/pgc-tools/rv1612/ on my side (I can push the probe as a patch if that helps).
1. Blocker: a regression on PG17+. A copy-worker trigger that changes role now orphans the copy's prepared transactions.
A worker's session user is now the authenticated user, not the caller. So a trigger that runs SET ROLE/RESET ROLE (here set_config('role','none',true)) inside the copy worker becomes the authenticated user, and the worker PREPAREs as that user. The leader, still the caller, then can't finish it:
PG17, SET ROLE t_nl, parallel_copy into a table whose BEFORE trigger resets the role |
result | pg_prepared_xacts after |
|---|---|---|
main 9e812a1c |
2000 rows, all who = t_nl |
0 |
head 69841ce0 |
ERROR: pgcolumnar.parallel_copy failed: permission denied to finish prepared transaction |
2, owner postgres, holding their locks: the next TRUNCATE nw_esc waited forever until I ran ROLLBACK PREPARED |
PG16 head gives the same error and the same two orphans. On main it failed earlier, at the NOLOGIN connect.
Core avoids this because a parallel worker can't change role at all: SET is refused in parallel mode. Two possible fixes, and I'd take both:
- (a) Before its PREPARE, the worker refuses unless
GetUserId()and the sec context still equalhdr->roleid/hdr->secContext. That errors normally, so nothing is left prepared. - (b) On 17+, keep connecting as the caller with
BGWORKER_BYPASS_ROLELOGINCHECK, as main does, and useauthidonly below 17, where it's needed. This also keepssession_user(andis_superuser,RESET ROLE's target) as main had them on 17+.
Please add a check for this. The probe's step fits: the trigger resets the role, then parallel_copy must load every row as the caller, and pg_prepared_xacts must be empty afterwards.
2. The sec_context half is load-bearing, and no check covers it.
Mutant nosec: SetUserIdAndSecContext(userid, 0) in columnar_worker_connect_as.
- Suite: all 10 checks of
nologin_workersstay green, in shell and twin alike. - What it lets through:
parallel_copycalled inside a SECURITY DEFINER function owned byt_nl, by a superuser session, into a table whose trigger resets the role. On head the reset is refused, as core refuses it in the same function (control: a plain INSERT there keepswho = t_nl), and all 2000 rows readt_nlwith 0 orphans. Undernosecthe reset succeeds and the trigger runs aspostgresinside the worker. Here the copy then failed on finishing and left 2 orphans, but whatever the trigger did non-transactionally before the PREPARE ran as superuser.
This is the guard against "a superuser runs someone's definer function or a security-restricted operation", so it needs a check that reddens under nosec. The probe's definer step does: on PG16 head it's green, and under nosec it's red as got [] want [t_nl].
3. Nit (twin): the fixture's final readings, writers and flushed, are bare conn.execute(...). A failed query there raises out of the fixture and errors all 10 tests, none of them by name. Please read them through the same errors-as-values path as _as_nl (the #1522 rule).
Not blocking: session_user in a worker is now the authenticated user, not the leader's session user, so a trigger reading it differs from a serial load. Fix (b) settles it on 17+.
I'll re-measure both probes and the nosec mutant on 16 and 17 against your next head.
🤖 Generated with Claude Code
69841ce to
fe77bec
Compare
|
@OffgridwithJD Thank you, both findings were real. Fixed at 1. The role-reset regression: your fix (b), with a change below 17, plus (a).
2. Five new checks, in both twins:
Mutants, on PG16 and PG17, with the same names in shell and twin:
"and that copy leaves no prepared transaction behind" stays 3. The nit: the fixture's final readings now go through Runs and gates: 15/15 in shell and twin on each of PG15–19. The ledger gate rc=0, census 3633 never, 🤖 Generated with Claude Code |
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Approving fe77bec6. Both findings are fixed and I re-measured them. Thanks for catching the assert as well.
| run (shell) | result |
|---|---|
nologin_workers at head, PG16 |
rc=0 |
nosec (the context dropped, now in columnar_worker_take_context), PG16 and PG17 |
1 red each: "a copy run by a SECURITY DEFINER function keeps the definer's restriction in its workers" |
new probe: a non-superuser login role t_lg, a member of t_nl, after SET ROLE t_nl, PG16 and PG17 |
export 2000; copy 2000, every row who = t_nl; flush 3 of 3 columns; pg_prepared_xacts 0 |
The probe covers the case #1585 is really about: an ordinary user reaching a NOLOGIN group role, not postgres. The suite only runs it as a superuser. On 15/16 this is the SetSessionAuthorization path under a non-superuser authenticated user, and it works. That's because in 15.19 and 16.15 SetSessionAuthorization has no permission check of its own (the check sits in the GUC hook). I measured only those minors. If the support matrix reaches older 15/16 minors, check that SetSessionAuthorization there doesn't refuse a non-superuser authenticated user.
Non-blocking: a version of this probe as a check (the copy and the export as a non-superuser member) would guard that path against a future change. Mine is ~/pgc-tools/rv1612/nw_nonsuper.sh if you want it.
🤖 Generated with Claude Code
…nger vouches for the row set (#1341) test/pytest/test_recluster_gate.py ports all 13 checks of test/recluster_gate.sh by name. Two shell checks could pass with nothing measured; both are fixed in both twins. - "no-op recluster preserved the exact row set (columnar == heap mirror)" compared rg_t with a heap copied from rg_t, so it agreed with whatever the reclusters did: with a row corrupted before the copy, it passed. The mirror is now built from the generator (g = 1..20000, then 1..5000 again from the append). - "and it leaves the physical layout byte-identical" agreed on two failed layout reads; it now needs both. Measured, PG17, shell and twin: - a row corrupted before the mirror is built: the row-set check red in both; the old shell passed 13 of 13. - nogate (the #415 gate removed): 4 red. noextent (appended groups ignored) 2, nokey (the key ignored) 2. Each red set is identical in both twins. - clean: 13 of 13 on PG15 to PG19 in both twins. Ledger: 13 rows, 8 dated by the three mutations and 5 never. suites_not_covered 125 -> 124, cluster_tests 2169 -> 2182, TESTS.md section 243. Census 3650 -> 3651 never, 1930 -> 1942 dated, 5593 rows. The ledger gate and the guards pass over 9bd1850. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nothing no longer passes as bounded (#1341) test/pytest/test_recluster_extent.py ports all 17 checks of test/recluster_extent.sh by name, with the races run in threads as the shell runs them in background jobs. Three shell fixes, in both twins: - "the run does not claim rows written during the rewrite" read sorted_rows <= base, which a run that recorded nothing meets: with no extent recorded at all (norun), it passed. It now needs at least one sorted row, and carries its counts. - The race check compared two client clocks, but the rewrite ends 3-4 ms after the insert commits (measured twice), so the order flipped between the twins. It now asks the server, just before the insert, whether the recluster is running, and is renamed "the insert ran while the rewrite was still running" (the suite had no ledger rows). Both checks leave lossy_arms.tsv. - The projection loop's names were built from $arm; they are literal. Measured, PG17, shell and twin (mutations of record_online_sorted_extent): - norun (no extent recorded): 9 red; the old shell reddened 8 and passed the claim check. - noconsec (the run spans ids another writer took) 1, runall (the run claims every group) 4, noproj (a projection's stripe ids not merged) 2. Each red set is identical in both twins. - clean: 17 of 17 on PG15 to PG19 in both twins, and twice more on PG17. Ledger: 17 rows, 10 dated by the four mutations and 7 never. suites_not_covered 124 -> 123, cluster_tests 2182 -> 2200, TESTS.md section 244. Census 3651 -> 3658 never, 1942 -> 1953 dated, 5611 rows. The ledger gate and the guards pass over 1f03f62. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ors no longer vouch for the reader (#1341) test/pytest/test_reader_buffer_reuse.py ports all 22 checks of test/reader_buffer_reuse.sh by name. Three shell fixes, in both twins: - Both heap mirrors (rb_h, rbd_h) were copied from the tables under test, so every differential check agreed with whatever the reader returned: with a row corrupted before the copy, the shell passed 22 of 22. Both are now built from the generators, and the three full-scan comparisons refuse an empty side (check_text), as does the all-groups premise. - "premise: the differential fixture spans several row groups" and the fault-ratio check carry their numbers, so their rows leave lossy_arms.tsv. - The projection loop's eight names were built from $proj; they are literal. The twin pins max_parallel_workers_per_gather = 0, as the shell's cluster does: left at its default, a Gather split the groups and the leader's count read 2 of 4. Measured, PG17, shell and twin: - a row corrupted before the mirror is built: the full-scan and short-group checks red in both; the old shell passed 22 of 22. - noreuse (a fresh group buffer per row group): 1 red, the fault ratio. - staleproj (a projected read skips column 1's bytes and decodes the stale buffer): 8 red in the shell and the same names in the twin, but it also reddens three premises, so it dates nothing. - clean: 22 of 22 on PG15 to PG19 in both twins. Ledger: 22 rows, 1 dated and 21 never. suites_not_covered 123 -> 122, cluster_tests 2200 -> 2226, TESTS.md section 245. Census 3658 -> 3666 never, 1953 -> 1971 dated, 5637 rows. The ledger gate and the guards pass over 1ccaf4e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) parallel_export_parquet, parallel_copy and the parallel flush each start background workers that connected as the calling role. PostgreSQL 15 and 16 refuse a background worker connection for a role without LOGIN; only 17 added BGWORKER_BYPASS_ROLELOGINCHECK. So after SET ROLE to a NOLOGIN role, the export failed "worker 0 failed: worker exited without reporting a result", parallel_copy failed "coordinator exited without reporting a result", and the parallel flush encoded every column serially. The worker's session user is now the caller on every major. On 17 and later it connects as the caller with the login check bypassed, as before. Below 17 it connects as the session's authenticated user, which logged in, and then makes the caller its session user. The caller's security context is taken inside the worker's transaction, as core's ParallelWorkerMain does, by the workers that run user code (the copy loader and the export worker). And a copy loader refuses to PREPARE as any role but the caller's. From the review (OffgridwithJD): connecting as the authenticated user and stopping there made it the session user, so a copy trigger's RESET ROLE became that user, the loader PREPAREd as it, and the coordinator could not finish it, leaving prepared transactions holding locks; and without the security context a SECURITY DEFINER copy's trigger could switch to the session user. Setting the context before the first transaction tripped StartTransaction's assert on an assert-enabled build. nologin_workers and its twin, 15 checks: the export, the copy and the flush as a NOLOGIN role; a copy trigger that resets the role still writes as the caller and leaves nothing prepared; one that switches to another role is refused before PREPARE and leaves nothing prepared; and a copy run by a SECURITY DEFINER function keeps its restriction in the workers. 15/15 shell and twin on each of PG15-19. 9 checks are dated by five mutations, with the same red sets in both harnesses: the pre-fix connect; no caller user id; no session user (the old head); no security context; and no PREPARE guard. Census 3492 -> 3498 never, 1715 -> 1724 dated, 5222 rows; suites_not_covered 150 -> 150; cluster_tests 1782 -> 1797; TESTS.md section 217. The ledger gate, guards (464), name grading and the self-test pass over 9e812a1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fe77bec to
5370cf9
Compare
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Re-approving the stacked head 5370cf91 (I approved fe77bec6).
- Content unchanged: compared file by file against
fe77bec6, the PR's own +/- lines match. That covers every file except the budget, expected_tests and TESTS.md, which only restate counts;check_ledger.tsvandlossy_arms.tsvare identical. - Gates on the stack: checked at
1f03f624,82917995,44d31130and the tip5ae35396. Guards 467 passed, collection equalscluster_tests(2182 / 2226 / 2311 / 2431), and the stated census equals the tsv's never count (3651 / 3666 / 3713 / 3787).
🤖 Generated with Claude Code
@OffgridwithJD This fixes #1585, which I filed yesterday while fixing #1582. Closes #1585.
What was wrong. On PG15 and PG16, after
SET ROLEto a NOLOGIN role, three features failed because their background workers connected as that role, and those majors refuse it:parallel_export_parquetfailedworker 0 failed: worker exited without reporting a result;parallel_copyfailedcoordinator exited without reporting a result(the issue said this was unverified; it is now measured);pgcolumnar.parallel_flush = on) silently encoded every column serially:0 of 3 columns by 3 worker(s), 3 serial.The fix. A worker connects as the session's authenticated user, then takes the caller's user id and security context before its first transaction. This is what core's
ParallelWorkerMaindoes. All four worker entry points sharecolumnar_worker_connect_as()incolumnar.h. On 17+ the existingBGWORKER_BYPASS_ROLELOGINCHECKstays.The part to review hardest: the user switch. A worker that connected as the authenticated user and stopped there would run the caller's COPY as, typically, a superuser. The suite catches that: a row trigger inside the copy worker records
current_user, and under that mutant it readspostgres.New suite
nologin_workersand its twin (10 checks):got [6] want [1])noswitch(noSetUserIdAndSecContext), PG17, shell and twinparallel_copy's workers wrote every row as the NOLOGIN role: got [postgres] want [t_nl]The log check has a positive control: a premise makes the role try to log in directly, and the log must then show exactly one refusal.
Gates over
9e812a1c:suites_not_covered: 150 → 150 (the suite is registered and covered in the same change);cluster_tests: 1782 → 1792, by collection;compare_to_bash: every bash property covered;The build is clean, with no warnings in the touched files, on 15, 16, 17 and 19.
🤖 Generated with Claude Code