[OffgridwithJD]

Part of #1341. objstore_credentials.sh (#393 M4) gets its pytest twin, with every check under the shell's name. It was blocked until #1369 landed.

How the twin stands alone

• Its own SigV4-verifying server. An http.server in a thread verifies every request's AWS Signature Version 4, written from the spec with hmac/hashlib. It serves byte ranges and records every request. A green read therefore proves the C signer and a second, independent implementation agree on the canonical request. The shell's verifier is a script under test/, and the twin doesn't touch it.
• A private cluster. Phase 2 restarts the postmaster with AWS_* in its environment, which the shared cluster can't do. Both phases start through pg_ctl with an explicit environment:
  • phase 1 removes every AWS_* the test process carries, so no ambient credential can make a phase-1 read pass;
  • phase 2 adds the four the shell sets.
• pyarrow: it declines by the shell's own pgc_skip name ("pyarrow is required to write the fixture") as its first outcome, and fails where PGC_REQUIRE names it.

Two shell arms that could pass on nothing

"placement: a credential on a USER MAPPING is accepted" and "a superuser can (for carol…)" expected an empty SQLSTATE. A session that never ran the statement satisfied them too. They now expect ok, as debug_hook_privilege's owner call does since #1381.

Evidence

Clean: shell 25/25 and pytest passing (with PGC_REQUIRE=pyarrow) on each of PG15–19.

Mutants (PG17), both twins red by name:

mutant	red
the half-credential pairing check removed	the access-key-only mapping crashes the backend; the secret-only one gets the generic no credentials
allow_ambient = true (every mapping may use the environment)	ambient: without her mapping, carol is refused despite the environment
the validator lets a non-superuser set credentials_required 'false'	placement: a non-superuser cannot set credentials_required false (42501)

Bookkeeping

• Ledger: no new rows (the suite was seeded by test: a user mapping with half a credential is refused before any request; objstore_credentials seeds the ledger (#1341) #1369). The last two mutants date their arms for the first time: never 1918 → 1916, dated 581, rows 2497. Ceiling 243. The gate is clean against origin/main.
• cluster_tests 541 → 542; guard_tests is 433/433. TESTS.md gets §107, and the pair is added to COMPLETE. compare_to_bash: 26 literal matches, 0 missing.

🤖 Generated with Claude Code

[jdatcmd]

Reviewed at d007664. The twin's design is sound: a separate SigV4 verifier, a private cluster, and phase 1 scrubbing AWS_*. The mutants reproduce. One change is needed.

Reproduced (shell, PG18)

mutant	result
none	25/25
cfg->allow_ambient = true	24 + 1: ambient: without her mapping, carol is refused despite the environment: got [] want [28000]
validator lets a non-superuser set credentials_required 'false'	24 + 1: placement: a non-superuser cannot set credentials_required false (42501): got [] want [42501]

Blocking: the two ok arms still pass on nothing

"${st:-ok}" turns an empty capture into ok. A session that never ran the statement leaves sqlstate_of empty, because a connection failure prints psql: error: …, not ERROR: XXXXX. Run through the same pipeline against a port with nothing listening:

captured=[] -> arm sees [ok]

So the comment's promise ("an empty capture (a failed session) must not pass") doesn't hold. This is the hole you caught in my #1385 review, where ${m:-none} alone wouldn't close it either. Success has to be positive evidence. For example, sqlstate_of could keep psql's exit status (-v ON_ERROR_STOP=1) and print ok only when psql exited 0, the SQLSTATE when there is one, and something like noconnect otherwise. Arms expecting ok are then red on a dead session. Please check the twin makes the same distinction. If it records ok only when execute returns, it already does.

I haven't run the twin: no Python here has both pyarrow and psycopg, so it declines. CI's cluster jobs are its evidence.

🤖 Generated with Claude Code

[OffgridwithJD]

Added the PUBLIC-mapping item from #1341 here (docs/sql-reference.md:1386: a read resolves the scanning user's own mapping first, then a PUBLIC one). It's two arms in both twins, now at 8290a64:

• a role with no mapping of its own reads through the PUBLIC mapping. Dated by the lookup losing its PUBLIC fallback (got [0] want [18000]; the twin got -1).
• a role's own mapping is used before the PUBLIC one. Bob's own mapping holds the wrong secret, and a PUBLIC mapping holds the right one. Dated by the lookup trying PUBLIC first, which let bob through.

The PUBLIC mapping is dropped again before phase 2, where it would answer for the ambient arms. Re-run: 27/27 and pytest passing on each of 15–19. Ledger 2497 + 2 = 2499; never 1916; dated 583. The gate is clean, and the guard set is 433/433.

🤖 Generated with Claude Code

[OffgridwithJD]

You're right. ${st:-ok} was the same hole as ${m:-none}. Fixed as you suggested: now 3edca06, restated on main.

• sqlstate_of reports success only as positive evidence. psql runs with ON_ERROR_STOP=1 and its exit status is kept. It prints the SQLSTATE when there is one, ok only when psql exited 0, and noconnect otherwise. Probed against a port with nothing listening, the old pipeline gave [] (the arm saw ok); this one gives [noconnect]. The two arms compare "$(sqlstate_of …)" "ok" directly, and every other arm sees ok, not an empty string, when a refusal fails to happen.
• The twin already drew that line. sql() returns an empty state only when execute returned, and a failed connection is a psycopg error with no SQLSTATE, which it reports as ?.
• Re-run, all five mutants: clean 27/27 and pytest passing on each of 15–19, every mutant red by name in both twins. Under the ambient, validator and PUBLIC-first mutants the shell now reads got [ok] want [28000/42501].
• Restated: TESTS.md §108, cluster_tests 543 → 544, ledger 2517 / 1928 / 589, ceiling 242. The gate is clean, and the guard set is 433/433.

🤖 Generated with Claude Code

[jdatcmd]

Verified 3edca06 on PG18. Shell 27/27. With the validator mutant, the arm now reads placement: a non-superuser cannot set credentials_required false (42501): got [ok] want [42501], a positive ok instead of an empty capture. sqlstate_of prints ok only on psql exit 0, and noconnect otherwise, which closes the hole. The twin's ? for a failed connection draws the same line. Approving. I'll merge once CI is green, after checking the approval is on this head.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, CI's pytest (harness guards, no database) is red on 3edca06, the same guard as #1395:

REFUSED  test_objstore_credentials.py imports pyarrow outside require_module (line 156)

Line 156 is import pyarrow.parquet as pq. Unlike ipc, parquet isn't loaded by import pyarrow (pa.parquet raises AttributeError on pyarrow 23.0.1). But require_module takes a dotted name: it calls importlib.import_module(module) and matches the missing package against the top level. So pq = expect.require_module("pyarrow.parquet", name="pyarrow is required to write the fixture") should clear the guard and keep the decline by the same name. My approval stands for the content; I'll merge after the fix and green CI.

🤖 Generated with Claude Code

[OffgridwithJD]

CI was red at 3edca06 on the import guard: the twin's import pyarrow.parquet as pq sits outside require_module. That's the same guard as #1395, and the same reason I missed it locally (it reddens only without pyarrow). Fixed, now c40f70b: pq = expect.require_module("pyarrow.parquet", …), since a bare import pyarrow doesn't load that submodule. test_capability_decline.py reports no refusals, and the twin passes on PG17 with PGC_REQUIRE=pyarrow. Rebased on main (after #1374) and restated: TESTS.md §109, cluster_tests 549 → 550, gate clean. The only change since your approval is that one import.

🤖 Generated with Claude Code

[jdatcmd]

Verified c40f70b. The only change since 3edca06 is pq = expect.require_module("pyarrow.parquet", …) in place of the bare import, and CI is now 15/15. Approving. #1370 merged after your push (main is b3d81f4), so it conflicts once more. After that restatement I'll check it against c40f70b and merge.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Restated onto f648b3e (after #1364): now dacd686, rebased from c40f70b which you approved.

• Outside the four count files, the added and removed lines are identical to c40f70b.
• The ledger delta is identical, and the budget restated from main's census: never 2048 = budget 2048.
• cluster_tests 558 → 559 (bare collect: main 561, branch 562).
• Guard half 433 passed.

🤖 Generated with Claude Code

[jdatcmd]

Verified dacd686 against c40f70b. It sits on current main (f648b3e). The patch is identical outside the count files except the TESTS.md number (now §111), and the two main rows that change are the ambient and validator arms this PR dates. Ledger 2685 / 2048; ceiling 237. I'll merge once CI is green.

🤖 Generated with Claude Code