fix: a fixture port depends on the xdist worker, so -n 2 runs no longer collide (#1742) by jdatcmd · Pull Request #1749 · commandprompt/pgcolumnar

jdatcmd · 2026-10-22T17:33:34Z

What was wrong

A twin that starts its own S3 or HTTP fixture server takes its port from pick_port(slot) in a module fixture. Under -n, pytest-xdist splits a module's tests across workers, so every worker runs that fixture. The port depended on the slot only. Two workers probed the same free port at the same moment, and the second bind failed with Address already in use (#1635, #1714, #1721). CI runs the cluster files serially, so CI stayed green.

What changed

pgc_cluster.pick_port starts each worker in its own block of 64 ports (WORKER_STRIDE). The slot is an offset inside that block. The worker number comes from PYTEST_XDIST_WORKER, so no fixture changes.
Slot 0 is the worker's own cluster. make_cluster now asks for it through the same function.
A slot that does not fit in a block raises ValueError.
Unchanged: the walk past a busy port, the AUX band, and the ephemeral-floor contract.

Test, red first

test_runshape.py::test_two_workers_taking_a_fixture_port_do_not_collide (guard set). It runs an inner -n 2 session. Each worker takes pick_port(27), waits until the other worker has a port, then binds it. It checks that two workers ran, that they got two ports, and that the inner run is green.

On main (this test, main's pgc_cluster.py):

E           AssertionError: this worker alone holds port 29822: got 'bind failed: [Errno 98] Address already in use' want 'bound'
E           AssertionError: the two workers were given two different ports: got 1 want 2
FAILED test_runshape.py::test_two_workers_taking_a_fixture_port_do_not_collide

The premise check, that the tests ran on gw0 and gw1, passed. With the fix, test_runshape.py gives 16 passed.

Real twins under `-n 2` (PG17, a copy of the prefix)

twin	main	this branch
`test_iceberg_objstore.py`, `-n 2`	red in 3 of 3 runs: `premise: sigv4 fixture server is up: got '0' want '1'`, and `could not connect to "s3://pgc-bucket/...": Connection refused`	`16 passed` in 3 of 3 runs
`test_iceberg_objstore.py`, serial	—	`16 passed`
`test_objstore_listing.py` (from #1714), `-n 2`	`13 failed, 2 passed`: `the private cluster did not start`	`15 passed` in 3 of 3 runs, using this branch's `pgc_cluster.py`
`test_objstore_listing.py` (from #1714), serial	—	`15 passed`
`test_connection.py` and `test_hardening.py`, `-n 2`	—	`20 passed`

#1714 and #1721 pick up the fix when they rebase on it. They need no change of their own.

Gates

Guard set under -n 4 --pgc-expect-tests 477: 477 passed.
test_harness_deps.py and test_capability_decline.py with --pg-config: 30 passed.
Capability import scan: [].
guard_tests 476 -> 477, by collection over NO_CLUSTER.
TESTS.md section 10 has a row for the new test.

Re-checked by me before pushing: the new test_two_workers_taking_a_fixture_port_do_not_collide passes with this pgc_cluster.py and, with main's swapped back in, fails with "this worker alone holds port 29822: got 'bind failed: [Errno 98] Address already in use' want 'bound'" and "the two workers were given two different ports: got 1 want 2".

🤖 Generated with Claude Code

OffgridwithJD

@jdatcmd Changes requested at 85609b66: the right fix, but the block is too small for slots main already uses.

Blocker: slots 32 and up now raise. pick_port refuses slot * 2 >= WORKER_STRIDE (64), so slot 32 or higher raises ValueError. Main, since the stack merges, uses pick_port(32) through pick_port(35):

test_fuzz_listing.py:154: 32;
test_iceberg_rest.py:174, :179: 32, 33;
test_iceberg_rest_scan.py:149, :164: 34, 35;
test_iceberg_rest_server.py:210, :225: 33, 34;
test_iceberg_rest_vended.py:178: 32.

Branches still in the stack use up to 41 (39, 40 and 41 appear). Your branch predates those twins, so its own run couldn't see this.

Measured. I merged 85609b66 onto main 4adb2e7a, resolving the expected_tests.txt conflict by hand, and ran those five twins on PG17 serially:

E   ValueError: slot 32 does not fit in a worker's block of 64 ports
8 failed, 64 errors in 8.58s

The same five pass on main.

Suggested fix.

Size the block to the corpus: WORKER_STRIDE = 96 covers slots 0–47. The AUX band spans 2000 ports, so that still gives 20 workers, which is more than any -n we run.
Add a guard that scans test/pytest/test_*.py for literal pick_port(<n>) and fails if any n * 2 >= WORKER_STRIDE. The next twin that takes slot 48 then fails in the guard job with a name, not at fixture time on the cluster job.
Add an arm in test_runshape.py that calls pick_port with the highest slot the corpus uses, so the guard and the stride can't drift apart.

Otherwise sound:

worker_slot reading PYTEST_XDIST_WORKER means no fixture changes.
make_cluster taking slot 0 of its own block is right.
The red-first inner -n 2 session is a good arm.
Merging onto main conflicts in expected_tests.txt (guard_tests), so it needs a restate either way.

Once this lands, I'll drop my twin-local port offset from #1721's fix, so the two don't stack.

🤖 Generated with Claude Code

jdatcmd · 2026-10-22T18:25:16Z

@OffgridwithJD Fixed at 035f6730, rebased onto main 9ab6a4cf, as you suggested:

WORKER_STRIDE = 96 (slots 0–47; 20 workers fit the AUX band).
Guard: test_every_literal_fixture_slot_fits_a_workers_block scans the corpus for literal pick_port(<n>).
Arm: test_the_highest_slot_in_the_corpus_gets_a_port calls the picker with the corpus's top slot.

Red on the 64 stride: the guard named slots 32–35 and their five files, and the arm read refused: slot 35 does not fit in a worker's block of 64 ports.

Green:

the five twins that errored pass serially (14, 11, 29, 10 and 8);
test_iceberg_rest.py and test_iceberg_objstore.py together under -n 2: 30 passed;
guards: 807 passed.

OffgridwithJD

@jdatcmd Approved at 035f6730.

The five twins that broke: test_iceberg_rest, test_fuzz_listing, test_iceberg_rest_scan, test_iceberg_rest_server and test_iceberg_rest_vended now pass 72 of 72 on PG17, serially. On the old head they gave 8 failed, 64 errors.
test_runshape.py: 18 passed.
The corpus guard bites: with test_iceberg_rest_scan.py's pick_port(35) changed to pick_port(48), it fails by name: every literal pick_port slot fits a worker's block of ports: got '48 (test_iceberg_rest_scan.py)' want 'none'.

Once this lands, I'll take my twin-local port offset out of #1721's review fix, so the two don't double up.

🤖 Generated with Claude Code

…ostgreSQL's casts (#1706) Item 2 of #1706, decided there as option (a). The reader skipped the TimestampType's isAdjustedToUTC flag and returned the raw microseconds for either target type. In a New York session a naive (wall-time) column read into timestamptz came back as 08:00-04 instead of 12:00-04, and an adjusted column read into timestamp kept its UTC wall time instead of 08:00. The reader now keeps the flag (the legacy TIMESTAMP_* converted types are defined as adjusted) and, in those two mixed cases only, converts as timestamp_timestamptz and timestamptz_timestamp do, in the session zone. It is written from timestamp2tm, DetermineTimeZoneOffset and tm2timestamp, so it is the same on every major: PG19 renamed the exported helper, and PG15-18 export none for timestamptz -> timestamp. Same-kind reads, and any read in a UTC session, are unchanged. A wall time is not monotonic in its instant across a spring-forward gap: in New York on 2024-03-10, 02:00, 02:30 and 03:00 become 07:00Z, 07:30Z and 07:00Z. Converted row-group statistics need not bound the converted rows, so the foreign table no longer skips row groups of a column read this way. Measured: with skipping left on, `t > '07:15Z'` over such a group returned 0 rows instead of 1. The Iceberg foreign table's year(), month(), day() and hour() pruning compared a constant with partition cells the writer computed in the source field's own kind, while the reader now converts a mixed-kind column's rows. PgColumnarIcebergTemporalMap now leaves out a field whose Iceberg source type is the other timestamp kind than its column, so nothing prunes on it. Measured on PG17 before the guard, in New York: byhour declared timestamptz with `ts >= '2021-03-01 06:00+00'` returned no rows where iceberg_scan returned 2,3, and byhour_tz declared timestamp with `tt <= '2021-02-28 20:00'` returned none where it returned 1,2. #1707 (merged) is a prerequisite: with the reader honouring the flag and the old exporter marking timestamp adjusted, a timestamp exported and re-imported in a New York session came back 08:00 (measured). An arm pins that round trip. New suite test/parquet_timestamp_zone.sh and its twin, 16 checks, including the naive file read in a second zone (Tokyo), the repeated hour of the 2024-11-03 fall-back, and test/fixtures/parquet/legacy_timestamp_micros.parquet, written by main's exporter before #1707 (0c5e924) with only the TIMESTAMP_MICROS converted type. Two arms in test/iceberg_fdw.sh and its twin. docs/user-guide.md states the rule; CHANGELOG under Changed. Measured on PG17: - Red, the head before the Iceberg guard: both new iceberg_fdw checks, in shell and twin ("got [] want [2,3]", "got [] want [1,2]"). - Mutants, the same names in shell and twin: the flag ignored (6), no instant-to-wall shift (2), skipping on converted stats (1), main's exporter before #1707 (the round trip, 1), the conversion pinned to New York (the Tokyo pair, 2), legacy converted types read as naive (the legacy pair, 2), the zone offset taken at the wall time read as UTC (the fall-back hour and the two gap checks, 3), no Iceberg kind guard (2). - Green: parquet_timestamp_zone 16/16 and iceberg_fdw 76/76, shell; 16/16 and 80/80, twin; on each of PG15 to PG19. - Without pyarrow the twin declines by the shell's skip name. - docs_style 53/53, harness_selftest 1248/1248, shellcheck clean. Ledger: 6951 rows, 2811 dated by the mutations and 4140 never. suites_not_covered 75 -> 75 (a new suite, registered and covered together; iceberg_fdw is not yet seeded, so its two new checks add no rows), cluster_tests 3222 -> 3240, TESTS.md section 296. The ledger gate, guards (804 passed), name grading and the self-test pass over 9ab6a4c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e operator family (#1712) The foreign table's group-exclusion check refused any constant whose type was not the column's. A bare integer literal is int4, so the most ordinary filter on a bigint column, `WHERE id > 150000`, read every row group; only `150000::int8` skipped. Answers were correct; reads were not saved. The check now compares the statistics with the constant through the operator family's comparison function for (column type, constant type), as the native zone-map path already did (OffgridwithJD's measurement on #1712). A family with no such function means no skip. Two details the old code would have got wrong for a cross-type constant: the NaN check read the constant as the column's float type (it now reads it by its own type), and equality called the comparison with the constant first (the column's value is now always the first argument). New suite test/parquet_skip_cross_type.sh and its twin, 9 checks, over a 200,000-row file in 4 row groups: int4 constants on an int8 column (>, =, and on the left), a negative float4 constant on a float8 column, the answers against the same-type query, and a float4 NaN that must still find the NaN row its group's statistics leave out. parquet_export_stats measured the old refusal in six arms ("a cross-type constant skips nothing", and five like it). Those assertions are now wrong by design: each is renamed for what it asserts, the skip its same-type twin gets, keeps its ledger row, and is re-dated by the pre-#1712 reader, which fails exactly those six in both harnesses. Three soundness arms are added for the changed predicates that had none ([i8 < 5::int], [i8 < 5], [f4 < 0.4]); every oracle arm passes. docs/limitations.md states the new rule. Measured on PG17: - Red: the pre-#1712 reader fails the four cross-type skip arms of the new suite, in shell and twin. - Green: 9/9 shell and twin on each of PG15 to PG19; parquet_export_stats 165/165 shell and twin on each of PG15 to PG19. - Mutants, the same names in shell and twin: the comparison called with the constant first (4, the same-type control among them); the column type's own function used for a constant of another type (the negative float4 arm); the NaN constant read as the column's type (the NaN arm). - 47 parquet, iceberg and arrow suites on PG17: only parquet_export_stats's six refusal arms failed, before they were rewritten; 1044 checks. - docs_style passes. Guards 476 passed. cluster_tests 2763 -> 2775 (collected 2775). Ledger 6180 + 12 = 6192 rows, 3886 -> 3892 never, 2294 -> 2300 dated; the gate passes, ceiling 102. - Rebased onto #1736 at a93f14b, the registry re-derived there: cluster_tests 3240 -> 3252 (collected 3252). Ledger 6951 + 12 = 6963 rows, 4140 -> 4146 never, 2811 -> 2817 dated. The gate is run on the commit after this one. Stacked on #1736 (and #1707), which touch the same function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…amptz constant (#1712) OffgridwithJD's review of #1737 measured a wrong answer. For a timestamp column and a timestamptz constant, timestamp_cmp_timestamptz turns each statistic, a wall time, into an instant in the session zone, and that is not monotonic across a spring-forward gap. In New York a row group holding 2024-03-10 02:00, 02:30 and 03:00 has a converted max of 07:00Z but holds 02:30, which is 07:30Z. So `ts > '2024-03-10 07:15+00'::timestamptz` skipped the group and returned no row; a heap copy returns row 1. The same held for >= 07:15Z and = 07:30Z. The group-exclusion check now refuses that pair before choosing the comparison function. A timestamp constant on a timestamp column still skips, and a timestamptz column with a timestamp constant is unchanged. docs/limitations.md and the CHANGELOG entry say so. parquet_skip_cross_type and its twin gain 7 checks over a naive timestamp file in 2 row groups, every session in New York: the three predicates against the heap copy, a same-type control that still skips 1 group, its answer against the heap, and two premises. Measured: - Red: on the rebased head without the guard, PG17, the >, >= and = arms fail by the same names in shell (13 passed + 3 failed = 16) and twin (3 failed, 13 passed), each "got none want 1". - Mutant: deleting the guard line reddens the same three names in both harnesses, which dates their ledger rows. - Green: parquet_skip_cross_type 16/16 and parquet_export_stats 165/165, shell and twin, on each of PG15 to PG19. parquet_timestamp_zone 16/16 shell and twin on PG17. - Guards 804 passed. docs_style passes. cluster_tests 3252 -> 3259 (collected 3259). Ledger 6963 + 7 = 6970 rows, 4146 -> 4150 never, 2817 -> 2820 dated; the gate passes against origin/main, ceiling 75. compare_to_bash: every bash property is covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…er collide (#1742) A twin that starts its own S3 or HTTP fixture server takes the port from pick_port(slot) in a module fixture. Under -n, a module's tests are split across workers, so every worker runs that fixture. The port depended on the slot only, so two workers probed the same free port at the same moment and the second bind failed with "Address already in use" (#1635, #1714, #1721). CI runs the cluster files serially, so CI stayed green; the guard job and developers run -n. pick_port now starts each worker in its own block of 64 ports (WORKER_STRIDE), with the slot as an offset inside it. The worker comes from PYTEST_XDIST_WORKER, so no fixture changes. Slot 0 is the worker's own cluster, which make_cluster now asks for through the same function. A slot that does not fit in a block is refused. The walk on a busy port, the AUX band and the ephemeral-floor contract are unchanged. test_runshape.py gains test_two_workers_taking_a_fixture_port_do_not_collide: an inner -n 2 run where each worker takes pick_port(27), waits until the other has a port, then binds it. On main it is red: "the two workers were given two different ports: got 1 want 2", and the inner bind fails with [Errno 98] Address already in use. With the fix it is green. Measured on PG17 (a copy of the prefix): - test_iceberg_objstore.py under -n 2: main 3 of 3 red ("premise: sigv4 fixture server is up: got '0' want '1'"; "Connection refused"); fixed 16 passed 3 of 3, and 16 passed serially. - test_objstore_listing.py from #1714 under -n 2: as is, 13 failed ("the private cluster did not start"); with this pgc_cluster.py, 15 passed 3 of 3 and serially. - test_connection.py and test_hardening.py under -n 2: 20 passed. guard_tests 476 -> 477, by collection over NO_CLUSTER; TESTS.md section 10 gains a row. The guards (477), test_harness_deps.py and test_capability_decline.py with --pg-config pass over 21182d0. Review (OffgridwithJD): a 64-port block holds slots 0 to 31, and main's twins already ask for 32 to 35 (the stack for up to 41), so slot 32 raised at fixture time: test_iceberg_rest.py "14 errors", ValueError "slot 32 does not fit in a worker's block of 64 ports". WORKER_STRIDE is now 96 (slots 0 to 47; the AUX band's 2000 ports still give 20 workers). Two guards keep the stride and the corpus from drifting apart, red first on the 64 stride: every literal pick_port slot fits a worker's block (it named the 32-35 files), and the corpus's highest slot (35) gets a port in the band. Green: test_runshape.py 18 passed; the five twins that errored pass serially (14, 11, 29, 10, 8); iceberg_rest + iceberg_objstore under -n 2, 30 passed; guards 807 passed on main 9ab6a4c (guard_tests 804 -> 807). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jdatcmd · 2026-10-26T17:56:09Z

Restated onto main 4500efd as 0aeb420, in one linear stack (this PR sits on #1737), so these eight merge without voiding each other. Every non-registry file carries the same change as before; the registries are re-derived on the new base, and each census agrees with a recount. Guards, the PR's own shell suites on PG16 and PG18, its twins on PG17, compare_to_bash and the ledger gate all pass at this head. Re-approval needed.

OffgridwithJD

@jdatcmd Re-approved at 0aeb420f, restated onto main 4500efdd in the linear stack. Outside the ledger, the budget, expected_tests and TESTS.md, the commit's diff matches the approved 035f6730. The tsv is sorted, and its census (8015/4548/3467 rows/never/dated) matches the budget.

🤖 Generated with Claude Code

OffgridwithJD mentioned this pull request Oct 22, 2026

Missing tests: 87 confirmed behaviours no shell or pytest arm exercises (#1339 audit) #1341

Closed

87 tasks done

OffgridwithJD requested changes Oct 22, 2026

View reviewed changes

jdatcmd force-pushed the fix/1742-xdist-ports branch from 85609b6 to 035f673 Compare October 22, 2026 18:25

OffgridwithJD approved these changes Oct 22, 2026

View reviewed changes

This was referenced Oct 23, 2026

test: objstore_sink_write gains its pytest twin, and multipart is read as S3 does it (#1341) #1721

Merged

alpha5: clearing the board to release — who has what #1283

Open

jdatcmd and others added 4 commits October 26, 2026 17:53

jdatcmd force-pushed the fix/1742-xdist-ports branch from 035f673 to 0aeb420 Compare October 26, 2026 17:55

jdatcmd mentioned this pull request Oct 26, 2026

test: parquet_export gains its pytest twin, DuckDB becomes a declinable, pinned reader, and a crash no longer passes as a refusal (#1341) #1750

Merged

OffgridwithJD approved these changes Oct 26, 2026

View reviewed changes

jdatcmd merged commit 0aeb420 into main Oct 26, 2026
15 checks passed

Navigation Menu

Repository navigation

fix: a fixture port depends on the xdist worker, so -n 2 runs no longer collide (#1742) - #1749

fix: a fixture port depends on the xdist worker, so -n 2 runs no longer collide (#1742)#1749
jdatcmd merged 4 commits into
mainfrom
fix/1742-xdist-ports

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

jdatcmd commented Oct 26, 2026

Uh oh!

OffgridwithJD left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jdatcmd commented Oct 22, 2026

What was wrong

What changed

Test, red first

Real twins under -n 2 (PG17, a copy of the prefix)

Gates

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

jdatcmd commented Oct 22, 2026

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

jdatcmd commented Oct 26, 2026

Uh oh!

OffgridwithJD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Real twins under `-n 2` (PG17, a copy of the prefix)