Repository navigation
test: parquet_export gains its pytest twin, DuckDB becomes a declinable, pinned reader, and a crash no longer passes as a refusal (#1341) - #1750
Conversation
e11f608 to
533b237
Compare
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Approved at 533b237f.
All runs are PG17 unless stated, on ~/pg/17-g built from this head's tree (no src change). Main's shell is 4adb2e7a's test/parquet_export.sh. DuckDB 1.5.6 sits in a separate target dir, not my base venv. For the shell's four DuckDB arms I used a stand-in duckdb CLI on PATH that runs the same SQL through that package.
Clean runs
| run | shell | twin |
|---|---|---|
| head, DuckDB present, PG15–19 | 14/14 on each | 14/14 on each |
| head, DuckDB absent | 10 passed + 4 skipped = 14 |
10 pass + 4 declined = 14 (rc 0) |
The refusal holes, red first
mutant (src/columnar_parquet.c) |
main's shell | head shell | head twin |
|---|---|---|---|
nocolcheck: the columnar-relation check skipped |
14/14 pass | FAIL reject non-columnar table: got [other error: ERROR: unsupported columnar format version …] |
reject non-columnar table red |
typecrash: abort() before the unsupported-type refusal |
reject unsupported type passes; 5 extended checks red, collateral of the recovery |
reject unsupported type: got [other error: server closed the connection unexpectedly], plus the same 5 |
the same 6 tests fail |
typeothererr: the refusal reworded |
14/14 pass | reject unsupported type red |
the same name red |
So "a crash no longer passes as a refusal" holds in both harnesses. Under typecrash the twin names reject unsupported type as a FAIL record. Its five collateral tests fail as refusals (a failed query on both sides), so they show as failed tests but not as named records. That's #1735's gap, not this PR's.
Also checked
- Oracles:
t_heapis filled from the generator, andt_colis copied from it. The id sets are pinned to the generator's (1..80000 ∪ {100001,100002,100003};1..4for the extended arm), with the file's row count beside them, andduckdb non-null count(b)is compared with the generator's 80002, not witht_col. - Capability machinery: with main's
pgc_vacuity.pyrestored, the four newtest_capability_decline.pytests fail (4 failed, 9 passed). On head they pass (13 passed)._capability_import_sitesover everytest_*.pyprints[]; the twin reachesduckdbonly throughexpect.require_module. - The pin: PyPI has
duckdb-1.5.6-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl, which fits ubuntu-latest's glibc. nightly.yml: its cluster job installsrequirements-test.txt, soduckdbis present there and the four tests run. It sets noPGC_REQUIRE, as before for pyarrow too, so this PR changes nothing there.- Skip rather than unrunnable: agreed. The four names are real checks, recorded by
check_skip(which the grader reads), always SKIP in the suites job and always required through the twin underPGC_REQUIRE. They're notpgc_skipcapability names, so their ledger rows are right. - Ledger: 6949 rows, 4137 never and 2812 dated, which matches the budget. It's C-sorted,
suites_not_coveredis 74, andunported_suites.txtdropsparquet_export.git merge-treeagainst main4adb2e7ais clean.
Not blocking
- Both refusals pin the message but not the SQLSTATE (
42809for the wrong object type,0A000for the unsupported type). The messages are unique enough that I measured no gap, but asserting the code too would match the other ports. - With no DuckDB CLI in the suites job, the shell's four DuckDB arms never run in CI. The twin carries them under
PGC_REQUIRE, as the PR says. If the shell should run them too, the CLI install is the separate change you named.
🤖 Generated with Claude Code
…ostgreSQL's casts (#1706) Item 2 of #1706, decided there as option (a). The reader skipped the TimestampType's isAdjustedToUTC flag and returned the raw microseconds for either target type. In a New York session a naive (wall-time) column read into timestamptz came back as 08:00-04 instead of 12:00-04, and an adjusted column read into timestamp kept its UTC wall time instead of 08:00. The reader now keeps the flag (the legacy TIMESTAMP_* converted types are defined as adjusted) and, in those two mixed cases only, converts as timestamp_timestamptz and timestamptz_timestamp do, in the session zone. It is written from timestamp2tm, DetermineTimeZoneOffset and tm2timestamp, so it is the same on every major: PG19 renamed the exported helper, and PG15-18 export none for timestamptz -> timestamp. Same-kind reads, and any read in a UTC session, are unchanged. A wall time is not monotonic in its instant across a spring-forward gap: in New York on 2024-03-10, 02:00, 02:30 and 03:00 become 07:00Z, 07:30Z and 07:00Z. Converted row-group statistics need not bound the converted rows, so the foreign table no longer skips row groups of a column read this way. Measured: with skipping left on, `t > '07:15Z'` over such a group returned 0 rows instead of 1. The Iceberg foreign table's year(), month(), day() and hour() pruning compared a constant with partition cells the writer computed in the source field's own kind, while the reader now converts a mixed-kind column's rows. PgColumnarIcebergTemporalMap now leaves out a field whose Iceberg source type is the other timestamp kind than its column, so nothing prunes on it. Measured on PG17 before the guard, in New York: byhour declared timestamptz with `ts >= '2021-03-01 06:00+00'` returned no rows where iceberg_scan returned 2,3, and byhour_tz declared timestamp with `tt <= '2021-02-28 20:00'` returned none where it returned 1,2. #1707 (merged) is a prerequisite: with the reader honouring the flag and the old exporter marking timestamp adjusted, a timestamp exported and re-imported in a New York session came back 08:00 (measured). An arm pins that round trip. New suite test/parquet_timestamp_zone.sh and its twin, 16 checks, including the naive file read in a second zone (Tokyo), the repeated hour of the 2024-11-03 fall-back, and test/fixtures/parquet/legacy_timestamp_micros.parquet, written by main's exporter before #1707 (0c5e924) with only the TIMESTAMP_MICROS converted type. Two arms in test/iceberg_fdw.sh and its twin. docs/user-guide.md states the rule; CHANGELOG under Changed. Measured on PG17: - Red, the head before the Iceberg guard: both new iceberg_fdw checks, in shell and twin ("got [] want [2,3]", "got [] want [1,2]"). - Mutants, the same names in shell and twin: the flag ignored (6), no instant-to-wall shift (2), skipping on converted stats (1), main's exporter before #1707 (the round trip, 1), the conversion pinned to New York (the Tokyo pair, 2), legacy converted types read as naive (the legacy pair, 2), the zone offset taken at the wall time read as UTC (the fall-back hour and the two gap checks, 3), no Iceberg kind guard (2). - Green: parquet_timestamp_zone 16/16 and iceberg_fdw 76/76, shell; 16/16 and 80/80, twin; on each of PG15 to PG19. - Without pyarrow the twin declines by the shell's skip name. - docs_style 53/53, harness_selftest 1248/1248, shellcheck clean. Ledger: 6951 rows, 2811 dated by the mutations and 4140 never. suites_not_covered 75 -> 75 (a new suite, registered and covered together; iceberg_fdw is not yet seeded, so its two new checks add no rows), cluster_tests 3222 -> 3240, TESTS.md section 296. The ledger gate, guards (804 passed), name grading and the self-test pass over 9ab6a4c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e operator family (#1712) The foreign table's group-exclusion check refused any constant whose type was not the column's. A bare integer literal is int4, so the most ordinary filter on a bigint column, `WHERE id > 150000`, read every row group; only `150000::int8` skipped. Answers were correct; reads were not saved. The check now compares the statistics with the constant through the operator family's comparison function for (column type, constant type), as the native zone-map path already did (OffgridwithJD's measurement on #1712). A family with no such function means no skip. Two details the old code would have got wrong for a cross-type constant: the NaN check read the constant as the column's float type (it now reads it by its own type), and equality called the comparison with the constant first (the column's value is now always the first argument). New suite test/parquet_skip_cross_type.sh and its twin, 9 checks, over a 200,000-row file in 4 row groups: int4 constants on an int8 column (>, =, and on the left), a negative float4 constant on a float8 column, the answers against the same-type query, and a float4 NaN that must still find the NaN row its group's statistics leave out. parquet_export_stats measured the old refusal in six arms ("a cross-type constant skips nothing", and five like it). Those assertions are now wrong by design: each is renamed for what it asserts, the skip its same-type twin gets, keeps its ledger row, and is re-dated by the pre-#1712 reader, which fails exactly those six in both harnesses. Three soundness arms are added for the changed predicates that had none ([i8 < 5::int], [i8 < 5], [f4 < 0.4]); every oracle arm passes. docs/limitations.md states the new rule. Measured on PG17: - Red: the pre-#1712 reader fails the four cross-type skip arms of the new suite, in shell and twin. - Green: 9/9 shell and twin on each of PG15 to PG19; parquet_export_stats 165/165 shell and twin on each of PG15 to PG19. - Mutants, the same names in shell and twin: the comparison called with the constant first (4, the same-type control among them); the column type's own function used for a constant of another type (the negative float4 arm); the NaN constant read as the column's type (the NaN arm). - 47 parquet, iceberg and arrow suites on PG17: only parquet_export_stats's six refusal arms failed, before they were rewritten; 1044 checks. - docs_style passes. Guards 476 passed. cluster_tests 2763 -> 2775 (collected 2775). Ledger 6180 + 12 = 6192 rows, 3886 -> 3892 never, 2294 -> 2300 dated; the gate passes, ceiling 102. - Rebased onto #1736 at a93f14b, the registry re-derived there: cluster_tests 3240 -> 3252 (collected 3252). Ledger 6951 + 12 = 6963 rows, 4140 -> 4146 never, 2811 -> 2817 dated. The gate is run on the commit after this one. Stacked on #1736 (and #1707), which touch the same function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…amptz constant (#1712) OffgridwithJD's review of #1737 measured a wrong answer. For a timestamp column and a timestamptz constant, timestamp_cmp_timestamptz turns each statistic, a wall time, into an instant in the session zone, and that is not monotonic across a spring-forward gap. In New York a row group holding 2024-03-10 02:00, 02:30 and 03:00 has a converted max of 07:00Z but holds 02:30, which is 07:30Z. So `ts > '2024-03-10 07:15+00'::timestamptz` skipped the group and returned no row; a heap copy returns row 1. The same held for >= 07:15Z and = 07:30Z. The group-exclusion check now refuses that pair before choosing the comparison function. A timestamp constant on a timestamp column still skips, and a timestamptz column with a timestamp constant is unchanged. docs/limitations.md and the CHANGELOG entry say so. parquet_skip_cross_type and its twin gain 7 checks over a naive timestamp file in 2 row groups, every session in New York: the three predicates against the heap copy, a same-type control that still skips 1 group, its answer against the heap, and two premises. Measured: - Red: on the rebased head without the guard, PG17, the >, >= and = arms fail by the same names in shell (13 passed + 3 failed = 16) and twin (3 failed, 13 passed), each "got none want 1". - Mutant: deleting the guard line reddens the same three names in both harnesses, which dates their ledger rows. - Green: parquet_skip_cross_type 16/16 and parquet_export_stats 165/165, shell and twin, on each of PG15 to PG19. parquet_timestamp_zone 16/16 shell and twin on PG17. - Guards 804 passed. docs_style passes. cluster_tests 3252 -> 3259 (collected 3259). Ledger 6963 + 7 = 6970 rows, 4146 -> 4150 never, 2817 -> 2820 dated; the gate passes against origin/main, ceiling 75. compare_to_bash: every bash property is covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er collide (#1742) A twin that starts its own S3 or HTTP fixture server takes the port from pick_port(slot) in a module fixture. Under -n, a module's tests are split across workers, so every worker runs that fixture. The port depended on the slot only, so two workers probed the same free port at the same moment and the second bind failed with "Address already in use" (#1635, #1714, #1721). CI runs the cluster files serially, so CI stayed green; the guard job and developers run -n. pick_port now starts each worker in its own block of 64 ports (WORKER_STRIDE), with the slot as an offset inside it. The worker comes from PYTEST_XDIST_WORKER, so no fixture changes. Slot 0 is the worker's own cluster, which make_cluster now asks for through the same function. A slot that does not fit in a block is refused. The walk on a busy port, the AUX band and the ephemeral-floor contract are unchanged. test_runshape.py gains test_two_workers_taking_a_fixture_port_do_not_collide: an inner -n 2 run where each worker takes pick_port(27), waits until the other has a port, then binds it. On main it is red: "the two workers were given two different ports: got 1 want 2", and the inner bind fails with [Errno 98] Address already in use. With the fix it is green. Measured on PG17 (a copy of the prefix): - test_iceberg_objstore.py under -n 2: main 3 of 3 red ("premise: sigv4 fixture server is up: got '0' want '1'"; "Connection refused"); fixed 16 passed 3 of 3, and 16 passed serially. - test_objstore_listing.py from #1714 under -n 2: as is, 13 failed ("the private cluster did not start"); with this pgc_cluster.py, 15 passed 3 of 3 and serially. - test_connection.py and test_hardening.py under -n 2: 20 passed. guard_tests 476 -> 477, by collection over NO_CLUSTER; TESTS.md section 10 gains a row. The guards (477), test_harness_deps.py and test_capability_decline.py with --pg-config pass over 21182d0. Review (OffgridwithJD): a 64-port block holds slots 0 to 31, and main's twins already ask for 32 to 35 (the stack for up to 41), so slot 32 raised at fixture time: test_iceberg_rest.py "14 errors", ValueError "slot 32 does not fit in a worker's block of 64 ports". WORKER_STRIDE is now 96 (slots 0 to 47; the AUX band's 2000 ports still give 20 workers). Two guards keep the stride and the corpus from drifting apart, red first on the 64 stride: every literal pick_port slot fits a worker's block (it named the 32-35 files), and the corpus's highest slot (35) gets a port in the band. Green: test_runshape.py 18 passed; the five twins that errored pass serially (14, 11, 29, 10, 8); iceberg_rest + iceberg_objstore under -n 2, 30 passed; guards 807 passed on main 9ab6a4c (guard_tests 804 -> 807). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le, pinned reader, and a crash no longer passes as a refusal (#1341) test/pytest/test_parquet_export.py ports all 14 checks of test/parquet_export.sh by name. Five kinds of shell check could pass with nothing measured, or vanish; all are fixed in both twins. - "reject non-columnar table" and "reject unsupported type" passed on any error. With the columnar check removed, exporting a heap failed on "unsupported columnar format version" and passed; with the type refusal formatting the typmod, it failed on "cache lookup failed" and passed. Each refusal now needs its own message. - "parquet values match heap oracle" and "extended values match heap oracle" agreed on two empty readings: with the generator's rows never inserted, both passed. They now need the generator's ids on both sides and the file's row count. - "duckdb non-null count(b)" compared DuckDB's count with t_col, the table under test: with half of t_col deleted before the export, it passed. It is now held to the generator's 80002, by check_num. - Without the duckdb CLI the shell left its four DuckDB checks out without a word (10 = 10). It now skips them by name (10 passed + 4 skipped = 14). A skip, not check_unrunnable: the suites job installs no DuckDB CLI, and an INCOMPLETE suite fails its major there. DuckDB in the twin, as decided on #1341 (option 1 with option 2's mechanism): the duckdb Python package is pinned at 1.5.6 in requirements-test.txt; duckdb joins pyarrow in DECLINABLE_CAPABILITIES, held by four new tests in test_capability_decline.py, written red first; and the cluster job sets PGC_REQUIRE=pyarrow,duckdb. The twin reaches duckdb only through require_module, under the shell's check names. Measured, PG17, shell and twin: - duckdb absent: the four DuckDB tests decline by name and the run is green (10 pass + 4 declined); with PGC_REQUIRE=duckdb the same four fail. - the columnar check removed, and the type refusal formatting the typmod: the refusal red in both; main's shell passed 14 of 14. - the generator's rows never inserted: 8 red in both; main's shell 5. - half of t_col deleted: 4 red in both; main's shell 3. - the type check removed (the backend aborts): the refusal red in both, with the same 5 collateral reds in both. - nine more mutations of src/columnar_parquet.c: each red set identical in both twins and in main's shell. DECIMAL written little-endian and a negative int4 written as null are red in both readers, pyarrow and DuckDB; row-group num_rows one high is red in DuckDB only. - clean: 14 of 14 on PG15 to PG19 in both twins. Ledger: 14 rows, 14 dated by the 11 mutations and 0 never. suites_not_covered 75 -> 74, cluster_tests 3222 -> 3236, guard_tests 804 -> 808, TESTS.md section 296. The ledger gate, guards (808), name grading and the self-test pass over 4adb2e7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
533b237 to
cef61cc
Compare
|
Restated onto main 4500efd as cef61cc, in one linear stack (this PR sits on #1749), so these eight merge without voiding each other. Every non-registry file carries the same change as before; the registries are re-derived on the new base, and each census agrees with a recount. Guards, the PR's own shell suites on PG16 and PG18, its twins on PG17, compare_to_bash and the ledger gate all pass at this head. Re-approval needed. |
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Re-approved at cef61cc0, restated onto main 4500efdd in the linear stack. Outside the ledger, the budget, expected_tests and TESTS.md, the commit's diff matches the approved 533b237f. The tsv is sorted, and its census (8029/4548/3481 rows/never/dated) matches the budget.
🤖 Generated with Claude Code
@OffgridwithJD
test: parquet_export gains its pytest twin, DuckDB becomes a declinable, pinned reader (#1341)
test/pytest/test_parquet_export.pyports all 14 checks oftest/parquet_export.sh, each under the shell's name. One commit on21182d05. It implements your DuckDB decision on #1341 ("option 1, with option 2's mechanism").DuckDB, as decided
duckdb==1.5.6intest/pytest/requirements-test.txt(the file uses no hashes). Released 2026-09-28; wheels for cp312 (CI's venv) and cp314 (here).DECLINABLE_CAPABILITIES = ("pyarrow", "duckdb"). Four new tests intest_capability_decline.pyhold it, written red first: the exact set, a by-name decline that stays green,PGC_REQUIREturning it into a failure, and the import scan namingimport duckdb.import duckdband setsPGC_REQUIRE=pyarrow,duckdb. No test file is named in.github/.duckdbpackage, reached only throughexpect.require_module("duckdb", name="<the check's name>"). The import scan prints[].check_skip), not left out.One departure from your comment: skip, not
check_unrunnable. An unrunnable check makes the suite INCOMPLETE (exit 67), andpgc_verdict_fails_majorfails a major on INCOMPLETE. The suites job installs no DuckDB CLI, socheck_unrunnablewould turn that job red on every major. A named skip is the shell's counterpart of the twin's DECLINED, aspgc_skipalready is for pyarrow. CI still requires the four checks, through the twin andPGC_REQUIRE. If you want the shell to require them too, the suites job needs the DuckDB CLI installed. That is a separate change.Measured, PG17:
duckdbnot installed10 pass + 0 fail + 0 unrun + 4 declined = 14, each decline named (DECLINED ... duckdb: duckdb is not installed: duckdb row count, and the other three)duckdbnot installed,PGC_REQUIRE=duckdb10 pass + 4 fail, eachPGC_REQUIRE names duckdb, so it must be present hereduckdb1.5.6 installed,PGC_REQUIRE=pyarrow,duckdbduckdbCLI offPATH10 passed + 0 failed + 0 unrunnable + 4 skipped = 14duckdbCLI offPATH10 passed + ... = 10, so the four checks disappear without a wordA corrupted export must be red in both readers. Two mutations show this, in both twins:
extended values match heap oracle(pyarrow) and induckdb reads decimal.parquet values match heap oracle(pyarrow) and induckdb non-null count(b).One mutation is red in DuckDB only. With row-group
num_rowswritten one high, all three DuckDB counts go red and pyarrow passes.What was fixed in the shell
reject non-columnar table,reject unsupported typeparquet values match heap oracle,extended values match heap oracleduckdb non-null count(b)t_col, the table under test.check_num.Main's shell, this shell and the twin under each change (PG17, DuckDB present)
Each cell is the number of red checks. "Same names" means the twin reddened exactly this shell's names.
nocolcheck)typemsgwrong)h-norows)t_coldeleted before export (h-colloses)notypecheck)retplus1)noint16anno)dateoff)boolflip)nodecimal)decle)negint4null)num_rowsone high (rgrows)namechop)Under
notypecheck, main's shell sawserver closed the connection unexpectedlyand counted it as the refusal.Found on the way
My first version of the new import-scan guard ran
import duckdbinside pytester, in-process. pytester'ssys.modulessnapshot then removed the module, but duckdb's C extension stayed loaded. In one session with--pg-config, the twin's four DuckDB tests then failed withNo module named '_duckdb._sqltypes'(4 failed, 44 passed). The guard now puts the import in a function that never runs, and the AST scan still finds it. The same session is now 48 passed. The guard still reds withduckdbtaken out of the list.Ledger and gates
notypecheck(a crash, with collateral reds) is not merged. Census: never 3885 -> 3885, dated 2311 -> 2325, rows 6196 -> 6210.suites_not_covered101 -> 100.cluster_tests2779 -> 2793 andguard_tests476 -> 480, both by collection. TESTS.md section 269, plus four rows in section 91.21182d05.1673 pass + 0 fail).test_harness_deps.py,test_capability_decline.pyand the twin, with--pg-configandPGC_REQUIRE=pyarrow,duckdbcompare_to_bash.py[]shellcheck -S errorharness_selftest.shorigin/mainhas moved 49 commits past21182d05since I started. The only overlap with this commit's files is aci.ymlstep in another job, so the registries need a restate and nothing more.Restated by me onto main
4adb2e7a: only the three count files conflicted. The ledger rows merged cleanly; the census (4137 never),suites_not_covered75 -> 74,guard_tests804 -> 808,cluster_tests3222 -> 3236 (collected 3236), and TESTS.md §296 are re-derived on main. Guards 808 passed; the ledger gate passes.🤖 Generated with Claude Code