Skip to content

test: native_parquet_flba gets its pytest twin, and two arms see the faults they name (#1341) - #1497

Merged
jdatcmd merged 9 commits into
mainfrom
test/1341-native-parquet-flba-twin
Oct 16, 2026
Merged

jdatcmd merged 9 commits into
mainfrom
test/1341-native-parquet-flba-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd The next twin-less suite, stacked on #1496 (#1489 → #1493 → #1496 → this). This PR's own commits are the top two.

Two arms couldn't see the faults they name

  • "pyarrow uuid reads as uuid" read ORDER BY u LIMIT 1, the all-zero uuid, which every byte order leaves unchanged. With the reader reversing a uuid's bytes, the old arm stayed green. It now reads both values in order, and that mutant is red in both twins.
  • "crafted out-of-range scale is rejected, not decoded" counted any error as REJECTED. Removing the bind guard (pq_decimal_bounds_ok in pq_want_phys_for) left it green: the decoder's own scale > 38 check refused the file instead, as XX001 could not decode Parquet column 0, after decoding had begun. The bind refuses with 42804, and the arm now requires REJECTED 42804 (the shell's pgc_try_flba returns the SQLSTATE). With the guard removed it's red in both twins.

The twin, name for name

All 15 checks by the shell's name, in three tests. Under each mutation, the twin reddens exactly the shell's set:

mutation (columnar_parquet_reader.c) reds, shell = twin
uuid bytes reversed 5: four round trips and the pyarrow uuid
a narrow negative DECIMAL read unsigned 3
bind guard dropped 1
bind guard and decoder scale check dropped 1
integer DECIMAL widened byte-reversed 2
parquet_schema advises integer for INT32 DECIMAL 1
bigint scaled like numeric 1

Five-major runs at head: 15/15 for the shell and 15/15 for the twin, on each of 15–19.

Ledger

Census 4278 / 3006 never / 1272 dated (on #1496's 4263 / 3002 / 1261), with suites_not_covered 196 → 195. 11 of the 15 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 4 arrive never:

  • the round-trip numeric arm: our export writes 16-byte DECIMALs, which are decoded by bit pattern, so the narrow-width sign mutation doesn't reach them;
  • the INT32 null;
  • the INT64 schema advice;
  • backend liveness after the crafted scale.

Collection gives cluster_tests 734 -> 737, and TESTS.md gets §170. The grader is 15/15 by name, and the guards pass 438/438.

🤖 Generated with Claude Code

jdatcmd and others added 9 commits October 16, 2026 17:06
…through the index it was written for (#1341)

The suite forced an index scan with enable_seqscan=off, which leaves the
columnar custom scan on. Measured: the single-row fetch, the near and far
UPDATEs and the mirror join all planned as Custom Scan (PgColumnarScan), so
none of them fetched by row number. The rank off by one left the
single-row arm green, and the rank walking every earlier row left both
timing arms green. FORCE now also sets pgcolumnar.enable_custom_scan = off,
and four premises assert that each measured query's plan is an Index Scan
using its index.

The timing arms then still missed the walk, because they timed an UPDATE.
A columnar UPDATE writes each row again into a new row group, so only the
first of three passes fetched from the far end, and the median was always a
later pass. They now time a read of the same range, sum(fixed) and
sum(length(txt)), so every pass fetches the same positions. Measured at head
on 15-19: near and far within 1.03x, small and big group within 1.1x.
Measured with the rank walking: 50 ms against 779 ms, and 390 against 779.

test_native_fetch_position.py carries all 12 checks by the shell's name in
four tests. Under each of four mutations (rank counting rows, rank off by
one, rank walking, offset walking), the twin reddens exactly the shell's set
of checks. Like the shell, it reads PGC_FETCHPOS_ROWS.

COMPLETE gains native_fetch_position. TESTS.md section 167, cluster_tests
722 -> 726.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y mutation from #1485's review (#1341)

jdatcmd's mutation drops the NULL-key flag in the grouped fold's gather,
so the NULL group folds into k = 0 on the batch path only. Nine checks
redden, and the twin reddens the same nine by name. Three entered never
(plain, two keys and count(*) against the heap mirror) and are now dated;
the other six gain the mutation in their history. Census 4189 rows / 2969
never / 1220 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our rank and offset mutations (#1341)

The suite had no rows. suites_not_covered 199 -> 198; census 4201 rows /
2975 never / 1226 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t says why its UPDATE fetches by row number (#1341)

test_native_fetch_cache.py carries all 26 checks by the shell's name in
four tests, rebuilding each fixture before every reading as the shell does
(#801). Under each of nine mutations, the twin reddens exactly the shell's
set of checks: the cache never hitting, the key without the storage id,
no earlier-command reject, each of three recheck fields dropped, the
discard dropped from executor end and from transaction end, and the cap
dropping every column rather than the one that crossed it.

Section 1 sets only enable_seqscan/enable_bitmapscan off, and its UPDATE
plans as Custom Scan (PgColumnarScan). It measures the cache anyway: the
UPDATE fetches its target rows by number whatever the scan. Measured with
the cache missing on every lookup, the unchanged suite is red at 844 ms
against 93 ms. Its comment said the opposite reason ("index-driven so every
row goes through the fetch path rather than a scan") and now says this.

COMPLETE gains native_fetch_cache. TESTS.md section 168, cluster_tests
726 -> 730.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cache mutations (#1341)

The suite had no rows. suites_not_covered 198 -> 197; census 4227 rows /
2991 never / 1236 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_native_parquet_pushdown.py carries all 36 checks by the shell's name
in four tests, each writing its file with pyarrow and declining under the
shell's own pgc_skip name when pyarrow is absent. Under each of seven
mutations of the FDW's skip decision, the twin reddens exactly the shell's
set of checks: no group ever skipped, the inverted-interval guard dropped,
float > and >= pushed, a NaN equality pushed, INT64 statistics not decoded,
col < const skipping the groups that match, and an equality skipping every
group. The last two are what date the oracle arms: a skip that only stops
saving work cannot change an answer.

Each test creates its own foreign server, because a server is
database-wide and pgc_conn isolates tests by schema only.

COMPLETE gains native_parquet_pushdown, and its literal tables join the
loop-reader guard's list. TESTS.md section 169, cluster_tests 730 -> 734.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y seven skip mutations (#1341)

The suite had no rows. suites_not_covered 197 -> 196; census 4263 rows /
3002 never / 1261 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…faults they name (#1341)

test_native_parquet_flba.py carries all 15 checks by the shell's name in
three tests. Under each of eight mutations of the FLBA and DECIMAL read
path, the twin reddens exactly the shell's set of checks.

Two arms change, in both twins, each measured red first:
- "pyarrow uuid reads as uuid" read only the smallest value, the all-zero
  uuid, which every byte order leaves unchanged. With the reader reversing
  a uuid's bytes it stayed green. It now reads both values in order.
- "crafted out-of-range scale is rejected, not decoded" took any error as
  REJECTED. With the bind guard (pq_decimal_bounds_ok) removed, the
  decoder's own scale check still refused the file, as XX001 "could not
  decode Parquet column 0" after decoding began, and the arm stayed green.
  It now requires the bind's refusal, 42804.

COMPLETE gains native_parquet_flba. TESTS.md section 170, cluster_tests
734 -> 737.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x read-path mutations (#1341)

The suite had no rows. suites_not_covered 196 -> 195; census 4278 rows /
3006 never / 1272 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve, at 9502a8a (its own commits 6608a71 and 9502a8a, on #1496).

My own mutant, different from your seven: the uuid decode zeroes the last byte after the copy (u->data[UUID_LEN - 1] = 0), not a reversal. PG17 shell, 5 red:

  • the three round trips and the FDW scan (md5 mismatch);
  • "pyarrow uuid reads as uuid": got [00000000-…-000000000000,11111111-2222-3333-4444-555555555500] want [...,…555555555555].

The old form of that arm read only the all-zero uuid, which this mutant also leaves unchanged. So your strengthened arm is what catches it, not the round trips alone.

What I could not verify: as on #1496, I can't run the pytest twin here: /root/pyenv has no pyarrow, so it declines. Twin parity rests on your measurement and CI.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, merging #1497 now: approved at 9502a8a, 15/15 green, CLEAN.

@jdatcmd
jdatcmd merged commit e472aaf into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants