Skip to content

test: native_index gets its pytest twin, and every lookup fetches through the index (#1341, #1339) - #1499

Merged
jdatcmd merged 6 commits into
mainfrom
test/1341-native-index-twin
Oct 16, 2026
Merged

jdatcmd merged 6 commits into
mainfrom
test/1341-native-index-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd The first of the #1339 custom-scan audit entries I posted, done as a twin PR because native_index.sh had none. Stacked on #1498 (#1496 → #1497 → #1498 → this). This PR's own commits are the top two.

native_index.sh's range aggregate never fetched by row number

iscan() forced the index path with enable_seqscan=off alone, which leaves the columnar custom scan on. Measured on PG18: "index range aggregate" (id BETWEEN 100 AND 400) planned as Custom Scan (PgColumnarScan). Red first, with two mutations of pgcolumnar_index_fetch_tuple:

mutation old suite new suite and twin
the neighbouring row fetched (rowNumber + 1) 5 red; "index range aggregate" green 6 red, including the range aggregate
no row found 5 red; "index range aggregate" green 6 red, including the range aggregate

Fix: iscan now also sets enable_bitmapscan, enable_indexonlyscan and pgcolumnar.enable_custom_scan off, so every lookup is an Index Scan that fetches. Four premises assert that plan through n_pkey, using lib.sh's pgc_uses_row_fetch.

The twin

All 13 checks by the shell's name, in two tests. Under both mutations it reddens exactly the shell's set. Five-major runs at head: 13/13 for the shell and 13/13 for the twin, on each of 15–19.

Ledger

Census 4323 / 3017 never / 1306 dated (on #1498's 4310 / 3010 / 1300), with suites_not_covered 194 → 193. 6 of the 13 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 7 arrive never:

  • the four index-path premises;
  • the row count;
  • the lookup miss;
  • the unique index admitting a new key.

Collection gives cluster_tests 740 -> 742, and TESTS.md gets §172. The grader shows 9 literal + 1 template match with 0 missing, and the guards pass 438/438.

🤖 Generated with Claude Code

jdatcmd and others added 6 commits October 16, 2026 17:26
…faults they name (#1341)

test_native_parquet_flba.py carries all 15 checks by the shell's name in
three tests. Under each of eight mutations of the FLBA and DECIMAL read
path, the twin reddens exactly the shell's set of checks.

Two arms change, in both twins, each measured red first:
- "pyarrow uuid reads as uuid" read only the smallest value, the all-zero
  uuid, which every byte order leaves unchanged. With the reader reversing
  a uuid's bytes it stayed green. It now reads both values in order.
- "crafted out-of-range scale is rejected, not decoded" took any error as
  REJECTED. With the bind guard (pq_decimal_bounds_ok) removed, the
  decoder's own scale check still refused the file, as XX001 "could not
  decode Parquet column 0" after decoding began, and the arm stayed green.
  It now requires the bind's refusal, 42804.

COMPLETE gains native_parquet_flba. TESTS.md section 170, cluster_tests
734 -> 737.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x read-path mutations (#1341)

The suite had no rows. suites_not_covered 196 -> 195; census 4278 rows /
3006 never / 1272 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_native_parquet_multifile.py carries all 32 checks by the shell's name
in three tests. Under each of nine mutations of the directory walk and glob
expansion, the twin reddens exactly the shell's set of checks: an empty
directory and an unmatched glob reading as nothing, an unstat-able path
dropped, a FIFO opened, _ and . names walked, a symlinked directory
descended, the files unsorted, no depth bound, and every extension read.

A read that could block runs on its own connection in a thread with a
deadline, as the shell runs psql under timeout. A cancel is no substitute:
a backend blocked in open(2) on a FIFO resumes the open after a signal. On a
timeout the twin opens the FIFO for writing, which releases the reader. The
FIFO arm is its own test, so it can decline under the shell's "the FIFO
case" when mkfifo is unavailable.

COMPLETE gains native_parquet_multifile. TESTS.md section 171,
cluster_tests 737 -> 740.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by nine walk and glob mutations (#1341)

The suite had no rows. suites_not_covered 195 -> 194; census 4310 rows /
3010 never / 1300 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ough the index (#1341, #1339)

native_index.sh forced its index path with enable_seqscan=off alone, which
leaves the columnar custom scan on. Measured: the range aggregate planned
as Custom Scan (PgColumnarScan), so it fetched nothing by row number. With
the index fetch reading the neighbouring row, or finding nothing, the old
suite's "index range aggregate" stayed green. iscan now also turns off the
custom scan, bitmap and index-only scans, and each lookup first asserts that
its plan is an Index Scan through n_pkey (pgc_uses_row_fetch). Both
mutations now redden it.

test_native_index.py carries all 13 checks by the shell's name in two
tests, and under both mutations it reddens exactly the shell's set.

COMPLETE gains native_index. TESTS.md section 172, cluster_tests 740 -> 742.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fetch mutations (#1341)

The suite had no rows. suites_not_covered 194 -> 193; census 4323 rows /
3017 never / 1306 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve, at 0c4edf8 (its own commits 7cd05c4 and 0c4edf8, on #1498).

My own mutant, different from your two: the deferred decode (pgcolumnar_slot_getsomeattrs, the #157 path an index fetch hands its slot to) returns k as NULL on every 7th row number. PG17:

  • shell: index range aggregate: got [301|644140|row-100|row-400] want [301|752500|row-100|row-400];
  • pytest: the same.

That's the arm your custom-scan fix moved onto the fetch path, so the fix is what makes it see this.

My own miss, for the record: my first attempt put the mutant at the end of pgcolumnar_index_fetch_tuple and stayed green. The slot is deferred there, so the later decode overwrote what I set: an inert mutant, not a gap in the suite. Moving it into getsomeattrs is what reached the runtime.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, merging #1499 now: approved at 0c4edf8, 15/15 green, CLEAN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants