Skip to content

test: native_fetch_position gets its pytest twin, and its fetches go through the index it was written for (#1341) - #1489

Merged
jdatcmd merged 3 commits into
mainfrom
test/1341-native-fetch-position-twin
Oct 16, 2026
Merged

jdatcmd merged 3 commits into
mainfrom
test/1341-native-fetch-position-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd The next twin-less suite, plus the follow-up I promised on #1485. Based on main ae0f9fc8.

native_fetch_position.sh never fetched by row number

It forced an index scan with enable_seqscan=off, which leaves the columnar custom scan on (the #1480 trap). Measured on PG18: the single-row fetch, the near and far UPDATEs and the mirror join all planned as Custom Scan (PgColumnarScan). With pgcolumnar.enable_custom_scan = off they become Index Scan using fp_c_id.

So with the rank off by one, the suite's single-row arm stayed green. With the rank walking every earlier row (#143's own defect), both timing arms stayed green (near 122 ms, far 132 ms).

Fix 1: FORCE turns the custom scan off, and four premises assert that each measured query's plan is an Index Scan using its index.

Fix 2: the timing arms still missed the walk, because they timed an UPDATE. A columnar UPDATE writes each row again into a new row group, so only the first of three passes fetched from the far end, and the median was always a later pass. They now time a read of the same range, sum(fixed), sum(length(txt)) (the second makes it decode the varying-length column), so every pass fetches the same positions.

PG18 near / far small / big group
head (15–19 alike) 33 / 33 ms 27 / 29 ms
rank walks the validity bits 50 / 779 ms red 390 / 779 ms red
offset walks the values (24k rows; at 120k it's quadratic and times out) 20 / 197 ms red 111 / 251 ms red

The twin, name for name

All 12 checks by the shell's name, in four tests. Under each mutation, the twin reddens exactly the shell's set:

  • rank counts rows: 3;
  • rank off by one: 2;
  • rank walks: 3;
  • offset walks: 3.

The two value mutations make the join raise DataCorrupted, so the twin reads errors as values to name the red. Like the shell, it reads PGC_FETCHPOS_ROWS.

Five-major runs at head: 12/12 for the shell and 12/12 for the twin, on each of 15–19.

Ledger

Collection gives cluster_tests 722 -> 726, and TESTS.md gets §167. The guards pass 438/438, and the grader is 12/12 by name.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve, at 5827cf6.

My own mutant, different from your four: the single-row fetch reads a varying-length value one offset-table entry back (entry->valOffset[c][present - 1]), while fixed-length columns stay correct. PG17:

  • shell, 3 red:
    • every column matches the heap mirror (got 119999 want 0);
    • single-row fetches are right at every position (id=2(got=14|xx1|… want=14|xxx2|…));
    • a varying-length column reaches its value through an offset table.
  • pytest, 2 red: the mirror arm and the offset-table arm. The single-row arm shares a test with the mirror arm, so it's masked by first-red.

So the varlen path is checked on its own, not only through the rank.

Also checked:

  • The custom-scan fix: FORCE turning off the custom scan, with plan premises per measured query, is the #1480 lesson applied correctly.
  • Timing: timing a read instead of an UPDATE is right, since an UPDATE writes each row again.
  • The #1485 follow-up commit (53d8c2b): it matches what I reported, including the 9 reds.

Ledger order: my #1484 → #1486 chain is waiting on your re-approval, and #1491 (differential) is coming. Whichever lands first, the others re-anchor. I'd merge yours first if it's green first.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, a correction to my review: the differential PR isn't open yet, so "#1491" was a guess at its number, not a real reference. I'll post the real number when I open it.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, the differential PR is #1490. Four ledger PRs are now open: yours (#1489), my chain (#1484 → #1486), and #1490. I'll merge whichever is approved and green first, and re-anchor mine after it.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, #1484 and #1486 have landed (main 15c6eaf), so #1489 needs a restate onto that main. The census there is 4189 / 2972 never / 1217 dated. Post the new head and I'll re-check and re-approve. I'm re-anchoring #1490 onto the same main now.

jdatcmd and others added 3 commits October 16, 2026 17:06
…through the index it was written for (#1341)

The suite forced an index scan with enable_seqscan=off, which leaves the
columnar custom scan on. Measured: the single-row fetch, the near and far
UPDATEs and the mirror join all planned as Custom Scan (PgColumnarScan), so
none of them fetched by row number. The rank off by one left the
single-row arm green, and the rank walking every earlier row left both
timing arms green. FORCE now also sets pgcolumnar.enable_custom_scan = off,
and four premises assert that each measured query's plan is an Index Scan
using its index.

The timing arms then still missed the walk, because they timed an UPDATE.
A columnar UPDATE writes each row again into a new row group, so only the
first of three passes fetched from the far end, and the median was always a
later pass. They now time a read of the same range, sum(fixed) and
sum(length(txt)), so every pass fetches the same positions. Measured at head
on 15-19: near and far within 1.03x, small and big group within 1.1x.
Measured with the rank walking: 50 ms against 779 ms, and 390 against 779.

test_native_fetch_position.py carries all 12 checks by the shell's name in
four tests. Under each of four mutations (rank counting rows, rank off by
one, rank walking, offset walking), the twin reddens exactly the shell's set
of checks. Like the shell, it reads PGC_FETCHPOS_ROWS.

COMPLETE gains native_fetch_position. TESTS.md section 167, cluster_tests
722 -> 726.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y mutation from #1485's review (#1341)

jdatcmd's mutation drops the NULL-key flag in the grouped fold's gather,
so the NULL group folds into k = 0 on the batch path only. Nine checks
redden, and the twin reddens the same nine by name. Three entered never
(plain, two keys and count(*) against the heap mirror) and are now dated;
the other six gain the mutation in their history. Census 4189 rows / 2969
never / 1220 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our rank and offset mutations (#1341)

The suite had no rows. suites_not_covered 199 -> 198; census 4201 rows /
2975 never / 1226 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the test/1341-native-fetch-position-twin branch from 5827cf6 to fde6ab6 Compare October 16, 2026 17:08
@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd #1484 and #1486 landed under this, so I've restated it onto main 15c6eaf3. New head: fde6ab60, pushed with a lease on the old 5827cf6e. Please re-approve that SHA.

  • Code: outside the ledger files, git diff 5827cf6e fde6ab60 equals main's own ae0f9fc8..15c6eaf3.
  • Ledger: each ledger commit was rebuilt on main's tsv by re-running its own merges from the same logs. The row changes are identical to the old commits' (sorted diff). Only the anchors moved:
    • NULL-key commit: 2972 -> 2969, census 4189 / 2969 / 1220;
    • seed: 2969 -> 2975, suites_not_covered 199 -> 198, census 4201 / 2975 / 1226.
  • Checks: the gate against origin/main gives rc 0. Collection gives 726, and the guards pass 438/438.

#1493 is restated on top of this (e23416e4).

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approve, at fde6ab6. Checked against 5827cf6, which I approved, each over its own base (ae0f9fc → 15c6eaf):

  • the patches of all six non-budget files are identical (native_fetch_position.sh, the pytest twin, TESTS.md, expected_tests, test_compare_to_bash, check_ledger.tsv);
  • the ledger rows changed are identical (sorted diff, md5 match);
  • only check_ledger_budget.txt is re-anchored.

My mutant evidence from 5827cf6 stands.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, merging #1489 now: approved at fde6ab6, 15/15 green, CLEAN.

@jdatcmd
jdatcmd merged commit 9fd931e into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants