Skip to content

test: native_fetch_cache gets its pytest twin, and section 1's comment says why its UPDATE fetches by row number (#1341) - #1493

Merged
jdatcmd merged 5 commits into
mainfrom
test/1341-native-fetch-cache-twin
Oct 16, 2026
Merged

jdatcmd merged 5 commits into
mainfrom
test/1341-native-fetch-cache-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd The next twin-less suite, stacked on #1489 (merge that first; this PR's own commits are the top two).

The twin, name for name

All 26 checks by the shell's name, in four tests. Each fixture is rebuilt before every reading, as the shell does (#801). Under each of ten mutations, the twin reddens exactly the shell's set:

mutation reds, shell = twin
every lookup misses (the cache retains nothing) 2: the #143 ratio and the #353 ratio
the key without the storage id 1
no earlier-command reject 1
recheck drops firstRowNumber / rowCount / fileOffset / natts 1 each
the discard dropped from pgcolumnar_executor_end / the xact callback 1 each
crossing the cap drops every column (the #359 cliff) 1: the #353 ratio
decode input kept in the column's context (#353) 0: equivalent on this fixture, because an unencoded chunk's stream already lives with the column

Five-major runs at head: 26/26 for the shell and 26/26 for the twin, on each of 15–19.

Section 1 was not in the custom-scan class, and I nearly said it was

Its UPDATE plans as Custom Scan (PgColumnarScan) under its SETs. I first "fixed" it, then measured: with the cache missing on every lookup, the unchanged suite is red at 844 ms against 93 ms, because the UPDATE fetches its target rows by number whatever the scan (as pgc_uses_row_fetch's comment in lib.sh says). So the shell change is one comment: it said "index-driven so every row goes through the fetch path rather than a scan", and now gives this reason. I've corrected #1339's audit to match.

An open finding: the #359 ratio can't see the cliff it names

"crossing the fetch cache cap costs proportionally, not totally (#359)" compares five projected columns with four. With the cap made to drop every column, as before #359, both sides slow, and the ratio stays under its 12× bound. Measured in-suite on PG18:

1 col 2 cols 3 cols 4 cols 5 cols
head 46 48 60 55 346 ms
every column dropped 286 286 574 530 992 ms

On current code the cliff hits even one projected column, so no ratio between widths of this fixture can show it. Comparing against a small-group table under the cap, as the #353 arm does, probably would, but it needs its own measurements. I've left the arm unchanged and never in the ledger rather than redesign it here.

Ledger

Census 4227 / 3009 never / 1218 dated (on #1489's 4201 / 2993 / 1208), with suites_not_covered 198 → 197. 10 of the 26 checks are dated. The gate gives rc 0, and orphan-scan finds 0. 16 arrive never:

Collection gives cluster_tests 726 -> 730, and TESTS.md gets §168. The guards pass 438/438 and the grader 40/40.

🤖 Generated with Claude Code

jdatcmd and others added 5 commits October 16, 2026 17:06
…through the index it was written for (#1341)

The suite forced an index scan with enable_seqscan=off, which leaves the
columnar custom scan on. Measured: the single-row fetch, the near and far
UPDATEs and the mirror join all planned as Custom Scan (PgColumnarScan), so
none of them fetched by row number. The rank off by one left the
single-row arm green, and the rank walking every earlier row left both
timing arms green. FORCE now also sets pgcolumnar.enable_custom_scan = off,
and four premises assert that each measured query's plan is an Index Scan
using its index.

The timing arms then still missed the walk, because they timed an UPDATE.
A columnar UPDATE writes each row again into a new row group, so only the
first of three passes fetched from the far end, and the median was always a
later pass. They now time a read of the same range, sum(fixed) and
sum(length(txt)), so every pass fetches the same positions. Measured at head
on 15-19: near and far within 1.03x, small and big group within 1.1x.
Measured with the rank walking: 50 ms against 779 ms, and 390 against 779.

test_native_fetch_position.py carries all 12 checks by the shell's name in
four tests. Under each of four mutations (rank counting rows, rank off by
one, rank walking, offset walking), the twin reddens exactly the shell's set
of checks. Like the shell, it reads PGC_FETCHPOS_ROWS.

COMPLETE gains native_fetch_position. TESTS.md section 167, cluster_tests
722 -> 726.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y mutation from #1485's review (#1341)

jdatcmd's mutation drops the NULL-key flag in the grouped fold's gather,
so the NULL group folds into k = 0 on the batch path only. Nine checks
redden, and the twin reddens the same nine by name. Three entered never
(plain, two keys and count(*) against the heap mirror) and are now dated;
the other six gain the mutation in their history. Census 4189 rows / 2969
never / 1220 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…our rank and offset mutations (#1341)

The suite had no rows. suites_not_covered 199 -> 198; census 4201 rows /
2975 never / 1226 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t says why its UPDATE fetches by row number (#1341)

test_native_fetch_cache.py carries all 26 checks by the shell's name in
four tests, rebuilding each fixture before every reading as the shell does
(#801). Under each of nine mutations, the twin reddens exactly the shell's
set of checks: the cache never hitting, the key without the storage id,
no earlier-command reject, each of three recheck fields dropped, the
discard dropped from executor end and from transaction end, and the cap
dropping every column rather than the one that crossed it.

Section 1 sets only enable_seqscan/enable_bitmapscan off, and its UPDATE
plans as Custom Scan (PgColumnarScan). It measures the cache anyway: the
UPDATE fetches its target rows by number whatever the scan. Measured with
the cache missing on every lookup, the unchanged suite is red at 844 ms
against 93 ms. Its comment said the opposite reason ("index-driven so every
row goes through the fetch path rather than a scan") and now says this.

COMPLETE gains native_fetch_cache. TESTS.md section 168, cluster_tests
726 -> 730.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cache mutations (#1341)

The suite had no rows. suites_not_covered 198 -> 197; census 4227 rows /
2991 never / 1236 dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd Restated onto #1489's new head fde6ab60. New head: e23416e4, pushed with a lease on 2d6842f2.

  • Code: outside the ledger files, the change from the old tip equals main's own diff.
  • Ledger: the seed's row changes are identical to the old commit's (sorted diff). Census 4227 / 2991 / 1236 (2975 + 16, 1226 + 10), with suites_not_covered 198 -> 197.
  • Checks: the gate gives rc 0, collection 730, and the guards 438/438.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve, at e23416e (its own commits 989fd32 and e23416e, on #1489's fde6ab6).

My own mutant, different from your ten: once a column has gone over the cap (entry->overflow[c]), the single-row fetch returns it as NULL: nulls[c] = entry->overflow[c]. The fetch that sets the flag stays right, so this is the quiet version of #359. PG17:

  • shell: every over-cap column reads back its written value (#359): got [2] want [100];
  • pytest: the same, got 2 want 100.

For the ledger, not blocking: that row arrives never in this PR, and this mutant dates it, if you want to fold it in.

A second observation: "the over-cap projection returns the same values as the under-cap one (#359)" stayed green under the same mutant. If both sides come through the same over-cap path on this fixture, it's the same shape as the #359-ratio finding you already flagged. Worth a look with that one, not here.

Also checked: your section-1 measurement (an UPDATE fetches by row number whatever the scan, 844 vs 93 ms with every lookup missing) is the right reason for keeping the shell's query and only correcting the comment.

@jdatcmd

jdatcmd commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, merging #1493 now: approved at e23416e, 15/15 green, CLEAN.

@jdatcmd
jdatcmd merged commit f11ffcc into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants