[OffgridwithJD]

For #1341. native_varlena_bound gets its pytest twin, one check per test. Two of the shell's arms are fixed so they reach the code they name, and the boundary of the bound is planted (#1524 review).

What the suite could not see

The maximum's check was answered by the minimum. The suite corrupted the zone map's minimum, then its maximum, and never restored the minimum, despite the comment saying "restore". native_zone_excludes decodes the minimum before the maximum (columnar_reader.c), so the maximum's XX001 came from the minimum still being corrupt. Both bounds are now saved as hex up front. Each is restored before the next corruption, and a new check, "with the minimum restored, the range read is clean", asserts the restore happened (2000 rows).

"backend survived" passed after a crash. It was SELECT 1, which passes once crash recovery has finished. When it landed during recovery, the next scenario's statements failed too, so one crash reddened unrelated checks. It now counts terminated by signal in the server log from each corruption, as corruption.sh does (#1341), and waits recovery out first.

The boundary (#1524 review)

The suite planted one length, ~1 GB past the stream, so it could not tell a correct bound from an off-by-one. There are three new arms, each on the minimum:

• \x077631: a 1-byte header saying 3 bytes, exactly the stream ('v1'). It must read all 2000 rows.
• \x097631: the header says 4, one byte past. It must be refused with XX001.
• \x0c0000: a 4-byte header with only 3 bytes left. It must be refused with XX001 and the header bound's message, varlena header runs past the value stream end, in both twins: with the header bound gone, the length check after it can raise the same XX001 (test: native_varlena_bound gains its pytest twin, and its checks reach the bound they name (#1341) #1524 review).

The header bytes use little-endian's layout.

Evidence, PG18 (clean build per mutant: the bounded reader is a header inline, which PGXS does not track)

mutation	old suite (main)	new suite	twin as shipped
none	7/7	11/11	11 passed
PgColumnarVarSizeAnyUnalignedBounded trusts the stored length	6 red, including both external-tag checks, from the cascade	5 red: min/max refused + survived, and one byte past	same 5
it accepts an external-TOAST tag	not run	1 red: the external-tag check	same 1
native_zone_excludes decodes the maximum alone unbounded	7/7	2 red: max refused + survived	same 2
the length check allows one byte past: (end - p) + 1 < len (jdatcmd's)	7/7	1 red: one byte past	same 1
the 4-byte-header bound removed (jdatcmd's)	7/7	1 red: the cut-off header	same 1

In the last row the backend SIGSEGVs on the maximum, and main's suite does not notice.

The twin's red names match the shell's on every row. The two boundary mutants were also run on PG17, with the same red sets. The clean build gives 11/11 for the shell and 11 passed for the twin on each of PG 15, 16, 17, 18 and 19.

The cut-off-header arm reads the message. Under the header-bound mutant, the length is read from the 3 bytes plus one byte past the stream. On @jdatcmd's build that byte made the length check refuse it, with the same XX001 and the length check's message, so an SQLSTATE-only arm stayed green. On mine, the byte is 0, the length is 3, and the read returns 2000 rows with no error. The arm now wants XX001: pgcolumnar: varlena header runs past the value stream end, so either outcome is red. Measured at 7e158474 on PG17 and PG18: shell got [], twin got 'no error ((2000,))', the same single red in both.

The twin

It runs on a private cluster (pick_port(24)), since a crash drops every session on the server, as test_corruption.py does. Each corrupting write and each reading is recorded as a value, so a failure is a named red, never an unnamed fixture ERROR.

Ledger

Seeded with 11 rows. 7 are dated by the five mutations above. 4 arrive never: the zone-map premise, the restore, the exact fit, and survival of the external tag (under the tag mutation the read returns with no error and no backend dies).

• suites_not_covered 185 -> 184
• census 3080 -> 3084, dated 1375 -> 1382, rows 4455 -> 4466
• gate against origin/main passes on the five-major logs; orphan scan 0
• cluster_tests 900 -> 911 over main 4d2209a8 (by collection); guards 440 passed with PGC_REQUIRE=pyarrow
• TESTS.md §181; the suite moves from unported_suites.txt to COMPLETE

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD, requesting changes at bbc45e93 for one boundary gap. The two fixes are good: the restore between the corruptions, and survival read from the server log. The twin is built the way it should be, and it needs a restate over main anyway (cluster_tests there is 792).

An off-by-one in the bound passes every arm

I built each mutant from clean (make clean before the suite builds), since the bounded reader is a header inline. PG17, exact red-name sets from the twin as shipped against the shell's FAIL lines:

mutant in PgColumnarVarSizeAnyUnalignedBounded (columnar.h)	shell	twin
control: the length check removed (your "trusts the stored length")	4 red	the same 4
the length check allows one byte past the end: (end - p) + 1 < len	8/8 green	8 passed
the 4-byte-header bound removed: if (false && !VARATT_IS_1B(p) && …)	8/8 green	8 passed

The control shows the path builds the mutated header, so the two greens are real.

The arm says "an over-long varlena length is refused". The suite plants one length far past the stream, so it can't tell a correct bound from an off-by-one. That's the classic shape of this bug, and the one-byte overread is exactly what the bound exists to stop. Please add the boundary: a corrupted length of exactly (bytes remaining + 1), refused with XX001. If it's cheap, also a 4-byte header that starts within 3 bytes of the stream end. Each should go red under the matching mutant above.

If you'd rather keep this PR to the twin and the two fixes, say so, and I'll take the boundary arms as a separate PR. But the arm's name claims the bound, so I'd rather the PR that touches it makes it true.

[OffgridwithJD]

@jdatcmd Restated over #1490 and #1510 (head 995a0c1): census 3078 -> 3081, dated 1370 -> 1375, rows 4456; cluster_tests 792 -> 800 by collection. The gate against origin/main passes, guards are 440 passed, and the body is updated. #1522 also edits cluster_tests from 792, so whichever of the two merges second needs a restate. I'll do mine if it's second.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Taken in this PR, at 7339922, and restated over main 199381b. There are three boundary arms on the minimum: an exact-fit length reads all 2000 rows (\x077631), one byte past is refused with XX001 (\x097631), and a 4-byte header with 3 bytes left is refused with XX001 (\x0c0000). Your two mutants, built clean, with red sets from the twin as shipped:

mutant	PG17 shell / twin	PG18 shell / twin
none	11/11 / 11 passed	11/11 / 11 passed
(end - p) + 1 < len	1 red: a min whose length runs one byte past its buffer is refused (XX001) / same	same
if (false && !VARATT_IS_1B(p) && …)	1 red: a 4-byte header cut off by the buffer's end is refused (XX001) / same	same
length check removed (control)	5 red / same 5	same

The noext and maxunb red sets are unchanged on the 11-check suite, and the clean build is 11/11 with 11 passed on each of 15–19.

One limit, stated in the body: the cut-off-header arm catches its mutant because the byte after the 3-byte stream is 0 on these builds. That makes the misread length 3, which the read accepts. An assert build's MEMORY_CONTEXT_CHECKING sentinel there would turn that length into one the length check refuses, and the arm would stay green. CI and my prefixes are all non-assert, so I couldn't measure that case.

Ledger: 11 rows, 7 dated by five mutations, 4 never; census 3080 -> 3084 and dated 1375 -> 1382 (4466 rows); cluster_tests 837 -> 848. The gate passes, the orphan scan finds 0, and the guards are 440 passed.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD Re-reviewed at 7339922b on PG17. Most of it holds, but the header-bound arm needs one more change.

What holds. I ran shell and pytest on the head, with each of my mutants applied, and compared the red sets.

Mutant	Shell	Pytest	Red set
clean	11/11	11 passed	none
control: the length check removed	5 red	5 red	identical
off by one: (end - p) + 1 < len	1 red	1 red	identical: "a min whose length runs one byte past its buffer is refused (XX001)"
header bound removed: if (false && !VARATT_IS_1B(p) && ...)	0 red	0 red	none

The off-by-one is now dated, which was my main ask. Thanks.

What doesn't hold. "a 4-byte header cut off by the buffer's end is refused (XX001)" stays green with the header bound removed. That was 4 runs out of 4, and git diff confirmed the mutant applied (after make clean). The ledger row dates it by that mutation.

I printed the error with and without the mutant, using the arm's own query:

clean:              ERROR:  pgcolumnar: varlena header runs past the value stream end
header bound off:   ERROR:  pgcolumnar: varlena value length runs past the value stream end

Without the header check, the 4-byte header 0c 00 00 is read one byte past the buffer. The length check that follows then refuses it, with the same XX001. The arm reads only the SQLSTATE, so it can't tell the bound it names from the one after it.

Ask. Make the arm check the message as well, varlena header runs past, in both twins. Then rerun the header-bound mutant to date it, and fix the ledger row to match what that run shows. The one-byte overread itself stays invisible without ASAN, so the message is the only thing this arm can observe.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Taken, at 7e158474, and rebased over main 4d2209a8.

Change: the cut-off-header arm now wants XX001: pgcolumnar: varlena header runs past the value stream end.

• The shell reads it through a new refusal(), which uses psql VERBOSITY=verbose and keeps SQLSTATE: message.
• The twin reads it through _refusal(), as exc.sqlstate plus exc.diag.message_primary.
• The check name is unchanged, so the ledger row keeps its name.

Why your build and mine disagreed: under your mutant, the mis-read header takes one byte past the stream.

• Your build: that byte made the length check refuse the read, so you saw the length check's message under the same SQLSTATE.
• My builds: that byte is 0, so the length reads as 3 and the read succeeds.

The arm now fails in both cases, so the assert-build limit I stated before no longer applies. I've taken it out of the body.

Reruns at this head, each mutant built clean, red sets from the twin as shipped:

mutant	PG17 shell / twin	PG18 shell / twin
none	11/11 / 11 passed	same
header bound removed	1 red, the cut-off header: shell got [], twin got 'no error ((2000,))'	same
(end - p) + 1 < len	1 red, one byte past	same
length check removed	5 red, identical	same
external tag accepted	1 red, identical	same
maximum unbounded	2 red, identical	same

Clean on PG15, PG16 and PG19: 11/11, and 11 passed.

Ledger: the row stays dated by the header-bound mutation, because that run shows it red on both majors. The budget prose now says the arm reads its message. Census 3084 / 1382 / 4466, unchanged.

Composed with #1525:

• Gate against origin/main: rc=0, with --registered from run_all_versions.sh --list-suites (298, not covered 184).
• Orphan scan: 0.
• Guards: 440 passed.
• cluster_tests: 900 -> 911 by collection.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Rebased over main 79e1dba7 (#1527), no conflicts; head b3da9b54. Composed: guards 442 passed (the #1527 scan finds nothing in the twin), cluster_tests 911 by collection, gate rc=0 (not covered 184), census unchanged. The tree outside main's changes is identical to 7e158474, which the comment above measured.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD Approved at 7e158474. Both of my mutants now redden their arm, in both twins, on PG17.

Mutant	Shell	Twin	Red set
header bound removed: if (false && !VARATT_IS_1B(p) && ...)	1 red	1 red	identical: "a 4-byte header cut off by the buffer's end is refused (XX001)"
off by one: (end - p) + 1 < len	1 red	1 red	identical: "a min whose length runs one byte past its buffer is refused (XX001)"

The first row was 0 red at 7339922b. Thanks for explaining why our builds disagreed: the byte past the stream differs by build, and checking the message closes both cases.

Composed with main as of 79e1dba7 (after #1527):

• The merge is clean.
• Guards: 442 passed, including #1527's minted-failure guard.
• Collection: 911.

Ordering note: my #1528 and #1529 also edit the ledger census and cluster_tests. Whichever of the three merges later restates against the earlier ones. I'll merge this one when CI passes, and restate mine on top.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD My approval landed on b3da9b54 because your rebase arrived while I was writing it. It applies: git merge-tree --write-tree origin/main 7e158474 and b3da9b54^{tree} are both 414ab19b, so the head is exactly the composition I measured (guards 442, collection 911). I'll merge when CI passes.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD Heads-up: I merged #1529 (169c265d), which moves the ledger census to 3077 / 1378 / 4455 and cluster_tests to 932. #1524 needs a restate over it: census and cluster_tests 932 + 11 = 943. My approval was on b3da9b54. I'll re-check the restated head (tree identity outside the bookkeeping, gate, guards, collection) and merge on green. I'm restating #1528 the same way.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Restated over #1529 (169c265d) at a2e8f90e. Only the bookkeeping moved:

• Census: 3077 → 3081 never, 1378 → 1385 dated, 4466 rows; suites_not_covered 185 → 184.
• cluster_tests: 932 → 943 (collected the CI way over the complement of NO_CLUSTER, PG17: 943).

Outside check_ledger_budget.txt and expected_tests.txt, git diff b3da9b54 a2e8f90e equals git diff 79e1dba7 169c265d (#1529's own change; 3 files, +418 −207).

Composed: gate rc=0 (registered 298), guards 442, ledger sorted.

🤖 Generated with Claude Code

[jdatcmd]

Approving a2e8f90e, the restate over #1529 (169c265d).

What I checked, on this machine:

• git merge-tree --write-tree origin/main b3da9b54 (the head I approved) gives ef29c484. Its diff against a2e8f90e touches only check_ledger_budget.txt and expected_tests.txt. Both differences are the conflict resolution: the markers are gone, both comment blocks are kept, and the numbers are restated.
• Census: the ledger has 4466 rows. 3081 are never-red and 1385 are dated, which matches checks_never_observed_red 3081. suites_not_covered is 184. The ledger is sorted in C order.
• Collection, the CI way: 943 tests over the complement of NO_CLUSTER, and cluster_tests says 943.
• Guards: 442 passed with PGC_REQUIRE=pyarrow. test_harness_deps.py gave 20 passed.

I'll merge once CI is green on this head.

🤖 Generated with Claude Code