Skip to content

test: corruption gets its pytest twin, and a backend crash can no longer pass for a refusal (#1341 item 25) - #1446

Merged
jdatcmd merged 4 commits into
mainfrom
test/1341-corruption-twin
Oct 16, 2026
Merged

jdatcmd merged 4 commits into
mainfrom
test/1341-corruption-twin

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

#1341 item 25, its third and last suite: a corrupt catalog row is refused cleanly, or degrades safely, and no backend dies. corruption.sh had no pytest twin. This PR adds one, every check by the shell's name, and seeds the suite into the ledger. The port found three kinds of arm in the shell that couldn't fail as claimed, and they are fixed in both twins.

Stacked on #1444. Merge #1444 first; this branch is #1444's head plus two commits.

What changes

  • test/pytest/test_corruption.py, the twin. It has seven tests, one per corrupted field, on a private cluster, since a crash drops every session. compare_to_bash.py grades it as covering every shell name, and it joins COMPLETE.
  • A backend crash could pass for a clean refusal.
    • Why: errors said yes to any failure of psql, and a crash is one. alive connected after the postmaster's crash recovery, so it passed whenever recovery finished first.
    • Measured: with the bad-descriptor refusal replaced by abort(), the old suite caught the crash only because alive reached the server mid-recovery. With alive 3s later, it was 12/12.
    • The fix: a refusal must now be a clean XX001, read from psql's VERBOSITY verbose output, or from exc.sqlstate in the twin. A crash prints no SQLSTATE. alive also requires that no backend has died since the scenario began, counted from "terminated by signal" in the server log, which doesn't depend on timing.
  • Each corrupting UPDATE was assumed to change a row.
    • Measured: with every UPDATE matching nothing (a storage id that doesn't exist), the old suite failed its four refusal arms, but the value_count, column_index and bloom scenarios passed on tables nobody had corrupted.
    • The fix: each scenario now asserts premise: the <field> corruption changed a catalog row.
  • The bloom arm couldn't see a false negative. It looked up an absent value, which is 0 rows whether the corrupt filter is ignored or wrongly says "absent".
    • Measured: with a malformed header read as "absent", which loses rows, the old suite was 12/12.
    • The fix: a new arm looks up a value the table holds: corrupt bloom: a value the table holds is still found.
  • Ledger: corruption is seeded with 20 rows from five-major runs, and 11 of the 20 rows dated by the mutants below. suites_not_covered goes 220 → 219, and the census is 3592 + 20 = 3612 rows, 2664 + 9 = 2673 never and 928 + 11 = 939 dated, matching the tsv. The gate gives rc=0 against origin/main.

Red first

Each mutation was run one at a time on PG18, through both twins, with the .so md5 printed every time.

mutation corruption.sh test_corruption.py the arms that went red
none (the head) 20/20 20/20 none
the group-span check removed 19/20 1 of 7 tests fails byte_length: got [no error (20000|…)] want [XX001]
the span refusal → abort() 18/20 1 fails byte_length's refusal and its alive: no error (server closed the connection unexpectedly), no (1 backend crashes)
the chunk-bound refusal → abort() (both sites) 17/20 1 fails page_length's refusal and alive, and value_count's alive
the row accounting and the value-stream bound, both removed 18/20 1 fails row_count's refusal and alive: a real SIGSEGV, as the source comment at the accounting check describes
the descriptor refusal → abort() 18/20 1 fails the descriptor's refusal and alive
a malformed bloom header read as "absent" 19/20 1 fails corrupt bloom: a value the table holds is still found: got [0] want [1] (old suite: 12/12)
the bloom length check removed 18/20 1 fails the held-value lookup and bloom's alive (a crash)

Each one stays green alone:

  • the row accounting check removed;
  • both chunk bounds removed;
  • the descriptor header check removed;
  • the column_index bounds removed.

Each of these corruptions meets a second guard that still refuses it cleanly. That's why the refusal → abort() mutants are the ones that test the arms: they replace the refusal that actually fires.

Harness mutations:

  • every corrupting UPDATE matching nothing: the 7 premises go red in both twins;
  • alive 3s after the abort mutant: still red, where the old suite was 12/12.

Never red (9):

  • the seven premises, which go red only under the harness mutation;
  • column_index's liveness, which no single bound reddens;
  • the absent-value bloom lookup.

Runs

major corruption.sh test_corruption.py
15 20/20 20/20
16 20/20 20/20
17 20/20 20/20
18 20/20 20/20
19 20/20 20/20

Also measured: the guard half (438 tests). cluster_tests goes 644 → 651 by collection, and TESTS.md gets §138.

🤖 Generated with Claude Code

@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

Also measured at this head, which contains #1444: harness_selftest 1245/1245 and docs_style PASSED, on PG18.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 8a4d96e.

At the head, PG17: corruption.sh 20/20; test_corruption.py 20 checks, 7 tests. The census at this head (3612 / 2673 / 939) matches the budget, and suites_not_covered is 219.

Your two central claims, with mutants of my own:

mutant shell pytest
the "unrecognized native encoding descriptor" refusal → abort() 18/20: bad descriptor errors, got [no error (server closed the connection unexpectedly)]; alive after bad descriptor, got [no (1 backend crashes)] bad descriptor errors, no error (consuming input failed: …)
both malformed-header returns in PgColumnarBloomProbe read as "absent" (return false) 19/20: corrupt bloom: a value the table holds is still found, got [0] want [1] the same arm, got 0 want 1

So a crash can no longer pass for a refusal, and a corrupt filter that loses rows is now caught. I restored the source with a make clean, and git status was clean.

As in the earlier twins, not a blocker: pytest stops at the first failed expect, so under the abort mutant it never reaches alive (19 checks run). The shell records both.

I didn't repeat your other six mutants; I took them from your table.

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving 81f0901, a restate of 8a4d96e (which I approved) onto main e04e1e0. Every file's patch is identical to the approved head's, with TESTS.md section numbers masked, since main moved only by #1445, which touches no ledger file. The census at this head matches its budget paragraph.

jdatcmd and others added 4 commits October 16, 2026 01:48
… must follow a write (#1341 item 25)

test_native_backend_crash.py asserts every shell check by name; compare_to_bash grades
it as covering all of them, and it joins COMPLETE. It runs on a private cluster,
since the reinit drops every session on the server, and a session a failed check
leaves running is terminated before its connection closes.

The mid-write kill waited a fixed 0.5s, and nothing showed the INSERT had written a
page by then. With the inserting session's stripe_row_limit above its row count, so
that the write buffers whole until its end, the kill landed on a write that had
produced nothing and the suite stayed 25/25. The kill now waits for the relation to
grow, and each crash asserts that its INSERT had written pages first; under the same
setting the mid-write arms go red, as the kill can no longer land mid-write.

cluster_tests 643 -> 646 by collection; TESTS.md gets section 137.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… unwritten WAL record (#1341)

28 rows from five-major runs (28/28 on each of 15-19). Dated on PG18 by a stripe
data page written with no WAL record: the three "committed content in c matches the
heap mirror" arms. The sentinel, second-reader, kill-phase and relation-grew arms
redden only under harness mutations and stay never, as do the MVCC arms.
suites_not_covered 221 -> 220; the census is 3599 rows, 2669 never, 930 dated, restated on main a4eef27.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ger pass for a refusal (#1341 item 25)

test_corruption.py asserts every shell check by name; compare_to_bash grades it as
covering all of them, and it joins COMPLETE. It runs on a private cluster, since a
crash drops every session.

Three kinds of arm could not fail as claimed:
- `errors` said yes to any failure of psql, and a crash is one; `alive` connected
  after crash recovery and passed whenever recovery won the race. With the
  bad-descriptor refusal replaced by abort() and `alive` 3s later, the suite was
  12/12. A refusal must now be a clean XX001, and `alive` also requires that no
  backend died since the scenario began, read from the server log.
- each corrupting UPDATE was assumed to change a row. With every UPDATE matching
  nothing, the value_count, column_index and bloom scenarios passed on tables nobody
  corrupted. Each now asserts that it changed a catalog row.
- the bloom arm looked up an absent value, which is 0 rows even when a corrupt
  filter wrongly says "absent". With a malformed header read as "absent", the suite
  was 12/12; a value the table holds must now still be found.

cluster_tests 646 -> 653 by collection; TESTS.md gets section 138.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions of the reader and bloom probe (#1341)

20 rows from five-major runs (20/20 on each of 15-19). Dated on PG18 by: the group-
span check removed (1); the span (2), chunk-bound (3) and descriptor (2) refusals
each replaced by a crash; the row accounting and value-stream bound removed (2); a
malformed bloom header read as absent (1); the bloom length check removed (2).
suites_not_covered 220 -> 219; the census is 3619 rows, 2678 never, 941 dated, restated on main a4eef27.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the test/1341-corruption-twin branch from 81f0901 to ea88468 Compare October 16, 2026 01:51

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving ea88468, a restate onto a4eef27 (#1449). Every test and suite file's patch is identical to the head I approved, with TESTS.md section numbers masked. Only the budget and expected_tests.txt moved, re-anchored on #1449's numbers. The census at this head matches its budget paragraph, and cluster_tests steps 643 → 646 → 653 → 660 along the chain.

@jdatcmd
jdatcmd merged commit ff1f6aa into main Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants