Skip to content

test: native_backend_crash's post-flush kill runs first, so a crashed write's leftover blocks cannot hide its write (#1757) - #1760

Merged
jdatcmd merged 1 commit into
mainfrom
test/1444-crash-reservation-flush
Oct 26, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
test/1444-crash-reservation-flush

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd This fixes the one remaining red on #1757. #1759's dispatch (run 40376459928) went green on the capability sweep. Its only failure was pytest (cluster tests, PG 17), on one premise in my #1444 twin:

FAILED test_native_backend_crash.py::test_a_kill_after_the_stripes_flushed - AssertionError:
crashB: the INSERT had written pages before the kill (the relation grew): got 'False' want 'True'

Cause: the scenario order, not the product

  • A killed write's stripe reservation (the metapage highwater, PgColumnarReserveOffset) is WAL-logged but unflushed, while its pages already extend the file.
  • If the walwriter hasn't flushed it by the SIGSEGV, redo restores the old highwater over blocks that physically exist.
  • The next write fills those blocks before it extends anything. crashA kills a 3M-row INSERT once the relation grows, and crashB's 100k rows then fit entirely inside crashA's leftover, so the relation never grows.

Reusing blocks no durable reservation covers is correct: every content, row-group and mirror check passed in that run.

Measured on local PG 17: crashA's pattern, then a committed 100k-row INSERT.

wal_writer_delay relation grew after the 100k INSERT
10s (unflushed at the kill) no, 3 of 3 runs
200ms (default) yes, 3 of 3 runs

Fix, in both suites

  • Pin wal_writer_delay = 10s (its maximum) in the private cluster, so the race always goes the losing way instead of depending on runner load.
  • Run crashB first, where its only predecessor is the committed baseline, whose commit flushed its reservation. crashA and crashC write 3M rows, which outgrow any leftover.

Evidence (PG 17)

state shell pytest
pin only, old order (red-first) 27 passed + 1 failed, crashB premise 19 pass + 1 fail, crashB premise
pin + reorder 5 of 5 green, 28 checks 5 of 5 green, 28 checks
  • Tried and rejected: a WAL flush from an observer just before the kill (committing an xid). The writer can reserve another stripe between the flush and the kill, and the shell went red 1 time in 3.
  • Gates: guards 847 passed. Collection 4332 equals cluster_tests. Census 4567 equals the tsv. Ledger gate rc=0. Harness self-test passed.
  • Ledger: no check is added, renamed or redated, so the ledger and census are untouched.

Once this merges, a dispatch of nightly.yml on main should close #1757.

🤖 Generated with Claude Code

… write's leftover blocks cannot hide its write (#1757)

The nightly on PG 17 (run 40376459928) failed one premise in the pytest twin:
"crashB: the INSERT had written pages before the kill (the relation grew)", got
False. The cause is the scenario order, not the product.

A killed write's stripe reservation (the metapage highwater) is WAL-logged but
not yet flushed, while its pages already extend the file. If the walwriter has
not flushed it by the SIGSEGV, redo restores the old highwater over blocks that
exist, and the next write fills those blocks before it extends anything. crashA
kills a 3,000,000-row INSERT once the relation grows, and crashB's 100,000 rows
then fitted entirely inside crashA's leftover. Reusing blocks that no durable
reservation covers is correct, and every content check passed.

Measured on a local PG 17 with crashA followed by a committed 100,000-row INSERT:
- wal_writer_delay=10s: the relation did not grow, 3 runs of 3.
- wal_writer_delay=200ms: it grew, 3 runs of 3.

Both suites now:
- hold the WAL writer off (wal_writer_delay = 10s, its maximum), so the race
  always goes the losing way instead of depending on runner load;
- run crashB first, where it follows only the committed baseline. crashA and
  crashC write 3,000,000 rows, which outgrow any leftover.

With the pin alone, on the old order, both are red on crashB's premise: the shell
27 passed + 1 failed, pytest 19 pass + 1 fail. With the reorder, both are green
5 runs of 5 (28 checks each).

A flush before the kill (committing an xid from an observer) was tried and does
not close it. The writer can reserve a further stripe between the flush and the
kill; the shell went red once in 3.

No check is added, renamed or redated, so the ledger and census are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on 3723d97c.

First, a correction. On #1283 I wrote that space reuse is not the cause. I was wrong. My probe ran with the default wal_writer_delay (200 ms). The walwriter flushed the reservation before every kill, so I saw no gap in 12 crashes and dropped the idea. Your 10 s pin is the control I lacked.

What I reproduced (local PG 18, pytest twin, main 6ae11a38 plus this PR):

state result
main's file plus the wal_writer_delay = '10s' pin only, old order red 4 of 4: crashB: the INSERT had written pages before the kill (the relation grew): got 'False' want 'True', the same message as the nightly
this PR's file green 5 of 5, 3 passed each

What I checked by reading:

  • The order holds in CI. pytest-cluster runs pytest without -n, so module order is collection order. The shell runs top to bottom.
  • crashA and crashC still prove their premise after the reorder. Each waits for pg_relation_size to pass size0, and a 3M-row write outgrows any leftover from an earlier kill. So their premise checks still mean what they say.
  • The pin is set once in the private cluster (nbc / the suite's own setup), so no other suite inherits it.

Not verified by me: the shell twin. The only PG 17 install here is root-owned and holds an older build. I'm relying on CI's suites jobs for it, and I'll merge only at 15/15.

On the follow-up: yes, please surface the INSERT's error in both twins. It wasn't the cause this time, but a swallowed error would leave the next red unexplained.

@jdatcmd
jdatcmd merged commit 5d5d55f into main Oct 26, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly deep gate is red

2 participants