Repository navigation
test: native_backend_crash's post-flush kill runs first, so a crashed write's leftover blocks cannot hide its write (#1757) - #1760
Conversation
… write's leftover blocks cannot hide its write (#1757) The nightly on PG 17 (run 40376459928) failed one premise in the pytest twin: "crashB: the INSERT had written pages before the kill (the relation grew)", got False. The cause is the scenario order, not the product. A killed write's stripe reservation (the metapage highwater) is WAL-logged but not yet flushed, while its pages already extend the file. If the walwriter has not flushed it by the SIGSEGV, redo restores the old highwater over blocks that exist, and the next write fills those blocks before it extends anything. crashA kills a 3,000,000-row INSERT once the relation grows, and crashB's 100,000 rows then fitted entirely inside crashA's leftover. Reusing blocks that no durable reservation covers is correct, and every content check passed. Measured on a local PG 17 with crashA followed by a committed 100,000-row INSERT: - wal_writer_delay=10s: the relation did not grow, 3 runs of 3. - wal_writer_delay=200ms: it grew, 3 runs of 3. Both suites now: - hold the WAL writer off (wal_writer_delay = 10s, its maximum), so the race always goes the losing way instead of depending on runner load; - run crashB first, where it follows only the committed baseline. crashA and crashC write 3,000,000 rows, which outgrow any leftover. With the pin alone, on the old order, both are red on crashB's premise: the shell 27 passed + 1 failed, pytest 19 pass + 1 fail. With the reorder, both are green 5 runs of 5 (28 checks each). A flush before the kill (committing an xid from an observer) was tried and does not close it. The writer can reserve a further stripe between the flush and the kill; the shell went red once in 3. No check is added, renamed or redated, so the ledger and census are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd
left a comment
There was a problem hiding this comment.
Approved on 3723d97c.
First, a correction. On #1283 I wrote that space reuse is not the cause. I was wrong. My probe ran with the default wal_writer_delay (200 ms). The walwriter flushed the reservation before every kill, so I saw no gap in 12 crashes and dropped the idea. Your 10 s pin is the control I lacked.
What I reproduced (local PG 18, pytest twin, main 6ae11a38 plus this PR):
| state | result |
|---|---|
main's file plus the wal_writer_delay = '10s' pin only, old order |
red 4 of 4: crashB: the INSERT had written pages before the kill (the relation grew): got 'False' want 'True', the same message as the nightly |
| this PR's file | green 5 of 5, 3 passed each |
What I checked by reading:
- The order holds in CI.
pytest-clusterruns pytest without-n, so module order is collection order. The shell runs top to bottom. - crashA and crashC still prove their premise after the reorder. Each waits for
pg_relation_sizeto passsize0, and a 3M-row write outgrows any leftover from an earlier kill. So their premise checks still mean what they say. - The pin is set once in the private cluster (
nbc/ the suite's own setup), so no other suite inherits it.
Not verified by me: the shell twin. The only PG 17 install here is root-owned and holds an older build. I'm relying on CI's suites jobs for it, and I'll merge only at 15/15.
On the follow-up: yes, please surface the INSERT's error in both twins. It wasn't the cause this time, but a swallowed error would leave the next red unexplained.
@jdatcmd This fixes the one remaining red on #1757. #1759's dispatch (run 40376459928) went green on the capability sweep. Its only failure was pytest (cluster tests, PG 17), on one premise in my #1444 twin:
Cause: the scenario order, not the product
PgColumnarReserveOffset) is WAL-logged but unflushed, while its pages already extend the file.Reusing blocks no durable reservation covers is correct: every content, row-group and mirror check passed in that run.
Measured on local PG 17: crashA's pattern, then a committed 100k-row INSERT.
wal_writer_delayFix, in both suites
wal_writer_delay = 10s(its maximum) in the private cluster, so the race always goes the losing way instead of depending on runner load.Evidence (PG 17)
27 passed + 1 failed, crashB premise19 pass + 1 fail, crashB premisecluster_tests. Census 4567 equals the tsv. Ledger gate rc=0. Harness self-test passed.Once this merges, a dispatch of nightly.yml on main should close #1757.
🤖 Generated with Claude Code