Skip to content

test: native_backend_crash gets its pytest twin, and a mid-write kill must follow a write (#1341 item 25) #1444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jdatcmd merged 2 commits into from
Oct 16, 2026
Merged

test: native_backend_crash gets its pytest twin, and a mid-write kill must follow a write (#1341 item 25) #1444

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fa44283
test: native_backend_crash gets its pytest twin, and a mid-write kill…
jdatcmd Oct 16, 2026
0115c61
test: native_backend_crash seeds the ledger, 3 of 28 arms dated by an…
jdatcmd Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 35 additions & 7 deletions test/check_ledger.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2226,6 +2226,34 @@ native_agg native_agg sum(int4) 15;16;17;18;19 never -
native_agg native_agg sum(int4,nulls) 15;16;17;18;19 never -
native_agg native_agg sum/min/max uses the metadata agg node 15;16;17;18;19 never -
native_agg native_agg the deletes removed rows at both ends, the same rows from each 15;16;17;18;19 never -
native_backend_crash native_backend_crash a second concurrent reader was also dropped by the crash 15;16;17;18;19 never -
native_backend_crash native_backend_crash crashA: kill lands mid-write (backend not yet in pg_sleep) 15;16;17;18;19 never -
native_backend_crash native_backend_crash crashA: the INSERT had written pages before the kill (the relation grew) 15;16;17;18;19 never -
native_backend_crash native_backend_crash crashB: kill lands after the stripes flushed (backend in pg_sleep) 15;16;17;18;19 never -
native_backend_crash native_backend_crash crashB: the INSERT had written pages before the kill (the relation grew) 15;16;17;18;19 never -
native_backend_crash native_backend_crash crashC: kill lands mid-write (backend not yet in pg_sleep) 15;16;17;18;19 never -
native_backend_crash native_backend_crash crashC: the INSERT had written pages before the kill (the relation grew) 15;16;17;18;19 never -
native_backend_crash native_backend_crash mid-write: cluster accepts new connections 15;16;17;18;19 never -
native_backend_crash native_backend_crash mid-write: committed content in c matches the heap mirror 15;16;17;18;19 2026-10-16 columnar_storage.c: a stripe data page written with no WAL record
native_backend_crash native_backend_crash mid-write: postmaster survived, same pid 15;16;17;18;19 never -
native_backend_crash native_backend_crash mid-write: the crash reinitialized the cluster (a live session was dropped) 15;16;17;18;19 never -
native_backend_crash native_backend_crash mid-write: the killed insert left no partial row group 15;16;17;18;19 never -
native_backend_crash native_backend_crash mid-write: the killed insert left no visible rows 15;16;17;18;19 never -
native_backend_crash native_backend_crash mid-write: the mirror itself is unchanged 15;16;17;18;19 never -
native_backend_crash native_backend_crash post-flush: cluster accepts new connections 15;16;17;18;19 never -
native_backend_crash native_backend_crash post-flush: committed content in c matches the heap mirror 15;16;17;18;19 2026-10-16 columnar_storage.c: a stripe data page written with no WAL record
native_backend_crash native_backend_crash post-flush: postmaster survived, same pid 15;16;17;18;19 never -
native_backend_crash native_backend_crash post-flush: the crash reinitialized the cluster (a live session was dropped) 15;16;17;18;19 never -
native_backend_crash native_backend_crash post-flush: the killed insert left no partial row group 15;16;17;18;19 never -
native_backend_crash native_backend_crash post-flush: the killed insert left no visible rows 15;16;17;18;19 never -
native_backend_crash native_backend_crash post-flush: the mirror itself is unchanged 15;16;17;18;19 never -
native_backend_crash native_backend_crash with-reader: cluster accepts new connections 15;16;17;18;19 never -
native_backend_crash native_backend_crash with-reader: committed content in c matches the heap mirror 15;16;17;18;19 2026-10-16 columnar_storage.c: a stripe data page written with no WAL record
native_backend_crash native_backend_crash with-reader: postmaster survived, same pid 15;16;17;18;19 never -
native_backend_crash native_backend_crash with-reader: the crash reinitialized the cluster (a live session was dropped) 15;16;17;18;19 never -
native_backend_crash native_backend_crash with-reader: the killed insert left no partial row group 15;16;17;18;19 never -
native_backend_crash native_backend_crash with-reader: the killed insert left no visible rows 15;16;17;18;19 never -
native_backend_crash native_backend_crash with-reader: the mirror itself is unchanged 15;16;17;18;19 never -
native_chunk_length_bound native_chunk_length_bound a sequential scan of the same poisoned chunk is refused (XX001) 15;16;17;18;19 never -
native_chunk_length_bound native_chunk_length_bound an index fetch of a chunk whose page_length is 2^32 too large is refused (XX001) 15;16;17;18;19 never -
native_chunk_length_bound native_chunk_length_bound backend survived the sequential refusal 15;16;17;18;19 never -
Expand Down Expand Up @@ -2352,6 +2380,9 @@ native_join_runtime_filter native_join_runtime_filter a (SELECT false) gate over
native_join_runtime_filter native_join_runtime_filter a (SELECT false) gate over a heap dimension equals heap 15;16;17;18;19 2026-10-15 the join runtime filter took a join carrying a gating qual and failed on the Result over it (the tree before #1437)
native_join_runtime_filter native_join_runtime_filter a (SELECT true) gate over a columnar dimension equals heap 15;16;17;18;19 2026-10-15 the join runtime filter took a join carrying a gating qual and failed on the Result over it (the tree before #1437)
native_join_runtime_filter native_join_runtime_filter a (SELECT true) gate over a heap dimension equals heap 15;16;17;18;19 2026-10-15 the join runtime filter took a join carrying a gating qual and failed on the Result over it (the tree before #1437)
native_join_runtime_filter native_join_runtime_filter an empty build from a large dimension answers as heap 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter an empty build on one rescan does not empty the next 15;16;17;18;19 2026-10-16 the rescan kept the first execution's runtime filter state: prepared not reset on rescan (mutation, #1447)
native_join_runtime_filter native_join_runtime_filter an empty build reads no fact chunk group 15;16;17;18;19 2026-10-16 an inner join whose build was empty ran on, and the fact scan rejected every row one at a time (the tree before #1447)
native_join_runtime_filter native_join_runtime_filter baseline core Hash Join 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter baseline reads all groups 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter build rows omit NULL 15;16;17;18;19 never -
Expand All @@ -2377,8 +2408,12 @@ native_join_runtime_filter native_join_runtime_filter join runtime filter defaul
native_join_runtime_filter native_join_runtime_filter plan has build tap 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter plan has runtime coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter plan retains core Hash Join 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: and its build is empty 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: heap's answer under a true gate over a columnar dimension holds 200 rows 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: heap's answer under a true gate over a heap dimension holds 200 rows 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: heap's empty-then-full rescan answer holds the 200 keys 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: the empty-then-full rescan runs under the coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: the large-dimension empty build runs under the coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: with no gate, the columnar-dimension join has the coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: with no gate, the heap-dimension join has the coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter projection join equals heap 15;16;17;18;19 never -
Expand Down Expand Up @@ -3562,10 +3597,3 @@ vm_privilege vm_privilege vm entry point is covered by this boundary test: vm_se
vm_privilege vm_privilege vm_is_visible refuses a heap relation even for its owner (#748) 15;16;17;18;19 never -
vm_privilege vm_privilege vm_selftest did not report success to the unprivileged role 15;16;17;18;19 2026-10-14 vm_selftest without its owner check
vm_privilege vm_privilege vm_selftest refuses a heap relation even for its owner (#748) 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: the large-dimension empty build runs under the coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: and its build is empty 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter an empty build reads no fact chunk group 15;16;17;18;19 2026-10-16 an inner join whose build was empty ran on, and the fact scan rejected every row one at a time (the tree before #1447)
native_join_runtime_filter native_join_runtime_filter an empty build from a large dimension answers as heap 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: the empty-then-full rescan runs under the coordinator 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter premise: heap's empty-then-full rescan answer holds the 200 keys 15;16;17;18;19 never -
native_join_runtime_filter native_join_runtime_filter an empty build on one rescan does not empty the next 15;16;17;18;19 2026-10-16 the rescan kept the first execution's runtime filter state: prepared not reset on rescan (mutation, #1447)
21 changes: 19 additions & 2 deletions test/check_ledger_budget.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
#
# Adding a check to a suite that is already covered does not move it, which is
# what makes it safe to bound.
suites_not_covered 221
suites_not_covered 220
#
# checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
# one.
Expand Down Expand Up @@ -2886,4 +2886,21 @@ suites_not_covered 221
# never 2639 + 5 = 2644
# dated 925 + 2 = 927
# 2644 + 927 == 3571
checks_never_observed_red 2644
#
# #1341, native_backend_crash seeded, with its twin (OffgridwithJD): 2644 -> 2669,
# and suites_not_covered 221 -> 220. The suite had no rows. It enters with all 28
# checks from five-major runs (28/28 on each of 15-19). 3 are dated by a stripe data
# page written with no WAL record: each scenario's "committed content in c matches
# the heap mirror", which the reinit's crash recovery cannot rebuild. Twenty-five
# arrive never. The no-visible-rows and no-partial-row-group arms are MVCC's: no
# mutation of the extension leaves an aborted transaction's catalog rows visible.
# The sentinel, second-reader, kill-phase and relation-grew arms go red only under
# harness mutations (a signal that kills nothing, an INSERT whose stripes buffer
# whole), which observe the suite rather than the code. The rest hold of the heap
# mirror, the postmaster pid and a connection.
#
# ledger rows 3571 + 28 = 3599
# never 2644 + 25 = 2669
# dated 927 + 3 = 930
# 2669 + 930 == 3599
checks_never_observed_red 2669
15 changes: 13 additions & 2 deletions test/native_backend_crash.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,9 @@ open_sentinel() {
# backend is parked in the trailing pg_sleep, between the last flush and the
# commit that never comes. The phase is asserted, not assumed, from wait_event.
crash_one_backend() {
local rows="$1" marker="$2" phase="$3" bg pid i we src
local rows="$1" marker="$2" phase="$3" bg pid i we src size0
open_sentinel || echo "-- sentinel did not connect"
size0="$(q "SELECT pg_relation_size('c');")"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \
-d "$PGC_DB" -c "BEGIN; INSERT INTO c /* $marker */ SELECT g, 'k'||g, g
FROM generate_series(1, $rows) g; SELECT pg_sleep(60);" >/dev/null 2>&1 &
Expand All @@ -131,8 +132,18 @@ crash_one_backend() {
sleep 0.1
done
else
sleep 0.5
# Until the write has reached the relation. A fixed 0.5s stood here, and
# nothing showed the INSERT had written a page by then: with its stripes
# buffered whole (stripe_row_limit above the row count) the kill landed on
# a write that had produced nothing, and every mid-write arm still passed
# (#1341). So the kill now waits for the relation to grow, and asserts it.
for i in $(seq 1 600); do
[ "$(q "SELECT pg_relation_size('c') > $size0;")" = t ] && break
sleep 0.1
done
fi
check "$marker: the INSERT had written pages before the kill (the relation grew)" \
"$(q "SELECT pg_relation_size('c') > $size0;")" "t"
we="$(q "SELECT coalesce(wait_event,'') FROM pg_stat_activity WHERE pid = $pid;")"
if [ "$phase" = sleep ]; then
check "$marker: kill lands after the stripes flushed (backend in pg_sleep)" "$we" "PgSleep"
Expand Down
15 changes: 15 additions & 0 deletions test/pytest/TESTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ behaviour, the source of that number is named.
- [134. test_unique_conc.py: concurrent inserters of one unique key serialize](#134-test_unique_concpy-concurrent-inserters-of-one-unique-key-serialize)
- [135. test_update_conc.py: two writers of one row serialize](#135-test_update_concpy-two-writers-of-one-row-serialize)
- [136. test_recovery.py: committed writes survive a crash, uncommitted ones leave nothing](#136-test_recoverypy-committed-writes-survive-a-crash-uncommitted-ones-leave-nothing)
- [137. test_native_backend_crash.py: one backend dies and committed data is intact](#137-test_native_backend_crashpy-one-backend-dies-and-committed-data-is-intact)

## 1. How to read a test in here

Expand Down Expand Up @@ -7296,3 +7297,17 @@ A private cluster, stopped with `-m immediate` for each crash. Each crash is pro
| `test_the_pair_through_three_crashes` | rows committed after a checkpoint survive; an open transaction's checkpointed INSERT leaves no row; committed deletes survive; each against the heap pair |
| `test_a_cleared_visibility_bit_stays_cleared` | a bit set and checkpointed, then cleared by a DELETE, stays cleared after the crash; a clean block's bit stays set; the rows match heap |
| `test_a_projection_survives_a_crash` | a projection copy committed after the checkpoint replays, and reproduces the base's rows and a scan through it |

## 137. test_native_backend_crash.py: one backend dies and committed data is intact

The pytest twin of `test/native_backend_crash.sh` (#217, ported for #1341), every check by the shell's name. A SIGSEGV in one backend makes the postmaster drop every session and reinitialize, with crash recovery, while it stays up itself. Committed columnar rows must then match a heap mirror, and an INSERT killed before its commit must leave no row and no row group.

The kill lands mid-write, once the INSERT has written pages to the relation and before it reaches its trailing `pg_sleep`, and after every stripe has flushed, while it sleeps; both are asserted. A sentinel session opened before each kill must be dropped by the reinit, which proves the crash happened. A private cluster, since the reinit drops every session on the server.

### Every test

| test | what it holds |
| --- | --- |
| `test_a_kill_mid_write` | a 3,000,000-row INSERT killed after it has grown the relation: the cluster reinitialized, the postmaster kept its pid, and c holds exactly its committed rows and row groups |
| `test_a_kill_after_the_stripes_flushed` | the same for a 100,000-row INSERT killed in its trailing `pg_sleep` |
| `test_a_kill_with_a_second_reader` | the mid-write kill with a second parked reader, which the reinit drops too |
4 changes: 3 additions & 1 deletion test/pytest/expected_tests.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -741,4 +741,6 @@ guard_tests 438
# tests, by collection over the cluster files on this tree.
# cluster_tests 641 -> 643 for #1447's two tests in test_join_runtime_filter.py, by
# collection over the cluster files on this tree.
cluster_tests 643
# cluster_tests 643 -> 646 for #1341's test_native_backend_crash.py, the twin of
# native_backend_crash.sh: three tests, by collection over the cluster files on this tree.
cluster_tests 646
1 change: 1 addition & 0 deletions test/pytest/test_compare_to_bash.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@
"latest_reading",
"maintenance_crash",
"maintenance_rollback",
"native_backend_crash",
"native_chunk_length_bound",
"native_delete_vector_index",
"native_delete_visibility_paths",
Expand Down
Loading
Loading