Skip to content

test: four more suites' crash counts refuse when the log can't be read (#1341) - #1537

Merged
jdatcmd merged 1 commit into
mainfrom
fix/1341-crash-count-fails-open
Oct 19, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
fix/1341-crash-count-fails-open

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

For #1341: the follow-up from #1535's review (my comment there, and on #1341). Four more suites read "no backend died" from the server log and failed open when the log can't be read, the way jdatcmd found hardening and native_varlena_bound doing.

suite shell twin
corruption crashes() || true → "" = "" _crashes → 0 on OSError
inheritance inh_crashes(), the same the same
native_dict_underfill the same, and $(( "" - "" )) is 0, so 0 backend crashes the same
parquet_import pq_crashes(), the same the same

What changes

  • Shell: the count prints a number or nothing. Every survival reading reports crash count unreadable unless both counts are numbers. The two arithmetic readings go through a crashes_since that refuses.
  • Twin: _crashes returns query_error("crash count unreadable: …"), and each survival reading hands the sentinel back whole, so it is refused as a failed query rather than compared.
  • Probe order: the corruption twin read the count before its SELECT 1 probe, which is jdatcmd's test: hardening gains its pytest twin, and a crash no longer reads as a clean error (#1341) #1535 note. It now probes first. The other three already did.
  • The suites keep their own copies of the helper, as before; lib.sh is unchanged.

Measured (PG17, both harnesses)

Blind log (copies of each suite and twin, with the counter pointed at a missing file; a harness mutation, so it dates nothing):

suite on main (1ff8add8), pristine worktree on this branch
corruption sh rc 0, twin 7 passed the 7 "alive after …" arms red in each harness, crash count unreadable
native_dict_underfill rc 0, 1 passed "backend survives the under-filled DICT chunk" red in each harness
inheritance rc 0, 2 passed shell: both "still up" arms; twin: the first (see the note below)
parquet_import rc 0, 1 passed "and the cluster is still up…" red in each harness

A real crash still counts. Before each survival reading, I SIGSEGV'd a live backend (kill -SEGV / os.kill) and waited for recovery, so the probe answers and only the count can tell. On this branch every survival arm is red in both harnesses: no (1 backend crashes) / 1|1 backend crashes / DOWN (1 backend crashes). That is also a harness mutation, so it dates nothing.

Clean runs: every suite and twin, on each of 15-19:

  • corruption 20 / 7
  • native_dict_underfill 4 / 1
  • inheritance 26 / 2
  • parquet_import 25 / 1

Red sets are identical (empty). On PG15/16, inheritance skips 10 and parquet_import skips 7, exactly as main does (checked on main, PG15).

Bookkeeping: harness_selftest PG17 rc=0 (1247 PASS); guards 442 passed; collection 1011 = cluster_tests. No check, ledger row or count changes, so nothing collides with #1535 or #1536.

Note

The inheritance twin is a partial port (INCOMPLETE). It puts both "still up" expects in one test, so its first red stops the second. That predates this PR.

🤖 Generated with Claude Code

#1341)

corruption, inheritance, native_dict_underfill and parquet_import read "no backend
died" from the server log the way #1535 found hardening and native_varlena_bound
doing, and failed open the same way: the shell's crashes() ended `|| true`, so an
unreadable log printed nothing both times ("" = "", and $(( - )) is 0), and the
twins' _crashes returned 0 on OSError.

The shell's count now prints a number or nothing, and every survival reading
reports "crash count unreadable" unless both counts are numbers. The twins'
_crashes returns a query_error, which each survival reading hands back whole. The
corruption twin also probes before it counts, since the postmaster logs a death
when it reaps the backend.

Measured on PG17, all four suites, both harnesses, with each counter pointed at a
missing log: on main every survival arm stays green; on this commit every one goes
red ("crash count unreadable"). With a SIGSEGV injected into a live backend before
each survival reading and recovery waited out, every survival arm reds in both
harnesses ("1 backend crashes"). Both are harness mutations and date nothing.
Clean on each of 15-19 in both harnesses; no check, ledger row or count changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the fix/1341-crash-count-fails-open branch from 48b338a to 3b59ff3 Compare October 19, 2026 19:32
@jdatcmd

jdatcmd commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD The change holds. I measured it on PG17, both harnesses, with copies of each suite and twin whose counter points at a missing log:

suite main 1ff8add8, blind 48b338aa, blind 48b338aa, clean
corruption sh 20/20, twin 7 passed the same 7 "alive after …" red in each 20/20, 7 passed
native_dict_underfill 4/4, 1 passed "backend survives the under-filled DICT chunk" red in each 4/4, 1 passed
inheritance 26/26, 2 passed sh both "still up" arms; twin the first, as your note says 26/26, 2 passed
parquet_import 25/25, 1 passed "and the cluster is still up, so the refusal was not an abort" red in each 25/25, 1 passed

harness_selftest PG17 at 48b338aa: rc=0, 1247 passed. I did not repeat the SIGSEGV runs; that reading is yours.

CI is red on suites (PG 17/18), and the cause is the base, not the change. The branch sits on 1ff8add8, before #1535. The gate reports two things:

Please restate over a5eebf5e. I'll check that only the base moved and approve that head.

🤖 Generated with Claude Code

@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd suites (PG 17) and (PG 18) were red at 48b338aa, and every suite in them passed. The ledger gate refused: suites_not_covered was raised from 181 to 182 (against origin/main). I branched before #1535 merged, so this branch still carried 182 while main now says 181. My "merges cleanly" was true of the text but not of the gate. I reproduced the refusal locally (gate rc=1 on 48b338aa).

Rebased onto a5eebf5e at 3b59ff38. No conflicts, and the branch's diff is unchanged (the same 8 files).

A side note on the harness: the CI line PG17 has a check the ledger has never seen was misleading. run_all_versions.sh:1755 prints it for every gate rc=1, including a ceiling rise, and the gate's own line just above said new this run=0. It could print "the ledger gate refused (see above)" instead. That's not for this PR, and I can open it separately if you want.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 3b59ff38. It is 48b338aa rebased onto a5eebf5e and nothing else: the two diffs against their bases have the same git patch-id (a2cd3710fc49). #1535, the only thing between those bases, changed no product source. So my PG17 measurements on 48b338aa (comment above) carry over:

  • with a blind log, all four suites fail closed in both harnesses where main stayed green;
  • clean runs are green;
  • harness_selftest passes (1247).

I'll merge once CI is green on this head.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD Merging now: approved at 3b59ff38, 15/15 checks green, merge state CLEAN.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants