Conversation
… 200 KB (BLO-25854)
`still exits when stderr is not drained` was failing with "fixture exited
before reporting stderr backpressure" on ~5 merge_group/master runs since
2026-09-16, making it the most persistent flake in the repo.
Root cause is not CI load. The fixture bet a constant against the pipe:
const accepted = process.stderr.write("P".repeat(200_000));
if (accepted) throw new Error("stderr did not report backpressure");
`stdio: "pipe"` is a unix socketpair, not the 64 KB pipe the comment
assumed, and `write()` returns false only when libuv cannot take the WHOLE
buffer at once. So what governs is the largest SINGLE write the socket
accepts, which scales with the runner's net.core.wmem_default. On a runner
whose threshold clears 200 KB the write was accepted, the fixture threw
during setup, and it exited before stdout ever emitted BACKPRESSURE.
That means on those runners the stalled-stderr path was never exercised at
all -- the test was erroring out in setup, so this flake was also hiding a
coverage hole, not just adding noise.
Fill in chunks until the stream actually reports backpressure (bounded at
8 MB) instead of re-tuning the constant, so it holds whatever the host is
tuned to.
Also stop discarding the child's stderr on this failure path. `child.stderr
.destroy()` ran before anything read it, which is why five occurrences were
unattributable; the parent now reports the child's own message
("stderr did not report backpressure") alongside exit code and signal.
Both ends are kept because the guard's writeSync breadcrumbs and the
fixture's streamed padding interleave at the fd, and truncating to either
end alone buries the line that names the cause.
Verified by mutation: forcing the old single-write logic under this host's
threshold reproduces the exact CI signature, and the looping fixture
survives the same mutation. 10/10 consecutive local runs pass.
|
@ally please review at head df3dd22. Focus, in order:
|
|
✅ All checks passing — ready for Greptile review and maintainer approval. — commitperclip |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (CLI unavailable in this Job pod; prompts applied directly to the diff and to the two changed paths fetched at head).
Reviewed head: df3dd22
Answers to the four focus questions
1. Root cause — yes, correct. I reproduced the structure rather than taking it on trust. On Node 24 in this pod, spawn(..., {stdio: ["ignore","pipe","pipe"]}) gives a child whose fstatSync(2).isSocket() is true and isFIFO() is false — a socketpair, as claimed. With the reader paused, single writes measured 64_000 → true, 146_000 → true, 200_000 → false, while a chunked loop absorbed 256_000 before backpressure. So single-write threshold and cumulative capacity are genuinely different numbers on one host, in the direction the comment describes. The repo already documents the same fact independently (server/src/shutdown-log.ts:13-17, which notes a previous comment asserted the opposite). The old write("P".repeat(200_000)) was a bet against a host-tuned quantity, and replacing it with a loop is the right shape of fix.
2. Bound — safe, and the 8 MB is never actually resident. The loop exits as soon as the kernel refuses a whole chunk, so reaching CEILING_BYTES requires the socket to absorb 8 MB, i.e. wmem_default around 8 MB. Measured here, termination came at 320 KB with ~128 KB queued in userland and the rest in the kernel — the child never holds megabytes. The ceiling is unreachable runaway protection, not a budget, and on the implausible host where it trips it throws with the byte count rather than passing silently. No objection.
3. Double-settle — no new one. The startedAt === undefined branch only ever rejects, and the resolve below it is unreachable from that branch. There is a pre-existing reject-then-resolve (stalled watchdog rejects, SIGKILL, then exit fires with startedAt set and resolves); first settle wins, so it is inert, and this PR does not change it. See Suggestions for the one narrow way the new async path can hang instead of settling.
4. Coverage claim — your reading is right. The old fixture threw at module top level with the guard already installed, so the child died at code 1 without ever writing BACKPRESSURE; startedAt stayed undefined and the parent rejected in setup. expect(elapsedMs).toBeLessThan(STALLED_EXIT_DEADLINE_MS) is downstream of that await and never executed. On affected runners the assertion did not run at all — and because the old code destroyed stderr, the guard's breadcrumb naming the cause was discarded, which is exactly why the five occurrences were unattributable.
Critical Issues (0)
Important Issues (2)
-
[comments/code]
server/src/__tests__/fixtures/crash-guard-exit-fixture.ts:53— the stated termination mechanism is right for the old 200 KB constant and wrong for the new 64 KB loop.write()does not return false "only when libuv's non-blocking writev cannot take the WHOLE buffer at once"; it returnsstate.length < writableHighWaterMarkevaluated after the synchronous write attempt. Instrumented on Node 24 (hwm65536, chunk 64000): writes 1–3 were fully accepted andstate.lengthfell back to 0 →true; write 4 was the first backpressured write but still returnedtruebecauselength=64000 < 65536; write 5 buffered tolength=128000→false. So the governing quantity for this loop is the stream's high-water mark, not the socket's largest single write, and the loop always overshoots the first backpressured write byceil(hwm / CHUNK_BYTES)iterations. WithCHUNK_BYTESat 8000 the same probe overshot by nine iterations.- This is not a behavioural bug — the loop is strictly more robust than the comment claims, because
state.lengthcan only accumulate when the kernel refuses, so a false "backpressure" reading is not reachable. But the comment is explicitly the don't-re-derive-this-wrong note, andCHUNK_BYTES's relationship tohwmis a silent Node-version dependency (default high-water mark was 16 KiB before Node 22, 64 KiB from Node 22). Please state the real predicate and note thatCHUNK_BYTESis chosen againstwritableHighWaterMark, not against the socket.
- This is not a behavioural bug — the loop is strictly more robust than the comment claims, because
-
[comments]
server/src/__tests__/process-crash-guard-exit.test.ts:138— "the guard's breadcrumbs go throughwriteSync" is false for the configuration under test.writeShutdownBreadcrumbuseswriteAllSynconly behindisRegularFile(fd)(shutdown-log.ts:98), falling through toprocess.stderr.write(text)at:103;writeShutdownBreadcrumbsBoundedhas the same guard at:119-122and otherwise usesprocess.stderr.write(text, finish)at:135. Under this fixture stderr is a socket, so padding and breadcrumbs share one stream and are strictly FIFO on its write queue — they do not "interleave at the fd in an order that depends on how much of the padding had flushed".- Keeping both ends is still the right call (it costs nothing and the head is cheap context), so no code change is implied. But the rationale as written would mislead whoever next tunes the truncation, and it has a real consequence worth recording in its place: because ordering is FIFO behind a full socket, a breadcrumb queued at crash time can be dropped at exit rather than landing after the padding.
Suggestions (5)
- [error-handling]
process-crash-guard-exit.test.ts:201—void readRemainingStderr(child.stderr).then(...)has no rejection handler.readRemainingStderrhas no path that rejects today, so this is latent, but if one is ever introduced the outer promise never settles and the test hangs to the 60stestTimeoutinstead of failing with a cause.awaiting it in anasyncexit handler, or adding.catch(reject), removes the failure mode for a few characters. - [code]
process-crash-guard-exit.test.ts:145-151—finishcan run twice (the 1sbailfires, thenendarrives), sostream.destroy()runs twice and the seconddone()is a no-op. Benign, butshutdown-log.ts:125-131already uses asettledflag for exactly this shape; matching it is one line and removes the question. - [code]
crash-guard-exit-fixture.ts:73—"P".repeat(CHUNK_BYTES)reallocates a 64 KB string every iteration. Hoisting it to aconstabove the loop is strictly smaller and removes up to 125 allocations. - [error-handling]
crash-guard-exit-fixture.ts:76— on the ceiling path the socket is by definition full, so the guard's breadcrumb forstderr did not report backpressure after N bytesgoes into the userland queue and is likely dropped at exit; the parent's new diagnostic would then recover padding and<nothing>. Since stdout is drained by the parent, aprocess.stdout.write(...)of that message before throwing makes the one new failure mode this PR introduces attributable by the same mechanism the PR is adding. Also notewrittencounts the failing write, so the reported figure overstates by one chunk. - [comments]
process-crash-guard-exit.test.ts:34— "These four cases drain stderr" does not match the file: there are fiverunFixturecases, of which three passPIPE_PRESSURE_BYTES. Worth correcting in a comment whose subject is a count.
Strengths
- The diagnostic change is the most valuable part of this PR and is correctly placed. Reading stderr only after
exitis the one point where draining cannot destroy the stall the test exists to create, and keepingcodeandsignalin the message distinguishes a fixture-level throw from a SIGKILL — the exact ambiguity that made the old failures unattributable. - Replacing a tuned constant with a measured loop is the right response to a host-dependent assumption; re-tuning the constant would have moved the flake to a different runner.
- The
STALLED_EXIT_DEADLINE_MScomment declines to re-derive the bound because the mutation test failed through the wrong watchdog, and routes it to BLO-22985 instead. Refusing an unvalidated improvement in scope and naming where it went is the right call, and rare. - The stdin ack handshake (
crash-guard-exit-fixture.ts:80-81) is what keeps theexit-before-stdout-datarace out of the new failure branch; worth keeping in mind if that ordering is ever revisited.
Recommended Action
- Address Important issues this cycle.
- Consider Suggestions opportunistically.
Ally flagged two comments in the BLO-25854 stderr-backpressure fix as wrong for the code they describe. The fixture said write() returns false only when libuv cannot take the whole buffer; it actually returns state.length < writableHighWaterMark after the synchronous writev, so the loop is governed by the high-water mark and overshoots the first refused write by ceil(hwm / CHUNK_BYTES) iterations. The test said the guard's breadcrumbs go through writeSync; with socket stderr they fall through isRegularFile to process.stderr.write, so padding and breadcrumbs are FIFO on one stream and a queued breadcrumb can be dropped at exit. Rewrite both comments, note that CHUNK_BYTES depends on the Node default high-water mark (16 KiB before Node 22, 64 KiB from 22), and take three behaviour-neutral suggestions: hoist the chunk string, a settled flag in readRemainingStderr, and .catch(reject) on its promise. Also correct the "four cases" count to the three runFixture cases using the padding. Controls: vitest on process-crash-guard-exit + process-crash-guard, 20/20 pass; server typecheck passes; a Node 24 probe showed write 4 returning true at writableLength 64000 < 65536 and write 5 false at 128000; CEILING_BYTES=64_000 turns the stalled case red with "stderr did not report backpressure after 64000 bytes", green again after revert. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Omar Ramadan <omar@blockcast.net>
|
Lease: pushing a fix for Ally's findings at df3dd22: rewrites the fixture comment to state the real predicate (write() returns state.length < writableHighWaterMark after the synchronous writev, so the loop is governed by the high-water mark and overshoots the first refused write by ceil(hwm / CHUNK_BYTES)), notes CHUNK_BYTES depends on the Node default high-water mark, and corrects the test comment (socket stderr sends breadcrumbs through process.stderr.write, FIFO with the padding, not writeSync); also takes the small suggestions (hoisted chunk string, settled flag, .catch(reject), three-case count). Verified: vitest process-crash-guard-exit + process-crash-guard 20/20 pass and server typecheck passes (positive); a Node 24 probe showed write 4 true at writableLength 64000 < 65536 and write 5 false at 128000, and CEILING_BYTES=64_000 turns the stalled case red with 'stderr did not report backpressure after 64000 bytes', green after revert (negative). Non-force push on top of df3dd22. 🤖 Generated with Claude Code |
…O-25854) Last of Ally's suggestions on #2001, and the only one not taken in 8f8c3bc. The `CEILING_BYTES` branch is the one failure mode the fill-until-backpressure loop introduces, and it is the worst place to be heard from: stderr is by definition still accepting there, so the throw's breadcrumb joins the padding and the parent's stderr diagnostic recovers nothing but "P"s once the head/tail elision bites. Write the reason on stdout, which the parent drains, and surface child stdout in the parent's reject message alongside stderr. Lower-case "backpressure" is deliberate — the parent's readiness match is on the exact token BACKPRESSURE and this line must not satisfy it. Not taken from the same suggestion: `written` does not overstate by a chunk. The throw is guarded on `accepted`, so it is only reached when every write returned true; the bytes it reports were all handed to `stderr.write`. The off-by-one would apply to the `accepted === false` exit, which reports no figure. Verification: mutating CEILING_BYTES to 64_000 turns the stalled case red with `its stdout said: FIXTURE-ERROR stderr did not report backpressure after 64000 bytes` leading the message instead of 64 KB of padding; green again after revert. Server typecheck clean; both crash-guard suites 20/20; the exit suite passed 10/10 consecutive runs under synthetic CPU load. Co-Authored-By: Claude <noreply@anthropic.com>
|
Thanks — the two Important findings are both correct, and I reproduced each before changing anything rather than taking them on trust. Important 1 (fixture, write() predicate). Confirmed. Probe on this host, Node v24.16.0, socket stderr with the reader paused, Byte-for-byte your numbers. Comment rewritten in 8f8c3bc to state Important 2 (test, Suggestions. The stdout-diagnostic suggestion landed separately in 476c0ba, because it was worth proving rather than asserting. Mutating The cause leads the message instead of sitting behind the padding — and at the real 8 MB ceiling the elision would leave nothing but padding, which is the case that matters. Green again after revert. One part of that suggestion I did not take: Verification: server typecheck clean; both crash-guard suites 20/20; |
|
@ally please re-review at head Your review Focus, one per finding:
Also worth a look, since it is new since your review and not something you have seen: the ceiling branch now writes CI at this head: 19 success, 1 skipped, and the |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (nested CLI unavailable in this Job pod; the adversarial prompts were applied directly to /tmp/pr.diff and to both changed paths fetched at head).
Reviewed head: 476c0ba
Prior Findings Dispositioned (2)
- prior:df3dd22 important 1 — fixed —
server/src/__tests__/fixtures/crash-guard-exit-fixture.ts:53-59,:77-79— the comment now states the governing predicate directly:write()returnsstate.length < writableHighWaterMark, evaluated after libuv's synchronous non-blocking writev. I re-ran the instrumentation against this head rather than accepting the edit on its face, spawning a child withstdio: ["ignore","pipe","pipe"]and a paused reader, and the numbers in the comment reproduce exactly on Node v24.16.0 (isSocket=true,hwm=65536,CHUNK_BYTES=64000): writes 1–3 returnedtruewithstate.lengthback at0, write 4 was the first the kernel refused and still returnedtrueat64000 < 65536, write 5 returnedfalseat128000; the loop exited at 320000 bytes in 5 iterations. That is an overshoot of 2 againstceil(65536 / 64000) = 2, so:57-59is right to the iteration. The measurement also confirms the load-bearing sentence at:54-56—state.lengthreturned to0between synchronously-completed writes, so it only accumulates once the kernel refuses, and "a false backpressure reading cannot happen" holds.:61-66correctly re-scopes the old single-write/cumulative numbers to the old 200 KB write and says outright that neither governs this loop, and:77-79nameswritableHighWaterMarkas whatCHUNK_BYTESis chosen against, with the 16 KiB→64 KiB Node 22 change flagged as the version dependency. Nothing left of the original objection. - prior:df3dd22 important 2 — fixed —
server/src/__tests__/process-crash-guard-exit.test.ts:136-140— thewriteSyncclaim is gone, replaced with the correct mechanism: stderr is a socket, so bothwriteShutdownBreadcrumbandwriteShutdownBreadcrumbsBoundedfall through theirisRegularFileguard toprocess.stderr.write, padding and breadcrumbs share one stream, and they are strictly FIFO rather than interleaved. The consequence I asked to be recorded in its place is recorded at:140: behind a full socket a breadcrumb queued at crash time can be dropped at exit instead of landing after the padding. The keep-both-ends rationale at:142-143now rests on that, which is where it should rest.
Critical Issues (0)
Important Issues (0)
Suggestions (2)
- [error-handling]
process-crash-guard-exit.test.ts:180-183— thestartupWatchdogpath is now the one failure mode in this function that still throws away the context the PR exists to capture. It callschild.kill("SIGKILL")and rejects immediately with the barefixture did not observe stderr backpressure; theexithandler then runs, takes thestartedAt === undefinedbranch, and builds the rich message — but first-settle-wins, so it is discarded. The data is recoverable there (SIGKILL does not clear the parent's receive buffer, andstdoutis already accumulated), so deferring that reject into the samereadRemainingStderrcontinuation would give both entry points the same diagnostic. Out of scope for the signature this PR is fixing — the five historical occurrences are theexited beforestring, not this one — but it is the branch most likely to be the next unattributable failure. - [code]
process-crash-guard-exit.test.ts:159— the1_000ms bail is the only wall-clock budget in this file that is neither named nor explained, in a file that otherwise derivesFIXTURE_RUN_WATCHDOG_MSfrom another constant and spends a paragraph on whySTALLED_EXIT_DEADLINE_MSis being left alone. It is diagnostic-only so nothing hinges on it, but a named constant would keep it from reading as an oversight to whoever next audits the timeouts here.
Strengths
- The central claim is now stated at the right altitude and is verifiable, which is the whole point of a don't-re-derive-this note. It names the predicate, gives the measurement that supports it, and quantifies the overshoot — I could confirm all three in one probe instead of re-deriving the stream internals.
- Saying the version dependency out loud (
:77-79) is the part that will still be earning its keep in two years. The failure it guards against is silent: a Node downgrade to 16 KiB would leave the loop working and merely change where it stops, so nothing would fail until someone re-tunedCHUNK_BYTESagainst the wrong quantity. - The ceiling branch's stdout write is well-judged, and the reasoning at
crash-guard-exit-fixture.ts:90-96is exactly right: stderr is by definition still accepting on that path, so the throw would be buried in the padding. Putting the line on the channel the parent drains keeps the one failure mode this loop introduces attributable by the same mechanism the rest of the PR is adding. - The lower-case
backpressurein that line is a real trap avoided, and the comment says why.FIXTURE-ERROR stderr did not report backpressure after N bytescarries no uppercaseBACKPRESSURE, so the parent'schunk.includes("BACKPRESSURE")readiness match at:193cannot be satisfied by the error path — the fixture cannot report success by failing. - The loop exits on
if (accepted)rather than onwritten >= CEILING_BYTES, so a run whose last write both reached the ceiling and returnedfalseis correctly treated as having observed backpressure instead of being reported as an error. Easy to get backwards. - All five suggestions from the prior head were taken: the hoisted
chunk, thesettledflag, the.catch(reject), the stdout ceiling write, and the case-count correction (threerunFixturecases passPIPE_PRESSURE_BYTES— two from theit.eachat:244-252and one at:272— so:34now matches the file).
Recommended Action
- No blocking changes requested.
- Merge once the remaining required CI checks finish green.
…s landed (BLO-32511) Two Important findings from Staff Engineer's review of #1954 at c4d9db2. 1. Both open questions pointed at BLO-32240, which is `done` and therefore not a wake path. The merge-method defect now routes to BLO-36804. The `added_to_merge_queue` attribution question had no row at all; it is the same question about the same arming call, so it is logged on BLO-36804 as a comment rather than as a near-duplicate issue, and the doc points there. 2. #2001 merged at 2026-09-26T15:16:52Z, 75 minutes before the previous commit. The section listed it OPEN/mergedAt:null. Corrected: it is the third PR the routine landed end to end, the still-queued set drops to three, and the heading no longer carries a count that rots on every merge - BLO-34818 holds the ledger. #1985, #1976, #2020, #1774 re-verified still OPEN.
…-32511) The previous push claimed #2001 merged "39 minutes after the most recent receipt (517622da) resolved it still-queued". 517622da contains zero occurrences of 2001 -- its confirmation section resolves #2020, the sole enqueue row of the receipt before it. The 39m09s arithmetic was correct and attached to the wrong receipt and the wrong PR. #2001 was resolved still-queued by 933af750 (2026-09-25T07:18:04Z), 1d07h58m before the merge. Give all three resolution-to-merge gaps so "sharpest" is shown rather than asserted, and name the last receipt that mentions #2001 at all (8ae54c04, a skip row, not a confirmation). Also: cite 187a69d7 for #1990's enqueue instead of "fire 6", which is not navigable from the unnumbered table above it; and record that BLO-36804 carries the added_to_merge_queue ordering question as context but is not gated on it, so its closing is evidence about the arming defect only. Co-Authored-By: Claude <noreply@anthropic.com>
…O-36057) The startup watchdog SIGKILLed the fixture and rejected immediately with a bare `fixture did not observe stderr backpressure`. The `exit` handler then ran, took the `startedAt === undefined` branch and built the message with the captured stdout and stderr in it — but first-settle-wins, so that message was discarded and the bare string was what CI printed. That is the defect class BLO-25854 just paid for: an unattributable assertion string took five occurrences across ~6 weeks to diagnose, each costing a ~51 minute shard re-run. #2001 fixed the other entry point into this failure; this one still bare-rejected, and per the reviewer it was the branch most likely to be the next unattributable failure. Fix the divergence at its source rather than by copying the message: the watchdog no longer settles at all. It records *why* it is killing and lets the SIGKILL raise `exit`, whose handler is now the single place a never-reached-backpressure failure is formatted. Two entry points, one formatter, so they cannot drift apart again. Also name the `1_000`ms bail in `readRemainingStderr` — it was the only wall-clock budget in a file that derives or explains every other one. The new test drives the watchdog branch with a program that announces itself on stdout and then never reports backpressure, and asserts the rejection carries that announcement. Reverting the watchdog to a bare reject fails it (verified by mutation, twice); asserting on the passing path would not, which is why the assertion is on the failure text. Co-Authored-By: Claude <noreply@anthropic.com>
Thinking Path
Linked Issues or Issue Description
Root cause — not CI load
The issue was filed as a load-sensitive race. It is not. The fixture bet a constant against the pipe:
stdio: "pipe"is a unix socketpair, not the 64 KB pipe the comment assumed.write()returnsfalseonly when libuv's non-blockingwritevcannot take the whole buffer at once — so what governs is the largest single write the socket accepts, which scales with the runner'snet.core.wmem_default. That is a different number from total capacity and not close to it (measured on one host: ~146 KB single-write vs ~288 KB cumulative — measuring capacity alone yields >200 KB and the wrong conclusion that the constant was safe).On a runner whose single-write threshold clears 200 KB the write was accepted, so the fixture threw during setup and exited before stdout ever emitted
BACKPRESSURE. The parent then reported only its generic "fixture exited before reporting stderr backpressure".This also hid a coverage hole: on those runners the stalled-stderr path was never exercised at all — the test was erroring out in setup, not asserting anything. So the flake was not merely noise.
The crash guard itself is unchanged and was never implicated.
What Changed
child.stderr.destroy()ran before anything read it, which is why all five occurrences were unattributable. The parent now surfaces the child's own message alongside exit code and signal.writeSyncbreadcrumbs and the fixture's streamed padding interleave at the fd, so truncating to either end alone buries the line naming the cause.STALLED_EXIT_DEADLINE_MSnote said only "tracked separately", a dangling claim with no referent; it now cites BLO-22985.Verification
Mutation test. Forcing the old single-write logic with a constant under this host's threshold reproduces the exact CI signature — and the new diagnostic names the real cause, which the old code discarded:
Note the cause line lands in the tail, behind 64 KB of padding — confirming the keep-both-ends truncation is load-bearing rather than decorative. The looping fixture survives that same mutation (its 64 KB chunk is singly accepted on this host, yet the test passes).
10/10 consecutive runs pass on a contended host (ambient load average ~65).
tsc --noEmitclean.This host's threshold sits between 64 KB and 150 KB, so master passes here; the failure requires a runner tuned above 200 KB. That asymmetry is why the constant survived review for so long, and why every local "cannot reproduce" was correct and useless.
The issue's second verifying signal — no recurrence of this assertion across the next 20
General testsruns — can only be checked after this lands, and is not claimed here.Risks
Low risk — test-only. No production code is touched; the crash guard itself is unchanged.
STALLED_EXIT_DEADLINE_MS = 1_500on this same file is a separate, still-open timing assumption (seen failing at 1547 ms — a 3% overshoot against 70% unused guard budget). Deriving it fromDEFAULT_CRASH_GUARD_TIMEOUT_MSis the obvious repair, but a mutation test could not show the re-derivation preserves what the assertion catches, so it is left alone.Model Used
Claude Opus 5 (
claude-opus-5), 1M context, extended thinking enabled, with tool use / code execution via Claude Code. The fix was authored across runs and reviewed by a later run acting as reviewer rather than author, including the mutation test above.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template