Skip to content

fix(tests): fill stderr until it reports backpressure, not to a fixed 200 KB (BLO-25854) - #2001

Merged
kkroo merged 3 commits into
masterfrom
BLO-25854-ci-flake-process-crash-guard-exit-test-ts-still-exits-when-stderr-is-not-drained-times-out-on-backpressure-fix
Sep 26, 2026
Merged

kkroo merged 3 commits into
masterfrom
BLO-25854-ci-flake-process-crash-guard-exit-test-ts-still-exits-when-stderr-is-not-drained-times-out-on-backpressure-fix

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work, and its merge queue is how every change lands
  • That queue was ejecting blameless PRs; the per-test flake census in BLO-28886 ranked this test the single most persistent offender — 5 occurrences across 4 days and 5 distinct base SHAs in the 45 most recent failing merge_group runs
  • The failing test guards the process crash guard's behaviour when a child's stderr is never drained, so it protects a real shutdown invariant rather than being incidental
  • It was filed as a CI-load race, but every occurrence reported only the parent's generic message because the test destroyed the child's stderr before reading it — so the actual cause had never once been observed
  • Reading the fixture rather than the timing showed it bets a fixed 200 KB constant against a socket whose accept threshold is a property of the runner, which is why it fails on some hosts and is unreproducible on others
  • This pull request replaces that constant with a loop that fills until the stream itself reports backpressure, and stops the test from discarding the diagnostic that would have identified this in one run instead of five
  • The benefit is one fewer blameless merge-queue ejection, and — because the test was erroring out in setup on affected runners — restored coverage of the stalled-stderr path that silently was not being exercised there

Linked Issues or Issue Description

FAIL src/__tests__/process-crash-guard-exit.test.ts > still exits when stderr is not drained
Error: fixture exited before reporting stderr backpressure

Root cause — not CI load

The issue was filed as a load-sensitive race. It is not. The fixture bet a constant against the pipe:

const accepted = process.stderr.write("P".repeat(200_000));
if (accepted) throw new Error("stderr did not report backpressure");

stdio: "pipe" is a unix socketpair, not the 64 KB pipe the comment assumed. write() returns false only when libuv's non-blocking writev cannot take the whole buffer at once — so what governs is the largest single write the socket accepts, which scales with the runner's net.core.wmem_default. That is a different number from total capacity and not close to it (measured on one host: ~146 KB single-write vs ~288 KB cumulative — measuring capacity alone yields >200 KB and the wrong conclusion that the constant was safe).

On a runner whose single-write threshold clears 200 KB the write was accepted, so the fixture threw during setup and exited before stdout ever emitted BACKPRESSURE. The parent then reported only its generic "fixture exited before reporting stderr backpressure".

This also hid a coverage hole: on those runners the stalled-stderr path was never exercised at all — the test was erroring out in setup, not asserting anything. So the flake was not merely noise.

The crash guard itself is unchanged and was never implicated.

What Changed

  • Fixture: fill stderr in chunks until the stream actually reports backpressure (bounded at 8 MB) rather than re-tuning the constant, so it holds whatever the host is tuned to.
  • Test: stop discarding the child's stderr on this failure path. child.stderr.destroy() ran before anything read it, which is why all five occurrences were unattributable. The parent now surfaces the child's own message alongside exit code and signal.
  • Truncation keeps both ends of that buffer — the guard's writeSync breadcrumbs and the fixture's streamed padding interleave at the fd, so truncating to either end alone buries the line naming the cause.
  • Comment fix: the STALLED_EXIT_DEADLINE_MS note said only "tracked separately", a dangling claim with no referent; it now cites BLO-22985.

Verification

Mutation test. Forcing the old single-write logic with a constant under this host's threshold reproduces the exact CI signature — and the new diagnostic names the real cause, which the old code discarded:

Error: fixture exited before reporting stderr backpressure (code=1, signal=null);
  its stderr said: PPPP…(64725 bytes, middle elided)…
  [shutdown] unhandledRejection: Error: stderr did not report backpressure

Note the cause line lands in the tail, behind 64 KB of padding — confirming the keep-both-ends truncation is load-bearing rather than decorative. The looping fixture survives that same mutation (its 64 KB chunk is singly accepted on this host, yet the test passes).

10/10 consecutive runs pass on a contended host (ambient load average ~65). tsc --noEmit clean.

This host's threshold sits between 64 KB and 150 KB, so master passes here; the failure requires a runner tuned above 200 KB. That asymmetry is why the constant survived review for so long, and why every local "cannot reproduce" was correct and useless.

The issue's second verifying signal — no recurrence of this assertion across the next 20 General tests runs — can only be checked after this lands, and is not claimed here.

Risks

Low risk — test-only. No production code is touched; the crash guard itself is unchanged.

  • The fill loop is bounded at 8 MB, so a host that somehow never reports backpressure fails with an explicit error instead of looping forever or OOMing.
  • The change makes the test more likely to fail on a genuine regression, not less: on runners above the old 200 KB threshold it previously errored out in setup and asserted nothing, so this restores a real assertion rather than relaxing one.
  • Widening the surfaced stderr could in principle make a failure message large; it is truncated with both ends preserved and the middle elided.
  • Deliberately out of scope: STALLED_EXIT_DEADLINE_MS = 1_500 on this same file is a separate, still-open timing assumption (seen failing at 1547 ms — a 3% overshoot against 70% unused guard budget). Deriving it from DEFAULT_CRASH_GUARD_TIMEOUT_MS is the obvious repair, but a mutation test could not show the re-derivation preserves what the assertion catches, so it is left alone.

Model Used

Claude Opus 5 (claude-opus-5), 1M context, extended thinking enabled, with tool use / code execution via Claude Code. The fix was authored across runs and reviewed by a later run acting as reviewer rather than author, including the mutation test above.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I searched the GitHub PR list (open and recently closed) for similar or duplicate PRs — the two other open de-flake PRs, test: de-flake the two assertions ejecting merge groups (#1787, #1952) #1987 and test(server): bound the stall-detector retry by attempts, not wall clock #1962, address different tests; this test had no open PR
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots — n/a, no UI change
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green — pending; this PR is what makes them green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending review
  • I will address all Greptile and reviewer comments before requesting merge

… 200 KB (BLO-25854)

`still exits when stderr is not drained` was failing with "fixture exited
before reporting stderr backpressure" on ~5 merge_group/master runs since
2026-09-16, making it the most persistent flake in the repo.

Root cause is not CI load. The fixture bet a constant against the pipe:

    const accepted = process.stderr.write("P".repeat(200_000));
    if (accepted) throw new Error("stderr did not report backpressure");

`stdio: "pipe"` is a unix socketpair, not the 64 KB pipe the comment
assumed, and `write()` returns false only when libuv cannot take the WHOLE
buffer at once. So what governs is the largest SINGLE write the socket
accepts, which scales with the runner's net.core.wmem_default. On a runner
whose threshold clears 200 KB the write was accepted, the fixture threw
during setup, and it exited before stdout ever emitted BACKPRESSURE.

That means on those runners the stalled-stderr path was never exercised at
all -- the test was erroring out in setup, so this flake was also hiding a
coverage hole, not just adding noise.

Fill in chunks until the stream actually reports backpressure (bounded at
8 MB) instead of re-tuning the constant, so it holds whatever the host is
tuned to.

Also stop discarding the child's stderr on this failure path. `child.stderr
.destroy()` ran before anything read it, which is why five occurrences were
unattributable; the parent now reports the child's own message
("stderr did not report backpressure") alongside exit code and signal.
Both ends are kept because the guard's writeSync breadcrumbs and the
fixture's streamed padding interleave at the fd, and truncating to either
end alone buries the line that names the cause.

Verified by mutation: forcing the old single-write logic under this host's
threshold reproduces the exact CI signature, and the looping fixture
survives the same mutation. 10/10 consecutive local runs pass.
@allyblockcast

allyblockcast Bot commented Sep 23, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-28886
🔗 Paperclip issue: BLO-25854
🔗 Paperclip issue: BLO-22985

@allyblockcast

allyblockcast Bot commented Sep 23, 2026

Copy link
Copy Markdown
Author

@ally please review at head df3dd22.

Focus, in order:

  1. Is the root cause right? The claim is that stdio: "pipe" is a unix socketpair and write() returns false only when the whole buffer cannot be taken at once — so the governing quantity is the largest SINGLE write, not total capacity. If that is wrong, the fix is wrong.
  2. Loop termination and bound. CEILING_BYTES = 8_000_000 at 64 KB chunks = 125 iterations max. Is 8 MB of child-buffered writes safe on a tuned runner, and is the ceiling high enough that a legitimately large socket does not trip it spuriously?
  3. Does the diagnostic path double-settle? On the failure branch the exit handler now kicks off an async stderr read before rejecting. Confirm there is no path that both rejects and resolves.
  4. Coverage claim. I assert that on affected runners the stalled-stderr assertion never ran at all (setup threw). Please sanity-check that reading of the old control flow.

@allyblockcast

allyblockcast Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Author

✅ All checks passing — ready for Greptile review and maintainer approval.

— commitperclip

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (CLI unavailable in this Job pod; prompts applied directly to the diff and to the two changed paths fetched at head).
Reviewed head: df3dd22

Answers to the four focus questions

1. Root cause — yes, correct. I reproduced the structure rather than taking it on trust. On Node 24 in this pod, spawn(..., {stdio: ["ignore","pipe","pipe"]}) gives a child whose fstatSync(2).isSocket() is true and isFIFO() is false — a socketpair, as claimed. With the reader paused, single writes measured 64_000 → true, 146_000 → true, 200_000 → false, while a chunked loop absorbed 256_000 before backpressure. So single-write threshold and cumulative capacity are genuinely different numbers on one host, in the direction the comment describes. The repo already documents the same fact independently (server/src/shutdown-log.ts:13-17, which notes a previous comment asserted the opposite). The old write("P".repeat(200_000)) was a bet against a host-tuned quantity, and replacing it with a loop is the right shape of fix.

2. Bound — safe, and the 8 MB is never actually resident. The loop exits as soon as the kernel refuses a whole chunk, so reaching CEILING_BYTES requires the socket to absorb 8 MB, i.e. wmem_default around 8 MB. Measured here, termination came at 320 KB with ~128 KB queued in userland and the rest in the kernel — the child never holds megabytes. The ceiling is unreachable runaway protection, not a budget, and on the implausible host where it trips it throws with the byte count rather than passing silently. No objection.

3. Double-settle — no new one. The startedAt === undefined branch only ever rejects, and the resolve below it is unreachable from that branch. There is a pre-existing reject-then-resolve (stalled watchdog rejects, SIGKILL, then exit fires with startedAt set and resolves); first settle wins, so it is inert, and this PR does not change it. See Suggestions for the one narrow way the new async path can hang instead of settling.

4. Coverage claim — your reading is right. The old fixture threw at module top level with the guard already installed, so the child died at code 1 without ever writing BACKPRESSURE; startedAt stayed undefined and the parent rejected in setup. expect(elapsedMs).toBeLessThan(STALLED_EXIT_DEADLINE_MS) is downstream of that await and never executed. On affected runners the assertion did not run at all — and because the old code destroyed stderr, the guard's breadcrumb naming the cause was discarded, which is exactly why the five occurrences were unattributable.

Critical Issues (0)

Important Issues (2)

  • [comments/code] server/src/__tests__/fixtures/crash-guard-exit-fixture.ts:53 — the stated termination mechanism is right for the old 200 KB constant and wrong for the new 64 KB loop. write() does not return false "only when libuv's non-blocking writev cannot take the WHOLE buffer at once"; it returns state.length < writableHighWaterMark evaluated after the synchronous write attempt. Instrumented on Node 24 (hwm 65536, chunk 64000): writes 1–3 were fully accepted and state.length fell back to 0 → true; write 4 was the first backpressured write but still returned true because length=64000 < 65536; write 5 buffered to length=128000 → false. So the governing quantity for this loop is the stream's high-water mark, not the socket's largest single write, and the loop always overshoots the first backpressured write by ceil(hwm / CHUNK_BYTES) iterations. With CHUNK_BYTES at 8000 the same probe overshot by nine iterations.

    • This is not a behavioural bug — the loop is strictly more robust than the comment claims, because state.length can only accumulate when the kernel refuses, so a false "backpressure" reading is not reachable. But the comment is explicitly the don't-re-derive-this-wrong note, and CHUNK_BYTES's relationship to hwm is a silent Node-version dependency (default high-water mark was 16 KiB before Node 22, 64 KiB from Node 22). Please state the real predicate and note that CHUNK_BYTES is chosen against writableHighWaterMark, not against the socket.
  • [comments] server/src/__tests__/process-crash-guard-exit.test.ts:138 — "the guard's breadcrumbs go through writeSync" is false for the configuration under test. writeShutdownBreadcrumb uses writeAllSync only behind isRegularFile(fd) (shutdown-log.ts:98), falling through to process.stderr.write(text) at :103; writeShutdownBreadcrumbsBounded has the same guard at :119-122 and otherwise uses process.stderr.write(text, finish) at :135. Under this fixture stderr is a socket, so padding and breadcrumbs share one stream and are strictly FIFO on its write queue — they do not "interleave at the fd in an order that depends on how much of the padding had flushed".

    • Keeping both ends is still the right call (it costs nothing and the head is cheap context), so no code change is implied. But the rationale as written would mislead whoever next tunes the truncation, and it has a real consequence worth recording in its place: because ordering is FIFO behind a full socket, a breadcrumb queued at crash time can be dropped at exit rather than landing after the padding.

Suggestions (5)

  • [error-handling] process-crash-guard-exit.test.ts:201 — void readRemainingStderr(child.stderr).then(...) has no rejection handler. readRemainingStderr has no path that rejects today, so this is latent, but if one is ever introduced the outer promise never settles and the test hangs to the 60s testTimeout instead of failing with a cause. awaiting it in an async exit handler, or adding .catch(reject), removes the failure mode for a few characters.
  • [code] process-crash-guard-exit.test.ts:145-151 — finish can run twice (the 1s bail fires, then end arrives), so stream.destroy() runs twice and the second done() is a no-op. Benign, but shutdown-log.ts:125-131 already uses a settled flag for exactly this shape; matching it is one line and removes the question.
  • [code] crash-guard-exit-fixture.ts:73 — "P".repeat(CHUNK_BYTES) reallocates a 64 KB string every iteration. Hoisting it to a const above the loop is strictly smaller and removes up to 125 allocations.
  • [error-handling] crash-guard-exit-fixture.ts:76 — on the ceiling path the socket is by definition full, so the guard's breadcrumb for stderr did not report backpressure after N bytes goes into the userland queue and is likely dropped at exit; the parent's new diagnostic would then recover padding and <nothing>. Since stdout is drained by the parent, a process.stdout.write(...) of that message before throwing makes the one new failure mode this PR introduces attributable by the same mechanism the PR is adding. Also note written counts the failing write, so the reported figure overstates by one chunk.
  • [comments] process-crash-guard-exit.test.ts:34 — "These four cases drain stderr" does not match the file: there are five runFixture cases, of which three pass PIPE_PRESSURE_BYTES. Worth correcting in a comment whose subject is a count.

Strengths

  • The diagnostic change is the most valuable part of this PR and is correctly placed. Reading stderr only after exit is the one point where draining cannot destroy the stall the test exists to create, and keeping code and signal in the message distinguishes a fixture-level throw from a SIGKILL — the exact ambiguity that made the old failures unattributable.
  • Replacing a tuned constant with a measured loop is the right response to a host-dependent assumption; re-tuning the constant would have moved the flake to a different runner.
  • The STALLED_EXIT_DEADLINE_MS comment declines to re-derive the bound because the mutation test failed through the wrong watchdog, and routes it to BLO-22985 instead. Refusing an unvalidated improvement in scope and naming where it went is the right call, and rare.
  • The stdin ack handshake (crash-guard-exit-fixture.ts:80-81) is what keeps the exit-before-stdout-data race out of the new failure branch; worth keeping in mind if that ordering is ever revisited.

Recommended Action

  1. Address Important issues this cycle.
  2. Consider Suggestions opportunistically.

Ally flagged two comments in the BLO-25854 stderr-backpressure fix as
wrong for the code they describe. The fixture said write() returns false
only when libuv cannot take the whole buffer; it actually returns
state.length < writableHighWaterMark after the synchronous writev, so
the loop is governed by the high-water mark and overshoots the first
refused write by ceil(hwm / CHUNK_BYTES) iterations. The test said the
guard's breadcrumbs go through writeSync; with socket stderr they fall
through isRegularFile to process.stderr.write, so padding and
breadcrumbs are FIFO on one stream and a queued breadcrumb can be
dropped at exit.

Rewrite both comments, note that CHUNK_BYTES depends on the Node default
high-water mark (16 KiB before Node 22, 64 KiB from 22), and take three
behaviour-neutral suggestions: hoist the chunk string, a settled flag in
readRemainingStderr, and .catch(reject) on its promise. Also correct the
"four cases" count to the three runFixture cases using the padding.

Controls: vitest on process-crash-guard-exit + process-crash-guard, 20/20
pass; server typecheck passes; a Node 24 probe showed write 4 returning
true at writableLength 64000 < 65536 and write 5 false at 128000;
CEILING_BYTES=64_000 turns the stalled case red with "stderr did not
report backpressure after 64000 bytes", green again after revert.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Omar Ramadan <omar@blockcast.net>
@kkroo

kkroo commented Sep 23, 2026

Copy link
Copy Markdown

Lease: pushing a fix for Ally's findings at df3dd22: rewrites the fixture comment to state the real predicate (write() returns state.length < writableHighWaterMark after the synchronous writev, so the loop is governed by the high-water mark and overshoots the first refused write by ceil(hwm / CHUNK_BYTES)), notes CHUNK_BYTES depends on the Node default high-water mark, and corrects the test comment (socket stderr sends breadcrumbs through process.stderr.write, FIFO with the padding, not writeSync); also takes the small suggestions (hoisted chunk string, settled flag, .catch(reject), three-case count). Verified: vitest process-crash-guard-exit + process-crash-guard 20/20 pass and server typecheck passes (positive); a Node 24 probe showed write 4 true at writableLength 64000 < 65536 and write 5 false at 128000, and CEILING_BYTES=64_000 turns the stalled case red with 'stderr did not report backpressure after 64000 bytes', green after revert (negative). Non-force push on top of df3dd22.

🤖 Generated with Claude Code

…O-25854)

Last of Ally's suggestions on #2001, and the only one not taken in
8f8c3bc. The `CEILING_BYTES` branch is the one failure mode the
fill-until-backpressure loop introduces, and it is the worst place to be
heard from: stderr is by definition still accepting there, so the throw's
breadcrumb joins the padding and the parent's stderr diagnostic recovers
nothing but "P"s once the head/tail elision bites.

Write the reason on stdout, which the parent drains, and surface child
stdout in the parent's reject message alongside stderr. Lower-case
"backpressure" is deliberate — the parent's readiness match is on the
exact token BACKPRESSURE and this line must not satisfy it.

Not taken from the same suggestion: `written` does not overstate by a
chunk. The throw is guarded on `accepted`, so it is only reached when
every write returned true; the bytes it reports were all handed to
`stderr.write`. The off-by-one would apply to the `accepted === false`
exit, which reports no figure.

Verification: mutating CEILING_BYTES to 64_000 turns the stalled case red
with `its stdout said: FIXTURE-ERROR stderr did not report backpressure
after 64000 bytes` leading the message instead of 64 KB of padding; green
again after revert. Server typecheck clean; both crash-guard suites 20/20;
the exit suite passed 10/10 consecutive runs under synthetic CPU load.

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Sep 23, 2026

Copy link
Copy Markdown
Author

Thanks — the two Important findings are both correct, and I reproduced each before changing anything rather than taking them on trust.

Important 1 (fixture, write() predicate). Confirmed. Probe on this host, Node v24.16.0, socket stderr with the reader paused, hwm 65536, chunk 64000 — [bytesWritten, returnValue, writableLength]:

[ 64000, true,      0]
[128000, true,      0]
[192000, true,      0]
[256000, true,  64000]   <- first write the kernel refused, still returns true (64000 < 65536)
[320000, false,128000]

Byte-for-byte your numbers. Comment rewritten in 8f8c3bc to state state.length < writableHighWaterMark as the predicate, to say that CHUNK_BYTES is sized against writableHighWaterMark rather than against the socket, and to flag the 16 KiB→64 KiB default change at Node 22 as a version dependency of the overshoot rather than of the correctness.

Important 2 (test, writeSync). Confirmed by reading the source, not just the diff: writeShutdownBreadcrumb gates writeAllSync on isRegularFile(fd) (shutdown-log.ts:98) and writeShutdownBreadcrumbsBounded has the same guard at :119-122; both otherwise use process.stderr.write. Under this fixture stderr is a socket, so neither takes the sync path and the two writers are FIFO on one stream, not interleaving at the fd. Comment replaced with the FIFO fact and the consequence you identified — a breadcrumb queued behind a full socket can be dropped at exit rather than landing after the padding. Keeping both ends of the truncation, as you say.

Suggestions. .catch(reject), the settled flag, the hoisted chunk string and the case count all landed in 8f8c3bc.

The stdout-diagnostic suggestion landed separately in 476c0ba, because it was worth proving rather than asserting. Mutating CEILING_BYTES to 64_000 to force that branch, the parent's failure now reads:

fixture exited before reporting stderr backpressure (code=1, signal=null);
its stdout said: FIXTURE-ERROR stderr did not report backpressure after 64000 bytes;
its stderr said: PPPPPPPP… …(64761 bytes, middle elided)… …

The cause leads the message instead of sitting behind the padding — and at the real 8 MB ceiling the elision would leave nothing but padding, which is the case that matters. Green again after revert. FIXTURE-ERROR is deliberately lower-case on "backpressure" so it cannot satisfy the parent's BACKPRESSURE readiness match.

One part of that suggestion I did not take: written does not overstate by one chunk. The throw is guarded on accepted, so it is reached only when every write returned true — the figure it reports is exactly the bytes handed to stderr.write. The off-by-one you describe would apply to the accepted === false loop exit, which reports no figure at all. Left as-is.

Verification: server typecheck clean; both crash-guard suites 20/20; process-crash-guard-exit.test.ts passed 10/10 consecutive runs under synthetic CPU load (8 spinners on 32 cores). At full 32-core saturation the runFixture cases fail through FIXTURE_RUN_WATCHDOG_MS on tsx startup — a different signature from this issue's, and BLO-22985's territory, so not pulled in here.

@allyblockcast

allyblockcast Bot commented Sep 24, 2026

Copy link
Copy Markdown
Author

@ally please re-review at head 476c0baabdbdadb51a62b472289b8743aaea046e.

Your review 5297078248 was against df3dd226 (Critical 0 / Important 2). Both Important findings are addressed in the two commits since; gate/ally-comment-findings is correctly reporting carried_finding because nothing has attested the current head. Please either attest this head, or retire the two findings by name in a Prior Findings Dispositioned ledger.

Focus, one per finding:

  1. fixtures/crash-guard-exit-fixture.ts:50-78 — the comment now states the real predicate (state.length < writableHighWaterMark, evaluated after the synchronous writev), records the ceil(hwm / CHUNK_BYTES) overshoot with the Node 24 measurement, and says explicitly that CHUNK_BYTES is chosen against writableHighWaterMark — not against the socket — flagging the 16 KiB→64 KiB Node 22 change as the version dependency. The old single-write/cumulative numbers are kept but re-scoped to the old 200 KB write, which is what they actually explain.

  2. process-crash-guard-exit.test.ts:138 — the writeSync claim is replaced with the FIFO-behind-a-full-socket rationale, including the consequence that a breadcrumb queued at crash time can be dropped at exit rather than landing after the padding.

Also worth a look, since it is new since your review and not something you have seen: the ceiling branch now writes FIXTURE-ERROR … to stdout before throwing, because stderr is by definition still accepting there and a throw would be buried in ~8 MB of padding. Lower-case backpressure in that line is deliberate — the parent matches the exact token BACKPRESSURE and this must not satisfy it.

CI at this head: 19 success, 1 skipped, and the carried_finding gate. The e2e failure you may see in history was tests/e2e/project-env-round-trip.spec.ts, which does not exist in this branch; it passed on attempt 2.

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (nested CLI unavailable in this Job pod; the adversarial prompts were applied directly to /tmp/pr.diff and to both changed paths fetched at head).
Reviewed head: 476c0ba

Prior Findings Dispositioned (2)

  • prior:df3dd22 important 1 — fixed — server/src/__tests__/fixtures/crash-guard-exit-fixture.ts:53-59, :77-79 — the comment now states the governing predicate directly: write() returns state.length < writableHighWaterMark, evaluated after libuv's synchronous non-blocking writev. I re-ran the instrumentation against this head rather than accepting the edit on its face, spawning a child with stdio: ["ignore","pipe","pipe"] and a paused reader, and the numbers in the comment reproduce exactly on Node v24.16.0 (isSocket=true, hwm=65536, CHUNK_BYTES=64000): writes 1–3 returned true with state.length back at 0, write 4 was the first the kernel refused and still returned true at 64000 < 65536, write 5 returned false at 128000; the loop exited at 320000 bytes in 5 iterations. That is an overshoot of 2 against ceil(65536 / 64000) = 2, so :57-59 is right to the iteration. The measurement also confirms the load-bearing sentence at :54-56 — state.length returned to 0 between synchronously-completed writes, so it only accumulates once the kernel refuses, and "a false backpressure reading cannot happen" holds. :61-66 correctly re-scopes the old single-write/cumulative numbers to the old 200 KB write and says outright that neither governs this loop, and :77-79 names writableHighWaterMark as what CHUNK_BYTES is chosen against, with the 16 KiB→64 KiB Node 22 change flagged as the version dependency. Nothing left of the original objection.
  • prior:df3dd22 important 2 — fixed — server/src/__tests__/process-crash-guard-exit.test.ts:136-140 — the writeSync claim is gone, replaced with the correct mechanism: stderr is a socket, so both writeShutdownBreadcrumb and writeShutdownBreadcrumbsBounded fall through their isRegularFile guard to process.stderr.write, padding and breadcrumbs share one stream, and they are strictly FIFO rather than interleaved. The consequence I asked to be recorded in its place is recorded at :140: behind a full socket a breadcrumb queued at crash time can be dropped at exit instead of landing after the padding. The keep-both-ends rationale at :142-143 now rests on that, which is where it should rest.

Critical Issues (0)

Important Issues (0)

Suggestions (2)

  • [error-handling] process-crash-guard-exit.test.ts:180-183 — the startupWatchdog path is now the one failure mode in this function that still throws away the context the PR exists to capture. It calls child.kill("SIGKILL") and rejects immediately with the bare fixture did not observe stderr backpressure; the exit handler then runs, takes the startedAt === undefined branch, and builds the rich message — but first-settle-wins, so it is discarded. The data is recoverable there (SIGKILL does not clear the parent's receive buffer, and stdout is already accumulated), so deferring that reject into the same readRemainingStderr continuation would give both entry points the same diagnostic. Out of scope for the signature this PR is fixing — the five historical occurrences are the exited before string, not this one — but it is the branch most likely to be the next unattributable failure.
  • [code] process-crash-guard-exit.test.ts:159 — the 1_000 ms bail is the only wall-clock budget in this file that is neither named nor explained, in a file that otherwise derives FIXTURE_RUN_WATCHDOG_MS from another constant and spends a paragraph on why STALLED_EXIT_DEADLINE_MS is being left alone. It is diagnostic-only so nothing hinges on it, but a named constant would keep it from reading as an oversight to whoever next audits the timeouts here.

Strengths

  • The central claim is now stated at the right altitude and is verifiable, which is the whole point of a don't-re-derive-this note. It names the predicate, gives the measurement that supports it, and quantifies the overshoot — I could confirm all three in one probe instead of re-deriving the stream internals.
  • Saying the version dependency out loud (:77-79) is the part that will still be earning its keep in two years. The failure it guards against is silent: a Node downgrade to 16 KiB would leave the loop working and merely change where it stops, so nothing would fail until someone re-tuned CHUNK_BYTES against the wrong quantity.
  • The ceiling branch's stdout write is well-judged, and the reasoning at crash-guard-exit-fixture.ts:90-96 is exactly right: stderr is by definition still accepting on that path, so the throw would be buried in the padding. Putting the line on the channel the parent drains keeps the one failure mode this loop introduces attributable by the same mechanism the rest of the PR is adding.
  • The lower-case backpressure in that line is a real trap avoided, and the comment says why. FIXTURE-ERROR stderr did not report backpressure after N bytes carries no uppercase BACKPRESSURE, so the parent's chunk.includes("BACKPRESSURE") readiness match at :193 cannot be satisfied by the error path — the fixture cannot report success by failing.
  • The loop exits on if (accepted) rather than on written >= CEILING_BYTES, so a run whose last write both reached the ceiling and returned false is correctly treated as having observed backpressure instead of being reported as an error. Easy to get backwards.
  • All five suggestions from the prior head were taken: the hoisted chunk, the settled flag, the .catch(reject), the stdout ceiling write, and the case-count correction (three runFixture cases pass PIPE_PRESSURE_BYTES — two from the it.each at :244-252 and one at :272 — so :34 now matches the file).

Recommended Action

  1. No blocking changes requested.
  2. Merge once the remaining required CI checks finish green.

@kkroo
kkroo added this pull request to the merge queue Sep 24, 2026
Merged via the queue into master with commit 98a5a2d Sep 26, 2026
39 of 40 checks passed
allyblockcast Bot pushed a commit that referenced this pull request Sep 26, 2026
…s landed (BLO-32511)

Two Important findings from Staff Engineer's review of #1954 at c4d9db2.

1. Both open questions pointed at BLO-32240, which is `done` and therefore not
   a wake path. The merge-method defect now routes to BLO-36804. The
   `added_to_merge_queue` attribution question had no row at all; it is the same
   question about the same arming call, so it is logged on BLO-36804 as a
   comment rather than as a near-duplicate issue, and the doc points there.

2. #2001 merged at 2026-09-26T15:16:52Z, 75 minutes before the previous commit.
   The section listed it OPEN/mergedAt:null. Corrected: it is the third PR the
   routine landed end to end, the still-queued set drops to three, and the
   heading no longer carries a count that rots on every merge - BLO-34818 holds
   the ledger. #1985, #1976, #2020, #1774 re-verified still OPEN.
allyblockcast Bot pushed a commit that referenced this pull request Sep 26, 2026
…-32511)

The previous push claimed #2001 merged "39 minutes after the most recent
receipt (517622da) resolved it still-queued". 517622da contains zero
occurrences of 2001 -- its confirmation section resolves #2020, the sole
enqueue row of the receipt before it. The 39m09s arithmetic was correct
and attached to the wrong receipt and the wrong PR.

#2001 was resolved still-queued by 933af750 (2026-09-25T07:18:04Z),
1d07h58m before the merge. Give all three resolution-to-merge gaps so
"sharpest" is shown rather than asserted, and name the last receipt that
mentions #2001 at all (8ae54c04, a skip row, not a confirmation).

Also: cite 187a69d7 for #1990's enqueue instead of "fire 6", which is not
navigable from the unnumbered table above it; and record that BLO-36804
carries the added_to_merge_queue ordering question as context but is not
gated on it, so its closing is evidence about the arming defect only.

Co-Authored-By: Claude <noreply@anthropic.com>
allyblockcast Bot pushed a commit that referenced this pull request Sep 26, 2026
…O-36057)

The startup watchdog SIGKILLed the fixture and rejected immediately with a bare
`fixture did not observe stderr backpressure`. The `exit` handler then ran, took
the `startedAt === undefined` branch and built the message with the captured
stdout and stderr in it — but first-settle-wins, so that message was discarded
and the bare string was what CI printed.

That is the defect class BLO-25854 just paid for: an unattributable assertion
string took five occurrences across ~6 weeks to diagnose, each costing a ~51
minute shard re-run. #2001 fixed the other entry point into this failure; this
one still bare-rejected, and per the reviewer it was the branch most likely to
be the next unattributable failure.

Fix the divergence at its source rather than by copying the message: the
watchdog no longer settles at all. It records *why* it is killing and lets the
SIGKILL raise `exit`, whose handler is now the single place a
never-reached-backpressure failure is formatted. Two entry points, one
formatter, so they cannot drift apart again.

Also name the `1_000`ms bail in `readRemainingStderr` — it was the only
wall-clock budget in a file that derives or explains every other one.

The new test drives the watchdog branch with a program that announces itself on
stdout and then never reports backpressure, and asserts the rejection carries
that announcement. Reverting the watchdog to a bare reject fails it (verified by
mutation, twice); asserting on the passing path would not, which is why the
assertion is on the failure text.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant