Repository navigation
fix(core): bound CDP sends so a stalled browser can't hang the CLI - #2401
Conversation
Browser.send and Session.send resolved only on a matching CDP reply or a close, with no deadline of their own. Chrome does not always reply: a renderer or browser process that is frozen, starved of CPU, or swapped out keeps its target attached and simply goes silent. Callers cannot defend against this either, because their deadlines are in-page timers living in the very renderer that stopped running -- which is why @percy/storybook's PERCY_STORY_RENDER_TIMEOUT never fires in these runs. The result is a total silent stall: no error, no log, no retry, and a build left unfinalized until the server force-closes it hours later. Add a Node-side deadline that nothing inside the page can defeat, on both the session-scoped and browser-scoped connections. Target creation and attachment run on the browser connection before any page exists, so bounding sessions alone still strands that path. Default 120s, floored above the longest legitimate in-page wait and auto-raised with PERCY_PAGE_LOAD_TIMEOUT / PERCY_STORY_RENDER_TIMEOUT; PERCY_CDP_TIMEOUT overrides it, and 0 disables it. The raw-message path stays untimed -- a session owns that reply. On expiry it rejects, so existing retries handle it. Reproduced end to end on a 2-core container: an unfixed run goes silent after "Page created" until the 5-minute monitoring timeout, while the same run with the deadline reports "Protocol error (Runtime.enable): Timed out after 30000ms" and retries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
73fd594 to
96a9ba9
Compare
restart() force-closes the browser first, and close() already clears every pending callback's deadline before emptying the map -- so the loop added with the browser-scoped deadline could never execute. It shared a line with the statement after it, which is why coverage read 99.24% statements against 100% lines and tripped the 100% global threshold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rishigupta1599
left a comment
There was a problem hiding this comment.
Claude Code Review (automated) — 6 inline finding(s). Full report in the PR comment below. Verdict: Passed.
|
|
||
| // Shared by both CDP deadlines - the session-scoped one below and the | ||
| // browser-scoped one in browser.js. | ||
| export function cdpTimeoutMessage(method, timeout) { |
There was a problem hiding this comment.
[Medium] CDP timeout errors are not recognised by the browser-restart retry heuristic
discovery.js:722 restarts the browser on retry only when the error message contains Browser not connected, Browser closed, Session closed or Session crashed. The message this function builds matches none of them, so a protocol timeout never triggers percy.browser.restart().
Recorded Medium rather than the High originally proposed: browser.page(...) is called inside the retried generator (discovery.js:640), so each retry already gets a fresh page and renderer. For the frozen-renderer case this PR targets, the retry escapes the wedged renderer without a restart. The gap is real only for a wedged browser process, where the snapshot fails with a clear error after ~3x the deadline instead of hanging forever — slower recovery, not incorrect behaviour.
Suggestion: export an isCdpTimeoutError(error) predicate here next to cdpTimeoutMessage and use it in the onRetry check, so the restart path also fires on a protocol timeout and the contract is not a duplicated substring match.
Reviewer: stack-code-reviewer
|
|
||
| // Resolved once per process, and memoized so a mid-run env change cannot make | ||
| // concurrent sessions disagree. `PERCY_CDP_TIMEOUT=0` disables the deadline. | ||
| export function cdpTimeout() { |
There was a problem hiding this comment.
[Low] Timeout resolves into process-global, first-caller-wins state
The resolved deadline is frozen on the Session class for the life of the process, so the first CDP send anywhere wins. Two Percy instances in one process with different config silently share the first value, and no shared test helper resets it — a latent cross-spec pollution vector (discovery.test.js:2137 sets PERCY_PAGE_LOAD_TIMEOUT=80000, which would feed inPageCeiling() if it were the first caller).
Recorded Low rather than Medium: this mirrors the pre-existing Page.TIMEOUT at page.js:75 / page.js:406-408 rather than introducing the pattern, and the new specs reset it in both beforeEach and afterEach.
Suggestion: scope the resolved value to the Browser/Percy instance, or export a reset hook wired into the shared setupTest(). Best done as a follow-up covering Page.TIMEOUT too.
Reviewer: stack-code-reviewer
| function inPageCeiling() { | ||
| return Math.max( | ||
| parseInt(process.env.PERCY_PAGE_LOAD_TIMEOUT, 10) || 0, | ||
| parseInt(process.env.PERCY_STORY_RENDER_TIMEOUT, 10) || 0 |
There was a problem hiding this comment.
[Low] PERCY_STORY_RENDER_TIMEOUT has no consumer in this repo
This is its only occurrence in the monorepo, so a reader cannot tell from here whether it is intentional or a leftover. It is intentional — the Storybook SDK sets it outside this repo, and keeping the Node deadline above that in-page wait is the whole point of inPageCeiling().
Suggestion: name the owning package in the comment, e.g. "set by @percy/storybook; not read anywhere else in this repo", so the reference explains itself.
Reviewer: stack-code-reviewer
| let timeout = cdpTimeout(); | ||
|
|
||
| if (timeout > 0) { | ||
| callback.timer = setTimeout(() => { |
There was a problem hiding this comment.
[Low] Deadline timers are not unref'd
Same at browser.js:308. For a programmatic consumer that finishes without closing the session or browser, an in-flight command with no reply can hold the event loop open for the full deadline. Moot for percy exec, which calls process.exit() explicitly.
Suggestion: callback.timer.unref?.(), matching the existing precedent at server.js:226 — a watchdog timer should never be the reason a process cannot exit.
Reviewer: stack-code-reviewer
|
|
||
| // browser.send() is awaited before the callback is registered, so give that | ||
| // microtask a chance to land before simulating a reply | ||
| const tick = () => new Promise(r => setTimeout(r, 10)); |
There was a problem hiding this comment.
[Low] Specs assert timing with real sleeps rather than fake timers
The 10ms tick() plus the 150/200/250ms waits add about a second of wall-clock time and make the "has not settled" assertions probabilistic under load rather than deterministic.
Suggestion: use jasmine.clock() for the scheduling assertions, keeping real timers only where the awaited browser.send() microtask genuinely needs them.
Reviewer: stack-code-reviewer
| // The browser process can go silent the same way a renderer can. Target | ||
| // creation and attachment happen on the browser-scoped connection, before any | ||
| // page exists, so a deadline on Session alone leaves that path unbounded. | ||
| describe('Unit / Browser CDP deadline', () => { |
There was a problem hiding this comment.
[Low] Browser-scope specs live in session.test.js
This block is here because the shared cdpTimeout/cdpTimeoutMessage exports live in session.js, which hurts discoverability for anyone looking for browser specs.
Suggestion: move the shared exports into a small cdp-timeout.js imported by both — which also decouples browser.js from Session's internals — and give the browser specs their own file.
Reviewer: stack-code-reviewer
Claude Code PR ReviewPR: #2401 • Head: b5c16e1 • Reviewers: stack-code-reviewer SummaryAdds a Node-side deadline to Review Table
FindingsNone of the findings below describe behaviour this PR makes worse than the status quo. Before the change, the failure mode is an unbounded silent hang; after it, every path is bounded and reported. The findings are unrealized improvements and polish.
Verdict: PASS — the deadline is correct and well-tested, and every finding is a follow-up improvement rather than a defect introduced here. |
Summary
Browser.sendandSession.sendresolved only on a matching CDP reply or a close — neither had a deadline of its own. Chrome does not always reply: a renderer or browser process that is frozen, starved of CPU, or swapped out keeps its target attached and simply goes silent.Callers cannot defend against this either, because their deadlines are in-page timers living in the very renderer that stopped running. That is why
@percy/storybook'sPERCY_STORY_RENDER_TIMEOUT(added in percy/percy-storybook#1354, shipped in 10.0.2) never fires in these runs.Net effect: a total silent stall — no error, no log, no retry, and a build left unfinalized until the server force-closes it ~8h later.
This adds a Node-side deadline that nothing inside the page can defeat, on both the session-scoped and browser-scoped connections — target creation and attachment run on the browser connection before any page exists, so bounding sessions alone still strands that path:
PERCY_PAGE_LOAD_TIMEOUT/PERCY_STORY_RENDER_TIMEOUTare raised, so tuning those can never cause a spurious protocol timeout.PERCY_CDP_TIMEOUToverrides it;PERCY_CDP_TIMEOUT=0disables it.withPageretry handles it — the same retry that already recovers a story whose preview page reloads mid-transition.Why the previous fix wasn't enough
This was previously diagnosed as the preview page reloading itself (
next/routercallingwindow.location.reload()), and fixed with an in-page render deadline. That fix is correct and shipped, but this is a different branch of the same failure — reported runs on 10.0.2 still stall at the same story with the 30s deadline never firing.Reproduced with the real CLI
Pagedriving the real shippedevalSetCurrentStory:next/router), 7 timingsSession crashed!Page.setWebLifecycleState)The last two rows match the reported signature exactly: total silence, no error, no timeout, CPU 0.5–5%, target still attached — consistent with a 2-core/8GB CI agent under memory pressure.
Test plan
packages/core/test/unit/session.test.js(11 specs): for each ofSession.sendandBrowser.send, the deadline fires and names the command, points at thePERCY_CDP_TIMEOUToverride, does not fire when a reply arrives, is cleared on reply so it cannot fire late, is cleared on close, and is disabled byPERCY_CDP_TIMEOUT=0; plus the raw-message path staying untimed.percy storybook(@percy/cli1.32.7 +@percy/storybook10.0.2, Chrome 143 linux-x64). Unfixed, the run goes completely silent afterPage createdat ~0.1% CPU until the 5-minute monitoring timeout — no error, no timeout, no retry:With the deadline backported onto that same 1.32.7 install, the identical run reports the hanging command and recovers via the existing retry:
That also identifies what was stranding the process —
Runtime.enable/Network.enable, the first commands sent to a new page's session. Core asset-discovery plumbing, unrelated to any SDK.PERCY_CDP_TIMEOUT=0) the same runs hang indefinitely again.eslintclean.Note on the browser-scoped deadline
The first version of this PR bounded
Session.sendonly, on the reasoning that a wedged renderer leaves the browser process responsive. The container run disproved that: it stranded inbrowser.page()→Target.createBrowserContext, before any page existed.Browser.sendhas the identical unbounded-promise shape, so it is bounded here too.