fix(instances): size the automatic warm-set cap by registered crews, not connected ones - #8573
Conversation
UX Review (Fable 5) — ✅ PASSUX-level review of UX-Verdict: PASS Backend-only cap fix plus logging; the user-visible outcome is one less phantom disconnect, and the updated settings copy honestly describes the new behavior. Suggestions
[UX-REVIEWED] 9beb358 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Root cause fixed at the right layer: the cap now derives from declared configuration, not from live state it was racing, and every doc, help text, and baseline moves in the same commit. The instrumentation half is proportionate to the failure class it targets (only reproduces across slow restarts, previously left zero evidence anywhere): it is a self-contained module plus logging-only call sites, fully reversible, and its untrusted-frame handling — fail-closed frame identity, aggregate write budget, newline/length sanitization, token redaction — treats pane output as the external-content boundary it is rather than opening a new one. The residual case (eviction past the auto ceiling of 8, or under an explicit tight cap, still reads as a disconnect at the pane) is disclosed in the docs and now leaves log evidence; making an evicted pane visually distinct from a disconnect is legitimate follow-up work outside this PR's scope. [DESIGN-REVIEWED] 9beb358 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All verification done — the contract's counts are in hand. Here is the review. First-Principles-Verdict: CONCERNS The cap fix is cause-level and self-sufficient; four-fifths of the diff is a permanent, always-on logging surface riding along in a What this change shipsIntent: stop one random crew pane from being silently evicted (and looking broken) whenever the dashboard polls before every tunnel finishes connecting — a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 9beb358 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've completed my analysis. Candidate 1 — Falsifying it: the claim requires (a) a URL reaching No other item in the diff grounds to the three-part bar (the No findings. [OPUS-REVIEWED] 9beb358 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- website/electron/frame-load-log.js:389 -- remote False positive or not applicable? A repository writer can comment: |
fd94ccf to
0e0cdbc
Compare
0e0cdbc to
01a629c
Compare
01a629c to
c7eb0a8
Compare
Legitimate and in scope: the spec is the reference for this module, and leaving it Three places carried the stale claim, and all three are now updated:
Every one of the three also states the automatic ceiling explicitly rather than Gates re-run after the edit: |
Legitimate, and the arithmetic is exactly right: Fixed in both places the finding names:
The help string is snapshotted, so Gates re-run: |
Taken seriously rather than waved off, because if it held, the whole Version, from the pinned dependency actually installed:
on(event: 'console-message', listener: (details: Event<WebContentsConsoleMessageEventParams>,
/** The log level, from 0 to 3 … @deprecated */
level: number,
/** The actual console message @deprecated */
message: string,
/** … @deprecated */
line: number,
/** @deprecated */
sourceId: string) => void): this;Argument 0 is the details object — it carries So Same for the navigation handlers. on(event: 'did-start-navigation', listener: (details: Event<WebContentsDidStartNavigationEventParams>,
/** @deprecated */ url: string,
/** @deprecated */ isInPlace: boolean,
/** @deprecated */ isMainFrame: boolean,
/** @deprecated */ frameProcessId: number,
/** @deprecated */ frameRoutingId: number) => void): this;
One thing the finding is right about in spirit, recorded rather than acted on: the Verified locally at head |
The finding's premise is that this PR touched the copy. It did not. The file Evidence at head The complete diff is 19 files: No path under Not dismissed as merely out of scope: if |
89e6617 to
d111944
Compare
Legitimate, and the reachability is not hypothetical — the call site says so in as Fixed on both axes the finding names.
Two choices worth stating, because they are where this could have gone wrong.
A pane's console errors are still journaled: a framing refusal or a refused One thing the finding did not raise, in the same class and fixed with it: the
Verified at head |
d111944 to
a8203c2
Compare
|
GPT 5.6 disposition — top-level remote navigation forging trusted pane records
The finding is right and I had the reasoning wrong. My previous round gated the What changed:
Tests, in
Whole electron package suite: 1657 passed, 1 skipped, 0 failed. |
|
GPT 5.6 disposition — distinct console errors bypassing the logging bound
The finding is correct. What changed:
The trade is explicit rather than hidden: past the budget a genuinely broken pane Tests, all four running the attack rather than the happy path:
The rewritten |
|
Opus disposition —
Confirmed against the code. The substring match itself is not the bug and stays: it is what catches function isSecretValue(key, value) {
return SECRET_KEYS.test(key) && typeof value !== 'boolean'
}
vitest ( |
a8203c2 to
454f066
Compare
|
GPT 5.6 disposition — untrusted frame navigation bypassing the logging bound (restructure round)
The finding is right. Last round I added Because this is the same span three rounds running, I stopped patching the symptom and
Made structural:
The budget stays charged only for lines actually written (suppressed repeats do not Tests added to
frame-load-log suite 45 passed; whole electron package suite 1662 passed, 1 skipped, If a fourth round lands on this span I will treat the trust-per-path model itself as |
…not connected ones The automatic cap (instances.warm_set_cap = 0) was resolved from the number of crews CONNECTED at the instant GET /api/instances was served. That made it a race against tunnel startup: with four crews configured and the fourth still connecting when the dashboard polled, the served cap came back 3, and the viewport dutifully evicted an LRU pane to honour it. Eviction is indistinguishable from a disconnect at the pane -- the iframe is unmounted, the token re-minted, and the remote SPA cold-boots on the next click -- so the user saw exactly one crew stuck loading forever. Which crew depended on which tunnel finished last, so the victim moved on every restart and read as a random failure rather than as a cap. Count registered crews instead. That is a function of configuration, not of live state, so it cannot race, and it widens by itself when a crew is added -- nobody has to remember to raise the cap alongside. WARM_SET_CAP_AUTO_CEILING (8) still bounds it, and an explicit cap >= 1 is still honoured verbatim. Also journals crew-pane load outcomes, because this class of bug only reproduces across a restart and left no evidence anywhere. A pane that never becomes a live document shows only "loading pane": the remote gateway configures no aiohttp access_log, the main window hooked no frame load events, and a packaged app has no devtools console to open, so the one question that decides the diagnosis -- did the frame ever navigate, and with what result -- had no answer. Two halves, joined in gateway-launch.log: - frame-load-log.js hooks did-start-navigation, did-frame-navigate and did-fail-load on the dashboard's webContents. Together they separate the three failures that look identical on screen: no start line at all means the frame was never pointed anywhere; a start with no commit means the request went out and never came back; a commit with status=403 means the remote refused the token. - src/lib/paneLog.ts journals the renderer's own view of the pane lifecycle (mint, warm, ready, timeout, retry, postMessage delivery). Its lines carry a [pane] prefix that the forwarder honours, so they interleave with the Chromium frame events they explain. The [pane] prefix is a marker, not a capability. The crew panes are cross-origin iframes OF the dashboard's webContents, so console-message fires for their documents too, and a prefix test alone would let a compromised remote gateway print entries into gateway-launch.log -- forging the very record that says whether its pane was ever requested, and buying the higher repeat cap and the exemption from the error-severity filter along with it. The forwarder therefore gates the prefix on frame identity, and identity takes two checks rather than one: Electron's console-message details carry the emitting WebFrameMain, so the forwarder requires BOTH parent === null (the top of the frame tree, a relationship Chromium owns and a nested page cannot claim) AND that the frame's origin equal the URL the window was loaded with. Position alone is not identity -- a cross-origin pane that gets a user click on a target="_top" link can navigate the top-level window, and the remote document then has parent === null too. sourceId is deliberately not used for either check -- any script can rewrite it with a //# sourceURL= comment. Both fail closed, so a runtime that supplies no frame, or a caller that configures no origin, loses the INFO-level journal rather than trusting an unverifiable claim. A pane's console ERRORS are still journaled, since a framing refusal inside the pane is exactly the diagnosis, but they are attributed to the emitting origin so a reader can tell pane-controlled text apart from the dashboard's own. All of that text -- message, sourceId, origin -- is escaped before it is written: gateway-launch.log is read by tailing it, so a raw newline would let a pane forge whole entries, and length is capped so one enormous message cannot scroll the lines around it out of the tail. The volume is bounded, and the bound is aggregate rather than per-path because a pane drives more than one path. The repeat counter is keyed by message TEXT, and text is what a compromised pane chooses: varying it defeats a per-message cap, and clearing the key map on overflow restarts the counting rather than holding the line. A pane can just as easily loop its own frame's NAVIGATIONS -- every did-start-navigation / did-frame-navigate / did-fail-load is another unconditional line -- so a limit on the console path alone is one the pane walks around by switching paths. Every emission therefore funnels through one writer (record), and every line attributable to an untrusted frame, console or navigation alike, is charged against a single per-attachment budget (UNTRUSTED_LOG_BUDGET = 100), independent of what the text says. It is charged only for lines actually written, and the last line it admits names the budget so the silence after it reads as the cap rather than as the pane recovering. A navigation carries only isMainFrame as a trust signal -- the positional events pass no frame or origin -- so a subframe navigation is the pane's and is budgeted, while a top-frame navigation is the dashboard's own handful and is not. Trusted and untrusted console repeat counters stay separate maps, so pane volume cannot evict the dashboard's own. Both redact ?token= while still recording that a token was present -- a token the remote rejected is the failure worth diagnosing; its value never is. Console output is otherwise filtered to errors, because renderer warnings arrive by the hundred per paint and would bury the load lines. The journal is always on rather than debug-gated: the failure is intermittent and can vanish on a restart, so the next occurrence has to leave a record without anyone having enabled anything beforehand. Also adds frame-load-log.js to build.files. electron-builder ships an explicit per-file allowlist, so an unlisted module works from source and is simply absent from the DMG -- the shell-contract drift guard caught this. The journal's redaction sentinels needed an i18n exemption. `<redacted>`, `<empty>`, `<unserializable>`, `?token=<redacted>` and `?<query>` are log tokens whose reader is whoever greps gateway-launch.log, so translating one would make the journal unsearchable in the incident it exists for -- and a renamed `<redacted>` would read as if the token had been printed. Two anchored shapes in eslint.i18n.config.js, both narrow enough that a placeholder inside a sentence stays reportable copy, with cases in i18nLintExemptions.test.ts pinning both directions.
454f066 to
9beb358
Compare
Problem / Motivation
Exactly one remote crew pane sits at "loading" forever. The tunnel is up the whole
time, the other crews are fine, and which crew is stuck changes on every
gateway restart — so it reads as a flaky SSH link rather than as a setting doing
what it was told.
Why it matters
The user has no way to attribute it. Nothing is red: the tunnel reports
connected, the registry looks right, and the only symptom is a spinner in onepane. The natural response is to go hunting through SSH config, tokens and
frame-ancestors— which is exactly what happened here — while the actual cause isa cap that came back one short. And because the victim moves each restart, every
"fix" appears to work once.
What changed (motivation → approach → change)
Symptom → one pane never becomes ready, victim rotates per restart.
Root cause → the automatic warm-set cap (
instances.warm_set_cap = 0, theshipped default) was resolved from the number of crews connected at the instant
GET /api/instanceswas served:That made the cap a race against tunnel startup. With four crews configured and
the fourth still connecting when the dashboard polled, the served cap came back
3, andInstancesViewport's K-cap effect dutifully evicted the LRU pane tohonour it.
Eviction is indistinguishable from a disconnect at the pane: the iframe is
unmounted, the token re-minted, and the remote SPA cold-boots on the next click.
So one crew looks broken, and which one depended on connection order — hence a
victim that moves on every restart.
Change → count registered crews instead (
len(items)from the registry).That is a function of configuration rather than of live state, so:
slot;
cap alongside. Forgetting that is precisely what reintroduces the eviction.
WARM_SET_CAP_AUTO_CEILING(8) still bounds automatic mode, so a large fleet stillevicts rather than mounting an unbounded number of dashboard SPAs in one renderer.
An explicit cap
>= 1is still honoured verbatim, including a value below theregistered count — a deliberately tight cap is the only knob that bounds renderer
cost, and silently widening it would defeat the operator's own trade.
The setting's help text moves off "connected" onto "configured" in the same commit,
and
config-baseline.json— the committed snapshot of the config schema, whichtest_config_baseline.pyasserts byte-identical against its generator — isregenerated to match. That one generated line is the whole of the baseline's diff.
docs/system-specs/modules/instances.mdmoves with the code: §2's warm-setparagraph, the
instances.warm_set_capconfig-table row and the LRU-evictiontroubleshooting row all described the cap as tracking the live connected count,
which this change makes untrue. Each now says registered, and each states the
automatic ceiling explicitly rather than promising that no configured crew is ever
evicted — past
WARM_SET_CAP_AUTO_CEILING(8) one is.The instrumentation half
The same commit adds a permanent journal of crew-pane load outcomes, because this
class of bug only reproduces across a restart and a restart is slow:
website/electron/frame-load-log.js(new) — hooksdid-start-navigation/did-frame-navigate/did-fail-load/console-messageon the dashboardwebContents, so a subframe that never commits, or commits a non-2xx, leaves a
line carrying the status or the net error code.
website/src/lib/paneLog.ts(new) plus ten call sites inInstancesViewport.tsx— a greppable one-line[pane]journal (mount, ready,unmount, re-mint, evict) written into the same
gateway-launch.log, so therenderer's view and the frame's view interleave on one timeline.
Tokens are redacted (
token=<redacted>) and the journal is unconditional — nodebug flag, because the failure is not reproducible on demand.
The
[pane]prefix is a marker, not a capability. The crew panes arecross-origin iframes of the dashboard's webContents, so
console-messagefiresfor their documents too: a prefix test on its own would let a compromised remote
gateway print entries into
gateway-launch.log, forging the very record that sayswhether its pane was ever requested. The forwarder gates the allowlist on frame
identity instead, and identity takes two checks — Electron's
console-messagedetails carry the emitting
WebFrameMain, so it requires both:parent === null— the top of the frame tree, a relationship Chromium owns and anested page cannot claim.
frame.originequal to the URL the window was loaded with (backendUrl, passedin at the single call site).
Neither alone is sufficient, because position is not identity: a cross-origin pane
that gets a user click on a
target="_top"link can navigate the top-level window,and the remote document then has
parent === nulltoo.sourceIdis deliberatelynot used for either check: any script can rewrite it with a
//# sourceURL=comment. Both fail closed, so a runtime that supplies no frame — or a caller that
configures no origin — loses the INFO-level journal rather than trusting an
unverifiable claim.
A pane's console errors are still journaled — a framing refusal inside the pane
is exactly the diagnosis — but attributed (
renderer console [error] (untrusted frame http://127.0.0.1:PORT): …) so a reader can tell pane-controlled text from thedashboard's own. Every field the emitting document controls is escaped before it is
written:
gateway-launch.logis read by tailing it, so a raw newline would let apane forge whole entries, and length is capped so one enormous message cannot
scroll the lines around it out of the tail.
The volume is bounded, and the bound is aggregate — because a pane drives more than
one path. The repeat counter is keyed by message text, and text is what a
compromised pane chooses: varying it defeats a per-message cap, and clearing the key
map on overflow restarts the counting rather than holding the line. A pane can just as
easily loop its own frame's navigations — every
did-start-navigation/did-frame-navigate/did-fail-loadis another unconditional line — so a limit onthe console path alone is one the pane walks around by switching paths. Every emission
therefore funnels through one writer (
record), and every line attributable to anuntrusted frame, console or navigation alike, is charged against a single
per-attachment budget (
UNTRUSTED_LOG_BUDGET = 100), independent of what the textsays. It is charged only for lines actually written, and the last line it admits names
the budget so the silence after it reads as the cap rather than as the pane
recovering. A navigation carries only
isMainFrameas a trust signal (the positionalevents pass no frame or origin), so a subframe navigation is the pane's and is
budgeted while a top-frame navigation is the dashboard's own handful and is not.
Trusted and untrusted console repeat counters stay separate maps, so pane volume
cannot evict the dashboard's own. The trade is explicit: past the budget a genuinely
broken pane stops explaining itself, which is acceptable because the first lines are
the diagnosis — a framing refusal repeats, it does not evolve — while an unbounded log
is a disk-exhaustion path on the user's machine.
A boolean presence flag is not a credential.
paneLog's redaction matches keynames as a substring so
authTokenandsession_secretare caught, which alsomatched the
hasTokenflag inremint-emptyandwarm-declined— redacting the onebit those lines exist to record while protecting nothing. Booleans are now exempt;
every other type under a secret-looking key is still replaced.
While wiring it up, the repo's own
shell-contractdrift guard caught a realpackaging bug:
frame-load-log.jswas missing frombuild.filesinwebsite/electron/package.json, so the DMG would have shipped without the moduleand window creation would have crashed. Fixed in the same commit.
website/electron/package.jsonadds no dependency. The only edit is one entryin the
build.filesallowlist — nodependencies/devDependencieschange, nolockfile change, no new third-party code. electron-builder ships an explicit
per-file allowlist, which is why an unlisted local module works from source and is
simply absent from the DMG.
The i18n exemption
The journal's redaction sentinels —
<redacted>,<empty>,<unserializable>,?token=<redacted>,?<query>— are log tokens, and thei18n:checkgate reportsuntranslated literals on added lines at zero tolerance. Translating one would make
gateway-launch.logunsearchable in the exact incident it exists for, and a localethat renamed
<redacted>would read as if the token had been printed. So they areexempted rather than wrapped, via two fully anchored
words.excludeshapes inwebsite/eslint.i18n.config.js:Both are deliberately separate from the existing
^[?&][a-z_]+=[a-z0-9]+$server-contract shape: that class admits no angle bracket, and widening it to reach
these would let a bracket into a pattern whose whole tightness argument is that it
carries only
[a-z0-9_=]. The anchors are what keep prose reportable —Enter <name> hereand?label=Save changesstill fail the lint, andwebsite/src/test/i18nLintExemptions.test.tspins both directions.Tests
test/test_warm_set_cap.py— rewritten for the registered-count semantics.New
TestAdmitsEveryRegisteredCrewclass pins the regression directly: a crewstill connecting does not shrink the cap, adding a crew widens it, and a
registered-but-never-connected crew still gets a slot.
test/test_instances.py—test_automatic_cap_covers_every_registered_crew_not_just_connected_ones(3 registered / 2 connected ⇒ cap 3, the exact shape that used to serve 2) and
test_adding_a_crew_widens_the_served_cap(1 → 2 with no config edit).website/src/lib/paneLog.test.ts(new, 16 cases) — the one-line format,credential redaction (including a wrapped key name like
authToken, and bothdirections of the boolean exemption:
hasToken=falsesurvives,hasToken: 'abc'is still redacted), dropped
undefinedfields, and thecross-origin/about:blank/no-elementbranches offrameDocumentState.website/electron/test/frame-load-log.test.js— 45 cases: started-vs-committednavigations, same-document SPA navigations staying silent, the
[pane]journal'shigher repeat cap (a re-mint loop is the finding), both
console-messageshapes,and a
frame-load-log console trust boundaryblock that runs each attack ratherthan describing it:
[pane] pane-ready id=nobita status=200produces noline at all, and neither does a remote document that navigated the top-level
window (
{ parent: null, origin: <remote> }) — thetarget="_top"case;normalizeTrustedOriginreduces the loaded URL to Chromium's serialization, and asame-prefix impostor (
http://localhost:54760) is refused, pinning that thecomparison is equality and not
startsWith;parent, a throwingorigin, and anunconfigured trusted origin all drop the journal;
UNTRUSTED_LOG_BUDGET + 50distinct pane errors produce exactlyUNTRUSTED_LOG_BUDGETlines with the last naming the budget; suppressed repeatsdo not consume it; a spent budget does not silence the dashboard's own errors; and
overflowing the untrusted key map does not reset the trusted counters;
did-fail-loadwithisMainFrame=false,UNTRUSTED_LOG_BUDGET + 50times) is bounded by the same budget, not just consoleoutput; console and navigation share one aggregate budget (70 + 70 across the
two paths still totals
UNTRUSTED_LOG_BUDGET, not 2×); and the dashboard's owntop-frame navigations (
isMainFrame=true) are never budgeted;untrusted frame <origin>, andboom\nframe navigated (subframe) status=200 …stays one record with its newlineescaped.
website/src/test/i18nLintExemptions.test.ts— 4 cases for the two newexemption shapes, in the file's existing both-directions style: quiet on the bare
sentinels and on the query forms, still reporting
Enter <name> here/Token <redacted>(a placeholder inside a sentence is copy) and still reporting?label=Save changes(the query shapes did not widen into "anything after a?").Whole file: 47 passed.
Local runs:
test_warm_set_cap.py+test_instances.py+test_config_superseded_defaults.py→ 358 passed, 2 skipped, 1 failed. Thefailure is unrelated:
TestForwarderPidHints::test_connect_persists_forwarder_identityasserts a non-empty
process_start_time(), which shells out tops -o lstart=onmacOS — blocked in my sandbox, so it returns
"". Electronnode --test(wholepackage suite): 1662 passed, 1 skipped, 0 failed. vitest (
paneLog+i18nLintExemptions): 63 passed.Manual verification
Partial, and stated plainly rather than claimed:
test_instances.pycases, which exercise the real registry and the realapi_instances_list.its bundled
backend-dist, so confirming the served integer in the UI (Settings→ Instances, "up to N instances stay warm") needs a
make desktop+reinstall. The reporting user is doing that run.
Screenshots / video
Why no screenshot: the frontend diff adds logging call sites only — no
component, layout, string or style change, so no pixel differs.
Related Issues
no linked issue: found while diagnosing a stuck pane in a live session, not filed
first.
Pattern harvest
Rule candidate:
review-promptPattern: deriving a budget from live state that the thing being budgeted is
still racing to enter. The cap counted connections while connections were still
being established, so it was systematically low during exactly the window it
governed. The general shape to flag: a limit computed from an observed runtime
count, applied to a set whose membership is declared — the two converge
eventually, and the gap is a silent, order-dependent failure. Prefer the declared
count (configuration) and bound it separately, rather than sampling the live one.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)constants.pydoc comments, the superseded-default display string, theinstancesmodule spec (§2, the config table and the evictiontroubleshooting row) and the committed
config-baseline.jsonsnapshot allmoved off "connected" onto "registered", each naming the automatic ceiling