Conversation
sirtimid
added this pull request to stack #1107
September 15, 2026 22:37
This was referenced Sep 15, 2026
`clearStorage`, `reset` and `stop` waited out the crank in flight and then wrote. That does not work: the run loop is synchronous from `endCrank` to the next `startCrank`, so it wins the race by construction, and the write lands inside the next crank's `delivery` savepoint for an ordinary abort to undo — after the caller has been told it succeeded. They now bring the loop to rest between cranks, where there is no transaction to land in, and start it again afterwards. `stop` does not, being the end of the kernel. `terminateAllVats` loses the wait it carried for the same reason, its only caller having already stopped the loop. All three destroy work the queue was holding, so they now reject the message results that go with it. A `queueMessage` awaiting a kernel promise that `reset` deleted used to hang forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up; three reviewers converged on two ways to hang the kernel. A stop request was read only at the top of the loop, so a crank that died while `stop` was waiting left `stopRunLoop` pending for good and the database never closed — a regression, since `waitForCrank` settled from `endCrank`'s `finally`. It is now settled however the loop leaves, and `stopRunLoop` re-reads the state afterwards so nobody tries to restart a loop that died. The request was also a single slot, so a second caller displaced the first and stranded it; worse the other way round, where the second found the loop already stopped, wrote unprotected, and had the first restart the loop underneath — this change's own defect, wearing a different hat. One shared request, and `Kernel` takes the window one turn at a time. Also: a stopped loop refuses work, rather than accepting messages nothing will drain; `discardQueuedWork` drops inbound remote arrivals, which would otherwise write pre-reset peer bookkeeping into a wiped store; `stop` discards before the teardown that can throw, and closes the database in a `finally`; and a loop that cannot be started says so instead of reporting a death to the embedder. `terminateAllVats` keeps its own `waitForCrank` after all. I had removed it believing `reset` was its only caller; it is on the kernel-control RPC surface and wired to a panel button. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sirtimid
force-pushed
the
sirtimid/stop-run-loop-before-direct-writes
branch
from
September 16, 2026 16:24
72ca9ad to
badc926
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1103's successor,
sirtimid/peer-incarnation-run-queue-item. This isthe last PR of the #1021–#1023 split, and the one that makes the invariant the
rest of Tier 3 was written against true: only the run loop writes the kernel
store.
Kernel.clearStorage,resetandstopeach didawait waitForCrank()andthen wrote. That does not work. The run loop is synchronous from
endCranktothe next
startCrank, so it wins that race by construction: the write landsinside the next crank's
deliverysavepoint, where an ordinary abort undoesit — after the caller has been told it succeeded. Waiting for a crank to end is
not the same as there being no crank.
They now bring the loop to rest between cranks, where there is no transaction to
land in, and start it again afterwards.
stopdoes not, being the end of thekernel. All three destroy work the queue was holding, so they reject the message
results that go with it: a
queueMessageawaiting a kernel promise thatresetdeleted used to hang forever.
@grypez's claim 1 in #1039 becomes true here. #1090's body promised the
crank's release would stay the intended commit point until this PR; that
promise is discharged. What #1090 could not do was stop something else opening a
transaction alongside the crank, and this is what stops it.
Deliberately not in this PR
Refusing an out-of-loop
createSavepointis a follow-up stacked on this one.It is enforcement rather than fix: after the two PRs below this one moved inbound
remote messages and peer incarnation changes onto the run queue,
grepfinds nocaller of
createSavepoint,rollbackSavepointorreleaseSavepointoutsidethe crank machinery at all. Splitting it keeps this PR to one mechanism, which
the plan authorised.
The detached transport failure handling in
RemoteHandlestill writes thestore outside every crank, as #1101 recorded: a give-up advances
startSeqanddeletes the pending message, and a later unrelated crank's
rollbackCrank('delivery')restores both, so on restart the kernel retransmits amessage it has already told its callers failed. It survives this PR. Fixing it
means routing
#rejectAllPendingthrough the run loop the way inbound messagesnow are — a mechanism of its own, in files two other PRs in this stack are
editing. It is the last known writer outside the loop.
Lane C's waiter findings on #1096 and #1097 are not reachable from here. The
restart and termination waiter arrays are on the vat-lifecycle chain, which is
not an ancestor of this branch —
VatManagerhere has no#restartWaitersandno queued-work items. When the two chains meet on
main,resetandclearStoragemust settle those arrays too;discardQueuedWorkis the place,and it is one line each.
Changes
KernelQueue.stopRunLoop(): the loop reads the request between cranks, records{ state: 'stopped' }and returns. The request is shared between callers andsettled however the loop leaves, including by dying — otherwise a crank that
died while
stopwaited would hang the shutdown, whichwaitForCrankdid not.run()is restartable and returnsPromise<void>; a loop that died stillcannot be run again.
Kernel.#withRunLoopStopped(work, { thenRestart }), taking the window one turnat a time, over
clearStorage,resetandstop.discardQueuedWorkrejects the message-result subscriptions and drops inboundremote arrivals, which would otherwise write pre-reset peer bookkeeping into a
wiped store.
drain.
@metamask/ocap-kernelchangelog entries.Testing
KernelQueue.run-loop-stop.test.tsis new and runs against a real nodejs SQLite:memory:store, becauseKernelQueue.test.tsmocksmakePromiseKitand socannot tell a loop that has come to rest from one that has merely been asked to
— a first draft of these tests was vacuous for exactly that reason. It asserts
isInCrank()is false when the caller regains control, that a write made in thewindow survives a later aborting crank, that a caller waiting on a loop that
died is let go, that two callers both get an answer, and that a stopped loop
refuses work. Each is mutation-checked; the two hangs fail as 2 s timeouts.
Kernel.test.tscovers the restart paths: aresetwhose work throws stillleaves the loop running, a
reseton a loop that already died does not try tostart it, and two concurrent direct writes do not interleave. Its
KernelQueuemock now models
idle/running/stoppedrather than always reportingrunning.Six mutations in all. Full
@metamask/ocap-kernelsuite green locally; eslint,constraintsandchangelog:validateclean.🤖 Generated with Claude Code
Note
High Risk
Changes kernel shutdown, reset, and storage clearing around run-loop and store transaction boundaries; incorrect sequencing could corrupt state or strand callers, though the PR adds explicit rejection of pending message results.
Overview
Control-plane store writes now stop the run loop between cranks instead of
waitForCrank(), which could still lose writes inside the next crank's delivery savepoint.Kernel.#withRunLoopStoppedserializesclearStorage,reset, andstopvia a one-at-a-time turn queue;reset/clearStoragerestart the loop afterward, whilestopdoes not.KernelQueueaddsstopRunLoop()(park between cranks, shared stop request, wake parked loops),discardQueuedWork()(reject message-result subscriptions and clear in-memory remote arrivals), and astoppedRunLoopStatusarm. The loop'srun()is restartable after a stop but not after failure;assertRunLoopAliverefuses work while stopped.Changelog documents the breaking status change and the fix; new integration tests cover the safe write window (real SQLite) and kernel stop/reset sequencing.
Reviewed by Cursor Bugbot for commit badc926. Bugbot is set up for automated code reviews on this repo. Configure here.