Skip to content

feat(ocap-kernel): stop the run loop before the control plane writes - #1105

Open
sirtimid wants to merge 3 commits into
sirtimid/peer-incarnation-run-queue-itemfrom
sirtimid/stop-run-loop-before-direct-writes
Open

sirtimid wants to merge 3 commits into
sirtimid/peer-incarnation-run-queue-itemfrom
sirtimid/stop-run-loop-before-direct-writes

Conversation

@sirtimid

@sirtimid sirtimid commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1103's successor, sirtimid/peer-incarnation-run-queue-item. This is
the last PR of the #1021#1023 split, and the one that makes the invariant the
rest of Tier 3 was written against true: only the run loop writes the kernel
store.

Kernel.clearStorage, reset and stop each did await waitForCrank() and
then wrote. That does not work. The run loop is synchronous from endCrank to
the next startCrank, so it wins that race by construction: the write lands
inside the next crank's delivery savepoint, where an ordinary abort undoes
it — after the caller has been told it succeeded. Waiting for a crank to end is
not the same as there being no crank.

They now bring the loop to rest between cranks, where there is no transaction to
land in, and start it again afterwards. stop does not, being the end of the
kernel. All three destroy work the queue was holding, so they reject the message
results that go with it: a queueMessage awaiting a kernel promise that reset
deleted used to hang forever.

@grypez's claim 1 in #1039 becomes true here. #1090's body promised the
crank's release would stay the intended commit point until this PR; that
promise is discharged. What #1090 could not do was stop something else opening a
transaction alongside the crank, and this is what stops it.

Deliberately not in this PR

Refusing an out-of-loop createSavepoint is a follow-up stacked on this one.
It is enforcement rather than fix: after the two PRs below this one moved inbound
remote messages and peer incarnation changes onto the run queue, grep finds no
caller of createSavepoint, rollbackSavepoint or releaseSavepoint outside
the crank machinery at all. Splitting it keeps this PR to one mechanism, which
the plan authorised.

The detached transport failure handling in RemoteHandle still writes the
store outside every crank
, as #1101 recorded: a give-up advances startSeq and
deletes the pending message, and a later unrelated crank's
rollbackCrank('delivery') restores both, so on restart the kernel retransmits a
message it has already told its callers failed. It survives this PR. Fixing it
means routing #rejectAllPending through the run loop the way inbound messages
now are — a mechanism of its own, in files two other PRs in this stack are
editing. It is the last known writer outside the loop.

Lane C's waiter findings on #1096 and #1097 are not reachable from here. The
restart and termination waiter arrays are on the vat-lifecycle chain, which is
not an ancestor of this branch — VatManager here has no #restartWaiters and
no queued-work items. When the two chains meet on main, reset and
clearStorage must settle those arrays too; discardQueuedWork is the place,
and it is one line each.

Changes

  • KernelQueue.stopRunLoop(): the loop reads the request between cranks, records
    { state: 'stopped' } and returns. The request is shared between callers and
    settled however the loop leaves, including by dying — otherwise a crank that
    died while stop waited would hang the shutdown, which waitForCrank did not.
  • run() is restartable and returns Promise<void>; a loop that died still
    cannot be run again.
  • Kernel.#withRunLoopStopped(work, { thenRestart }), taking the window one turn
    at a time, over clearStorage, reset and stop.
  • discardQueuedWork rejects the message-result subscriptions and drops inbound
    remote arrivals, which would otherwise write pre-reset peer bookkeeping into a
    wiped store.
  • A stopped loop refuses new work, rather than accepting messages nothing will
    drain.
  • @metamask/ocap-kernel changelog entries.

Testing

KernelQueue.run-loop-stop.test.ts is new and runs against a real nodejs SQLite
:memory: store, because KernelQueue.test.ts mocks makePromiseKit and so
cannot tell a loop that has come to rest from one that has merely been asked to
— a first draft of these tests was vacuous for exactly that reason. It asserts
isInCrank() is false when the caller regains control, that a write made in the
window survives a later aborting crank, that a caller waiting on a loop that
died is let go, that two callers both get an answer, and that a stopped loop
refuses work. Each is mutation-checked; the two hangs fail as 2 s timeouts.

Kernel.test.ts covers the restart paths: a reset whose work throws still
leaves the loop running, a reset on a loop that already died does not try to
start it, and two concurrent direct writes do not interleave. Its KernelQueue
mock now models idle/running/stopped rather than always reporting
running.

Six mutations in all. Full @metamask/ocap-kernel suite green locally; eslint,
constraints and changelog:validate clean.

🤖 Generated with Claude Code


Note

High Risk
Changes kernel shutdown, reset, and storage clearing around run-loop and store transaction boundaries; incorrect sequencing could corrupt state or strand callers, though the PR adds explicit rejection of pending message results.

Overview
Control-plane store writes now stop the run loop between cranks instead of waitForCrank(), which could still lose writes inside the next crank's delivery savepoint. Kernel.#withRunLoopStopped serializes clearStorage, reset, and stop via a one-at-a-time turn queue; reset/clearStorage restart the loop afterward, while stop does not.

KernelQueue adds stopRunLoop() (park between cranks, shared stop request, wake parked loops), discardQueuedWork() (reject message-result subscriptions and clear in-memory remote arrivals), and a stopped RunLoopStatus arm. The loop's run() is restartable after a stop but not after failure; assertRunLoopAlive refuses work while stopped.

Changelog documents the breaking status change and the fix; new integration tests cover the safe write window (real SQLite) and kernel stop/reset sequencing.

Reviewed by Cursor Bugbot for commit badc926. Bugbot is set up for automated code reviews on this repo. Configure here.

sirtimid and others added 3 commits September 16, 2026 18:20
`clearStorage`, `reset` and `stop` waited out the crank in flight and
then wrote. That does not work: the run loop is synchronous from
`endCrank` to the next `startCrank`, so it wins the race by
construction, and the write lands inside the next crank's `delivery`
savepoint for an ordinary abort to undo — after the caller has been told
it succeeded.

They now bring the loop to rest between cranks, where there is no
transaction to land in, and start it again afterwards. `stop` does not,
being the end of the kernel. `terminateAllVats` loses the wait it
carried for the same reason, its only caller having already stopped the
loop.

All three destroy work the queue was holding, so they now reject the
message results that go with it. A `queueMessage` awaiting a kernel
promise that `reset` deleted used to hang forever.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up; three reviewers converged on two ways to hang the
kernel.

A stop request was read only at the top of the loop, so a crank that
died while `stop` was waiting left `stopRunLoop` pending for good and
the database never closed — a regression, since `waitForCrank` settled
from `endCrank`'s `finally`. It is now settled however the loop leaves,
and `stopRunLoop` re-reads the state afterwards so nobody tries to
restart a loop that died.

The request was also a single slot, so a second caller displaced the
first and stranded it; worse the other way round, where the second found
the loop already stopped, wrote unprotected, and had the first restart
the loop underneath — this change's own defect, wearing a different hat.
One shared request, and `Kernel` takes the window one turn at a time.

Also: a stopped loop refuses work, rather than accepting messages
nothing will drain; `discardQueuedWork` drops inbound remote arrivals,
which would otherwise write pre-reset peer bookkeeping into a wiped
store; `stop` discards before the teardown that can throw, and closes
the database in a `finally`; and a loop that cannot be started says so
instead of reporting a death to the embedder.

`terminateAllVats` keeps its own `waitForCrank` after all. I had removed
it believing `reset` was its only caller; it is on the kernel-control
RPC surface and wired to a panel button.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sirtimid
sirtimid force-pushed the sirtimid/stop-run-loop-before-direct-writes branch from 72ca9ad to badc926 Compare September 16, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant