feat(ocap-kernel): carry out a vat termination on the run loop

[sirtimid]

Second PR of the "only the run loop writes the store" work, stacked on #1096. Read this diff as git diff sirtimid/restart-vat-run-queue-item...sirtimid/terminate-vat-through-run-loop.

A vat's death is a set of store writes. Made from outside the run loop they land in whichever crank happens to be open: the loop starts its next crank in the same turn it ends the last, so a caller that waited for a crank resumed inside the next one, and an ordinary rollback there undid a death the caller had already been told about.

Control-plane terminateVat now queues a terminateVat run queue item and waits on the same per-vat RAM waiter a restart uses — the queue-and-wait shape from #1096, factored out so both share it. The run loop carries the request out, so #retireVat's writes belong to the crank that made them. A teardown that fails settles the caller rather than throwing, since the run loop's catch would roll back whatever of the death did get written and restore the request.

Unlike a restart, a termination with no waiter left is still carried out: it is an instruction rather than a request, and a vat the next boot has already relaunched is exactly the one it names. A vat something else killed first — the in-crank crankResult.terminate path, or terminateAllVats — resolves its caller rather than reporting a failure for a vat that died as asked.

terminateAllVats deliberately keeps writing directly. It is part of tearing the kernel down, and reset has to work on a kernel whose run loop has died, which a queued request never could be. The narrow "run loop is not running, so direct writes are legal" mode that makes that safe is the last of the control-plane moves, not this one.

terminateSubcluster walks persisted membership, so it now retires a member the kernel has no handle for rather than skipping it and stranding the rest — the half of @grypez's #1030 that #1093 did not reach. It tries every member and, if any survived, keeps the subcluster record and says so: deleteSubcluster drops every member's vat-to-subcluster mapping, and getVatSubcluster is a Fail, so deleting the record over a live member would break getStatus for the whole kernel from then on. It also refuses outright on a dead run loop, as launchSubcluster already does. A failed launchSubcluster keeps the record on the same terms: its cleanup terminates each member it launched, and with termination queued, a run loop that died mid-launch leaves every one of them running.

Two limits worth stating. A restart asked for after a termination is queued cannot be ordered against it — waiters are pooled per vat, not bound to their item — so only one outstanding when the termination is requested is superseded; the other direction, a termination undone by a later restart, cannot happen. And reset can still destroy a queued item and strand its waiter; that belongs to the PR that stops the run loop before those writes.

Synchronous is still not atomic: a stopVat that fails part-way commits what it wrote. Now that the work happens inside a crank, a nested savepoint could roll just that back — the first time that has been possible — but the crank's savepoint structure is being reworked in a parallel PR, so it is left for afterwards.

Changes

• Add a terminateVat run queue item, dispatched through KernelRouter to a new VatManager.performVatTermination
• VatManager.terminateVat queues one and waits; the queue-and-wait and settle-the-waiters shapes are shared with restart
• terminateSubcluster refuses on a dead run loop, retires a member with no handle, and keeps the record if a member survives
• launchSubcluster's failure cleanup keeps the record if a member survives
• KernelQueue gains enqueueTerminateVat

Testing

VatManager.test.ts covers the queued path, the superseded restart, a teardown failure settling rather than throwing, a vat something else already killed, a request nobody is waiting for, and that terminateAllVats does not queue. SubclusterManager.test.ts covers the no-handle member, the surviving member (record kept, error raised), the dead run loop, a failed launch whose cleanup leaves a member running, and that cleanup waiting for the crank before it deletes the record.

Kernel.test.ts's mocked KernelQueue now captures the deliver function the kernel hands to run(), so enqueueRestartVat/enqueueTerminateVat dispatch the item through the real KernelRouter on a later turn. That is what lets its existing end-to-end terminate and terminateSubcluster cases keep working through the new route, and it restores the restarts a vat case #1096 had to delete.

Twelve mutations checked, each failing only its own cases: terminateVat writing directly, performVatTermination rethrowing, the already-dead guard removed, terminateAllVats routed through the queue, the queued restart not superseded, terminateSubcluster skipping a no-handle member, propagating one member's failure, deleting the record over a survivor in either terminateSubcluster or a failed launch's cleanup, that cleanup deleting the record without waiting for the crank to end, not checking the run loop, and enqueueTerminateVat always setting reason (which toHaveBeenCalledWith cannot see, since it ignores undefined-valued keys — the item's reason is exactOptional).

Full @metamask/ocap-kernel suite green after the rebase onto the current #1096. Before it, kernel-test vat-lifecycle, subclusters, cluster-launch and resume were green against a fresh dist. The full kernel-test suite aborts in a native better-sqlite3 teardown in this environment, which reproduces unchanged on origin/main.

Carries draft findings A2, A5 and A13.

🤖 Generated with Claude Code

---

Note

High Risk
Changes core vat lifecycle and crank transaction semantics (irrevocable terminations, subcluster teardown); incorrect rollback or waiter handling could leave inconsistent store state or stranded subclusters.

Overview
Control-plane terminateVat now queues a terminateVat run queue item (mirroring restartVat) so vat teardown store writes run inside the crank that performs them, not in whatever crank happens to be open when the control plane calls—avoiding rollbacks that undo a death the caller was already told succeeded.

VatManager enqueues via enqueueTerminateVat, waits with shared queue-and-wait helpers, and performVatTermination runs on the run loop; teardown failures settle callers instead of rejecting the crank, and successful terminations return CrankResult.irrevocable so later crank failures do not roll back the death. Pending restart requests for the same vat are superseded when termination is requested. terminateAllVats still calls stopVat directly (no queue) so kernel teardown works when the run loop is dead.

terminateSubcluster and failed launchSubcluster cleanup are tightened: refuse a dead run loop, retire persisted members without a live handle, try every member, and keep the subcluster record (and IO/name where appropriate) if any vat survives termination—deleting the record over live members would break kernel status.

Tests extend KernelQueue, KernelRouter, VatManager, SubclusterManager, and Kernel.test mocks so queued terminate/restart items dispatch through the router like a real crank.

^{Reviewed by Cursor Bugbot for commit 012e7ed. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	73.58% ⬆️ +0.25%	10088 / 13710
🔵	Statements	73.36% ⬆️ +0.24%	10217 / 13926
🔵	Functions	73.87% ⬆️ +0.12%	2350 / 3181
🔵	Branches	68.05% ⬆️ +0.36%	4175 / 6135

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/ocap-kernel/src/Kernel.ts	89.84% 🟰 ±0%	79.54% 🟰 ±0%	85.41% 🟰 ±0%	89.84% 🟰 ±0%	325-327, 398, 422, 497-507, 595, 663, 739-742, 755, 765-766, 819, 842
packages/ocap-kernel/src/KernelQueue.ts	98.42% ⬇️ -0.33%	91.66% ⬆️ +0.97%	96.15% ⬇️ -3.85%	98.94% ⬆️ +0.19%	181, 235, 702
packages/ocap-kernel/src/KernelRouter.ts	94.63% ⬆️ +0.15%	82.19% ⬆️ +0.50%	100% 🟰 ±0%	94.63% ⬆️ +0.15%	139, 202, 219, 321, 374, 434, 452, 455
packages/ocap-kernel/src/types.ts	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%
packages/ocap-kernel/src/vats/SubclusterManager.ts	96.03% ⬇️ -0.14%	91.57% ⬆️ +1.45%	100% 🟰 ±0%	95.97% ⬇️ -0.14%	158-161, 185-188, 275-278, 354, 433, 455, 472
packages/ocap-kernel/src/vats/VatManager.ts	98.31% ⬇️ -0.02%	96.15% ⬇️ -1.46%	97.36% ⬆️ +0.94%	98.31% ⬇️ -0.02%	236-239, 266, 353

Generated in workflow #5114 for commit 012e7ed by the Vitest Coverage Report Action

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 012e7ed. Configure here.}

[cursor]

Restart superseded if terminate enqueue fails

Medium Severity

terminateVat takes outstanding restart waiters and settles them with VatDeletedError before enqueueTerminateVat runs. If that enqueue throws — a dead run loop, or any later store failure — those restart callers are told the vat is gone and their queued restart is dropped as stale, while the vat is still running and was never queued for death. #awaitQueuedWork already enqueues before registering a waiter so a refusal stays that call's own error; the supersede does not follow the same order.

 

^{Reviewed by Cursor Bugbot for commit 012e7ed. Configure here.}