[CMGS]

Implements #443: POST /v1/sandboxes/{id}/move hands one running claim to a named peer with cocoon's live migration. The guest keeps its processes and memory and stops only for the final round.

evacuate stays the cold path and never uses this. move never hibernates. If it cannot go ahead it fails fast, cleans up, and leaves the claim where it was; the caller decides what next.

Needs a cocoon release containing cocoonstack/cocoon#303, which is merged but in no tag yet. MigrateCocoon is set to v0.7.7, and a dev build passes the check. It also needs the CH fork at 35b098c65260 or newer. Until a node has both, a move answers 409 and nothing else changes.

How it works

The move has its own record in the meta store (table sandboxd_moves), not an archive row. A checkpoint row says where a stored checkpoint lives; a move hands over a live guest. Only the compare-and-set code is shared, as metastore.CAS.

From	To	Who	When
(none)	pending	source	after the target admitted the move, before the baseline
pending	final	source	the final round paused the guest
final	committed	target	after the receive; the point of no return
committed	started	target	after it journaled the start
pending or final	aborted	source	before it resumes the guest

The rules that keep one guest:

• The target commits only from final.
• The source resumes only on proof: its own record, still open or aborted. A missing record or another node's means the claim moved on, so the source drops it.
• A target with a received claim starts it on committed or started, and drops it on aborted or a missing record.
• Only the source deletes a record, and only an ended one.
• An abort runs in the order aborted, resume, target drop, journal clear, record delete.
• Before the final round a replicated claim's row is consumed, its fields cleared and journaled, then the row deleted. No node adopts a tombstone, and a source cut off from the registry changes nothing, so its fence still finds a peer that adopted the claim. The target's first round puts a new row.

Counting. The target resolves the tenant, then admits a move like a new claim (node max_claims, max_running, tenant cap, memory_limit, move_concurrency) and holds a reservation until its commit swaps it for the claim under one lock. The source counts the claim until it drops it, so the cluster counts a moving claim twice, which can refuse earlier but never over-admits.

Config and API. move_concurrency (default 2, reloadable) counts both directions. max_downtime_ms is a request field, default 2000, floor above 400. Moves need meta_store with a cell, which holds the record, and mesh.cluster_key. They do not need archive_failover or an S3 checkpoint_store: the rounds go straight to the peer. Usage gains move_in and move_out.

Limits

• Refused with 409: egress lane, volumes, hibernated, archived, failed. The egress lane may come later: a receive leaves the guest paused, so the tap can be locked first, but the guest's network recovery with new MACs is unchecked.
• Both nodes need the same path strings in cocoon's root_dir and run_dir. A common symlink to different disks works. Different strings fail at the receive, after the pause, with a 502, and the guest resumes on the source.
• Same-model nodes: cocoon did not test TSC across hosts of different frequency.
• Connections do not survive; the client meets a 404 on the source and runs Lookup.
• docs/cluster.md lists the rest: a committed record whose target never returns stays as one row, and without sync_claims a host crash can roll back the journal, after which the source drops the claim.
• A crash or lost reply right after the row is consumed leaves the source's own tombstone. The guest is safe, but replica rounds are refused until the hourly sweep puts the row back on the replica, so a node loss in that gap loses the claim. S35 hit this with a SIGKILL in the 1 ms window: the row stayed consumed through 15 refused rounds and one restart. The unit test shows the sweep restoring it; it did not fire on hardware.
• A capture killed before the series id is journaled can leave the guest paused, the same window pool: accepted windows from the 2026-11 audit, with reopen triggers #433 records for hibernate.

Hot path

A claim gains len(m.incoming) in the claimed and running checks, under the lock it already holds; the tenant count loops only when a reservation exists. BenchmarkQuotaErr: 23.46-23.69 ns/op with none, 23.25-23.47 ns/op with two. No warm claim A/B was run on hardware.

Verification

Head 4a17e6a6, on main a08f5529. Gates ran on this head:

lint 0-issues lines: 10
gotest rc=0
asl -forwarder=false rc=0 on sandboxd, darwin and linux; the other modules rc=0 with forwarder on
moves/pg, archives/pg ok against PostgreSQL 17
rust darwin silkd 161/0/3, linux 162/0/3; boot/init darwin 15/0/0, linux 17/0/0
py sdk/python 278, sdk/openai 10, sdk/langchain 15

Hardware: two bare-metal nodes over a 57 MB/s link, cocoon master-3b885b8, CH 35b098c65260, a common path string through a symlink.

Run, tree	What	Result
S35, 4a17e6a6	replicated claim, both ways	200, 3 rounds, downtime_ms 237 and 272; consumed, then row deleted 1 ms later, then final 19 ms after that; new row 32.9 s after the answer
S35	SIGKILL target at final, replicated	502, aborted, guest intact on the source, new row and replica 22 s later
S35	SIGKILL source after the row delete; after the consume	claim kept and guest intact both times; row back 22 s later after the delete; stuck consumed after the consume (see limits)
S35	PostgreSQL cut from the source only, past the lease, during a move	the target node adopts at +12.8 s and serves; on restore the source drops the claim and removes its VM within about 1 s; one guest serves
S34, 759047d1	no archive_failover, no S3: config load, one move each way	loads, refuses move_concurrency without a cell or cluster_key; 200, 3 rounds, downtime_ms 367 and 279
S34	same config, SIGKILL target at committed, SIGKILL source at final	intact on the target, started; resumes on the source, aborted
S34	tenant at its cap on a target with a cold tenant cache	429 in 53 ms, guest intact on the source, no residue; at cap 2 the same move answers 200
S33, 5ab54e7f	one move each way	200, 3 rounds, downtime_ms 257 and 227
S33	SIGTERM source in rounds, exit 5.6 s	restart writes aborted, guest intact on the source
S33	SIGTERM source at final, exit 0.8 s	the in-flight move finished, one guest on the target, no cold boot
S33	SIGTERM target in rounds, exit 5.6 s	502 at 7.7 s, guest intact on the source, target clean after restart
S33	SIGTERM target around its commit, 12 tries	every claim intact, no restart, no failure; see below
S32, c2485d3e	idle guest, both ways, sparse 10 GiB disk	200, 3 rounds, downtime_ms 172 and 221, about 14 s per call
S32	8 MiB/s fsync writer, both ways	200, downtime_ms 194 and 234
S32	hot writer (300 MiB loop), default budget, 3 runs	503 at about 41 s, never went final, guest intact
S32	VMM SIGKILL during rounds	409, no cold boot until the move ended, no false success
S31, 70911ac5	SIGKILL target at committed, 2 runs	claim intact on the target, record reaches started
S31	SIGKILL source at final and at committed	resumes on the source; drops, guest on the target
S31	PostgreSQL stopped after final	503, paused on both; on return exactly one guest runs
S31	fork, hibernate, PATCH volumes during a move	503 in under 2 ms
S30, d8d1ac8d	both nodes killed at committed, both restart orders	guest intact on the target
S30	restart with the claim journaled and the VM running, 3 runs	vm start harmless, guest intact
S29, 25e1f08e	counts on both nodes, refusal at max_claims, replicated claim's row	match the table; 429 in 12 ms with no residue; row gone before final, new row after one interval

A client running exec every 20 ms saw a longest gap of 352 ms against a downtime_ms of 280 (S29).

Rerun decision after rebasing on #444. #444 makes sandboxd wait for running cocoon commands on stop and refuse new ones after that, and every earlier fault leg used SIGKILL. So S33 ran graceful stops on the rebased tree. The rows from S29-S34 did not rerun on this head; the commit since S34 changes only the replicated claim's row handling, which S35 ran.

One interplay with #444 is fixed here but not proved on hardware. A target stopping between its commit and its vm start used to read the refused command as a guest that cannot start, and failed an intact claim. It now keeps the journaled claim for its restart, and TestMoveTargetStoppingBeforeItsStartKeepsTheGuestForItsRestart fails without the fix. S33 hit that window 0 of 12 times: stop waits 5 s for the commit handler before the engine refuses commands, so the start ran first each time. Two routes could reach it and neither ran: a stop under about 0.6 s after a restart with a journaled claim, and a stop about 5 s before a receive ends.

Not run on hardware: lost replies on the row consume and delete, the hourly sweep restoring a stuck row (unit tests), the refused start above, a committed record whose target never returns, a same-source retry racing an old abort, a chain of three nodes (those are unit tests), the hourly sweep (it did not fire in a run), and a reload of move_concurrency under load. A real egress or moved-guest network path is out of scope with the lane refused.

Two adversarial reads and hardware together found real bugs on the way, including one that lost the claim on both nodes and one that resumed a guest the target was running. Each fix has a regression test that fails without it, and the safety rules above were mutation-tested.