sandboxd: an abrupt death during a capture leaves the guest paused after restart #446

If sandboxd dies abruptly while a capture is running, the next start can treat the claim as running even though the capture is still going. Tracking it; no work asked for now.

What happens

A clean stop is fine: since #444 sandboxd waits for running cocoon commands before it exits. This covers a crash, or a SIGKILL past TimeoutStopSec, during vm hibernate.

1. The claim's intent is journaled as PendingSnap, and cocoon is still capturing.
2. The next start runs Reconcile. While the capture runs, the record reads running, so Reconcile clears PendingSnap and HibernateSnap and treats the claim as running.
3. cocoon dies on its next log line, because the read end of its stderr is gone. The guest stays Paused, the record reads running, and every exec gets 502 until someone runs vm start. Memory is intact. Liveness does not see it, since the VMM process is alive.

cocoon keeps this behaviour: killing the CLI mid-capture is the same window as SIGKILL, and recovery belongs to the caller. So do not hand cocoon a stderr that outlives sandboxd, such as a log file, without fixing Reconcile first: the hibernate would then finish behind Reconcile's back, and liveness would cold-boot the guest, losing its memory while exec answers 200.

Reconcile adopts the claim as hibernated only when cocoon already got past the point where the VMM stops, which a dead pipe usually prevents.

Evidence

Hardware, cocoon v0.7.6, sandboxd stopped during vm hibernate before #444:

The second row is why a log file alone is worse: in both runs Reconcile ran before the capture had finished.

Possible fix

Not a bare vm start from Reconcile. While the capture CLI is alive, vm start waits (about 150 s in two hardware runs) and then returns rc 0 after cold-booting the guest: new boot id, memory gone, snapshot still listed. rc 0 does not mean the guest survived.

So first wait for the CLI to exit, by matching the unique snapshot name in PendingSnap against exact process arguments. Do not bound that wait tightly, since a process in an fsync exits late. Then read cocoon's rendered state, which vm list and vm inspect already report and sandboxd already reads:

During a dump vm start fails after 30 s and leaves the guest alone, so it needs retries sized for a slow disk: on hardware, with a dump of about 150 s, four attempts failed and the fifth resumed the guest intact. A VMM left in CH state Created by a kill after vm.shutdown reads running, and vm start returns rc 0 and changes nothing, which is why the probe is required. A killed capture also leaves a memory-sized snapshot-* under the live VM's run dir until cocoon's gc passes 24 hours. The same steps after a failed capture would also cover #442. A snapshot save journals no intent, so a save interrupted the same way needs one before this can see it.

Cost of waiting

It needs an abrupt death during a capture, and a restart before that capture ends. Related: #442, a capture that outlives every timeout.