nvx-rs slimming 2.3.3: a VM restores from a snapshot, through a sans-I/O restore machine #1103

Parent: #1081, in #1076. Needs: #1099, #1101, #1102. Unblocks: #1113, #1130.

Goal. A VM restores from a snapshot. A gate first proves, on Linux, that concurrent restores of one snapshot are safe under a per-snapshot lock that the caller owns, and that a restore whose caller dies can't complete. Then host::vm::create with from: Some restores a VM from a snapshot and a VmConfig, with the ready listener bound before any spawn; RestoreMachine runs a restore's rules without I/O or a clock; and a failed restore reports its facts in RestoreError.

Notes.

• The gate has no fallback. If it fails, an OpenVMM change that restores any snapshot from any location becomes a prerequisite of SL2.16.
• The lock is the caller's. On Linux, the caller holds a per-snapshot lock from create(from: Some), which adds the hard links, through the peer-verified restore-ready event or the VM's verified exit, and around removing a restore's <vm_dir>/openvmm/. A restore whose caller dies can't complete: OpenVMM checks the snapshot's files before it connects the restore-ready endpoint, once and without retry, so the caller keeps no durable record of its restores. create's docs state these rules; NVX holds no lock, and Windows needs none.

PRs, in order. Each PR has one goal, leaves dev green, and names in its description each old test of its area as kept, rewritten or retired, with each change that check_test_changes.py reports. Its tests follow the repository's test layout: a module's unit tests, and the code that only they use, sit in its tests.rs or tests/ and move with it. The estimates count core lines (code, not comments, docs or tests) and counted files; each PR records its measured values. Each PR does what its Scope says and no more: a case, input or layout that its Scope doesn't name, and whatever "Not in this PR" lists, is out of scope. Address every review comment within the scope, and ask before acting on one that asks for more. A PR whose measured core lines pass the stop that its Estimate names (twice the estimate, at most 600) stops and asks before continuing.

1. SL2.15: Gate: concurrent Linux restores obey the caller's exclusion, and a dead caller's restore can't complete.

   • Objective: Linux restore views change hard-link counts on snapshot files, which OpenVMM checks while starting a restore. This test-only gate proves the caller's per-snapshot exclusion is sufficient and proves a restore cannot complete after its controller dies before the restore-ready connection.
   • Contents: A real-VM test, with the test adapter's guest: OpenVMM restores from a hard-linked <vm_dir>/openvmm/, with its control socket in the VM directory; several restores of one snapshot start while views are added and removed under a per-snapshot lock; a link added without the lock while a restore starts fails it, as the ctime check predicts. A controller killed before OpenVMM connects the restore-ready endpoint leaves a VMM that fails that connect and exits; once OpenVMM has connected it, a link change no longer fails the restore, so its file checks are over. If the gate fails, an OpenVMM change that restores any snapshot from any location becomes a prerequisite of SL2.16; there is no fallback.
   • Scope:
     • One Linux real-VM test uses the runtime adapter's guest and restores from a hard-linked <vm_dir>/openvmm/ with the control socket in the VM directory.
     • Several restores of one snapshot start while restore views are added and removed under the caller's per-snapshot lock.
     • Adding a link without the lock while a restore starts makes that restore fail through OpenVMM's ctime check.
     • Killing the controller before OpenVMM connects the restore-ready endpoint leaves a VMM that fails the connection and exits.
     • After OpenVMM connects the restore-ready endpoint, changing a link no longer fails the restore, proving its snapshot-file checks are complete.
   • Not in this PR:
     • Production restore code, a lock implementation or lock storage; SL2.16 documents the caller-owned rule.
     • Durable in-flight restore records; the dead-controller case proves callers do not need them.
     • A Windows variant; Windows does not need the hard-link exclusion.
     • A copy-based view, snapshot-parent layout or fallback when the gate fails; an OpenVMM change becomes the prerequisite instead.
   • Approach: Extend the existing real-VM restore fixtures with controlled hard-link changes and process termination around the restore-ready listener. Use the caller's ordinary file lock and existing runtime-adapter guest; add no production hooks, timing framework or synthetic OpenVMM substitute. Treat any failure of the required ordering as a blocking gate, not a reason to weaken the assertions.
   • Done when:
     • Concurrent restores succeed while all view additions and removals obey the per-snapshot lock.
     • A link addition outside the lock makes a concurrently starting restore fail as predicted.
     • A controller killed before the restore-ready connection leaves a VMM that fails the connection and exits.
     • A link change after the restore-ready connection does not fail that restore.
   • Needs: SL2.14, in nvx-rs slimming 2.3.2: snapshots have one strict format, and a capture finishes into it #1102.
   • Estimate: 0 core lines (a test), 1 counted file.

2. SL2.16: A VM restores from a snapshot and a VmConfig.

   • Objective: host::vm::create currently refuses snapshots, leaving callers on the store-coupled clone launch. This PR prepares a restore from a Snapshot and VmConfig, with all endpoints bound before spawn and the effective devices and policy exposed through the common launch types.
   • Contents: create with from: Some: the refusal of boot, accepts, the restore view, the restore arguments (internal, from nvx-rs/src/host/template/clone/launch.rs), the console's attachment, the NIC from Snapshot::nic for Prepared::nic and activation_record, the ready listener bound before any spawn, Launched::ready and VmmCommand::ready, with the external-launcher example's restore. The restore arguments install the restore's own network policy with --restore-network-policy, and activation_record carries its DNS, publication and proxy. Its docs state the per-snapshot lock rule, and that a restore whose caller dies can't complete, so no durable record is needed.
   • Scope:
     • host::vm::create with from: Some(snapshot) refuses boot, calls Snapshot::accepts and builds a hard-linked <vm_dir>/openvmm/ restore view.
     • Internal restore arguments reuse the current clone launch rules, including the permitted console attachment, processors, memory and --restore-network-policy for the restore's own policy.
     • The snapshot NIC feeds Prepared::nic and activation_record; the record carries the restore policy's DNS, publication and proxy.
     • The restore-ready listener is bound before any spawn and is handed through Launched::ready and VmmCommand::ready without reopening it.
     • The external-launcher example gains a restore, and create documents the Linux caller-owned lock and why no durable in-flight record is needed.
   • Not in this PR:
     • Owning or acquiring the per-snapshot lock: Linux callers hold it from create through the peer-verified restore-ready event or the VM's verified exit, and around view removal.
     • Durable in-flight restore records or recovery indexes; a restore whose caller dies cannot complete.
     • Copying snapshot files, restoring across filesystems or a snapshot-parent fallback; hard links require the VM directory on the same filesystem.
     • NIC re-addressing or adding a console listener absent from the capture; Snapshot::accepts refuses both.
     • Driving activation to Ready or deciding retries and cancellation; SL2.18a supplies the sans-I/O machine.
     • Snapshot tiers other than Fresh: this release creates only the hard-linked restore view for the Fresh snapshots that Snapshot::accepts admits.
   • Approach: Fold the existing clone launch's argument rendering into the new host::vm restore branch, but use Snapshot, VmConfig and common Prepared ownership instead of preserving store and lease abstractions. Create fixed hard links, bind the ready listener first, and hand all open objects forward; do not add a restore store, lock manager or alternate file-transfer path.
   • Done when:
     • create(from: Some) refuses boot, an incompatible VM and a restore view that would cross filesystems.
     • Restore arguments carry the console attachment that the snapshot allows, processors, memory and the restore's own network policy; a snapshot captured Disconnected refuses a listener.
     • Prepared::nic and activation_record use the snapshot NIC and the restore policy's guest-applied settings.
     • The ready listener exists before spawn and reaches both Launched and VmmCommand.
     • The compiled external-launcher example restores a VM.
   • Needs: SL2.6, in nvx-rs slimming 2.3.1: activation record v2 carries only what the guest applies #1101; SL2.15, above; SL2.17, in nvx-rs slimming 2.2.1: a VM is created in a private directory, with its restore-ready listener, then spawned #1099.
   • Estimate: 420 core lines, 4 counted files. Past 600 core lines, stop and ask before continuing.

3. SL2.18a: A restore's rules run without I/O or a clock.

   • Objective: The current clone restore sequence owns blocking I/O, deadlines, records and workload startup, which prevents callers from integrating their own runtime. RestoreMachine retains only the ordered restore rules and ends at Ready while callers supply every event.
   • Contents: restore::{RestoreMachine, RestoreInput, RestoreAction}: RestoreMachine::new(&ActivationRecord); the same record on every session; the transaction echo; OperationFailed { phase, evidence, source } with RestorePhase and RestoreEvidence; Connect again after SessionLost; Expired and Cancelled; and the end at Ready, from nvx-rs/src/host/template/clone/restore.rs.
   • Scope:
     • host::restore::{RestoreMachine, RestoreInput, RestoreAction, RestorePhase, RestoreEvidence} is extracted from host/template/clone/restore.rs; its transition cases move from host/template/clone/tests/restore.rs into host/restore/tests.rs.
     • RestoreMachine::new(&ActivationRecord) sends the same canonical record on every session and validates the transaction echoed by Ready.
     • Inputs cover the restore-ready event, connection and session progress, SessionLost, OperationFailed { phase, evidence, source }, Expired, Cancelled and verified Exited.
     • After SessionLost, the next action offers Connect again; a final operation failure preserves its phase, evidence and source.
     • The machine ends at Ready without sending the mapping table or Start.
   • Not in this PR:
     • Sockets, transports, blocking or async drivers, peer checks or clocks; callers perform I/O and feed results to the machine.
     • Built-in timeout durations, retry delays or a maximum session count; the caller owns budgets and retry policy.
     • Sending mappings or starting a workload; those remain the caller's protocol steps after Ready.
     • Stopping the VM, reading its outcome or persisting progress; callers retain the process and storage.
     • Failure records, quarantine and store attribution; SL2.18b exposes only restore failure facts.
   • Approach: Extract the transition logic and canonical record bytes from the existing clone restore sequence into a small deterministic state machine. Represent every external wait or operation as an action and every result as an input, preserving phase and evidence directly; do not abstract over runtimes, embed callbacks, retain clone-store records or simulate a clock.
   • Done when:
     • Every session receives identical record bytes, and Ready with another transaction is refused.
     • SessionLost returns the machine to a Connect action without changing the accepted record.
     • OperationFailed, Expired, Cancelled and Exited are handled in every pending phase with the expected facts.
     • The successful action sequence ends at Ready and never emits a mapping-table or Start action.
   • Needs: SL2.5b, in nvx-rs slimming 2.3.1: activation record v2 carries only what the guest applies #1101; SL2.17, in nvx-rs slimming 2.2.1: a VM is created in a private directory, with its restore-ready listener, then spawned #1099.
   • Estimate: 450 core lines, 2 counted files. Past 600 core lines, stop and ask before continuing.

4. SL2.18b: A failed restore reports its facts.

   • Objective: Restore failures currently arrive wrapped in clone records, teardown and quarantine policy. Callers instead receive the facts needed to choose their own response, while retaining ownership of the VM process after failure.
   • Contents: RestoreError: the phase, Attribution, the evidence, whether the guest may have run, a message and a source, from nvx-rs/src/host/template/clone/failure.rs, without its records or quarantine. RestoreAction::Fail returns the error without stopping the VM: a failed restore leaves the VM's process to the caller.
   • Scope:
     • RestoreError reports the restore phase, Attribution, evidence, whether the guest may have run, a message and an error source.
     • The fact classification is derived from the existing clone failure and restore progress rules in nvx-rs/src/host/template/clone/failure.rs, with its cases adapted from failure/tests.rs into host/restore/tests.rs.
     • RestoreAction::Fail returns the error without stopping, killing or detaching the VM.
     • Restore-machine tests cover each exposed fact and the process-ownership contract.
   • Not in this PR:
     • Clone failure records, progress journals, teardown records or quarantine; callers own persistence and policy.
     • Automatically stopping the VM or reading its outcome report; the failed launch remains with the caller.
     • A retry, reporting or snapshot-disposition policy based on attribution; the error contains facts only.
     • A general crate-wide error redesign; this PR adds the restore domain's error.
   • Approach: Keep the useful phase, evidence and source classification from the existing failure code, rename its policy-free attribution value and discard store, lease, record, quarantine and teardown machinery. Construct RestoreError at the machine's failure transitions and return it unchanged; do not add callbacks, automatic cleanup or a durable failure format.
   • Done when:
     • RestoreError exposes phase, attribution, evidence, guest-may-have-run, message and source for each machine failure class.
     • Tests distinguish failures before and after the guest may have run.
     • RestoreAction::Fail leaves the VM process running or exited as it already is, for the caller to handle.
     • No restore error path writes records or quarantines a snapshot.
   • Needs: SL2.18a, above.
   • Estimate: 200 core lines, 3 counted files. Past 400 core lines, stop and ask before continuing.

Tests. The gate, on real VMs with the test adapter's guest: restores of one snapshot start while views are added and removed under the lock; a link added without the lock while a restore starts fails it; a controller killed before OpenVMM connects the restore-ready endpoint leaves a VMM that fails that connect and exits; and once OpenVMM has connected it, a link change no longer fails the restore. create(from: Some) refuses boot and a VM that the snapshot doesn't accept; builds the hard-linked view, and fails clearly when the view would cross filesystems; renders the restore arguments with the console attachment that the snapshot allows, so a snapshot captured Disconnected can't restore with a listener; resolves the NIC and the activation record against the snapshot; binds the ready listener before the spawn; and hands it over through both Launched::ready and VmmCommand::ready. The external-launcher example restores. RestoreMachine sends the same record on every session, checks the transaction echo, offers Connect again after SessionLost, keeps the phase, evidence and source of OperationFailed, handles Expired and Cancelled, and Exited in every pending phase, and ends at Ready without sending the mapping table or Start. RestoreError carries its phase, attribution, evidence, whether the guest may have run, message and source, and a failure leaves the VM's process to the caller. The clone path's launch, retry and failure tests that cover these behaviors are kept or rewritten by name.

Done when: every PR above has merged, with dev green.

Existing restore-regression coverage to preserve

SL2.15 extends the existing real-VM restore fixtures. Preserve their assertions that OpenVMM consumes no host input before repair acknowledgement and that disk reads obey the restore invariants. The hard-link/exclusion gate supplements those checks; it does not replace them or weaken their assertions. This is coverage preservation, not a new restore protocol or production instrumentation requirement.