cli: restore managed sandboxes from named snapshots (SL3.8a2)

doc/usage.md

@@ -383,7 +383,60 @@ evidence and reports an error. Unknown files or nonempty unjournaled capture-slo
 contents are kept for inspection, not deleted; an empty launch lock alone is
 harmless. Known host and OpenVMM atomic staging files are recovered only with
 a valid launch journal. Captures in one store serialize; snapshot locks remain
-separate from their immutable directories. Restore commands are not added here.
+separate from their immutable directories. [`snapshot restore`](#snapshot-restore)
+starts managed sandboxes from them.
+
+### `snapshot restore`
+
+```sh
+nvx-rs/target/release/nvx-rs snapshot restore baseline \
+    --store-dir /private/snapshots --state-dir /private/sandboxes/restored \
+    --layer distro,/images/distro.erofs,UUID --upper ram --hypervisor kvm
+```
+
+`snapshot restore` provisions a managed sandbox in `--state-dir` with the
+options of [`sandbox provision`](#sandbox), but for those that only an
+execution or a cold boot takes, `--cwd`, `--environment`, `--environment-file`,
+`--inherit-default-environment`, `--exec-timeout-ms`, `--wait`,
+`--outcome-report`, `--cmdline`, and `--dry-run`, then starts it from the named
+snapshot of `--store-dir`. The sandbox's configuration records the snapshot,
+its store, and the sandbox's workload, `--entrypoint` with its `--arg` values,
+and each later `sandbox start` of the sandbox restores the snapshot again. Each
+restore runs the workload over the control protocol, as `run` and `exec` do,
+not from the kernel command line, so its entrypoint and arguments may contain
+whitespace and double quotes, which `sandbox provision` refuses; the command
+checks them as `exec` checks its workload before it provisions anything. The
+state directory must be outside the store. The guest of a restore accepts only
+a fresh scratch image, whose upper no earlier root wrote, such as a new empty
+ext4 image, so a later start of a restored sandbox with a scratch image refuses
+once an earlier start has created the sandbox's view of the snapshot, as each
+start does before its guest can use the image: deprovision the sandbox and
+restore the snapshot again with a fresh one. A missing snapshot provisions
+nothing; a sandbox whose start fails stays provisioned, as after
+`sandbox provision`.
+
+A restore attaches the sandbox's image and upper in place of the snapshot's
+placeholder image and dormant scratch: the image must fit the placeholder,
+with its block sizes, and a scratch upper must be as long as the dormant
+scratch. The VM keeps the processors that the snapshot had online, and
+`--memory-mib` must be a multiple of 128 between the snapshot's boot memory and
+its capacity. The snapshot fixes the NIC: one that `snapshot capture` captured
+has none, so its restores take no `--net`. A restore of a snapshot with a NIC
+installs the restore's own network policy, whose DNS server and listener
+announcements the guest applies as it activates.
+
+Each start holds the snapshot's exclusive lock, which it waits for up to
+`--timeout`, from before it links the snapshot's files into the sandbox's view
+of it, `openvmm/` in the state directory, until the start ends or the VM has
+exited, and `sandbox deprovision` holds it while it removes the view, so it
+needs the snapshot's store: concurrent restores of one snapshot start one at a
+time. Once the guest is ready, it maps the sandbox's shares and runs the
+workload, whose output the command forwards, and waits for it to exit with
+status 0 within `--timeout`, as the default workload, `/bin/sh`, does at once;
+any other end fails the start, which ends the VM. The sandbox then runs as a
+started one does, for `sandbox exec` and `sandbox stop`. A start journals its
+launch as a cold start does, so a restore whose command dies before the guest
+is ready never completes, and the next command of the sandbox ends its VM.
 
 ### `download`
 

nvx-rs/README.md

@@ -2560,6 +2560,7 @@ nvx-rs [-h] [--root PATH] egress compile PATH
 nvx-rs [-h] [--root PATH] run [OPTIONS]
 nvx-rs [-h] [--root PATH] sandbox [run|provision|start|exec|stop|deprovision] [OPTIONS]
 nvx-rs [-h] [--root PATH] snapshot capture NAME [OPTIONS]
+nvx-rs [-h] [--root PATH] snapshot restore NAME [OPTIONS]
 ```
 
 ```bash
@@ -2617,7 +2618,24 @@ exit before finishing and publishing exactly the strict snapshot's three
 entries. The next capture using that store ends an interrupted attempt through
 ordinary launch recovery, never resuming it. See
 [`snapshot capture`](../doc/usage.md#snapshot-capture) for its resource options
-and recovery behavior. Restore commands are separate follow-ups.
+and recovery behavior.
+
+`snapshot restore NAME --store-dir PATH --state-dir PATH` provisions a managed
+sandbox, with the options of `sandbox provision` that a restore takes, whose
+configuration also records the snapshot, its store, and the sandbox's workload,
+then starts it: that start, and each later one, creates the VM with
+`host::vm::create` from the snapshot, through the store's launch journal, and
+drives its restore with the blocking restore driver, which activates the guest
+with the start's own network policy, maps the sandbox's shares, and runs the
+workload, whose output the start forwards until it exits. A restore's guest
+accepts only a fresh upper, so a later start refuses a scratch image once an
+earlier start has created its view, as each start does before its guest can use
+the image. Each start holds the snapshot's exclusive lock from before it links
+the snapshot's files into the sandbox's `openvmm/` view until the start ends or
+the VM has exited, and `deprovision` holds it while it removes the view; a start
+whose command dies leaves a journal whose recovery ends the VM. No index of a
+snapshot's sandboxes or record of a restore in progress exists. See
+[`snapshot restore`](../doc/usage.md#snapshot-restore).
 
 ### Artifact root
 
@@ -2868,7 +2886,7 @@ them:
 
 | File | Contents |
 | --- | --- |
-| `config.json` | The provisioned sandbox, in format 9 only: `image` holds its format, path and identity; `upper` is tagged as scratch with its path or RAM with `max_memory_percent`. `distro_binding` binds the image, and `scratch_binding` binds scratch or is `null` for RAM. The export and network are typed, `null` when absent. An atomic pretty-JSON write uses mode 0600, and start reads the same strict typed fields without a legacy reader. |
+| `config.json` | The provisioned sandbox, in format 9 only: `image` holds its format, path and identity; `upper` is tagged as scratch with its path or RAM with `max_memory_percent`. `distro_binding` binds the image, and `scratch_binding` binds scratch or is `null` for RAM. The export and network are typed, `null` when absent. A sandbox that `snapshot restore` provisioned also records its snapshot's name under `source_snapshot`, the absolute path of the snapshot's store under `source_store`, and the program and arguments of the workload that each restore runs under `workload`; one that boots cold records none of them. An atomic pretty-JSON write uses mode 0600, and start reads the same strict typed fields without a legacy reader. |
 | `sandbox.lock` | The store's lock, which serializes the lifecycle transitions |
 | `exec.lock` | The exec lock, which an `exec` holds from before it finds the running VM until its workload ends, so that one `exec` at a time runs in the sandbox; it stays from the first `exec` until `deprovision` |
 | `launch.json` | The store's journal of a start in progress, or of one that was interrupted |
@@ -2877,6 +2895,8 @@ them:
 | `control.sock` | The control endpoint on POSIX hosts |
 | `openvmm.log` | OpenVMM's output, which each start replaces |
 | `outcome.json` | OpenVMM's outcome report of the last start |
+| `openvmm/` | A restored sandbox's view of its snapshot, hard links to the files of the snapshot's `openvmm/`, which a start creates and `deprovision` removes, each under the snapshot's lock |
+| `restore-ready.sock` | On Linux, the endpoint of a restore's restore-ready event, which the next start of a restored sandbox, or its `deprovision`, removes under the snapshot's lock |
 
 `provision`, `start`, `stop`, and `deprovision` hold the store's lock for their
 whole transition, and `exec` only while it finds the running VM, so that a long

nvx-rs/src/host/cli/mod.rs

@@ -7,6 +7,7 @@
 //! nvx-rs [-h] [--root PATH] run [OPTIONS]
 //! nvx-rs [-h] [--root PATH] sandbox [run|provision|start|exec|stop|deprovision] [OPTIONS]
 //! nvx-rs [-h] [--root PATH] snapshot capture NAME [OPTIONS]
+//! nvx-rs [-h] [--root PATH] snapshot restore NAME [OPTIONS]
 //! ```
 //!
 //! - `artifacts` prints the paths of the artifacts of the artifact root as JSON.
@@ -19,7 +20,8 @@
 //!   exit statuses of the `nvx.py run` that it replaced.
 //! - `sandbox` runs or manages a sandbox, with the options, outputs, exit statuses, outcome
 //!   reports, and state directories of the `nvx.py sandbox` that it replaced.
-//! - `snapshot capture` captures a blockless VM into an immutable named snapshot.
+//! - `snapshot capture` captures a blockless VM into an immutable named snapshot, and
+//!   `snapshot restore` starts a new managed sandbox from one.
 //!
 //! The artifact root is `--root`, or else the release or repository checkout that holds the
 //! executable, and the `NVX_OPENVMM` environment variable names another OpenVMM executable.
@@ -126,7 +128,8 @@ positional arguments:
     run                 run an OpenVMM microVM
     sandbox             run or manage workloads over EROFS layers and private
                         ext4 scratch
-    snapshot            capture an immutable named snapshot
+    snapshot            capture an immutable named snapshot, or restore a
+                        managed sandbox from one
 
 options:
   -h, --help            show this help message and exit
@@ -345,8 +348,8 @@ enum Command {
     Run(Box<run::Options>),
     /// `sandbox ...`.
     Sandbox(Box<sandbox::Options>),
-    /// `snapshot capture ...`.
-    Snapshot(snapshots::capture::Options),
+    /// `snapshot capture ...` or `snapshot restore ...`.
+    Snapshot(snapshots::Command),
 }
 
 /// The top-level options.
@@ -403,8 +406,8 @@ fn run(arguments: &[OsString]) -> Result {
             (Command::Sandbox(Box::new(options)), unrecognized)
         }
         "snapshot" => {
-            let (options, unrecognized) = snapshots::capture::parse(rest)?;
-            (Command::Snapshot(options), unrecognized)
+            let (command, unrecognized) = snapshots::parse(rest)?;
+            (Command::Snapshot(command), unrecognized)
         }
         other => {
             return Err(usage(
@@ -430,7 +433,7 @@ fn run(arguments: &[OsString]) -> Result {
         Command::EgressCompile(path) => egress::run(&path),
         Command::Run(options) => run::run(root, *options),
         Command::Sandbox(options) => sandbox::run(root, *options),
-        Command::Snapshot(options) => snapshots::capture::run(root, options),
+        Command::Snapshot(command) => snapshots::run(root, command),
     }
 }
 

nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs

@@ -31,7 +31,7 @@ use crate::host::cli::artifacts::{Located, locate};
 use crate::host::cli::deadline::RETRY_INTERVAL;
 use crate::host::cli::sandbox::lock::LifecycleLock;
 use crate::host::cli::sandbox::state::{
-    Config, RecordedDistro, RecordedScratch, read_config, serialize_launch, write_json,
+    Config, RecordedDistro, RecordedScratch, Source, read_config, serialize_launch, write_json,
 };
 use crate::host::cli::sandbox::{Artifacts, error_with_cause};
 use crate::host::cli::session::Session;
@@ -76,12 +76,14 @@ pub(crate) const REQUIRED_FEATURES: GuestFeatures = GuestFeatures::LAYERED_SANDB
 
 /// Records a launch and its distro layer, at the path at which it was verified, with the layer's
 /// binding, in a new state directory, as `provision` does, with the binding of its scratch image,
-/// if its upper is on scratch, at the path that it records for it.
+/// if its upper is on scratch, at the path that it records for it, and `source`, the named
+/// snapshot that each start of a sandbox of `snapshot restore` restores, or `None`.
 pub(crate) fn provision(
     state_path: &str,
     launch: &SandboxLaunch,
     distro: &RecordedDistro,
     config: &ProvisionConfig<'_>,
+    source: Option<&Source>,
     timeout: f64,
 ) -> Result {
     let state_dir = prepare_state_directory(state_path, true)?;
@@ -98,7 +100,7 @@ pub(crate) fn provision(
     let scratch = RecordedScratch::new(launch)?;
     write_json(
         &config_path,
-        &serialize_launch(launch, distro, scratch.as_ref(), config, None)?,
+        &serialize_launch(launch, distro, scratch.as_ref(), config, source)?,
     )
 }
 

nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs

@@ -9,13 +9,17 @@ use serde_json::{Value, json};
 
 use super::super::*;
 use crate::activation::{ActivationRecord, Image, Network, Publication, ReadyActivation, Upper};
+use crate::host::artifacts::{
+    ArtifactDir, Guest, INITRAMFS_PROVENANCE_NAME, KERNEL_PROVENANCE_NAME, OPENVMM_PROVENANCE_NAME,
+};
 use crate::host::cli::hex;
 use crate::host::cli::sandbox::lifecycle::{ProvisionConfig, deprovision};
 use crate::host::cli::sandbox::state::{RecordedDistro, serialize_launch, write_json};
-use crate::host::cli::snapshots::Name;
+use crate::host::cli::snapshots::{self, Name};
 use crate::host::cli::store::StateDir;
 use crate::host::filesystem::{Bind, ExportChild, ExportPlan, Owner};
 use crate::host::network::{Access, NetworkPolicy};
+use crate::host::paths::resolve;
 use crate::host::process::{Opened, Process, ProcessRef};
 use crate::host::sandbox::distro::DistroBinding;
 use crate::host::sandbox::layer::{LayerRole, SandboxLayer};
@@ -600,6 +604,102 @@ fn a_restored_start_refuses_a_share_of_its_snapshot_store() {
     assert!(fixture.spawned().is_empty());
 }
 
+/// Returns an artifact root with the fakes and provenance whose presence a start checks before
+/// it launches anything, which no start of these tests runs.
+fn artifact_root(root: &Path) -> String {
+    let dir = ArtifactDir::from_dir(root).unwrap();
+    for (artifact, provenance, field) in [
+        (dir.openvmm(), OPENVMM_PROVENANCE_NAME, "executable_sha256"),
+        (dir.kernel(), KERNEL_PROVENANCE_NAME, "kernel_sha256"),
+        (
+            dir.initramfs(Guest::Alpine),
+            INITRAMFS_PROVENANCE_NAME,
+            "initramfs_sha256",
+        ),
+    ] {
+        for (path, contents) in [
+            (artifact, "fake".to_owned()),
+            (
+                dir.artifact(provenance),
+                json!({ "format": 1, field: "ab".repeat(32) }).to_string(),
+            ),
+        ] {
+            fs::create_dir_all(path.parent().unwrap()).unwrap();
+            fs::write(path, contents).unwrap();
+        }
+    }
+    root.to_str().unwrap().to_owned()
+}
+
+/// `snapshot restore` provisions a sandbox whose configuration records the snapshot, the store as
+/// the command resolved it, and the workload, an exec request that may hold whitespace and
+/// quotes, which no boot token can, and starts it as a restore of that snapshot: a start with
+/// memory that the snapshot refuses, which only a restore checks, fails before it launches or
+/// links anything, and leaves the sandbox provisioned.
+#[test]
+fn snapshot_restore_records_its_source_and_starts_a_restore_of_it() {
+    let fixture = Fixture::new(false);
+    let root = artifact_root(&fixture.state("checkout"));
+    let layer = fixture.state("distro.erofs");
+    fs::write(&layer, "distro").unwrap();
+    let store = fixture.root.path().join("store");
+    let state = fixture.state("command");
+    let workload = ["/bin/echo", "a \"restored\" sandbox"];
+    let arguments = [
+        "restore",
+        "source",
+        "--store-dir",
+        store.to_str().unwrap(),
+        "--state-dir",
+        state.to_str().unwrap(),
+        "--layer",
+        &format!(
+            "distro,{},11111111-1111-1111-1111-111111111111",
+            layer.display()
+        ),
+        "--upper",
+        "ram",
+        "--hypervisor",
+        if cfg!(windows) { "whp" } else { "kvm" },
+        "--memory-mib",
+        "2176",
+        "--entrypoint",
+        workload[0],
+        "--arg",
+        workload[1],
+        "--timeout",
+        "10",
+    ]
+    .map(str::to_owned);
+    let (command, unrecognized) = snapshots::parse(&arguments).unwrap();
+    assert!(unrecognized.is_empty(), "{unrecognized:?}");
+    let error = snapshots::run(Some(&root), command).unwrap_err();
+    assert_eq!(error.code(), Some(ErrorCode::Unsupported), "{error:?}");
+    let message = error.message().unwrap_or_default();
+    assert!(
+        message.contains("snapshot memory must be from boot"),
+        "{message}"
+    );
+    let record = read_config(&state.join(CONFIG_NAME)).unwrap();
+    assert_eq!(record["source_snapshot"], "source");
+    assert_eq!(record["workload"], json!(workload));
+    assert_eq!(
+        Config::new(record).unwrap().source().unwrap(),
+        Some(Source {
+            snapshot: name(),
+            store: resolve(&store).unwrap(),
+            workload: workload.map(str::to_owned).to_vec(),
+        })
+    );
+    assert_eq!(fixture.view("command"), 0);
+    assert!(fixture.snapshot_free());
+    let artifacts = Artifacts {
+        root: Some(root.as_str()),
+    };
+    deprovision(state.to_str().unwrap(), &artifacts, 5.0).unwrap();
+    assert!(!state.exists());
+}
+
 /// Reads one record of the control endpoint, or `None` once the host has gone.
 fn read_record(stream: &mut impl Read) -> Option<(u8, Vec<u8>)> {
     let mut header = [0; OUTER_HEADER_LEN];

nvx-rs/src/host/cli/sandbox/mod.rs

@@ -44,10 +44,11 @@ pub(super) use self::lifecycle::{copy_openvmm_outcome, validate_outcome_destinat
 pub(super) use self::one_shot::run_openvmm;
 pub(super) use self::options::sandbox_network;
 pub(super) use self::options::{
-    HYPERVISORS, MAX_MEMORY_MIB, MOUNT_OWNERS, hypervisor, parse, path_text,
+    HYPERVISORS, MAX_MEMORY_MIB, MOUNT_OWNERS, OPTIONS, hypervisor, parse, path_text,
 };
 pub(crate) use self::options::{Options, path_string};
 use self::state::RecordedDistro;
+pub(crate) use self::state::Source;
 
 /// The artifacts of a sandbox, found when an operation first needs them.
 pub(crate) struct Artifacts<'a> {
@@ -133,6 +134,25 @@ pub(crate) fn error_with_cause(error: crate::Error) -> Exit {
 
 /// Runs the operation of `options`, as `command_sandbox` did.
 pub(super) fn run(root: Option<&str>, options: Options) -> Result {
+    run_from(root, options, None)
+}
+
+/// Provisions the sandbox of `options`, whose operation is `provision`, with `source`, the named
+/// snapshot that each of its starts restores, then starts it, as `snapshot restore` does.
+pub(super) fn restore(root: Option<&str>, options: Options, source: Source) -> Result {
+    let (state_dir, timeout) = (options.state_dir.clone(), options.timeout);
+    run_from(root, options, Some(&source))?;
+    lifecycle::start(
+        state_dir
+            .as_deref()
+            .expect("provision required --state-dir"),
+        &Artifacts { root },
+        timeout,
+    )
+}
+
+/// Runs the operation of `options`, recording `source` with the sandbox that it provisions.
+fn run_from(root: Option<&str>, options: Options, source: Option<&Source>) -> Result {
     let operation = options.operation;
     if !matches!(
         operation,
@@ -198,9 +218,9 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result {
             workload_identity: options.workload_user,
             ..SandboxSettings::default()
         };
-        // A run's workload is its exec request, which no boot token carries, and which its control
-        // session checks.
-        if operation == SandboxOperation::Run {
+        // The workload of a run, and that of a restore, is an exec request, which no boot token
+        // carries: a run's control session checks it, and `snapshot restore` a restore's.
+        if operation == SandboxOperation::Run || source.is_some() {
             let workload = SandboxSettings::default();
             (settings.entrypoint, settings.args) = (workload.entrypoint, workload.args);
         }
@@ -283,6 +303,7 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result {
                     cmdline: &options.cmdline,
                     ram_upper_percent: options.ram_upper_percent,
                 },
+                source,
                 options.timeout,
             )
         }