diff --git a/nvx-rs/README.md b/nvx-rs/README.md index bf3756829..d12fac71e 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -124,7 +124,7 @@ changes only when the module's purpose does. | [`nvx::Hypervisor`](#nvxhypervisor) | KVM, MSHV, or WHP, with its serde form behind the `serde` feature | | [`nvx::exec`](#nvxexec) | The execution engine: the streams, credit, and outcome of an execution's two sides, without I/O, for any framing | | [`nvx::host::runtime`](#nvxhostruntime) | `Runtime`: what every launch shares, the runtime's files and their identity, the hypervisor and its probe, and the launch options | -| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold and capture preparation, owned cold-launch command, verified detached launch, confirmed stop, and caller-saved reattachment | +| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold, capture and restore preparation, owned launch commands, verified detached launch, confirmed stop, and caller-saved reattachment | | [`nvx::host::snapshot`](#nvxhostsnapshot) | Strict private snapshots, post-exit capture finishing, verified metadata, VM compatibility, captured NIC addresses, and time eligibility | | [`nvx::host::config`](#nvxhostconfig) | `VmConfig` and `CaptureConfig`: pure configuration, cold-boot kernel/OpenVMM arguments, portable length checks, and capture identity | | [`nvx::host::restore`](#nvxhostrestore) | A caller-driven restore to Ready without I/O or a clock, and its peer-verified restore-ready listener | @@ -513,13 +513,38 @@ failure never hides confirmed exit. `vm::create(&runtime, &config, None, &dir)` creates an inspectable cold `Prepared` launch without starting a process or binding a listener. It requires -`config.boot`, refuses a snapshot input, rechecks the directory's privacy, and +`config.boot`, rechecks the directory's privacy, and creates a new private log. Existing log, outcome and Linux endpoint entries are refused rather than replaced. An ordinary cold boot creates only the log before a spawn; OpenVMM creates its remaining outputs when it runs. A failed preparation closes and removes only its new files, preserving all pre-existing entries and caller-owned inputs, and reports a cleanup failure with the original error. +With `Some(snapshot)`, creation refuses `boot`, checks `Snapshot::accepts` +at the real `HostTime::now`, and hard-links only the existing OpenVMM files +into a new `/openvmm/`. A different filesystem fails; there is no +copy or snapshot-parent fallback. The restore's arguments follow the clone +launch's rules: the devices at the snapshot's processor capacity, +`--restore-network-policy` with the restore's own policy in place of `--net` +and the guest's address, which OpenVMM takes from the snapshot, +`--restore-absent-scratch` for an upper in RAM, `--restore-processors` only +when the snapshot records its boot processors, `--restore-memory`, the +console attachment that the snapshot allows, the snapshot's backend and CPU +profile, and the launch options, with no kernel, initramfs, command line or +`--memory`. Creation binds the restore-ready listener before any spawn and +retains that same owned listener through `Launched::ready` or +`VmmCommand::ready`, never in the child's inheritance allowlist. + +On Linux, the caller holds its per-snapshot lock from creation, including +adding links, through a peer-verified restore-ready event or verified VM +exit, and around view removal. NVX takes no lock; Windows needs none. +A restore whose caller dies before OpenVMM connects the restore-ready +endpoint cannot complete: OpenVMM checks the snapshot files before that one +connection, which it does not retry, and the dead caller's endpoint no longer +listens. After the connection OpenVMM no longer checks those files, so no +durable in-flight restore record is needed; the caller keeps the VM's process +as for any launch. A successful view remains caller-owned. + With `boot.capture`, creation requires a blockless root and refuses every filesystem export. It creates private sparse `capture-image.raw` and `capture-scratch.raw` files, verifies their actual complete geometries, and @@ -566,13 +591,14 @@ selects the network's presence and IPv4 and derived IPv6 gateways, while the launch's policy supplies DNS, publication and proxy. An omitted limit takes the record's maximum bound. It reads no host path and opens no snapshot. A blockless root or memory that cannot fit the record's byte count fails with -`ErrorKind::Unsupported`. These projections do not enable the restore caller; -`create` still accepts cold launches only. +`ErrorKind::Unsupported`. A restore resolves these settings against the +captured NIC and its own requested policy. `Prepared::into_command()` moves the complete launch into `VmmCommand`: its program, arguments, working directory, capability, outputs, inherited resources, -opened log and Windows breakaway setting. It neither reopens nor duplicates -resources, renders again nor rediscovers paths. `InheritedHandle` owns its +opened log, Windows breakaway setting, and any already-bound ready listener. +It neither reopens nor duplicates resources, renders again nor rediscovers +paths. `InheritedHandle` owns its resource, converts from and back to `OwnedFd` on Unix or `OwnedHandle` on Windows, and also accepts an opened `File`. The command documents the caller's spawn contract, including standard streams, allowlisted inheritance, Linux detachment @@ -589,9 +615,17 @@ outcome with complete teardown. Build with kernel, initramfs, new VM directory, and the three artifact SHA-256 digests from their installation metadata, in that order. Artifact paths may be relative to the caller's directory. +Append a snapshot directory and EROFS image to run its restore-ready +demonstration instead, using a snapshot captured with a disconnected boot +console, as `nvx-rs snapshot capture` captures it. It uses the public +peer-verified receiver, then intentionally stops and verifies the VM's exit +without driving guest activation. Linux callers of the example coordinate on +the snapshot's `key.bin` lock, taken before opening the snapshot and retained +through view removal. `vm::spawn(prepared)` consumes that preparation and returns `Launched` with -its verified `process`, the same `outputs`, and the same `capability`. The log +its verified `process`, the same `outputs`, the same `capability`, and a +restore's same bound `ready` listener. The log and inherited resources are passed as already opened objects, never reopened by name. The capability reaches OpenVMM only on standard input, then end-of-file, not in arguments, environment variables, or a file. A failure @@ -673,9 +707,9 @@ the supplied `HostTime` is `Unavailable`. The caller keeps the directory and every entry private and immutable while using the snapshot; opening checks them once and retains no file handles -or lock. `Snapshot` remains an opaque input of a future restore: -`vm::create` still refuses `Some(snapshot)` with `ErrorKind::Unsupported`; -only cold boots are supported. +or lock. `vm::create(..., Some(snapshot), ...)` checks compatibility and +current-time eligibility, then creates the fixed hard-linked view and +binds the caller-owned ready listener described above. `CaptureContext` records a prepared capture's checked configuration and effective devices independently of runtime paths or network policy. @@ -844,7 +878,7 @@ Start. Terminal results are offered once and absorb later inputs. `ReadyListener` owns a restore-ready endpoint: a mode-0600 Unix socket in the private VM directory on Linux, or a fresh first-instance named pipe with an -owner-only protected access control list on Windows. The library's receiver +owner-only protected access control list on Windows. Its public `receive` checks the connecting client's process ID against the held VM process. On Linux it also checks the socket's original peer through `SO_PEERPIDFD`, so a reaped client's queued event cannot impersonate a VM that reuses its ID. @@ -861,8 +895,8 @@ socket peer through `SO_PEERPIDFD`, refuses an exited peer, and matches its credentials against the held VM process: opening a pidfd from the cached numeric PID is not authentication. On Windows it checks the named-pipe client process ID. After matching the peer, it checks `Process::has_exited` and -refuses an exited VM. Listener construction remains internal until restore -launch preparation supplies it. +refuses an exited VM. Only `vm::create` constructs a listener, for a restore, +and hands it over through `VmmCommand::ready` or `Launched::ready`. The Linux PID-reuse regression runs a separate helper in private PID and mount namespaces with Bash as its reaper. If the host refuses unprivileged user diff --git a/nvx-rs/examples/external-launch.rs b/nvx-rs/examples/external-launch.rs index 12060e616..c168e53ff 100644 --- a/nvx-rs/examples/external-launch.rs +++ b/nvx-rs/examples/external-launch.rs @@ -1,28 +1,41 @@ -//! A caller-owned cold launch, using the strict low-level spawn primitive without a store. +//! A caller-owned launch, using the strict low-level spawn primitive without a store. //! //! Run the compiled example with a backend, OpenVMM, kernel, initramfs, new VM directory, and //! the three artifact SHA-256 digests from their installation metadata, in that order. +//! Append SNAPSHOT-DIR IMAGE.erofs for a restore-ready demonstration: it verifies the ready +//! event, then intentionally stops and verifies the VM's exit without driving guest activation. +//! The snapshot must have captured a disconnected boot console, as `nvx-rs snapshot capture` +//! does. use std::error::Error; +#[cfg(target_os = "linux")] +use std::fs::File; use std::io::{self, Write}; use std::path::Path; use std::time::{Duration, Instant}; use nvx::Hypervisor; use nvx::activation::{MachineIdMode, View}; -use nvx::host::config::{Boot, ConsoleAttachment, RootConfig, VmConfig}; +use nvx::host::config::{ + Boot, ConsoleAttachment, LayeredRoot, NetworkConfig, RootConfig, UpperConfig, VmConfig, +}; +use nvx::host::network::{Access, NetworkPolicy}; use nvx::host::platform::Transport; use nvx::host::runtime::{ArtifactIdentity, LaunchOptions, Runtime, RuntimeIdentity}; use nvx::host::supervisor::{self, Inherited}; -use nvx::host::{artifacts, report, session, vm}; +use nvx::host::{artifacts, report, session, snapshot, vm}; +use nvx::protocol::root::RootImage; + +/// One byte past OpenVMM's 25-byte restore-ready event, so that a longer event fails to parse. +const READY_LIMIT: usize = 26; fn main() -> Result<(), Box> { let args: Vec<_> = std::env::args_os().collect(); - if args.len() != 9 { + if args.len() != 9 && args.len() != 11 { return Err(io::Error::new( io::ErrorKind::InvalidInput, "usage: external-launch BACKEND OPENVMM KERNEL INITRAMFS VM-DIR \ - OPENVMM-SHA256 KERNEL-SHA256 INITRAMFS-SHA256", + OPENVMM-SHA256 KERNEL-SHA256 INITRAMFS-SHA256 [SNAPSHOT-DIR IMAGE.erofs]", ) .into()); } @@ -52,7 +65,19 @@ fn main() -> Result<(), Box> { windows_breakaway: false, options: LaunchOptions::default(), }; - let config = VmConfig { + let snapshot_dir = (args.len() == 11).then(|| Path::new(&args[9])); + // Every caller of this snapshot must take this same lock before it opens the snapshot, and + // hold it through the restore-ready event or the VM's exit, and the view's removal. + #[cfg(target_os = "linux")] + let snapshot_lock = snapshot_dir + .map(|dir| File::open(dir.join(snapshot::KEY_FILE))) + .transpose()?; + #[cfg(target_os = "linux")] + if let Some(lock) = &snapshot_lock { + lock.lock()?; + } + let snapshot = snapshot_dir.map(snapshot::Snapshot::open).transpose()?; + let mut config = VmConfig { processors: 1, memory_mib: 256, root: RootConfig::Blockless, @@ -69,6 +94,29 @@ fn main() -> Result<(), Box> { capture: None, }), }; + if let Some(snapshot) = &snapshot { + let key = snapshot.key(); + config.processors = match key.boot_processors() { + 0 => key.processors(), + boot => boot, + }; + config.memory_mib = key.memory() / (1 << 20); + config.root = RootConfig::Layered(LayeredRoot { + image: artifacts::absolute(Path::new(&args[10]))?, + format: RootImage::erofs(None), + gpt_first_partition_bound: None, + upper: UpperConfig::Ram { + max_memory_percent: 50, + }, + overlay: nvx::activation::OverlayMode::TrustedLowers, + }); + config.network = snapshot.nic().map(|_| NetworkConfig { + guest_network: None, + policy: NetworkPolicy::new(Access::Deny), + }); + config.console = ConsoleAttachment::Disconnected; + config.boot = None; + } vm::create_private_dir(Path::new(&args[5]))?; let dir = vm::VmDir::open(Path::new(&args[5]))?; let vm::VmmCommand { @@ -80,7 +128,8 @@ fn main() -> Result<(), Box> { inherit, log, breakaway_from_job, - } = vm::create(&runtime, &config, None, &dir)?.into_command(); + mut ready, + } = vm::create(&runtime, &config, snapshot.as_ref(), &dir)?.into_command(); let command = supervisor::VmmCommand { program, arguments, @@ -88,10 +137,46 @@ fn main() -> Result<(), Box> { breakaway_from_job, inherit: inherit.iter().map(Inherited::new).collect(), }; - let mut launched = supervisor::spawn_vm(&command, capability.as_bytes(), log)?; + let mut launched = match supervisor::spawn_vm(&command, capability.as_bytes(), log) { + Ok(launched) => launched, + Err(error) => { + if snapshot.is_some() + && let Err(cleanup) = remove_view(dir.path()) + { + return Err(io::Error::other(format!( + "{error}; cannot remove restore view after spawn failure: {cleanup}" + )) + .into()); + } + return Err(error.into()); + } + }; let result = (|| -> Result<(), Box> { let process = launched.process()?; let deadline = Instant::now() + Duration::from_secs(60); + if let Some(listener) = &mut ready { + let mut event = Vec::new(); + let mut open = true; + while open && event.len() < READY_LIMIT { + let timeout = deadline.saturating_duration_since(Instant::now()); + if timeout.is_zero() { + return Err( + io::Error::new(io::ErrorKind::TimedOut, "restore-ready timed out").into(), + ); + } + open = listener.receive(&process, &mut event, READY_LIMIT, timeout)?; + } + nvx::protocol::restore_ready::parse_ready_event(&event)?; + process.kill()?; + if process.wait(deadline)?.is_none() || launched.wait_timeout(Duration::ZERO)?.is_none() + { + return Err(io::Error::other("restored VM exit is unconfirmed").into()); + } + println!( + "external restore-ready peer verified; demonstration VM intentionally stopped" + ); + return Ok(()); + } let console = outputs.console.as_deref().ok_or_else(|| { io::Error::new(io::ErrorKind::InvalidInput, "no prepared boot console") })?; @@ -136,16 +221,36 @@ fn main() -> Result<(), Box> { )) .into()); } + if snapshot.is_some() + && let Err(cleanup) = remove_view(dir.path()) + { + return Err(io::Error::other(format!( + "{error}; cannot remove restore view: {cleanup}" + )) + .into()); + } return Err(error); } launched.detach(); + if snapshot.is_some() { + remove_view(dir.path())?; + } println!( - "external cold launch completed; outcome={}", + "external launch completed; outcome={}", outputs.outcome.display() ); Ok(()) } +/// Removes only the successful fixed restore view, while the caller still holds its lock. +fn remove_view(directory: &Path) -> io::Result<()> { + let view = directory.join(snapshot::OPENVMM_DIR); + for name in [snapshot::MANIFEST_NAME, "memory.bin", "state.bin"] { + std::fs::remove_file(view.join(name))?; + } + std::fs::remove_dir(view) +} + /// Prints console output through a marker, retaining only enough bytes to match across reads. fn read_until(console: &mut dyn Transport, marker: &[u8], deadline: Instant) -> io::Result<()> { let mut pending = Vec::new(); diff --git a/nvx-rs/public-api/linux.txt b/nvx-rs/public-api/linux.txt index e9fa8fdf9..e4a8135f9 100644 --- a/nvx-rs/public-api/linux.txt +++ b/nvx-rs/public-api/linux.txt @@ -2486,6 +2486,7 @@ variant nvx::host::restore::Attribution::Snapshot variant nvx::host::restore::Attribution::Unknown struct nvx::host::restore::ReadyListener method nvx::host::restore::ReadyListener::into_std +method nvx::host::restore::ReadyListener::receive enum nvx::host::restore::RestoreAction variant nvx::host::restore::RestoreAction::AwaitReadyEvent variant nvx::host::restore::RestoreAction::Connect @@ -3780,6 +3781,7 @@ struct nvx::host::vm::Launched field nvx::host::vm::Launched::capability field nvx::host::vm::Launched::outputs field nvx::host::vm::Launched::process +field nvx::host::vm::Launched::ready struct nvx::host::vm::Prepared method nvx::host::vm::Prepared::activation_record method nvx::host::vm::Prepared::arguments @@ -3819,6 +3821,7 @@ field nvx::host::vm::VmmCommand::inherit field nvx::host::vm::VmmCommand::log field nvx::host::vm::VmmCommand::outputs field nvx::host::vm::VmmCommand::program +field nvx::host::vm::VmmCommand::ready field nvx::host::vm::VmmCommand::working_dir fn nvx::host::vm::create fn nvx::host::vm::create_private_dir diff --git a/nvx-rs/public-api/windows.txt b/nvx-rs/public-api/windows.txt index 655540281..ad44f0575 100644 --- a/nvx-rs/public-api/windows.txt +++ b/nvx-rs/public-api/windows.txt @@ -1072,6 +1072,7 @@ variant nvx::host::restore::Attribution::Snapshot variant nvx::host::restore::Attribution::Unknown struct nvx::host::restore::ReadyListener method nvx::host::restore::ReadyListener::into_std +method nvx::host::restore::ReadyListener::receive enum nvx::host::restore::RestoreAction variant nvx::host::restore::RestoreAction::AwaitReadyEvent variant nvx::host::restore::RestoreAction::Connect @@ -2365,6 +2366,7 @@ struct nvx::host::vm::Launched field nvx::host::vm::Launched::capability field nvx::host::vm::Launched::outputs field nvx::host::vm::Launched::process +field nvx::host::vm::Launched::ready struct nvx::host::vm::Prepared method nvx::host::vm::Prepared::activation_record method nvx::host::vm::Prepared::arguments @@ -2404,6 +2406,7 @@ field nvx::host::vm::VmmCommand::inherit field nvx::host::vm::VmmCommand::log field nvx::host::vm::VmmCommand::outputs field nvx::host::vm::VmmCommand::program +field nvx::host::vm::VmmCommand::ready field nvx::host::vm::VmmCommand::working_dir fn nvx::host::vm::create fn nvx::host::vm::create_private_dir diff --git a/nvx-rs/src/host/config/mod.rs b/nvx-rs/src/host/config/mod.rs index bc92eaa28..f7152cbde 100644 --- a/nvx-rs/src/host/config/mod.rs +++ b/nvx-rs/src/host/config/mod.rs @@ -583,31 +583,7 @@ impl VmConfig { ))); } } - let console = match (self.console, console_endpoint) { - (ConsoleAttachment::Listener, Some(console_endpoint)) => { - check_endpoint(console_endpoint, "console_endpoint")?; - if console_endpoint == endpoint - || (runtime.hypervisor == Hypervisor::Whp - && console_endpoint.eq_ignore_ascii_case(endpoint)) - { - return Err(Error::unsupported( - "boot and control console listeners require distinct endpoints", - )); - } - format!("listen={console_endpoint}") - } - (ConsoleAttachment::Disconnected, None) => "none".to_owned(), - (ConsoleAttachment::Listener, None) => { - return Err(Error::unsupported( - "a console listener needs console_endpoint", - )); - } - (ConsoleAttachment::Disconnected, Some(_)) => { - return Err(Error::unsupported( - "a disconnected console takes no console_endpoint", - )); - } - }; + let console = self.console_backend(runtime, endpoint, console_endpoint)?; let mut arguments = self.device_arguments()?; let capture = self.boot.as_ref().and_then(|boot| boot.capture.as_ref()); arguments.extend(boot_arguments( @@ -621,34 +597,76 @@ impl VmConfig { arguments.extend(capture.openvmm_arguments()); } arguments.extend(console_arguments(&console, endpoint, report)); - if let Some(profile) = &runtime.cpu_profile { - if profile.is_empty() { - return Err(Error::unsupported("Runtime.cpu_profile must not be empty")); + arguments.extend(runtime_arguments(runtime)?); + check_argument_nuls(&arguments)?; + fit_process_command_line(runtime.openvmm.as_os_str(), &arguments, 0)?; + Ok(arguments) + } + + /// Renders the console attachment shared by cold boots and restores. + pub(crate) fn console_backend( + &self, + runtime: &Runtime, + endpoint: &str, + console_endpoint: Option<&str>, + ) -> Result { + match (self.console, console_endpoint) { + (ConsoleAttachment::Listener, Some(console_endpoint)) => { + check_endpoint(console_endpoint, "console_endpoint")?; + if console_endpoint == endpoint + || (runtime.hypervisor == Hypervisor::Whp + && console_endpoint.eq_ignore_ascii_case(endpoint)) + { + return Err(Error::unsupported( + "boot and control console listeners require distinct endpoints", + )); + } + Ok(format!("listen={console_endpoint}")) } - arguments.extend(["--cpu-profile".into(), profile.as_str().into()]); + (ConsoleAttachment::Disconnected, None) => Ok("none".to_owned()), + (ConsoleAttachment::Listener, None) => Err(Error::unsupported( + "a console listener needs console_endpoint", + )), + (ConsoleAttachment::Disconnected, Some(_)) => Err(Error::unsupported( + "a disconnected console takes no console_endpoint", + )), } - let shutdown = match runtime.options.guest_shutdown { - GuestPowerAction::Exit => "exit", - GuestPowerAction::Halt => "halt", - GuestPowerAction::Reset => "reset", - }; - arguments.extend(["--guest-shutdown-action".into(), shutdown.into()]); - if let Some(timeout) = runtime.options.control_auth_timeout { - if timeout < Duration::from_millis(1) - || timeout > Duration::from_secs(60) - || timeout.subsec_nanos() % 1_000_000 != 0 - { + } + + /// Renders checked devices at snapshot capacity, replacing only the captured NIC's policy. + pub(crate) fn restore_device_arguments( + &self, + capacity: u32, + nic: Option, + ) -> Result> { + self.check()?; + if self.boot.is_some() { + return Err(Error::unsupported("a snapshot restore does not take Boot")); + } + let mut config = self.clone(); + config.processors = capacity; + match (config.network.as_mut(), nic) { + (Some(network), Some(nic)) => network.guest_network = Some(nic.guest_network), + (None, None) => {} + _ => { return Err(Error::unsupported( - "LaunchOptions.control_auth_timeout must be whole milliseconds from 1 to 60000", + "snapshot NIC presence differs from the requested VM's", )); } - arguments.extend([ - "--microvm-control-auth-timeout-ms".into(), - timeout.as_millis().to_string().into(), - ]); } - check_argument_nuls(&arguments)?; - fit_process_command_line(runtime.openvmm.as_os_str(), &arguments, 0)?; + let mut arguments = config.device_arguments()?; + if let Some(at) = arguments.iter().position(|argument| argument == "--net") { + arguments.splice(at..at + 2, ["--restore-network-policy".into()]); + } + if matches!( + self.root, + RootConfig::Layered(LayeredRoot { + upper: UpperConfig::Ram { .. }, + .. + }) + ) { + arguments.push("--restore-absent-scratch".into()); + } Ok(arguments) } @@ -1188,8 +1206,40 @@ pub(crate) fn console_arguments(console: &str, endpoint: &str, report: &Path) -> .into() } +/// Runtime options common to cold boots and restores. +pub(crate) fn runtime_arguments(runtime: &Runtime) -> Result> { + let mut arguments = Vec::new(); + if let Some(profile) = &runtime.cpu_profile { + if profile.is_empty() { + return Err(Error::unsupported("Runtime.cpu_profile must not be empty")); + } + arguments.extend(["--cpu-profile".into(), profile.as_str().into()]); + } + let shutdown = match runtime.options.guest_shutdown { + GuestPowerAction::Exit => "exit", + GuestPowerAction::Halt => "halt", + GuestPowerAction::Reset => "reset", + }; + arguments.extend(["--guest-shutdown-action".into(), shutdown.into()]); + if let Some(timeout) = runtime.options.control_auth_timeout { + if timeout < Duration::from_millis(1) + || timeout > Duration::from_secs(60) + || timeout.subsec_nanos() % 1_000_000 != 0 + { + return Err(Error::unsupported( + "LaunchOptions.control_auth_timeout must be whole milliseconds from 1 to 60000", + )); + } + arguments.extend([ + "--microvm-control-auth-timeout-ms".into(), + timeout.as_millis().to_string().into(), + ]); + } + Ok(arguments) +} + /// Refuses caller-selected endpoint names that OpenVMM's serial options cannot hold literally. -fn check_endpoint(endpoint: &str, name: &str) -> Result<()> { +pub(crate) fn check_endpoint(endpoint: &str, name: &str) -> Result<()> { if endpoint.is_empty() || endpoint.contains(['\0', ',', '=']) { return Err(Error::unsupported(format!( "{name} must name a nonempty endpoint without a NUL, comma, or equals sign", @@ -1199,7 +1249,7 @@ fn check_endpoint(endpoint: &str, name: &str) -> Result<()> { } /// Checks OpenVMM arguments for NULs that cannot be passed to a process. -fn check_argument_nuls(arguments: &[OsString]) -> Result<()> { +pub(crate) fn check_argument_nuls(arguments: &[OsString]) -> Result<()> { if arguments .iter() .any(|argument| argument.to_string_lossy().contains('\0')) diff --git a/nvx-rs/src/host/mod.rs b/nvx-rs/src/host/mod.rs index de25eda26..a41a2dcbe 100644 --- a/nvx-rs/src/host/mod.rs +++ b/nvx-rs/src/host/mod.rs @@ -50,9 +50,10 @@ //! [`PreparedLaunch`](sandbox::PreparedLaunch) hands the VMM the files that those checks //! opened, which it inherits through the [`supervisor`]'s allowlist. //! - [`vm`] holds a caller-owned VM directory, private to the current user, the fixed names of -//! its files and endpoints, and a checked cold [`Prepared`](vm::Prepared) launch with its -//! opened log and control credential, which the library never writes. The opaque -//! [`Snapshot`](snapshot::Snapshot) input of a future restore is refused by cold creation. +//! its files and endpoints, and a checked cold, capture or restore [`Prepared`](vm::Prepared) +//! launch with its opened log and control credential, which the library never writes. A +//! restore from a [`Snapshot`](snapshot::Snapshot) hard-links the snapshot's OpenVMM files into +//! the VM directory and binds its restore-ready listener before any spawn. //! - [`restore`] orders caller-driven activation to Ready without I/O or a clock, and listens for //! OpenVMM's restore-ready event from the held VM process. //! - `cli` (feature `cli`) is the `nvx-rs` command-line interface, which runs microVMs and diff --git a/nvx-rs/src/host/restore.rs b/nvx-rs/src/host/restore.rs index 244509d94..b55c118a1 100644 --- a/nvx-rs/src/host/restore.rs +++ b/nvx-rs/src/host/restore.rs @@ -19,7 +19,7 @@ use std::{fmt, io}; use super::platform; use super::process::{Process, ProcessRef}; use super::report::Report; -use super::vm::VmDir; +use super::vm_directory::VmDir; use crate::activation::{ActivationRecord, ReadyActivation}; use crate::protocol::restore_ready::parse_ready_event; use crate::protocol::status; @@ -521,10 +521,6 @@ impl ReadyListener { /// Replaces a previous Linux endpoint. Fails with [`ErrorKind::Io`](crate::ErrorKind::Io) /// if binding fails or the directory is no longer private, and with /// [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) on unsupported hosts. - #[cfg_attr( - not(all(test, any(target_os = "linux", windows))), - expect(dead_code, reason = "SL2.16 binds a restore's endpoint (#1103)") - )] pub(crate) fn bind(directory: &VmDir) -> Result { let directory = VmDir::open(directory.path())?; let endpoint = directory.restore_ready_endpoint()?; @@ -540,10 +536,6 @@ impl ReadyListener { } /// Returns the endpoint for OpenVMM's `--restore-ready-path`. - #[cfg_attr( - not(all(test, any(target_os = "linux", windows))), - expect(dead_code, reason = "SL2.16 renders the restore endpoint (#1103)") - )] pub(crate) fn endpoint(&self) -> &str { &self.endpoint } @@ -558,14 +550,7 @@ impl ReadyListener { /// Linux authenticates the original socket peer through `SO_PEERPIDFD` (Linux 6.5 or a /// backport), failing with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) /// if the host cannot provide it, rather than falling back to a numeric PID. - #[cfg_attr( - not(all(test, any(target_os = "linux", windows))), - expect( - dead_code, - reason = "SL4.3's blocking driver receives the restore-ready event" - ) - )] - pub(crate) fn receive( + pub fn receive( &mut self, process: &Process, received: &mut Vec, diff --git a/nvx-rs/src/host/snapshot.rs b/nvx-rs/src/host/snapshot.rs index 6f65cefbf..694539b4a 100644 --- a/nvx-rs/src/host/snapshot.rs +++ b/nvx-rs/src/host/snapshot.rs @@ -99,17 +99,14 @@ pub struct CaptureContext { pub(crate) cpu_profile: Option, } -/// A checked snapshot directory and its verified metadata, for a future VM restore. +/// A checked snapshot directory and its verified metadata, which +/// [`vm::create`](crate::host::vm::create) restores a VM from. /// /// The caller must keep the directory and its entries private and immutable while using this /// value. Opening checks them once; it neither watches them nor holds their files open. #[derive(Debug)] pub struct Snapshot { /// The absolute directory path, with its components preserved rather than its links resolved. - #[cfg_attr( - not(test), - expect(dead_code, reason = "SL2.15 uses the snapshot directory (#1103)") - )] pub(crate) dir: PathBuf, /// The bounded canonical key checked against the manifest. key: SnapshotKey, @@ -190,6 +187,31 @@ impl Snapshot { self.nic } + /// Links the fixed snapshot inventory into a newly created restore view. + /// + /// The caller owns `destination` and records each successful link for rollback. Hard-link + /// errors, including a different filesystem, retain their source; nothing is copied. + pub(crate) fn link_restore_view( + &self, + destination: &Path, + created: &mut Vec, + ) -> Result<()> { + for name in SNAPSHOT_ARTIFACTS { + let source = self.dir.join(OPENVMM_DIR).join(name); + let path = destination.join(name); + fs::hard_link(&source, &path).map_err(|error| { + Error::io(format!( + "cannot hard-link snapshot artifact {} into restore view {}", + source.display(), + path.display() + )) + .with_source(error) + })?; + created.push(path); + } + Ok(()) + } + /// Checks whether this snapshot can restore `config` under `runtime` at `now`. /// /// Requires a layered root, processors from the effective boot count to capacity in diff --git a/nvx-rs/src/host/snapshot/tests.rs b/nvx-rs/src/host/snapshot/tests.rs index 23b8352f3..c99645c03 100644 --- a/nvx-rs/src/host/snapshot/tests.rs +++ b/nvx-rs/src/host/snapshot/tests.rs @@ -1428,15 +1428,50 @@ fn snapshot_description_needs_no_data_read_access() { assert_eq!(Snapshot::open(&fixture.dir).unwrap().key(), &fixture.key); } -struct RestoreFixture { +pub(in crate::host) struct RestoreFixture { snapshot: SnapshotFixture, - config: VmConfig, - runtime: Runtime, + pub(in crate::host) config: VmConfig, + pub(in crate::host) runtime: Runtime, now: super::HostTime, } impl RestoreFixture { - fn new(nic: bool) -> Self { + #[cfg(any(target_os = "linux", windows))] + pub(in crate::host) fn current(nic: bool) -> Self { + let mut fixture = Self::new(nic); + fixture.retime(super::HostTime::now().unwrap()); + fixture + } + + #[cfg(any(target_os = "linux", windows))] + pub(in crate::host) fn retime(&mut self, now: super::HostTime) { + self.now = now; + let time = self.snapshot.spec.time(); + time.capture_utc_ns = now.utc_ns; + time.capture_monotonic_ns = now.monotonic_ns; + time.host_clock = now.clock.as_str().to_owned(); + time.host_id = now.host_id.to_vec(); + time.boot_id = now.boot_id.as_bytes().to_vec(); + time.cut_to_anchor_ns = Some(0); + self.write_metadata(); + } + + pub(in crate::host) fn all_processors_online(&mut self) { + self.snapshot.spec.contract().boot_online_vp_count = 0; + self.config.processors = self.snapshot.spec.vp_count; + self.write_metadata(); + } + + pub(in crate::host) fn console_listener(&mut self) { + self.config.console = ConsoleAttachment::Listener; + let console = &mut self.snapshot.spec.contract().attachments[0]; + console.reconnect_policy = "recreate-listener".to_owned(); + console.identity_kind = "unix-socket".to_owned(); + console.identity = b"/captured/boot-console".to_vec(); + self.write_metadata(); + } + + pub(in crate::host) fn new(nic: bool) -> Self { let mut snapshot = SnapshotFixture::new(); snapshot.spec = if nic { let mut spec = ManifestSpec::with_network(); @@ -1555,7 +1590,7 @@ impl RestoreFixture { .unwrap(); } - fn opened(&self) -> Snapshot { + pub(in crate::host) fn opened(&self) -> Snapshot { Snapshot::open(&self.snapshot.dir).unwrap() } @@ -1569,6 +1604,79 @@ impl RestoreFixture { } } +#[test] +fn a_restore_view_hard_links_only_the_fixed_openvmm_inventory() { + let fixture = SnapshotFixture::new(); + let snapshot = Snapshot::open(&fixture.dir).unwrap(); + let destination = fixture.root.path().join("view"); + fs::create_dir(&destination).unwrap(); + let mut created = Vec::new(); + snapshot + .link_restore_view(&destination, &mut created) + .unwrap(); + assert_eq!(created.len(), super::SNAPSHOT_ARTIFACTS.len()); + assert_eq!(fs::read_dir(&destination).unwrap().count(), created.len()); + for path in &created { + let source = fixture + .dir + .join(OPENVMM_DIR) + .join(path.file_name().unwrap()); + assert_eq!( + fs::metadata(path).unwrap().len(), + fs::metadata(&source).unwrap().len() + ); + assert_eq!( + crate::host::platform::file_identity(path).unwrap(), + crate::host::platform::file_identity(&source).unwrap() + ); + #[cfg(target_os = "linux")] + { + use std::os::unix::fs::MetadataExt; + let original = fs::metadata(&source).unwrap(); + let linked = fs::metadata(path).unwrap(); + assert_eq!( + (original.dev(), original.ino()), + (linked.dev(), linked.ino()) + ); + assert_eq!(original.nlink(), 2); + } + } + assert!(!destination.join(KEY_FILE).exists()); + assert!(!destination.join(DESCRIPTION_FILE).exists()); +} + +#[test] +fn partial_restore_links_report_the_original_error_and_only_created_entries() { + use std::error::Error as _; + + let fixture = SnapshotFixture::new(); + let snapshot = Snapshot::open(&fixture.dir).unwrap(); + let destination = fixture.root.path().join("view"); + fs::create_dir(&destination).unwrap(); + let existing = destination.join("memory.bin"); + fs::write(&existing, b"caller-owned").unwrap(); + let mut created = Vec::new(); + let error = snapshot + .link_restore_view(&destination, &mut created) + .unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io); + assert_eq!( + error + .source() + .unwrap() + .downcast_ref::() + .unwrap() + .kind(), + std::io::ErrorKind::AlreadyExists + ); + assert_eq!(created, [destination.join(MANIFEST_NAME)]); + for path in created { + fs::remove_file(path).unwrap(); + } + assert_eq!(fs::read(&existing).unwrap(), b"caller-owned"); + assert_eq!(fs::read_dir(&destination).unwrap().count(), 1); +} + #[test] fn snapshot_accepts_matching_real_disk_geometry_and_every_resource_boundary() { for nic in [false, true] { diff --git a/nvx-rs/src/host/vm.rs b/nvx-rs/src/host/vm.rs index b148774fc..080c301ba 100644 --- a/nvx-rs/src/host/vm.rs +++ b/nvx-rs/src/host/vm.rs @@ -26,18 +26,20 @@ use crate::activation::{ }; use crate::host::config::{ BlockGeometry, CaptureConfig, ConsoleAttachment, DiskCapture, NetworkConfig, NicConfig, - RootConfig, UpperConfig, VmConfig, fit_process_command_line, reject_disk_path, + RootConfig, UpperConfig, VmConfig, check_argument_nuls, check_endpoint, console_arguments, + fit_process_command_line, reject_disk_path, runtime_arguments, }; use crate::host::filesystem; use crate::host::network; use crate::host::paths::absolute; use crate::host::platform; use crate::host::report::{self, Report}; +use crate::host::restore::ReadyListener; use crate::host::runtime::{Runtime, RuntimeIdentity}; -use crate::host::snapshot::{CaptureContext, Snapshot}; +use crate::host::snapshot::{CaptureContext, HostTime, Snapshot}; use crate::host::spawn; use crate::host::transport::{self, HostConnection}; -use crate::host::vm_directory::{CONSOLE, CONTROL, io_error}; +use crate::host::vm_directory::{CONSOLE, CONTROL, RESTORE_READY, io_error}; pub use crate::host::vm_directory::{VmDir, VmOutputs, create_private_dir}; use crate::host::{ platform::inherit::Inherited, @@ -247,7 +249,7 @@ impl std::os::windows::io::AsHandle for InheritedHandle { } } -/// A complete cold launch handed to the caller's launcher, with all its resources owned. +/// A complete launch handed to the caller's launcher, with all its resources owned. /// /// The launcher supplies the capability on stdin through a pipe closed to writers before /// spawning, and uses the opened log for stdout and stderr. It allows only the standard streams @@ -259,8 +261,9 @@ impl std::os::windows::io::AsHandle for InheritedHandle { /// refuse a taken or untrusted job name, and honor `breakaway_from_job`. Windows callers /// coordinate their other spawns with the inheritance window. If setup fails after spawning, /// terminate and reap only that child before returning an error. The caller journals recovery -/// data and keeps the VM directory and runtime inputs stable; this value starts no process or -/// listener. +/// data and keeps the VM directory and runtime inputs stable; this value starts no process. +/// A restore retains its already-bound ready listener in `ready`, outside the child +/// inheritance allowlist. #[derive(Debug)] pub struct VmmCommand { /// The checked OpenVMM executable. @@ -279,12 +282,15 @@ pub struct VmmCommand { pub log: File, /// Whether Windows leaves the caller's job, which must permit breakaway. pub breakaway_from_job: bool, + /// The same restore-ready listener bound during creation, absent for a cold launch. + pub ready: Option, } -/// A checked cold launch, with its capability, chosen outputs, and already opened log. +/// A checked launch, with its capability, chosen outputs, and already opened log. /// -/// Creation starts no process and binds no listener. The caller keeps the VM directory and -/// runtime inputs stable, and may journal the capability and outputs before a later spawn. +/// Creation starts no process; a restore binds its ready listener. The caller keeps the VM +/// directory and runtime inputs stable, and may journal the capability and outputs before a +/// later spawn. /// Dropping a successful preparation closes its files but leaves its output paths to the caller. #[derive(Debug)] pub struct Prepared { @@ -312,6 +318,8 @@ pub struct Prepared { inherit: Vec, /// The checked context of a capture, absent for an ordinary cold launch. capture_context: Option, + /// The restore's listener, retained outside the child's inheritance allowlist. + ready: Option, } impl Prepared { @@ -453,10 +461,11 @@ impl Prepared { /// Transfers the complete checked launch to the caller's launcher. /// /// Moves the same opened log and inherited resources, without cloning, reopening, rendering - /// arguments again or rediscovering paths. Starts no process or listener. The caller follows - /// [`VmmCommand`]'s launch contract and may retain its capability and outputs for recovery. - /// Read [`nic`](Self::nic) or [`capture_context`](Self::capture_context), or build an - /// [`activation_record`](Self::activation_record), before consuming the preparation. + /// arguments again or rediscovering paths. Moves a restore's same bound listener, without + /// putting it in the child's inheritance allowlist. Starts no process or new listener. The + /// caller follows [`VmmCommand`]'s launch contract and may retain its capability and outputs + /// for recovery. Read [`nic`](Self::nic) or [`capture_context`](Self::capture_context), or + /// build an [`activation_record`](Self::activation_record), before consuming the preparation. pub fn into_command(self) -> VmmCommand { VmmCommand { program: self.program, @@ -467,13 +476,18 @@ impl Prepared { inherit: self.inherit, log: self.log, breakaway_from_job: self.windows_breakaway, + ready: self.ready, } } } -/// Creates an inspectable cold launch in `dir`, without starting a process or binding endpoints. +/// Creates an inspectable cold launch or snapshot restore in `dir`, without starting a process. /// -/// A cold launch needs [`VmConfig::boot`]. Restoring `from` a [`Snapshot`] is not supported yet. +/// A cold launch needs [`VmConfig::boot`]; a restore refuses it and calls [`Snapshot::accepts`] +/// with this host's actual time. A restore hard-links only the snapshot's OpenVMM files into the +/// new, fixed `openvmm/` view, requiring the same filesystem, and binds its ready listener before +/// any spawn. Both launch paths retain that same listener. Nothing is copied or restored from a +/// snapshot-parent fallback. The successful view and all output paths remain caller-owned. /// The caller keeps the directory and runtime inputs stable during creation and a later spawn. /// Caller-relative runtime and block-image paths become absolute during creation, without /// resolving links or `..` components; output paths come from `dir`. @@ -487,7 +501,17 @@ impl Prepared { /// checked capture context. OpenVMM captures to `openvmm/` under this VM directory, with a /// configured NIC under deny-all. No key, description or context file is written. /// -/// Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for a restore, a missing +/// On Linux the caller holds its per-snapshot lock from this call, including adding the hard +/// links, through a peer-verified restore-ready event or the VM's verified exit, and around +/// removing `openvmm/`. NVX acquires no lock; Windows needs none. A restore whose caller dies +/// before OpenVMM connects the restore-ready endpoint cannot complete: OpenVMM checks the +/// snapshot's files before that one connection, which it does not retry, and the dead caller's +/// endpoint no longer listens. After the connection OpenVMM no longer checks those files, so +/// the caller needs no durable record of its in-flight restores; it keeps the VM's process as it +/// does for any launch. +/// +/// Fails as [`Snapshot::accepts`] does, or with +/// [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported) for Boot on a restore, a missing /// boot, a refused configuration, unrepresentable Linux socket names, or a complete Windows /// command line beyond the portable limit. /// The final check includes the actual runtime and output paths, with no reserve, on every host. @@ -499,12 +523,14 @@ pub fn create( from: Option<&Snapshot>, dir: &VmDir, ) -> Result { - if from.is_some() { - return Err(Error::unsupported( - "creating a VM from a Snapshot is not supported yet", - )); - } - if config.boot.is_none() { + if let Some(snapshot) = from { + if config.boot.is_some() { + return Err(Error::unsupported( + "restoring a Snapshot does not take Boot", + )); + } + snapshot.accepts(config, runtime, &HostTime::now()?)?; + } else if config.boot.is_none() { return Err(Error::unsupported("creating a cold VM needs Boot")); } let capture = config.boot.as_ref().and_then(|boot| boot.capture.as_ref()); @@ -515,8 +541,16 @@ pub fn create( VmDir::open(dir.path())?; platform::check_working_dir(dir.path())?; let outputs = dir.outputs(config.console == ConsoleAttachment::Listener)?; + let ready_endpoint = if from.is_some() && !cfg!(windows) { + Some(dir.restore_ready_endpoint()?) + } else { + None + }; if !cfg!(windows) { - for endpoint in std::iter::once(&outputs.endpoint).chain(outputs.console.as_ref()) { + for endpoint in std::iter::once(&outputs.endpoint) + .chain(outputs.console.as_ref()) + .chain(ready_endpoint.as_ref()) + { if endpoint.len() >= 108 || endpoint.contains([',', '=']) { return Err(Error::unsupported(format!( "the VM socket {endpoint} must be shorter than 108 bytes and hold no ',' or '='", @@ -525,7 +559,10 @@ pub fn create( } let canonical = filesystem::canonicalize(dir.path()) .map_err(|error| io_error("resolve", dir.path(), error))?; - for name in std::iter::once(CONTROL).chain(outputs.console.as_ref().map(|_| CONSOLE)) { + for name in std::iter::once(CONTROL) + .chain(outputs.console.as_ref().map(|_| CONSOLE)) + .chain(ready_endpoint.as_ref().map(|_| RESTORE_READY)) + { let endpoint = canonical.join(name); if endpoint.to_str().is_none() || endpoint.as_os_str().len() >= 108 { return Err(Error::unsupported(format!( @@ -569,6 +606,9 @@ pub fn create( .with_source(error) })?; let mut created = Vec::new(); + let view = dir.path().join(OPENVMM); + let mut created_view = false; + let mut ready = None; let checked = (|| { #[cfg(unix)] { @@ -591,6 +631,31 @@ pub fn create( } } let capability = Capability(crate::host::instance::generate_capability()?); + if let Some(snapshot) = from { + unused_output(&view)?; + if let Some(endpoint) = &ready_endpoint { + unused_output(Path::new(endpoint))?; + } + // The private directory stays caller-stable; setup can fail after creating it. + created_view = true; + create_private_dir(&view)?; + snapshot.link_restore_view(&view, &mut created)?; + // Binding can leave its new socket if later listener setup fails. + if let Some(endpoint) = &ready_endpoint { + created.push(PathBuf::from(endpoint)); + } + let listener = ReadyListener::bind(dir)?; + let arguments = restore_arguments( + &runtime, + &config, + snapshot, + dir, + &outputs, + listener.endpoint(), + )?; + ready = Some(listener); + return Ok((capability, arguments, snapshot.nic(), None)); + } let mut arguments = config.openvmm_arguments( &runtime, &outputs.endpoint, @@ -659,9 +724,11 @@ pub fn create( nic, inherit: Vec::new(), capture_context, + ready, }), Err(mut error) => { drop(log); + drop(ready); for path in created.iter().rev() { if let Err(cleanup) = fs::remove_file(path) && cleanup.kind() != io::ErrorKind::NotFound @@ -669,13 +736,26 @@ pub fn create( error = Error::new( error.kind(), format!( - "{error}; cannot remove created capture file {}: {cleanup}", + "{error}; cannot remove created VM entry {}: {cleanup}", path.display() ), ) .with_source(error); } } + if created_view + && let Err(cleanup) = fs::remove_dir(&view) + && cleanup.kind() != io::ErrorKind::NotFound + { + error = Error::new( + error.kind(), + format!( + "{error}; cannot remove created restore view {}: {cleanup}", + view.display() + ), + ) + .with_source(error); + } match fs::remove_file(&outputs.log) { Ok(()) => Err(error), Err(cleanup) if cleanup.kind() == io::ErrorKind::NotFound => Err(error), @@ -692,6 +772,73 @@ pub fn create( } } +/// Renders a restore without cold-boot inputs or a store/lease dependency. +fn restore_arguments( + runtime: &Runtime, + config: &VmConfig, + snapshot: &Snapshot, + dir: &VmDir, + outputs: &VmOutputs, + ready: &str, +) -> Result> { + check_endpoint(&outputs.endpoint, "endpoint")?; + check_endpoint(ready, "restore-ready endpoint")?; + for (name, path) in [ + ("Runtime.openvmm", runtime.openvmm.as_path()), + ("report", outputs.outcome.as_path()), + ] { + if path.as_os_str().is_empty() || path.as_os_str().to_string_lossy().contains('\0') { + return Err(Error::unsupported(format!( + "{name} must name a path without a NUL" + ))); + } + } + for endpoint in std::iter::once(&outputs.endpoint).chain(outputs.console.as_ref()) { + if endpoint == ready + || (runtime.hypervisor == crate::Hypervisor::Whp + && endpoint.eq_ignore_ascii_case(ready)) + { + return Err(Error::unsupported( + "restore-ready and console endpoints must be distinct", + )); + } + } + let console = config.console_backend(runtime, &outputs.endpoint, outputs.console.as_deref())?; + let key = snapshot.key(); + let mut arguments = config.restore_device_arguments(key.processors(), snapshot.nic())?; + arguments.extend([ + "--single-process".into(), + "--hypervisor".into(), + key.hypervisor().as_str().into(), + "--restore-snapshot".into(), + dir.path().join(OPENVMM).into_os_string(), + ]); + if key.boot_processors() != 0 { + arguments.extend([ + "--restore-processors".into(), + config.processors.to_string().into(), + ]); + } + arguments.extend([ + "--restore-memory".into(), + format!("{}M", config.memory_mib).into(), + "--restore-ready-path".into(), + ready.into(), + ]); + arguments.extend(console_arguments( + &console, + &outputs.endpoint, + &outputs.outcome, + )); + arguments.extend(runtime_arguments(&Runtime { + cpu_profile: Some(key.cpu_profile()), + ..runtime.clone() + })?); + check_argument_nuls(&arguments)?; + fit_process_command_line(runtime.openvmm.as_os_str(), &arguments, 0)?; + Ok(arguments) +} + /// A detached VM owned by its caller. /// /// Dropping this value closes the process handle, not the VM. Its outputs remain caller-owned, @@ -704,6 +851,8 @@ pub struct Launched { pub outputs: VmOutputs, /// The control credential, never written to disk. pub capability: Capability, + /// The preparation's same restore-ready listener, absent for a cold launch. + pub ready: Option, /// Retains only a foreground caller's child for status and control attachment. #[cfg(feature = "cli")] child: ForegroundChild, @@ -742,7 +891,7 @@ impl Launched { } } -/// Starts a prepared cold launch and returns its verified, detached process. +/// Starts a prepared launch and returns its verified, detached process and any ready listener. /// /// The capability reaches OpenVMM only on standard input, followed by end-of-file. The opened /// log becomes standard output and error without reopening its path. Only the preparation's @@ -861,6 +1010,7 @@ fn spawn_verified_with_cleanup( log, arguments, inherit, + ready, .. } = prepared; let command = spawn::VmmCommand { @@ -915,6 +1065,7 @@ fn spawn_verified_with_cleanup( process, outputs, capability, + ready, #[cfg(feature = "cli")] child: ForegroundChild(child), }) diff --git a/nvx-rs/src/host/vm/tests.rs b/nvx-rs/src/host/vm/tests.rs index 7eecb2efe..53b42f697 100644 --- a/nvx-rs/src/host/vm/tests.rs +++ b/nvx-rs/src/host/vm/tests.rs @@ -824,6 +824,7 @@ fn cold_creation_is_inspectable_without_a_process_listener_or_capability_file() }; let prepared = create(&runtime, &config, None, &vm).unwrap(); assert!(prepared.capture_context().is_none()); + assert!(prepared.ready.is_none()); let outputs = prepared.outputs(); assert_eq!(outputs.log, vm.path().join("openvmm.log")); assert_eq!( @@ -851,7 +852,7 @@ fn cold_creation_is_inspectable_without_a_process_listener_or_capability_file() } #[test] -fn a_missing_boot_and_a_snapshot_are_refused_before_any_output_is_created() { +fn a_missing_boot_and_boot_with_a_snapshot_are_refused_before_any_output_is_created() { let (_directory, vm) = vm_dir(); let runtime = cold_runtime(); let mut config = cold_config(); @@ -869,6 +870,432 @@ fn a_missing_boot_and_a_snapshot_are_refused_before_any_output_is_created() { } } +#[cfg(any(target_os = "linux", windows))] +#[test] +fn snapshot_creation_checks_eligibility_and_real_host_time_before_creating_outputs() { + use crate::host::snapshot::tests::RestoreFixture; + + for refusal in [ + "processors", + "memory", + "runtime", + "backend", + "profile", + "NIC", + "address", + "console", + "early", + "expired", + ] { + let mut fixture = RestoreFixture::current(true); + let mut kind = ErrorKind::Unsupported; + match refusal { + "processors" => fixture.config.processors = 8, + "memory" => fixture.config.memory_mib = 129, + "runtime" => { + fixture.runtime.identity.kernel = ArtifactIdentity::new("other-kernel").unwrap() + } + "backend" => fixture.runtime.hypervisor = Hypervisor::Mshv, + "profile" => fixture.runtime.cpu_profile = Some("different".to_owned()), + "NIC" => fixture.config.network = None, + "address" => { + fixture.config.network.as_mut().unwrap().guest_network = + Some(Ipv4Cidr::parse("10.43.0.2/24").unwrap()) + } + "console" => fixture.config.console = ConsoleAttachment::Listener, + "early" | "expired" => { + let mut time = HostTime::now().unwrap(); + time.host_id[0] ^= 1; + let outside = crate::host::snapshot::MAX_DOWNTIME_NS + 1_000_000_000; + time.utc_ns = if refusal == "early" { + time.utc_ns + outside + } else { + time.utc_ns - outside + }; + fixture.retime(time); + kind = ErrorKind::Unavailable; + } + _ => unreachable!(), + } + let snapshot = fixture.opened(); + let (_directory, vm) = vm_dir(); + let error = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap_err(); + assert_eq!(error.kind(), kind, "{refusal}: {error}"); + assert_eq!(fs::read_dir(vm.path()).unwrap().count(), 0, "{refusal}"); + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn snapshot_creation_owns_the_fixed_view_and_uses_captured_nic_with_requested_policy() { + use crate::host::snapshot::tests::RestoreFixture; + + let mut published = NetworkPolicy::new(Access::Allow); + published.ingress = Access::Allow; + for (policy, expected) in [ + ( + published, + Network { + dns: Some("10.42.0.1".parse().unwrap()), + publication: Publication::Published, + proxy: None, + }, + ), + ( + NetworkPolicy::new(Access::Deny), + Network { + dns: None, + publication: Publication::Unpublished, + proxy: None, + }, + ), + ( + NetworkPolicy::proxy_only(NetworkProxy::new(3128).with_scheme(ProxyScheme::Http)), + Network { + dns: None, + publication: Publication::Unpublished, + proxy: Some("http://10.42.0.1:3128".to_owned()), + }, + ), + ] { + let mut fixture = RestoreFixture::current(true); + fixture.config.network.as_mut().unwrap().policy = policy; + let snapshot = fixture.opened(); + let (_directory, vm) = vm_dir(); + let input = vm.path().join("caller-input"); + fs::write(&input, b"preserved").unwrap(); + let prepared = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap(); + assert_eq!(prepared.nic(), snapshot.nic()); + assert!(prepared.capture_context().is_none()); + assert!(prepared.ready.is_some()); + let record = prepared + .activation_record(TransactionId::from_bytes([0x31; 16])) + .unwrap(); + assert_eq!(record.configuration().network, Some(expected)); + assert!(prepared.inherit.is_empty()); + for name in crate::host::snapshot::SNAPSHOT_ARTIFACTS { + assert_eq!( + platform::file_identity(&snapshot.dir.join(OPENVMM).join(name)).unwrap(), + platform::file_identity(&vm.path().join(OPENVMM).join(name)).unwrap() + ); + } + drop(prepared); + assert_eq!(fs::read_dir(vm.path().join(OPENVMM)).unwrap().count(), 3); + assert_eq!(fs::read(&input).unwrap(), b"preserved"); + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn snapshot_creation_never_replaces_existing_outputs_views_or_listeners() { + use crate::host::snapshot::tests::RestoreFixture; + + let mut names = vec![ + crate::host::vm_directory::LOG, + crate::host::vm_directory::OUTCOME, + OPENVMM, + ]; + if !cfg!(windows) { + names.extend([CONTROL, RESTORE_READY]); + } + for name in names { + let fixture = RestoreFixture::current(false); + let snapshot = fixture.opened(); + let (_directory, vm) = vm_dir(); + let path = vm.path().join(name); + let preserved = if name == OPENVMM { + fs::create_dir(&path).unwrap(); + path.join("caller-file") + } else { + path + }; + fs::write(&preserved, b"existing").unwrap(); + let error = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{name}: {error}"); + assert_eq!(fs::read(&preserved).unwrap(), b"existing"); + assert_eq!(fs::read_dir(vm.path()).unwrap().count(), 1); + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn snapshot_creation_rolls_back_partial_links_and_a_render_failure_after_binding() { + use crate::host::snapshot::tests::RestoreFixture; + use std::error::Error as _; + + for partial_link in [false, true] { + let mut fixture = RestoreFixture::current(false); + let snapshot = fixture.opened(); + if partial_link { + fs::remove_file(snapshot.dir.join(OPENVMM).join("memory.bin")).unwrap(); + } else { + fixture.runtime.openvmm = PathBuf::from("x".repeat(PROCESS_COMMAND_LINE_LIMIT)); + } + let (_directory, vm) = vm_dir(); + let input = vm.path().join("caller-input"); + fs::write(&input, b"preserved").unwrap(); + let error = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap_err(); + if partial_link { + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + assert_eq!( + error + .source() + .unwrap() + .downcast_ref::() + .unwrap() + .kind(), + io::ErrorKind::NotFound + ); + } else { + assert_eq!(error.kind(), ErrorKind::Unsupported, "{error}"); + assert!( + error.to_string().contains("Windows process command line"), + "{error}" + ); + } + assert_eq!(fs::read(&input).unwrap(), b"preserved"); + assert_eq!(fs::read_dir(vm.path()).unwrap().count(), 1); + } +} + +#[cfg(target_os = "linux")] +#[test] +fn the_same_prebound_listener_keeps_its_queue_through_both_launch_paths_without_inheritance() { + use crate::host::snapshot::tests::RestoreFixture; + use std::os::fd::AsRawFd; + use std::os::unix::net::UnixStream; + + for external in [false, true] { + let fixture = RestoreFixture::current(false); + let snapshot = fixture.opened(); + let (_directory, vm) = vm_dir(); + let mut prepared = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap(); + let endpoint = vm.path().join(RESTORE_READY); + let original = platform::file_identity(&endpoint).unwrap(); + let queued = UnixStream::connect(&endpoint).unwrap(); + let mut running = None; + let listener = if external { + let command = prepared.into_command(); + assert!(command.inherit.is_empty()); + command.ready.unwrap().into_std().unwrap() + } else { + (prepared.program, prepared.arguments) = stand_in_command(); + running = Some(RunningVm(spawn(prepared).unwrap())); + running + .as_mut() + .unwrap() + .0 + .ready + .take() + .unwrap() + .into_std() + .unwrap() + }; + assert_eq!(platform::file_identity(&endpoint).unwrap(), original); + let (accepted, _) = listener + .accept() + .expect("the connection queued before handoff remains"); + assert_eq!( + listener.local_addr().unwrap().as_pathname(), + Some(endpoint.as_path()) + ); + if let Some(running) = &running { + let socket = fs::read_link(format!("/proc/self/fd/{}", listener.as_raw_fd())).unwrap(); + let descriptors = format!("/proc/{}/fd", running.0.process.reference().pid); + for entry in fs::read_dir(descriptors).unwrap() { + match fs::read_link(entry.unwrap().path()) { + Ok(inherited) => assert_ne!( + inherited, socket, + "the caller's ready listener reached the child" + ), + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + Err(error) => panic!("cannot inspect the owned child descriptor: {error}"), + } + } + } + drop((accepted, queued, running)); + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn snapshot_creation_keeps_captured_console_attachment_and_zero_boot_processor_rules() { + use crate::host::snapshot::tests::RestoreFixture; + + for listener in [false, true] { + let mut fixture = RestoreFixture::current(false); + fixture.all_processors_online(); + if listener { + fixture.console_listener(); + } + let snapshot = fixture.opened(); + let (_directory, vm) = vm_dir(); + let prepared = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap(); + assert!( + !prepared + .arguments() + .iter() + .any(|arg| arg == "--restore-processors") + ); + let console = prepared + .arguments() + .windows(2) + .find(|pair| pair[0] == "--virtio-console") + .unwrap(); + assert_eq!( + console[1], + if listener { + OsString::from(format!( + "listen={}", + prepared.outputs().console.as_ref().unwrap() + )) + } else { + OsString::from("none") + } + ); + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +#[ignore = "requires an explicitly approved second filesystem in NVX_VM_TEST_OTHER_FILESYSTEM"] +fn snapshot_creation_refuses_a_true_cross_filesystem_view_without_copying() { + use crate::host::snapshot::tests::RestoreFixture; + use std::error::Error as _; + + let other = std::env::var_os("NVX_VM_TEST_OTHER_FILESYSTEM") + .expect("an explicitly approved second filesystem is required, not a skipped assertion"); + let fixture = RestoreFixture::current(false); + let snapshot = fixture.opened(); + let directory = tempfile::tempdir_in(other).unwrap(); + let path = directory.path().join("vm"); + create_private_dir(&path).unwrap(); + let vm = VmDir::open(&path).unwrap(); + let input = vm.path().join("caller-input"); + fs::write(&input, b"preserved").unwrap(); + assert_ne!( + platform::file_identity(&input).unwrap().0, + platform::file_identity(&snapshot.dir.join(OPENVMM).join("memory.bin")) + .unwrap() + .0, + "the test must actually cross filesystems" + ); + let error = create(&fixture.runtime, &fixture.config, Some(&snapshot), &vm).unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io, "{error}"); + let source = error.source().unwrap().downcast_ref::().unwrap(); + eprintln!( + "strict cross-filesystem refusal: native={:?}; source_fs={}; destination_fs={}; {error}", + source.raw_os_error(), + platform::file_identity(&snapshot.dir.join(OPENVMM).join("memory.bin")) + .unwrap() + .0, + platform::file_identity(&input).unwrap().0, + ); + #[cfg(target_os = "linux")] + assert_eq!(source.raw_os_error(), Some(nix::libc::EXDEV)); + #[cfg(windows)] + assert_eq!( + source.raw_os_error(), + Some(windows_sys::Win32::Foundation::ERROR_NOT_SAME_DEVICE as i32) + ); + assert_eq!(fs::read(&input).unwrap(), b"preserved"); + assert_eq!(fs::read_dir(vm.path()).unwrap().count(), 1); +} + +#[test] +fn restore_arguments_use_snapshot_capacity_current_policy_and_no_cold_inputs() { + use crate::host::snapshot::tests::RestoreFixture; + + for upper_in_ram in [false, true] { + for policy in [ + NetworkPolicy::new(Access::Allow), + NetworkPolicy::new(Access::Deny), + NetworkPolicy::proxy_only(NetworkProxy::new(3128)), + ] { + let mut fixture = RestoreFixture::new(true); + fixture.config.network.as_mut().unwrap().policy = policy.clone(); + fixture.config.processors = 4; + fixture.config.memory_mib = 768; + if upper_in_ram { + let RootConfig::Layered(root) = &mut fixture.config.root else { + unreachable!() + }; + root.upper = UpperConfig::Ram { + max_memory_percent: 50, + }; + } + let snapshot = fixture.opened(); + let (_directory, vm) = vm_dir(); + let outputs = vm.outputs(false).unwrap(); + let ready = vm.restore_ready_endpoint().unwrap(); + let arguments = restore_arguments( + &fixture.runtime, + &fixture.config, + &snapshot, + &vm, + &outputs, + &ready, + ) + .unwrap(); + let value = |name| { + arguments + .windows(2) + .find(|pair| pair[0] == name) + .map(|pair| pair[1].clone()) + }; + assert_eq!(value("--processors"), Some("4".into())); + assert_eq!(value("--restore-processors"), Some("4".into())); + assert_eq!(value("--restore-memory"), Some("768M".into())); + assert_eq!( + value("--restore-snapshot"), + Some(vm.path().join(OPENVMM).into_os_string()) + ); + assert_eq!(value("--restore-ready-path"), Some(ready.into())); + assert_eq!(value("--virtio-console"), Some("none".into())); + assert_eq!( + value("--cpu-profile"), + Some(snapshot.key().cpu_profile().into()) + ); + assert_eq!( + arguments + .iter() + .filter(|arg| *arg == "--restore-network-policy") + .count(), + 1 + ); + assert_eq!( + arguments + .iter() + .any(|arg| arg == "--restore-absent-scratch"), + upper_in_ram + ); + assert_eq!( + arguments + .iter() + .filter(|arg| *arg == "--microvm-sandbox-block") + .count(), + if upper_in_ram { 1 } else { 2 } + ); + for forbidden in [ + "--net", + "--kernel", + "--initrd", + "--cmdline", + "--memory", + "--snapshot-destination", + ] { + assert!(!arguments.iter().any(|arg| arg == forbidden), "{forbidden}"); + } + let proxy = policy.proxy.is_some(); + assert_eq!( + value("--network-proxy"), + proxy.then(|| "10.42.0.1:3128".into()) + ); + assert_eq!(fs::read_dir(vm.path()).unwrap().count(), 0); + } + } +} + #[cfg(target_os = "linux")] #[test] fn cold_creation_refuses_listener_delimiters_before_creating_outputs() { @@ -1359,6 +1786,7 @@ fn spawn_returns_the_same_outputs_and_capability_with_a_verified_process() { let launched = RunningVm(spawn(prepared).unwrap()); assert_eq!(launched.0.outputs, outputs); assert_eq!(launched.0.capability.as_bytes(), &capability); + assert!(launched.0.ready.is_none()); let reference = launched.0.process.reference(); assert_ne!(reference.pid, std::process::id()); let Opened::Running(reopened) = Process::open(reference).unwrap() else { @@ -1573,6 +2001,7 @@ fn a_library_vm_outlives_the_caller_that_spawned_it_without_retaining_its_output process, outputs: VmDir::open(&vm).unwrap().outputs(false).unwrap(), capability: Capability([0; CAPABILITY_LEN]), + ready: None, #[cfg(feature = "cli")] child: ForegroundChild(None), }); diff --git a/scripts/test-layout-allowlist.json b/scripts/test-layout-allowlist.json index 2b0a9cc65..a3211e68a 100644 --- a/scripts/test-layout-allowlist.json +++ b/scripts/test-layout-allowlist.json @@ -15,10 +15,8 @@ "nvx-rs/src/host/cli/store/mod.rs": 1, "nvx-rs/src/host/network/address.rs": 2, "nvx-rs/src/host/platform/windows/spawn.rs": 1, - "nvx-rs/src/host/restore.rs": 3, "nvx-rs/src/host/sandbox/plan.rs": 5, "nvx-rs/src/host/session.rs": 1, - "nvx-rs/src/host/snapshot.rs": 1, "nvx-rs/src/host/vm.rs": 2, "nvx-rs/src/protocol/app.rs": 6 }