Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions nvx-rs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,11 @@ path = "src/bin/nvx-rs.rs"
required-features = ["cli"]
doc = false

[[example]]
name = "external-launch"
path = "examples/external-launch.rs"
required-features = ["host"]

# The guest binaries. Their unit tests live in the library, so the binaries have no test harness.
[[bin]]
name = "nvx-console-pending"
Expand Down
93 changes: 79 additions & 14 deletions nvx-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,8 @@ changes only when the module's purpose does.
| [`nvx::Hypervisor`](#nvxhypervisor) | KVM, MSHV, or WHP, with its serde form behind the `serde` feature |
| [`nvx::exec`](#nvxexec) | The execution engine: the streams, credit, and outcome of an execution's two sides, without I/O, for any framing |
| [`nvx::host::runtime`](#nvxhostruntime) | `Runtime`: what every launch shares, the runtime's files and their identity, the hypervisor and its probe, and the launch options |
| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold preparation, verified detached launch, confirmed stop, and caller-saved reattachment |
| [`nvx::host::snapshot`](#nvxhostsnapshot) | Strict private snapshot directories, verified metadata, VM compatibility, captured NIC addresses, and time eligibility |
| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, owned cold/capture commands, verified detached launch, confirmed stop, and caller-saved reattachment |
| [`nvx::host::snapshot`](#nvxhostsnapshot) | Strict private snapshots, post-exit capture finishing, verified metadata, VM compatibility, captured NIC addresses, and time eligibility |
| [`nvx::host::config`](#nvxhostconfig) | `VmConfig` and `CaptureConfig`: pure configuration, cold-boot kernel/OpenVMM arguments, portable length checks, and capture identity |
| [`nvx::host::restore`](#nvxhostrestore) | A caller-driven restore to Ready without I/O or a clock, and its peer-verified restore-ready listener |
| [`nvx::host::artifacts`](#nvxhostartifacts) | Locating the OpenVMM executable and the guest artifacts that a VM runs, and their runtime identity |
Expand Down Expand Up @@ -515,11 +515,23 @@ failure never hides confirmed exit.
`Prepared` launch without starting a process or binding a listener. It requires
`config.boot`, refuses a snapshot input, rechecks the directory's privacy, and
creates a new private log. Existing log, outcome and Linux endpoint entries
are refused rather than replaced. Only the log is created before a spawn;
OpenVMM creates the remaining outputs when it runs. A failed preparation
closes and removes only its new log, preserving all pre-existing entries and
are refused rather than replaced. An ordinary cold boot creates only the log
before a spawn; OpenVMM creates its remaining outputs when it runs. A failed preparation
closes and removes only its new files, preserving all pre-existing entries and
caller-owned inputs, and reports a cleanup failure with the original error.

With `boot.capture`, creation requires a blockless root and refuses every
filesystem export. It creates private sparse `capture-image.raw` and
`capture-scratch.raw` files, verifies their actual complete geometries, and
targets `<vm_dir>/openvmm`. A configured NIC keeps its resolved address but
uses the complete deny-all policy, whatever its configured policy.
`Prepared::capture_context()` exposes a cloneable, serializable
`snapshot::CaptureContext` with the checked capture configuration and disk
geometries, NIC and console devices, runtime identity, hypervisor and CPU
profile. The caller chooses whether and how to persist it. Creation writes
no context file, `key.bin` or `snapshot.json`, and does not finish or publish
a capture.

On Linux, the generated control and boot-console socket names must be valid
Unicode and shorter than 108 bytes both as supplied and with the directory's
ancestor links resolved. Supplied names must hold no comma or equals sign,
Expand All @@ -528,7 +540,7 @@ these names before creating any output, including the log.

`Prepared::capability()` and `outputs()` return what the caller may journal
before a spawn. `log()` returns the same opened log, without reopening its
path. `inherit(file)` moves an already opened resource into the preparation's
path. `inherit(resource)` moves an already opened resource into the preparation's
inheritance allowlist and keeps it open. `arguments()` returns the whole
configuration's OpenVMM arguments. Creation makes caller-relative runtime
and block-image paths absolute without resolving links or `..` components,
Expand Down Expand Up @@ -557,6 +569,27 @@ blockless root or memory that cannot fit the record's byte count fails with
`ErrorKind::Unsupported`. These projections do not enable the restore caller;
`create` still accepts cold launches only.

`Prepared::into_command()` moves the complete launch into `VmmCommand`: its
program, arguments, working directory, capability, outputs, inherited resources,
opened log and Windows breakaway setting. It neither reopens nor duplicates
resources, renders again nor rediscovers paths. `InheritedHandle` owns its
resource, converts from and back to `OwnedFd` on Unix or `OwnedHandle` on Windows,
and also accepts an opened `File`. The command documents the caller's spawn
contract, including standard streams, allowlisted inheritance, Linux detachment
and umask, Windows log job and breakaway, and cleanup of a failed child.
Read the resolved NIC, build an activation record, or clone the capture context
before consuming `Prepared` into the command.

The compiled [external-launch example](examples/external-launch.rs) consumes
this command, uses `supervisor::spawn_vm` without a sandbox store, checks the
cold guest's boot marker and shell response, then asks it to exit cleanly.
It requires the VMM's actual exit status 0 and a successful
outcome with complete teardown. Build with
`cargo build --locked --example external-launch` and supply the backend, OpenVMM,
kernel, initramfs, new VM directory, and the three artifact SHA-256 digests from
their installation metadata, in that order. Artifact paths may be relative to
the caller's directory.

`vm::spawn(prepared)` consumes that preparation and returns `Launched` with
its verified `process`, the same `outputs`, and the same `capability`. The log
and inherited resources are passed as already opened objects, never reopened
Expand Down Expand Up @@ -586,8 +619,8 @@ responsive shell input, and a verified clean exit; it downloads no artifact.
### `nvx::host::snapshot`

`SnapshotKey` and `SnapshotManifest` own the compatibility key and strict
OpenVMM manifest reader. `SnapshotManifest::time` exposes the capture's
`TimeContract`, whose `eligibility` checks its restore window at a `HostTime`;
OpenVMM manifest reader. `SnapshotManifest::time` and `SnapshotKey::time` expose
the capture's `TimeContract`, whose `eligibility` checks its restore window at a `HostTime`;
`HostTime::now` samples the host's clock and identities. The existing
`host::template` names remain aliases of these same types.

Expand All @@ -596,6 +629,12 @@ The key's canonical encoding and identity use the snapshot names and omit
and no reader or migration of the prior encoding is supported. `CaptureClass`
still records the legacy capture inputs, including its guest build.

The persisted key includes the complete per-snapshot time contract and must
agree with the manifest's time. Its compatibility identity excludes capture
anchors, host and boot identities, lineage and cut-to-anchor time: captures
at different times can have different key bytes and the same `KeyIdentity`.
The declared rates and time ABI remain compatibility inputs.

`Snapshot::open` checks exactly `key.bin`, `snapshot.json` and
`openvmm/{manifest.bin,memory.bin,state.bin}`. Directories are private, files
are regular and private to their owner, artifact lengths equal the strict
Expand Down Expand Up @@ -638,6 +677,20 @@ or lock. `Snapshot` remains an opaque input of a future restore:
`vm::create` still refuses `Some(snapshot)` with `ErrorKind::Unsupported`;
only cold boots are supported.

`CaptureContext` records a prepared capture's checked configuration and
effective devices independently of runtime paths or network policy.
`vm::Prepared::capture_context` returns it for the caller to clone and
serialize; NVX owns no capture-context store or persistence format.

`finish_capture(&VmDir, &CaptureContext, &Exited)` requires the capture VM's
proved exit and successful outcome, checks the OpenVMM snapshot against the
context, and atomically writes `key.bin` and informational `snapshot.json`
beside `openvmm/`. It returns `CapturedSnapshot { key }`. The caller moves
only those three entries into its own private, immutable snapshot directory;
logs, outcomes and other VM files remain where they are. A failure can be
corrected and retried while the VM directory and capture context stay intact,
including when one metadata file was already written.

### `nvx::host::config`

`VmConfig`: the configuration of a VM as one value, its processors, its memory
Expand Down Expand Up @@ -715,12 +768,15 @@ The renderer emits neither `--microvm-lifecycle` nor
`--microvm-workload-identity`. Callers inspect the complete arguments through
`Prepared::arguments()`.

Both argument entry points refuse a `Boot` with a capture as `Unsupported`
rather than silently omit its devices and resource capacities.
They also refuse non-Unicode export child roots instead of lossily rendering
a different host path.
`VmConfig::check` and `kernel_command_line` retain their existing behavior for
valid capture configurations.
For a capture, `CaptureConfig` supplies the processor capacity, the boot-online
processor limit, and the boot memory and its capacity; the ordinary VM's cold
resource counts need not match. Boot memory must be aligned to 128 MiB.
The kernel text omits root, view, mapping and workload tokens and admits only
OpenVMM's named platform allowlist. Caller-supplied tier, processor, console,
device and network tokens are refused by name even when allowlisted: the
builder owns them. OpenVMM adds exactly one `nvx_snapshot_tier=platform` from
`--snapshot-tier platform`; callers may not supply or duplicate it.
Non-Unicode export child roots are still refused rather than lossily rendered.

`VmConfig::check_command_line()` checks the configuration's known arguments on
every host, before any host change, with 2048 UTF-16 code units reserved for the
Expand Down Expand Up @@ -1074,6 +1130,15 @@ after, and returns a `Starting` launch that the caller commits once the VMM
serves its control endpoint, or abandons. A launch that fails never leaves an
unrecorded VMM running.

`spawn_vm` starts a caller-owned command without a store, with the same strict
Windows log-job requirement as `vm::spawn`: an untrusted job name or a log that
cannot name a job is refused before child creation. The legacy `spawn` keeps
its fallback without a job for those logs and names. Other spawn behavior is
shared. If setup fails after a child is created, including failed process
verification, `Launched::kill_and_confirm_exit` waits without a deadline for
its exit and reaps it on Unix. The caller reports any cleanup error; the
legacy `kill_and_wait` remains time-bounded and may release a running child.

`VmmCommand::inherit` lists the descriptors (Linux) or handles (Windows) that
the VMM inherits under their numbers, besides its standard streams, and it
inherits nothing else of the caller's: on Linux, they must be close-on-exec, a
Expand Down
176 changes: 176 additions & 0 deletions nvx-rs/examples/external-launch.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
//! A caller-owned cold launch, using the strict low-level spawn primitive without a store.
//!
//! Run the compiled example with a backend, OpenVMM, kernel, initramfs, new VM directory, and
//! the three artifact SHA-256 digests from their installation metadata, in that order.

use std::error::Error;
use std::io::{self, Write};
use std::path::Path;
use std::time::{Duration, Instant};

use nvx::Hypervisor;
use nvx::activation::{MachineIdMode, View};
use nvx::host::config::{Boot, ConsoleAttachment, RootConfig, VmConfig};
use nvx::host::platform::Transport;
use nvx::host::runtime::{ArtifactIdentity, LaunchOptions, Runtime, RuntimeIdentity};
use nvx::host::supervisor::{self, Inherited};
use nvx::host::{artifacts, report, session, vm};

fn main() -> Result<(), Box<dyn Error>> {
let args: Vec<_> = std::env::args_os().collect();
if args.len() != 9 {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"usage: external-launch BACKEND OPENVMM KERNEL INITRAMFS VM-DIR \
OPENVMM-SHA256 KERNEL-SHA256 INITRAMFS-SHA256",
)
.into());
}
let hypervisor = match args[1].to_str() {
Some("kvm") => Hypervisor::Kvm,
Some("mshv") => Hypervisor::Mshv,
Some("whp") => Hypervisor::Whp,
_ => return Err(io::Error::new(io::ErrorKind::InvalidInput, "invalid backend").into()),
};
nvx::host::runtime::probe_hypervisor(hypervisor)?;
let identity = |index: usize| {
ArtifactIdentity::sha256(args[index].to_str().ok_or_else(|| {
io::Error::new(
io::ErrorKind::InvalidInput,
"artifact digest is not Unicode",
)
})?)
.map_err(Box::<dyn Error>::from)
};
let runtime = Runtime {
openvmm: artifacts::absolute(Path::new(&args[2]))?,
kernel: artifacts::absolute(Path::new(&args[3]))?,
initramfs: artifacts::absolute(Path::new(&args[4]))?,
identity: RuntimeIdentity::new(identity(6)?, identity(7)?, identity(8)?),
hypervisor,
cpu_profile: Some("auto".to_owned()),
windows_breakaway: false,
options: LaunchOptions::default(),
};
let config = VmConfig {
processors: 1,
memory_mib: 256,
root: RootConfig::Blockless,
view: View {
hostname: b"external".to_vec(),
machine_id: MachineIdMode::Image,
},
network: None,
filesystem: None,
console: ConsoleAttachment::Listener,
boot: Some(Boot {
command_line: "quiet loglevel=0".to_owned(),
capture: None,
}),
};
vm::create_private_dir(Path::new(&args[5]))?;
let dir = vm::VmDir::open(Path::new(&args[5]))?;
let vm::VmmCommand {
program,
arguments,
working_dir,
capability,
outputs,
inherit,
log,
breakaway_from_job,
} = vm::create(&runtime, &config, None, &dir)?.into_command();
let command = supervisor::VmmCommand {
program,
arguments,
working_dir,
breakaway_from_job,
inherit: inherit.iter().map(Inherited::new).collect(),
};
let mut launched = supervisor::spawn_vm(&command, capability.as_bytes(), log)?;
let result = (|| -> Result<(), Box<dyn Error>> {
let process = launched.process()?;
let deadline = Instant::now() + Duration::from_secs(60);
let console = outputs.console.as_deref().ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidInput, "no prepared boot console")
})?;
let mut console = session::connect(
console,
process.reference().pid,
process.reference().start_time,
deadline,
)?;
read_until(console.as_mut(), b"NVX-GUEST-BOOT-OK:", deadline)?;
console.write_all(
b"printf 'NVX-EXTERNAL-LAUNCH-OK\\n'; /sbin/nvx-exit 0\n",
Some(deadline.saturating_duration_since(Instant::now())),
)?;
read_until(console.as_mut(), b"\nNVX-EXTERNAL-LAUNCH-OK", deadline)?;
if process.wait(deadline)?.is_none() {
return Err(io::Error::new(io::ErrorKind::TimedOut, "VM did not exit").into());
}
match launched.wait_timeout(Duration::ZERO)? {
Some(0) => {}
Some(status) => {
return Err(io::Error::other(format!("VMM exited with status {status}")).into());
}
None => return Err(io::Error::other("VMM exit status is unavailable").into()),
}
let outcome = report::read(&outputs.outcome)?
.ok_or_else(|| io::Error::other("missing or invalid VM outcome"))?;
if outcome.category.as_deref() != Some("success")
|| outcome.status_code != Some(0)
|| outcome.teardown_complete != Some(true)
{
return Err(
io::Error::other(format!("VM did not shut down cleanly: {outcome:?}")).into(),
);
}
Ok(())
})();
if let Err(error) = result {
if let Err(cleanup) = launched.kill_and_confirm_exit() {
return Err(io::Error::other(format!(
"{error}; could not confirm exit of the launched VM: {cleanup}"
))
.into());
}
return Err(error);
}
launched.detach();
println!(
"external cold launch completed; outcome={}",
outputs.outcome.display()
);
Ok(())
}

/// Prints console output through a marker, retaining only enough bytes to match across reads.
fn read_until(console: &mut dyn Transport, marker: &[u8], deadline: Instant) -> io::Result<()> {
let mut pending = Vec::new();
let mut buffer = [0; 4096];
loop {
let remaining = deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
return Err(io::Error::new(
io::ErrorKind::TimedOut,
"console marker not received",
));
}
let count = console.read(&mut buffer, Some(remaining))?;
if count == 0 {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"boot console closed",
));
}
io::stdout().write_all(&buffer[..count])?;
pending.extend_from_slice(&buffer[..count]);
if pending.windows(marker.len()).any(|bytes| bytes == marker) {
return Ok(());
}
if pending.len() > marker.len() {
pending.drain(..pending.len() - marker.len());
}
}
}
Loading
Loading