From 30a7bafda676f7206a45a14e6934e91ee1ae8fcd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 19:42:44 -0800 Subject: [PATCH 1/3] guest: end a failed restored activation with its class's status A restored activation that failed in nvx-init, such as one whose root does not assemble over a zeroed scratch image, ended the VM with the generic status 125, as any other failure of PID 1. The host's restore then saw only the guest's exit: RestoreEvidence::reported classifies the statuses of nvx::protocol::status, and found none in 125, so it attributed the failure to nothing, as the old adapter's worker, whose failure_status picked the class, had not. nvx-init now ends the VM with the status of the failure's class, from the statuses that nvx::protocol::status already defines: once the control session has failed, and no helper ended, the session asks the restored activation for its failure, and Restored::failure_status returns its class: - the activation's own failure, which a deadline that has passed by then also is, by its cause, ACTIVATION_DEADLINE to ACTIVATION_ORDER. A failed application takes the class of the first cause that names one: the ApplyError of the record's application, or the step of nvx-init's own space that it marks, the workload's account and hostname and the announcer, or else the root's, 54, whose runtime file systems and switch the other steps are; - an activate request that Restored refuses before its activation takes it: ACTIVATION_PROTOCOL for one that changes the accepted request, before or after Ready, as the status's docs say, and ACTIVATION_RECORD for one that does not decode. A request while the record applies now fails the activation, as its state machine's table says, where it ended the session without it. An expiry that only the last check finds is reported on the console too; every other failure was reported where it happened. A cold boot, a ready activation and every other failure keep their statuses, 125 among them. Tests: a_failed_activation_ends_the_vm_with_the_status_of_its_class drives an activation to each cause, each class of application failure and each refused request, and a_failed_restored_activation_gives_init_the_status_of_its_class checks the session's wiring, with a request while the record applies. The live restore fixture's deadline and changed cases, which pinned 125, now expect ACTIVATION_DEADLINE and ACTIVATION_PROTOCOL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/guest-agent-guidelines.md | 5 +- nvx-rs/README.md | 8 +- nvx-rs/src/guest/init.rs | 13 +- nvx-rs/src/guest/init/boot.rs | 127 +++++++++- nvx-rs/src/guest/init/boot/tests.rs | 216 ++++++++++++++++++ nvx-rs/src/guest/managed_agent/dispatch.rs | 13 +- nvx-rs/src/guest/managed_agent/session.rs | 34 ++- .../src/guest/managed_agent/session/tests.rs | 49 ++++ nvx-rs/src/host/restore/tests/live_init.rs | 4 +- 9 files changed, 439 insertions(+), 30 deletions(-) diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md index d1310ce83..c2b42c089 100644 --- a/doc/guest-agent-guidelines.md +++ b/doc/guest-agent-guidelines.md @@ -430,7 +430,10 @@ gives the restore its network. console, then syncs and powers off with the status. - A failed activation, wholly or partly applied, never reports Ready: the init reports the failure and ends the VM. nvx-init writes it to the console, - after `NVX-INIT-ACTIVATION:`, and powers off with status 125. + after `NVX-INIT-ACTIVATION:`, and powers off with the status of the + failure's class, which `nvx::protocol::status` defines, such as 54, + `ACTIVATION_ROOT`, for a root that does not assemble over its upper, from + which the host attributes the failure. ## Replay diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 80a0dc8c3..8f4e86efc 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -2729,7 +2729,7 @@ disk's block device. | `nvx-ingress [inherit\|start] GATEWAY_IPV4` | Announces reachable TCP listeners and settled UDP sockets to the portable-network gateway, from a privileged source port that it binds, or, with `inherit`, from its standard input, a socket that the caller bound to one after the clone's process-free application; `start` binds such a socket and starts `inherit` on it in a session of its own, then exits once its program runs, or with status 1 if it does not start, for a guest agent that cannot hold the announcer; `test DIRECTORY` runs synthetic procfs snapshots | | `nvx-time [--report-only] COMMAND` | Enforces NVX time ABI v1: runs its conformance checks, repairs the wall clock after a restore, publishes the time state, and runs the discipline and violation-watcher daemon; `test NAME ...` runs its decoder and process test hooks (x86_64) | | `nvx-time-probe latency\|warp\|clock ...` | Diagnoses clock-read latency, cross-CPU TSC warp and offset, the clocks, CPUID, clock events, kernel messages, and optionally MSRs (x86_64) | -| `nvx-init TIME_DAEMON [ANNOUNCER]` | Takes PID 1 over from a layered root's early boot, with its time daemon and optional announcer: gives itself and them an out-of-memory score adjustment of -1000 and a nice value of -10, collects every child's status, and, after `sync`, ends the VM with status 125 when a helper ends or 0 on `SIGTERM`; elsewhere it exits with 2 (x86_64) | +| `nvx-init TIME_DAEMON [ANNOUNCER]` | Takes PID 1 over from a layered root's early boot, with its time daemon and optional announcer: gives itself and them an out-of-memory score adjustment of -1000 and a nice value of -10, collects every child's status, and, after `sync`, ends the VM with status 125 when a helper ends, 0 on `SIGTERM`, or the status of a failed restored activation's class; elsewhere it exits with 2 (x86_64) | Each binary's command line, exit statuses, and byte output are compatibility contracts; the [`nvx::guest`](src/guest/mod.rs) documentation describes each @@ -2741,7 +2741,11 @@ then completes and reports Ready. The host supplies the complete mapping table before the first valid EXEC starts the restored workload; later executions and byte-identical activation retries, also those that arrive while a workload runs, reuse the retained state, and an activation with other bytes ends the VM -as a failure. +as a failure. A failed activation, which a deadline that passes before Ready +and a refused activate request also are, ends the VM with the status of the +failure's class, which [`nvx::protocol::status`](#nvxprotocolstatus) defines, +such as 54, `ACTIVATION_ROOT`, for a root that does not assemble over its +upper, or 51, `ACTIVATION_PROTOCOL`, for an activation with other bytes. The accepted proxy replaces the four HTTP(S) proxy defaults, or clears them when the restore has no proxy. Per-EXEC layered entries still win, and exact environments still replace every default. diff --git a/nvx-rs/src/guest/init.rs b/nvx-rs/src/guest/init.rs index 00632ecc1..80ca4978f 100644 --- a/nvx-rs/src/guest/init.rs +++ b/nvx-rs/src/guest/init.rs @@ -30,8 +30,13 @@ //! nothing from it; //! - ends the VM itself, without another guest program: it writes every file system's modified //! data to its storage, with `sync`, and then stops the VM through OpenVMM's control port with a -//! status: 0 after a host `STOP` or `SIGTERM`, or 125, as `nvx-init-agent`'s failures do, after -//! a failure, which an `NVX-INIT-ERROR:` line on the console names first. +//! status: 0 after a host `STOP` or `SIGTERM`; after its restored activation failed, which a +//! deadline that passed before Ready and an activate request that it refused also are, the +//! status of `nvx::protocol::status` of the failure's class, such as 54, `ACTIVATION_ROOT`, for +//! a root that does not assemble over its upper, or 51, `ACTIVATION_PROTOCOL`, for an activate +//! request that changes the accepted one, which an `NVX-INIT-ACTIVATION:` line on the console +//! names first; or 125, as `nvx-init-agent`'s failures do, after another failure, which an +//! `NVX-INIT-ERROR:` line on the console names first. //! //! Outside PID 1, `nvx-init` changes nothing: it fails with status 2 and a message. @@ -62,8 +67,8 @@ pub(crate) use boot::{RestoreHandoff, Restored}; /// The process ID of the init process. const INIT_PROCESS: i32 = 1; -/// The VM's exit status after a failure of PID 1 or the end of a helper: that of -/// `nvx-init-agent`'s failures. +/// The VM's exit status after a failure of PID 1 or the end of a helper, but for a failed restored +/// activation's: that of `nvx-init-agent`'s failures. const FAILURE: u8 = 125; /// The out-of-memory score adjustment of PID 1 and its helpers, Linux's `OOM_SCORE_ADJ_MIN`, /// which exempts them from the out-of-memory killer. diff --git a/nvx-rs/src/guest/init/boot.rs b/nvx-rs/src/guest/init/boot.rs index 1623f4696..35429a636 100644 --- a/nvx-rs/src/guest/init/boot.rs +++ b/nvx-rs/src/guest/init/boot.rs @@ -1,6 +1,7 @@ //! The fixed cold-boot and restored-space setup of the VM's shared layered root. use std::ffi::{CString, OsStr}; +use std::fmt; use std::net::IpAddr; use std::os::unix::ffi::OsStrExt; use std::path::{Path, PathBuf}; @@ -9,8 +10,8 @@ use std::time::{Duration, Instant}; use crate::activation::{self, View}; use crate::gpt::Expected; use crate::guest::activation::{ - self as guest_activation, Activation, ActivationAction, ActivationInput, ActivationState, - Applied, RootTemplate, + self as guest_activation, Activation, ActivationAction, ActivationCause, ActivationFailure, + ActivationInput, ActivationState, Applied, ApplyError, RootTemplate, }; use crate::guest::mappings::{Access, Mapper, MappingTable, Reserved, SourceRoot, TargetRoot}; use crate::guest::root::{self, Assembly, Device, Root, ShareChild, ShareKind}; @@ -22,7 +23,7 @@ use crate::protocol::export::{EXPORT_DIRECTORY, HostExport}; use crate::protocol::root::{ DISTRO_MMIO_BASE, ImageFormat, RootLayout, RootUpper, SCRATCH_MMIO_BASE, }; -use crate::protocol::{self, ProtocolError}; +use crate::protocol::{self, ProtocolError, status}; use crate::{Error, Result}; mod account; @@ -331,6 +332,9 @@ fn share_child(value: &[u8]) -> Result { pub(crate) struct Restored { pub(crate) activation: Activation, payload: Option>, + /// The status of the class of an activate request that this refused before its activation + /// took the request, which ends the VM. + refused: Option, _applied: Option, } @@ -352,6 +356,7 @@ impl Restored { Self { activation: Activation::new(repaired, Duration::from_secs(30), now), payload: None, + refused: None, _applied: None, } } @@ -369,6 +374,8 @@ impl Restored { } /// Compares the complete retry payload before delegating record replay to the state machine. + /// A request that changes the accepted one, before or after Ready, is refused as the + /// protocol's failure, and one that does not decode as the record's. pub(crate) fn accept( &mut self, payload: &[u8], @@ -379,11 +386,15 @@ impl Restored { .as_deref() .is_some_and(|accepted| accepted != payload) { + self.refused = Some(status::ACTIVATION_PROTOCOL); return Err(Error::unsupported( "the activate request changed after acceptance", )); } - let (_, workload) = protocol::decode_activate_payload(payload).map_err(protocol_error)?; + let (_, workload) = protocol::decode_activate_payload(payload).map_err(|error| { + self.refused = Some(status::ACTIVATION_RECORD); + protocol_error(error) + })?; let action = self .activation .input(now, ActivationInput::Activate(&payload[9..])) @@ -394,6 +405,23 @@ impl Restored { Ok((action, workload)) } + /// Returns the status with which nvx-init ends the VM once this activation has failed: that + /// of a request that this refused, or else the one that [`class_status`] selects for the + /// activation's failure, or `None` while it has not failed. An activation that is not ready + /// fails once its deadline has passed at `now`, which this reports, as the session reports + /// every other failure where it happens. + pub(crate) fn failure_status(&mut self, now: Instant) -> Option { + if self.refused.is_some() { + return self.refused; + } + let reported = self.activation.state() == ActivationState::Failed; + let failure = self.activation.tick(now).err()?; + if !reported { + super::report_activation(&failure); + } + Some(class_status(&failure)) + } + /// Applies and switches one restored space, retaining its root and handing over its mapper. pub(crate) fn apply( &mut self, @@ -436,12 +464,15 @@ impl Restored { applied.root().path(), (workload.uid, workload.gid), workload.create_account, - )?; + ) + .map_err(classed(status::ACTIVATION_WORKLOAD))?; Ok(RestoreHandoff { workload, user, home, - hostname: CString::new(applied.hostname()).map_err(activation_error)?, + hostname: CString::new(applied.hostname()) + .map_err(activation_error) + .map_err(classed(status::ACTIVATION_WORKLOAD))?, proxy: applied.proxy().map(str::to_owned), table: u32::try_from(record.configuration().mappings) .ok() @@ -455,7 +486,8 @@ impl Restored { }, |applied| { if applied.publication() == activation::Publication::Published { - announce(applied.publication(), applied.gateway())?; + announce(applied.publication(), applied.gateway()) + .map_err(classed(status::ACTIVATION_NETWORK))?; } Ok(()) }, @@ -467,6 +499,87 @@ impl Restored { } } +/// Returns the status of `nvx::protocol::status` with which nvx-init ends the VM once its restored +/// activation failed with `failure`, that of the failure's class: +/// +/// - [`ACTIVATION_DEADLINE`](status::ACTIVATION_DEADLINE): the deadline passed before the activation +/// was ready; +/// - [`ACTIVATION_PROTOCOL`](status::ACTIVATION_PROTOCOL): the host reset the session again, sent a +/// request before the activation, or an Activate of another transaction, or one that changes +/// the accepted record; +/// - [`ACTIVATION_RECORD`](status::ACTIVATION_RECORD): the Activate's record does not decode, +/// breaks a limit, or is a cold boot's; +/// - [`ACTIVATION_RESOURCES`](status::ACTIVATION_RESOURCES): the record expects other resources +/// than the repaired VM has; +/// - for a failed application of the record, that of its first cause that names a class: the +/// [`ApplyError`] of the record's application, whose GPT disk and root's assembly are the +/// root's, [`ACTIVATION_ROOT`](status::ACTIVATION_ROOT), whose machine ID and view are the +/// workload's, [`ACTIVATION_WORKLOAD`](status::ACTIVATION_WORKLOAD), whose host export is +/// [`ACTIVATION_EXPORT`](status::ACTIVATION_EXPORT), and whose network settings are +/// [`ACTIVATION_NETWORK`](status::ACTIVATION_NETWORK); the workload's account and hostname, +/// `ACTIVATION_WORKLOAD`; the announcer of the network's listeners, `ACTIVATION_NETWORK`; or +/// else the root's, `ACTIVATION_ROOT`, whose runtime file systems and switch the other steps +/// are; +/// - [`ACTIVATION_ORDER`](status::ACTIVATION_ORDER): nvx-init's own order completed an activation +/// that did not run, or started the workload before the activation was ready. +fn class_status(failure: &ActivationFailure) -> u8 { + match failure.cause() { + ActivationCause::Expired => status::ACTIVATION_DEADLINE, + ActivationCause::Reset + | ActivationCause::Request + | ActivationCause::Transaction { .. } + | ActivationCause::Changed => status::ACTIVATION_PROTOCOL, + ActivationCause::Record(_) | ActivationCause::Cold => status::ACTIVATION_RECORD, + ActivationCause::Resources { .. } => status::ACTIVATION_RESOURCES, + ActivationCause::Failed(error) => { + let error: &(dyn std::error::Error + 'static) = &**error; + std::iter::successors(Some(error), |error| error.source()) + .find_map(|error| match error.downcast_ref::() { + Some(ApplyError::Gpt(_) | ApplyError::Assembly(_)) => { + Some(status::ACTIVATION_ROOT) + } + Some(ApplyError::MachineId { .. } | ApplyError::View { .. }) => { + Some(status::ACTIVATION_WORKLOAD) + } + Some(ApplyError::Export { .. }) => Some(status::ACTIVATION_EXPORT), + Some(ApplyError::Network { .. }) => Some(status::ACTIVATION_NETWORK), + None => error + .downcast_ref::() + .map(|classed| classed.status), + }) + .unwrap_or(status::ACTIVATION_ROOT) + } + ActivationCause::Completion | ActivationCause::Start => status::ACTIVATION_ORDER, + } +} + +/// The failure of a step of a restored space whose class is not the root's, with the status of +/// its class, which [`class_status`] finds among an application's causes. +#[derive(Debug)] +struct Classed { + /// The status of the step's class. + status: u8, + /// The step's failure, whose message and causes this keeps. + error: Error, +} + +impl fmt::Display for Classed { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + self.error.fmt(formatter) + } +} + +impl std::error::Error for Classed { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + std::error::Error::source(&self.error) + } +} + +/// Returns a step's failure with its message, classed with `status`. +fn classed(status: u8) -> impl FnOnce(Error) -> Error { + move |error| Error::new(error.code(), error.message()).with_source(Classed { status, error }) +} + /// Closes prepared handles before tearing down a space whose setup failed. fn restore_in_order( deadline: Option, diff --git a/nvx-rs/src/guest/init/boot/tests.rs b/nvx-rs/src/guest/init/boot/tests.rs index d8e75bec0..2fd8139cd 100644 --- a/nvx-rs/src/guest/init/boot/tests.rs +++ b/nvx-rs/src/guest/init/boot/tests.rs @@ -274,6 +274,222 @@ fn an_expired_activation_cannot_begin_or_advance_to_another_space_step() { assert!(error.to_string().contains("deadline")); } +#[test] +fn a_failed_activation_ends_the_vm_with_the_status_of_its_class() { + use crate::activation::{ActivationRecord, Configuration, RestoreSection, TransactionId}; + use crate::guest::activation::NetworkError; + use crate::protocol::status::*; + + let now = Instant::now(); + let (_, payload) = super::super::Capture::activation_fixture(); + let (record, _) = protocol::decode_activate_payload(&payload).unwrap(); + let encode = |restore, change: fn(&mut Configuration)| { + let mut configuration = record.configuration().clone(); + change(&mut configuration); + ActivationRecord::new(restore, configuration) + .unwrap() + .encode() + }; + let cold = encode(None, |_| {}); + let larger = encode(record.restore().copied(), |configuration| { + configuration.resources.cpus += 1 + }); + let other = encode( + Some(RestoreSection { + transaction: TransactionId::from_bytes([0x33; TransactionId::LEN]), + }), + |_| {}, + ); + let restricted = || { + super::super::Capture::activation_fixture() + .0 + .restored + .unwrap() + }; + let refused = |input: ActivationInput<'_>| { + let mut restored = restricted(); + restored.activation.input(now, input).unwrap_err(); + restored + }; + let activating = || { + let mut restored = restricted(); + assert!(matches!( + restored.accept(&payload, now).unwrap().0, + ActivationAction::Apply(_) + )); + restored + }; + // The record's application fails with `error`, as `Restored::apply` returns it. + let failed = |error: Error| { + let mut restored = activating(); + restored + .activation + .complete(now, Err(error.into())) + .unwrap_err(); + restored + }; + let applying = |error: ApplyError| failed(activation_error(error)); + let missing = || root::Error::Missing { + role: root::Role::Distro, + mmio: DISTRO_MMIO_BASE, + }; + let relative = RootTemplate { + runtime: "run/nvx".into(), + distro: Device { + mmio: DISTRO_MMIO_BASE, + sectors: None, + }, + scratch: Device { + mmio: SCRATCH_MMIO_BASE, + sectors: None, + }, + timeout: Duration::ZERO, + filesystem_tag: "microvm".into(), + gateway: None, + }; + let Err(assembly) = guest_activation::apply(&record, &relative, None) else { + panic!("a relative runtime directory was assembled"); + }; + let mut ordered = restricted(); + ordered.activation.start().unwrap_err(); + let mut completed = restricted(); + completed.activation.complete(now, Ok(())).unwrap_err(); + let mut changed = activating(); + changed + .activation + .input(now, ActivationInput::Activate(&other)) + .unwrap_err(); + // Requests that `Restored` refuses before its activation takes them. + let mut identity = payload.clone(); + identity[0] ^= 1; + let mut undecodable = restricted(); + undecodable.accept(&payload[..5], now).unwrap_err(); + let mut rewritten = activating(); + rewritten.accept(&identity, now).unwrap_err(); + let mut replayed = activating(); + replayed.activation.complete(now, Ok(())).unwrap(); + replayed.accept(&identity, now).unwrap_err(); + let cases = [ + ("an expired deadline", restricted(), ACTIVATION_DEADLINE), + ( + "a second reset", + refused(ActivationInput::Reset), + ACTIVATION_PROTOCOL, + ), + ( + "a request before the Activate", + refused(ActivationInput::Request), + ACTIVATION_PROTOCOL, + ), + ("another transaction", changed, ACTIVATION_PROTOCOL), + ("a changed request", rewritten, ACTIVATION_PROTOCOL), + ( + "a changed request once ready", + replayed, + ACTIVATION_PROTOCOL, + ), + ("an undecodable request", undecodable, ACTIVATION_RECORD), + ( + "an undecodable record", + refused(ActivationInput::Activate(&[0])), + ACTIVATION_RECORD, + ), + ( + "a cold boot's record", + refused(ActivationInput::Activate(&cold)), + ACTIVATION_RECORD, + ), + ( + "other resources", + refused(ActivationInput::Activate(&larger)), + ACTIVATION_RESOURCES, + ), + ( + "a root that does not assemble", + applying(assembly), + ACTIVATION_ROOT, + ), + ( + "a GPT disk", + applying(ApplyError::Gpt(missing())), + ACTIVATION_ROOT, + ), + ( + "a machine ID", + applying(ApplyError::MachineId { + error: std::io::Error::other("no machine ID"), + teardown: None, + }), + ACTIVATION_WORKLOAD, + ), + ( + "a view", + applying(ApplyError::View { + error: Error::io("no hostname"), + teardown: None, + }), + ACTIVATION_WORKLOAD, + ), + ( + "an export", + applying(ApplyError::Export { + error: missing(), + teardown: None, + }), + ACTIVATION_EXPORT, + ), + ( + "network settings", + applying(ApplyError::Network { + error: NetworkError::Gateway, + teardown: None, + }), + ACTIVATION_NETWORK, + ), + ( + "an account", + failed(classed(ACTIVATION_WORKLOAD)(Error::unsupported( + "the account is missing", + ))), + ACTIVATION_WORKLOAD, + ), + // A teardown's failure keeps the step's class among the causes. + ( + "an announcer", + failed( + Error::io("the announcer failed; cleanup failed").with_source(classed( + ACTIVATION_NETWORK, + )( + Error::io("the announcer failed"), + )), + ), + ACTIVATION_NETWORK, + ), + ( + "a switch", + failed(Error::io("failed to switch")), + ACTIVATION_ROOT, + ), + ("an early completion", completed, ACTIVATION_ORDER), + ("an early start", ordered, ACTIVATION_ORDER), + ]; + for (case, mut restored, expected) in cases { + let later = now + Duration::from_secs(3600); + assert_eq!( + restored.failure_status(later), + Some(expected), + "{case}: {:?}", + restored.activation + ); + assert_eq!(restored.failure_status(later), Some(expected), "{case}"); + } + // An activation fails only once its deadline has passed, and never once it is ready. + assert_eq!(restricted().failure_status(now), None); + let mut ready = activating(); + ready.activation.complete(now, Ok(())).unwrap(); + assert_eq!(ready.failure_status(now + Duration::from_secs(3600)), None); +} + fn tokens(extra: &str) -> Tokens { Tokens::parse(format!("{ROOT} {VIEW} {CLI} {extra}").as_bytes()).unwrap() } diff --git a/nvx-rs/src/guest/managed_agent/dispatch.rs b/nvx-rs/src/guest/managed_agent/dispatch.rs index d047af7c6..b4dd6cddb 100644 --- a/nvx-rs/src/guest/managed_agent/dispatch.rs +++ b/nvx-rs/src/guest/managed_agent/dispatch.rs @@ -79,7 +79,9 @@ impl AgentConfig { config.serve_init(runtime, control, HostMappings::default(), true) } - /// Borrows the one init runtime for either boot entry and returns its actual exit status. + /// Borrows the one init runtime for either boot entry and returns its actual exit status: a + /// failed restored activation, which its diagnostics named, ends the VM with the status of its + /// class. fn serve_init( &self, runtime: &mut Runtime, @@ -92,10 +94,11 @@ impl AgentConfig { let mut session = Session::with_runtime(control, runtime)?.with_capture(capture); run(&mut system, &mut session, self, &mut mappings) .map(|Outcome::Stopped| session.capture_status().unwrap_or_default()) - .map_err(|error| { - session - .runtime_failure() - .unwrap_or_else(|| format!("control session failed: {}", error.desc())) + .or_else(|error| match session.runtime_failure() { + Some(failure) => Err(failure), + None => session + .activation_status() + .ok_or_else(|| format!("control session failed: {}", error.desc())), }) } } diff --git a/nvx-rs/src/guest/managed_agent/session.rs b/nvx-rs/src/guest/managed_agent/session.rs index c4d618ed0..15aa65004 100644 --- a/nvx-rs/src/guest/managed_agent/session.rs +++ b/nvx-rs/src/guest/managed_agent/session.rs @@ -3,7 +3,7 @@ use std::io::{self, Read, Write}; use std::process::ExitStatus; use std::time::{Duration, Instant}; -use crate::guest::activation::{ActivationAction, ActivationState}; +use crate::guest::activation::{ActivationAction, ActivationInput, ActivationState}; use crate::guest::init::{self, Capture, CaptureStep, RestoreHandoff, Restored, Runtime}; use crate::guest::nvxs::{GuestConnection, GuestConnectionError}; use crate::guest::snapshot::{Captured, Disposition}; @@ -280,6 +280,16 @@ impl<'r> Session<'r> { self.capture.as_ref().and_then(Capture::shutdown_status) } + /// The status of the failed restored activation's class, which the PID-1 caller uses once + /// the session has failed, or `None` without a restored activation that failed. + pub(crate) fn activation_status(&mut self) -> Option { + self.capture + .as_mut()? + .restored + .as_mut()? + .failure_status(Instant::now()) + } + /// Refuses rooted callers before validating the flag or reserving workload cleanup. pub(crate) fn queue_capture( &mut self, @@ -457,15 +467,21 @@ impl<'r> Session<'r> { let retry = self .accept_request(&bytes) .map_err(DataFailure::into_error)?; - if retry.kind != protocol::APP_ACTIVATE - || !matches!( + if retry.kind != protocol::APP_ACTIVATE { + return Err(activation_failure( restored - .accept(retry.payload, Instant::now()) - .map_err(activation_failure)? - .0, - ActivationAction::Join - ) - { + .activation + .input(Instant::now(), ActivationInput::Request) + .expect_err("an activation takes no request yet"), + )); + } + if !matches!( + restored + .accept(retry.payload, Instant::now()) + .map_err(activation_failure)? + .0, + ActivationAction::Join + ) { return Err(Errno::EPROTO); } replies.push(retry.request_id); diff --git a/nvx-rs/src/guest/managed_agent/session/tests.rs b/nvx-rs/src/guest/managed_agent/session/tests.rs index d4147b2de..62232fd52 100644 --- a/nvx-rs/src/guest/managed_agent/session/tests.rs +++ b/nvx-rs/src/guest/managed_agent/session/tests.rs @@ -276,6 +276,55 @@ fn failed_application_never_reports_ready_or_admits_root_dependent_work() { ); } +#[test] +fn a_failed_restored_activation_gives_init_the_status_of_its_class() { + let (mut session, _peer) = connected(true); + assert_eq!(session.activation_status(), None); + let (mut session, mut peer, payload) = restored(); + assert_eq!(session.activation_status(), None); + let bytes = activate_bytes(&mut session, &mut peer, 68, &payload); + let request = session + .accept_request(&bytes) + .unwrap_or_else(|_| panic!("valid activate")); + peer.credit(); + let failure = crate::Error::io("the network settings cannot be applied").with_source( + crate::guest::activation::ApplyError::Network { + error: crate::guest::activation::NetworkError::Gateway, + teardown: None, + }, + ); + assert_eq!( + session + .activate_with::<()>(request, |_, _, _, _| Err(failure)) + .err(), + Some(Errno::EPROTO) + ); + assert_eq!( + session.activation_status(), + Some(protocol::status::ACTIVATION_NETWORK) + ); + // A request while the record applies fails the activation as the protocol's failure. + let (mut session, mut peer, payload) = restored(); + let bytes = activate_bytes(&mut session, &mut peer, 69, &payload); + let request = session + .accept_request(&bytes) + .unwrap_or_else(|_| panic!("valid activate")); + peer.credit(); + let result = session.activate_with(request, |_, _, _, _| { + peer.send(protocol::APP_PING, 70, &[]); + Ok(()) + }); + assert_eq!(result.err(), Some(Errno::EPROTO)); + assert_eq!( + session.activation_state_for_test(), + Some(ActivationState::Failed) + ); + assert_eq!( + session.activation_status(), + Some(protocol::status::ACTIVATION_PROTOCOL) + ); +} + fn short_deadline(session: &mut Session<'_>) -> Instant { let now = Instant::now(); let repaired = crate::guest::snapshot::repair::tests::repaired_fixture(); diff --git a/nvx-rs/src/host/restore/tests/live_init.rs b/nvx-rs/src/host/restore/tests/live_init.rs index ce6c83c6d..0b9aa4ec4 100644 --- a/nvx-rs/src/host/restore/tests/live_init.rs +++ b/nvx-rs/src/host/restore/tests/live_init.rs @@ -365,7 +365,7 @@ fn run() -> Result<(), AnyError> { "ORIGINAL_ACK_DEADLINE_ELAPSED_MS={}", native_ready.elapsed().as_millis() ); - expected_status = 125; + expected_status = i64::from(crate::protocol::status::ACTIVATION_DEADLINE); } else { let published = case != "unpublished"; let transaction = TransactionId::from_bytes([0x77; 16]); @@ -422,7 +422,7 @@ fn run() -> Result<(), AnyError> { .is_some(), "changed identity replay did not end VM", )?; - expected_status = 125; + expected_status = i64::from(crate::protocol::status::ACTIVATION_PROTOCOL); println!("CHANGED_IDENTITY_REPLAY_FATAL"); } else { connection.send( From 9879e419dd4216d9854b1e28463b8bef408b2def Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 19:43:08 -0800 Subject: [PATCH 2/3] cli: report the attribution of a restore that fails in its guest A restore whose guest failed before it was ready printed only how its VM ended, such as `OpenVMM exited during startup: guest-exit status 125`: once the VM had exited by itself, launch_locked replaced the restore driver's RestoreError, whose evidence attributes the failure, with the outcome report's category and status, or with the error of a report that it could not read, and a restore that failed otherwise printed the error's phase and evidence without its attribution. The start of a restored sandbox, which `snapshot restore`, a later `sandbox start` and `run --restore-snapshot` share, now keeps the restore driver's error and prints what its evidence attributes the failure to: after the VM's outcome when the VM exited by itself, as `OpenVMM exited during startup: guest-exit status 54; restore failed during Activate: UnusableRoot, attributed to the request`, after the error of an outcome report that it cannot read, and in the `snapshot restore failed` error otherwise. A cold start's messages and every error's code are unchanged. Tests: a_restore_that_fails_before_its_guest_is_ready_reports_its_attribution restores through the fake VMM, whose guest ends its VM with ACTIVATION_ROOT's status after an outcome report, or after one that cannot be parsed, or answers the Activate with a malformed Ready. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/usage.md | 10 +++ .../src/host/cli/sandbox/lifecycle/start.rs | 52 +++++++++++++-- .../sandbox/lifecycle/start/tests/restore.rs | 65 ++++++++++++++++++- 3 files changed, 120 insertions(+), 7 deletions(-) diff --git a/doc/usage.md b/doc/usage.md index 37758d1d6..331feefe0 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -494,6 +494,16 @@ is ready never completes, and the next command of the sandbox ends its VM. [`run --restore-snapshot`](#run) restores a snapshot once, in a sandbox that it removes. +A restore that fails before its guest is ready, in `snapshot restore`, a later +`sandbox start`, or `run --restore-snapshot`, fails with the restore's evidence +and what it attributes the failure to: the request, the host, the snapshot, the +guest's repair, or nothing that it establishes. A guest that cannot activate +the restore, such as one that cannot mount its scratch image as its upper, ends +its VM with the status of the failure's class, which `nvx::protocol::status` +defines, and the command reports the guest's exit with that evidence, such as +`OpenVMM exited during startup: guest-exit status 54; restore failed during +Activate: UnusableRoot, attributed to the request`. + ### `download` ```text diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs b/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs index 6125bc8bc..0ef21b4fe 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs @@ -38,7 +38,7 @@ use crate::host::config::{COMMAND_LINE_BUDGET, RootConfig, UpperConfig, VmConfig use crate::host::platform; use crate::host::process::KILL_TIMEOUT; use crate::host::report::Report; -use crate::host::restore::RestoreEvidence; +use crate::host::restore::{Attribution, RestoreError, RestoreEvidence}; use crate::host::restore_driver::{self, Cancel, ControlRestoreError, Workload}; use crate::host::runtime::{ArtifactIdentity, Runtime, RuntimeIdentity}; use crate::host::sandbox::distro::DISTRO_POLICY_REVISION; @@ -287,12 +287,14 @@ fn launch_locked( }; let maps = config.mapping_table(); let mut stopping = false; + // The restore driver's error of a restore that failed before its guest was ready. + let mut failure = None; let ready = starting .record_process(&saved.process) .map_err(error_with_cause) .and_then(|()| match (&activation, &restore) { (Some(record), Some(restore)) => { - restore_guest(&mut launched, record, restore, &maps, timeout) + restore_guest(&mut launched, record, restore, &maps, timeout, &mut failure) } _ => check_ready(&saved, runtime, timeout).and_then(|mut session| { if maps.is_empty() { @@ -340,9 +342,14 @@ fn launch_locked( lock.clear_runtime()?; // The VM has exited, which ends its claim of the scratch image and its use of the snapshot. drop((scratch, held)); - let report = stopped.report.map_err(error_with_cause)?; + let report = stopped + .report + .map_err(|report| with_attribution(error_with_cause(report), failure.as_ref()))?; if stopped.self_exited && !stopping && !matches!(error, Exit::Interrupted) { - return Err(exited_during_startup(report)); + return Err(with_attribution( + exited_during_startup(report), + failure.as_ref(), + )); } Err(error) } @@ -363,6 +370,32 @@ fn exited_during_startup(report: Option) -> Exit { ) } +/// Returns the error `error` of a restore that failed before its guest was ready, with the +/// restore driver's error `restore`, if it failed, and what its evidence, such as the guest's +/// status, attributes the failure to. +fn with_attribution(error: Exit, restore: Option<&RestoreError>) -> Exit { + match (error, restore) { + (Exit::Error { code, message }, Some(restore)) => { + Exit::error(code, format!("{message}; {}", attributed(restore))) + } + (error, _) => error, + } +} + +/// Returns the message of the restore driver's `error`, with what its evidence attributes the +/// restore's failure to: the request, the host, the snapshot, or the guest's repair, or nothing +/// that it establishes. +fn attributed(error: &RestoreError) -> String { + let attribution = match error.attribution { + Attribution::Request => "attributed to the request", + Attribution::Host => "attributed to the host", + Attribution::Snapshot => "attributed to the snapshot", + Attribution::Repair => "attributed to the guest's repair", + Attribution::Unknown => "of unknown attribution", + }; + format!("{error}, {attribution}") +} + /// Returns the error of a start whose distro layer or scratch image fails the check of its /// binding: a binding that no longer holds needs a new provision, which verifies the layer and /// binds the image again, while a scratch image that another sandbox claims, and a failure of the @@ -406,13 +439,16 @@ fn check_ready(saved: &SavedVm, runtime: &Runtime, timeout: f64) -> Result, maps: &[HostMap], timeout: f64, + failure: &mut Option, ) -> Result { let (uid, gid) = restore.identity; let workload = Workload { @@ -503,9 +539,13 @@ fn restore_guest( _ => ErrorCode::Protocol, }, }; + let message = match error { + ControlRestoreError::Machine(error) => attributed(failure.insert(error)), + error => error.to_string(), + }; Err(Exit::error( code, - format!("snapshot restore failed: {error}"), + format!("snapshot restore failed: {message}"), )) } diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs b/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs index d9a64a364..e6f939913 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs @@ -35,6 +35,7 @@ use crate::protocol::{ APP_ACTIVATE, APP_EXEC, APP_EXIT, APP_HEADER_LEN, APP_MAPS, APP_READY, APP_STDOUT, OUTER_DATA, OUTER_HEADER_LEN, OUTER_HOST_ATTACH, OUTER_MAX_PAYLOAD, OUTER_READY, decode_app, decode_outer_header, encode_app, encode_exec_payload, encode_maps_payloads, encode_outer, + status, }; /// The fake VMM and guest that the fixture's restores spawn in place of OpenVMM. @@ -155,7 +156,7 @@ impl Fixture { .unwrap(); let recipe = json!({ "endpoint": prepared.outputs().endpoint, "ready": ready, "behavior": behavior, - "trace": state_dir.with_extension("trace"), + "trace": state_dir.with_extension("trace"), "outcome": prepared.outputs().outcome, }); fs::write( state_dir.with_extension("json"), @@ -543,6 +544,47 @@ fn a_failed_restore_ends_its_vm_and_its_launch_before_its_snapshot_goes_free() { } } +/// A restore whose guest fails before it is ready reports what the restore's evidence attributes +/// the failure to: the request, for a guest that ends its VM with the status of a root that does +/// not assemble, after the VM's outcome, the host, for an outcome report that cannot be read, +/// after the report's error, and nothing that it establishes for a malformed Ready. +#[test] +fn a_restore_that_fails_before_its_guest_is_ready_reports_its_attribution() { + for (behavior, code, prefix, attribution) in [ + ( + "unusable-root", + ErrorCode::Exited, + "OpenVMM exited during startup: guest-exit status 54; restore failed during ", + ": UnusableRoot, attributed to the request", + ), + ( + "unreadable-report", + ErrorCode::Io, + "cannot parse outcome report ", + ": ReportUnreadable, attributed to the host", + ), + ( + "malformed", + ErrorCode::Protocol, + "snapshot restore failed: restore failed during ", + ": Malformed, of unknown attribution", + ), + ] { + let fixture = Fixture::new(false); + let error = fixture + .start("state", fixture.restore.config.clone(), behavior, 30.0) + .unwrap_err(); + assert_eq!(error.code(), Some(code), "{behavior}: {error:?}"); + let message = error.message().unwrap_or_default(); + assert!( + message.starts_with(prefix) && message.ends_with(attribution), + "{behavior}: {message}" + ); + assert!(fixture.exited(0), "{behavior}"); + assert!(fixture.snapshot_free(), "{behavior}"); + } +} + #[test] fn a_restore_whose_spawn_cannot_confirm_that_its_vm_ended_recovers_it_before_its_snapshot_goes_free() { @@ -943,6 +985,27 @@ fn serve(stream: &mut (impl Read + Write), capability: &[u8], epoch: u64, recipe } let replies = match request.kind { APP_ACTIVATE if recipe["behavior"] == "malformed" => vec![(APP_READY, 0, vec![0])], + // The guest cannot assemble its root, and powers off with the status of the failure's + // class, which OpenVMM reports, and then exits with, or reports unreadably. + APP_ACTIVATE + if matches!( + recipe["behavior"].as_str(), + Some("unusable-root" | "unreadable-report") + ) => + { + let code = status::ACTIVATION_ROOT; + let report = json!({ + "schema_version": 1, + "outcome": {"operation": "managed", "category": "guest-exit", "status_code": code}, + "teardown": {"workers": true}, + }); + let report = match recipe["behavior"].as_str() { + Some("unreadable-report") => "{ truncated".to_owned(), + _ => report.to_string(), + }; + fs::write(recipe["outcome"].as_str().unwrap(), report).unwrap(); + std::process::exit(code.into()); + } APP_ACTIVATE => { let ready = ReadyActivation::of(&activation(request.payload).0); vec![(APP_READY, 0, ready.encode())] From c1850747280ff203472173177e7331da4dc2e41c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 19:43:16 -0800 Subject: [PATCH 3/3] tests: check the attribution of a restore that fails in its guest snapshot-core's restore over a zeroed scratch image, which the guest cannot mount as its upper, required only that the CLI report the guest's exit with status 125, and the case matrix recorded the missing attribution as a known gap. The scenario now requires nvx-init's status of a root that does not assemble, 54, and that the CLI attribute the failure to the request from the restore's evidence, UnusableRoot, which gives back the coverage of the old adapter test a_clone_whose_activation_fails_ends_before_it_is_ready. Its fake CLI fails the scenario for a guest that ends with the generic status and for a failure attributed to another cause. The case matrix names the new tests in that test's row, and nvx-init's in the row of the adapter worker's test of each class's status; its Snapshots table describes the attributed failure in place of the known gap, which goes; and Run describes the scenario's new check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/acceptance-case-matrix.md | 14 ++++++-------- doc/run.md | 7 +++++-- scripts/nvx_tools/microvm_tests.py | 28 ++++++++++++++++++++++++---- scripts/test_microvm_tests.py | 29 +++++++++++++++++++++++++++-- 4 files changed, 62 insertions(+), 16 deletions(-) diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index 559def069..d37c2cacb 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -228,7 +228,7 @@ The adapter's README said what each proved. | `a_clone_runs_the_executions_that_name_their_identity_once_it_is_ready` | retired | Executions that name their identities leave with the exec engine (#1087) | | `a_clone_s_executions_find_the_proxy_of_its_policy` | rewritten | `nvx-rs/src/guest/managed_agent/launch/tests.rs::restored_proxy_defaults_replace_template_values_before_exec_overrides`; `test-microvm --scenario platform-network-policy` | | `a_clone_s_time_settles_after_it_is_ready` | rewritten | `test-microvm --scenario time-host-correction`; `test-microvm --scenario restore-downtime` | -| `a_clone_whose_activation_fails_ends_before_it_is_ready` | rewritten | `nvx-rs/src/host/restore/tests.rs::restore_failures_attribute_the_existing_policy_free_evidence_classes`; `nvx-rs/src/host/restore/tests.rs::restore_exit_reports_keep_guest_status_classification_and_actual_details`; `test-microvm --scenario snapshot-core`, whose restore over a zeroed scratch image ends its VM before its workload runs. Quarantine and leases retire (#1121) | +| `a_clone_whose_activation_fails_ends_before_it_is_ready` | rewritten | `nvx-rs/src/host/restore/tests.rs::restore_failures_attribute_the_existing_policy_free_evidence_classes`; `nvx-rs/src/host/restore/tests.rs::restore_exit_reports_keep_guest_status_classification_and_actual_details`; `nvx-rs/src/guest/init/boot/tests.rs::a_failed_activation_ends_the_vm_with_the_status_of_its_class`; `nvx-rs/src/guest/managed_agent/session/tests.rs::a_failed_restored_activation_gives_init_the_status_of_its_class`; `nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs::a_restore_that_fails_before_its_guest_is_ready_reports_its_attribution`; `test-microvm --scenario snapshot-core`, whose restore over a zeroed scratch image ends its VM before its workload runs, with nvx-init's status of an unusable root, 54, which the CLI attributes to the request. Quarantine and leases retire (#1121) | | `a_clone_whose_mapping_fails_ends_before_its_sandbox_starts` | rewritten | `nvx-rs/src/guest/mappings/mapper/tests.rs::a_failure_at_any_step_leaves_nothing_mounted`; `nvx-rs/src/guest/mappings/table/tests.rs::a_failed_mapping_keeps_those_before_it_and_latches_the_table` | | `a_clone_whose_view_cannot_be_set_ends_before_it_is_ready` | rewritten | `nvx-rs/src/guest/activation/state/apply/tests.rs::the_record_s_view_is_applied_once_with_its_own_dns_and_returned_machine_id`; `nvx-rs/src/guest/root/tests/view.rs::a_later_bind_failure_stays_owned_and_its_teardown_removes_the_earlier_bind` | | `a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released` | retired | Clone records and leases leave with the old clone path (#1121). A stop that finds an exit: `nvx-rs/src/host/vm/tests.rs::stop_preserves_report_results_after_a_real_self_exit` | @@ -275,7 +275,7 @@ checked the adapter's own code, and CI ran in its configurations of the adapter. | `src/template/tests.rs` | `template_mode_refuses_a_gateway_that_openvmm_would_not_render` | retired | The adapter's template mode; a NIC's address is checked as `NicConfig` is built | | `src/template/tests.rs` | `template_mode_refuses_quotes_and_the_delimiter` | rewritten | `nvx-rs/src/host/config/tests.rs::cold_boot_command_lines_refuse_native_control_console_syntax` | | `src/template/tests.rs` | `a_template_s_statuses_are_its_own`, `a_template_s_repair_delay_is_positive_milliseconds_of_at_most_half_a_minute` | retired | The adapter's template statuses and its clones' repair delay | -| `src/worker/activation/tests.rs` | `each_class_of_failure_ends_the_clone_with_its_own_status`, `the_activation_s_timeout_covers_the_wait_for_both_devices`, `the_activation_probe_reports_the_clone_s_events_after_the_acknowledgement` | retired | The adapter's guest worker. NVX's activation: `nvx-rs/src/guest/activation/state/tests.rs` and nvx-init's `nvx-rs/src/guest/init/tests/restore.rs` | +| `src/worker/activation/tests.rs` | `each_class_of_failure_ends_the_clone_with_its_own_status`, `the_activation_s_timeout_covers_the_wait_for_both_devices`, `the_activation_probe_reports_the_clone_s_events_after_the_acknowledgement` | retired | The adapter's guest worker. NVX's activation: `nvx-rs/src/guest/activation/state/tests.rs` and nvx-init's `nvx-rs/src/guest/init/tests/restore.rs`, and nvx-init's status of each class of failure: `nvx-rs/src/guest/init/boot/tests.rs::a_failed_activation_ends_the_vm_with_the_status_of_its_class` | | `src/worker/capture/tests.rs` | `closing_the_fence_acquires_it_until_it_opens`, `each_stage_of_a_capture_ends_the_worker_with_a_status_of_its_own`, `a_request_that_the_time_abi_could_not_repair_ends_with_the_violation_s_status`, `a_template_s_devices_are_those_of_a_root_and_the_host_filesystem`, `a_clone_assembles_its_root_with_the_template_s_configuration`, `the_worker_acknowledges_the_broker_s_reset_before_the_capture_and_reads_nothing_more`, `no_capture_runs_without_the_session_s_handshake` | retired | The adapter's guest worker and its admission fence, which leaves (#1087). NVX's capture: `nvx-rs/src/guest/snapshot/tests.rs` and nvx-init's `nvx-rs/src/guest/init/tests/capture.rs` | | `src/worker/entries/tests.rs` | `capabilities_are_named_without_their_prefix`, `status_fields_are_found_by_name`, `outcomes_name_the_error`, `refusals_name_what_refused_the_identity`, `unknown_probes_and_arguments_are_refused`, `absolute_paths_lose_their_leading_slash` | retired | The adapter's guest worker | | `src/worker/executions/tests.rs` | `a_cancellation_after_the_timeout_leaves_a_program_that_ended_before_it_its_own_end` | retired | The adapter's guest worker, over the exec engine that leaves (#1087) | @@ -398,14 +398,12 @@ capture`, whose VM nvx-init runs, and restore it with `nvx-rs snapshot restore` | --- | --- | | A real VM is captured, opened as a strict snapshot and restored with nvx-init | `snapshot-core`: two one-off restores and three managed ones, whose guests keep the snapshot's uptime, read the host's clock, and take fresh random values and machine IDs. `snapshot-tiers`: a managed restore of each image format, EROFS and GPT, over each upper, scratch and RAM | | Concurrent restores of one snapshot are serialized by the CLI's lock | `snapshot-core`: three restores at once, whose workloads end only when the scenario releases them. While one holds the lock, no other links its view of the snapshot, and a restore that cannot wait for the lock fails with `sandbox-busy` before its workload runs. The three then run at once, each with its own share, upper and machine ID | -| The selected failed restore reports its attribution | `snapshot-core`: a scratch image that the snapshot cannot serve is refused before any VM starts, with an error that names the lengths of the requested scratch and of the snapshot's, which attributes the failure to the request; the managed sandbox of the refused restore holds its configuration and its lock alone, with no trace of a launch. A restore that fails in its guest ends its VM, frees the lock, and reports `guest-exit status 125` | +| The selected failed restore reports its attribution | `snapshot-core`: a scratch image that the snapshot cannot serve is refused before any VM starts, with an error that names the lengths of the requested scratch and of the snapshot's, which attributes the failure to the request; the managed sandbox of the refused restore holds its configuration and its lock alone, with no trace of a launch. A restore over a zeroed scratch image fails in its guest, which cannot mount the image as its upper: its VM ends with nvx-init's status of an unusable root, 54, and frees the lock, and the CLI reports `guest-exit status 54` with the restore's evidence, `UnusableRoot`, which it attributes to the request | | A restore of a snapshot with a NIC installs a network policy of its own | `untiered-network-policy`: a snapshot that `snapshot capture --net` captures under the complete deny-all policy, restored four times with `snapshot restore` and `run --restore-snapshot`. Each guest names exactly its policy's DNS server and runs the announcer of its listeners only when its policy publishes them, reaches exactly the host services that its egress admits, and serves the host through a forward and through the port that its policy publishes, on host loopback alone or to the host's private networks. The CLI refuses a restore without `--network-profile`, and one with `--net`, before it provisions anything | -The CLI prints no attribution of a restore that fails in its guest beyond the guest's exit -status: nvx-init's status for each class of failure, and the CLI's report of the restore's -attribution, are a follow-up on #1118. A snapshot that `snapshot capture` captures without -`--net` has no NIC, so its restores take no network option; one captured with `--net` keeps its -NIC, and each of its restores installs the network policy of its own options on it. +A snapshot that `snapshot capture` captures without `--net` has no NIC, so its restores take no +network option; one captured with `--net` keeps its NIC, and each of its restores installs the +network policy of its own options on it. ## CI diff --git a/doc/run.md b/doc/run.md index 3c24659fb..0ba9e25f3 100644 --- a/doc/run.md +++ b/doc/run.md @@ -1154,8 +1154,11 @@ and the view of the snapshot, until `sandbox deprovision` removes it. A `run --restore-snapshot` over a zeroed scratch image of the dormant scratch's length, which the guest cannot mount as its upper, ends its VM before its workload runs and frees the snapshot's lock, which the restores that follow -take, and the run reports the guest's exit: -`OpenVMM exited during startup: guest-exit status 125`. +take: nvx-init ends the VM with the status of a root that does not assemble, +54, and the run reports the guest's exit with the restore's evidence and what +it attributes the failure to, the request, such as +`OpenVMM exited during startup: guest-exit status 54; restore failed during +Activate: UnusableRoot, attributed to the request`. Then three `snapshot restore` restores start at once, each with a host name and a read-only host share of its own, the last over a scratch upper and the diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index ba5144632..e49b145e2 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -2733,6 +2733,11 @@ def clean_up(self) -> list[Exception]: # How much longer than the capture, with the clocks' tolerance on either side, # the downtime before snapshot-core's first restore is. SNAPSHOT_CORE_DOWNTIME_MARGIN_SECONDS = 1.0 +# The status with which nvx-init ends the VM of a restore whose root does not +# assemble over its upper, ACTIVATION_ROOT of nvx::protocol::status, and how +# the CLI's error ends, with the restore's evidence and its attribution. +SNAPSHOT_CORE_UNUSABLE_ROOT_STATUS = 54 +SNAPSHOT_CORE_UNUSABLE_ROOT_ATTRIBUTION = ": UnusableRoot, attributed to the request" class SnapshotCoreAcceptance(SnapshotAcceptance): @@ -2748,7 +2753,9 @@ class SnapshotCoreAcceptance(SnapshotAcceptance): managed sandbox of the restore as provisioned, with an error that names the requested scratch, which attributes the failure to the request; a zeroed scratch image, which the guest cannot mount as its upper, ends - its VM before its workload runs, and the CLI reports the guest's exit. + its VM before its workload runs, with nvx-init's status of a root that + does not assemble, and the CLI reports the guest's exit with the restore's + evidence, which attributes the failure to the request. Three managed restores at once, each with a host share of its own, take turns at the snapshot's lock, and a one-off restore that cannot wait for it fails; the three then run at once, each with its own share, machine ID, and @@ -2832,8 +2839,11 @@ def check_failed_restores(self) -> None: and outcome report and the view of the snapshot. A one-off restore over a zeroed scratch image of the dormant scratch's length, which the guest cannot mount as its upper, ends its VM before its workload runs, - which the CLI reports as the guest's exit with status 125; the - restores that follow take the snapshot's lock, which it freed. + with nvx-init's status of a root that does not assemble, + SNAPSHOT_CORE_UNUSABLE_ROOT_STATUS, which the CLI reports as the + guest's exit with the restore's evidence of an unusable root, which it + attributes to the request; the restores that follow take the + snapshot's lock, which it freed. """ size = self.artifacts.scratch_template.stat().st_size short, zeroed = "short-scratch", "zeroed-scratch" @@ -2913,10 +2923,20 @@ def options(label: str, length: int) -> list[str]: *options(zeroed, size), expected=1, code="exited", - message="OpenVMM exited during startup: guest-exit status 125", + message=( + "OpenVMM exited during startup: guest-exit status " + f"{SNAPSHOT_CORE_UNUSABLE_ROOT_STATUS}; restore failed during " + ), ) if SNAPSHOT_PROBE_PREFIX in failed.stdout: raise RuntimeError(f"the {zeroed} restore ran its workload") + reported = nvx_rs.reported_error(failed.stderr) + assert reported is not None + if not reported.message.endswith(SNAPSHOT_CORE_UNUSABLE_ROOT_ATTRIBUTION): + raise RuntimeError( + f"the {zeroed} restore does not attribute its unusable root to the " + f"request: {reported.message!r}" + ) self.require_processes((), f"after the {zeroed} restore") def check_concurrent_restores(self) -> list[dict[str, str]]: diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 03ebf8d5a..cd4af4032 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3718,12 +3718,25 @@ def _start( os.link(directory / "openvmm" / "memory.bin", leftover) if zeroed: # The guest cannot mount the zeroed scratch as its upper, so its - # VM exits before the workload runs, and frees the lock. + # VM exits before the workload runs, and frees the lock, with + # nvx-init's status of a root that does not assemble, which the + # CLI reports with the restore's attribution. self.terminate(self._spawn()) if "failure-keeps-the-lock" in self.faults: locked = False + if "unclassified-guest-failure" in self.faults: + return self._error( + "exited", "OpenVMM exited during startup: guest-exit status 125" + ) + attribution = ( + "Io, attributed to the host" + if "misattributed-guest-failure" in self.faults + else "UnusableRoot, attributed to the request" + ) return self._error( - "exited", "OpenVMM exited during startup: guest-exit status 125" + "exited", + "OpenVMM exited during startup: guest-exit status 54; restore " + f"failed during Activate: {attribution}", ) guest = self._guest(values) self.guests[state] = guest @@ -4189,6 +4202,18 @@ def test_snapshot_core_detects_regressions_of_snapshots_and_their_restores(self) "zeroed-scratch: nvx-rs run --restore-snapshot returned 0, expected 1 " "with error[exited]" ), + # A guest that ends its VM with the generic status, or a CLI that + # attributes the failure to another cause. + "unclassified-guest-failure": ( + "zeroed-scratch: nvx-rs run --restore-snapshot returned 1, expected 1 " + "with error[exited]; stdout=''; stderr='error[exited]: OpenVMM exited " + "during startup: guest-exit status 125" + ), + "misattributed-guest-failure": ( + "the zeroed-scratch restore does not attribute its unusable root to the " + "request: 'OpenVMM exited during startup: guest-exit status 54; restore " + "failed during Activate: Io, attributed to the host'" + ), "unserialized-restores": ( "busy: nvx-rs run --restore-snapshot returned 0, expected 1 " "with error[sandbox-busy]"