diff --git a/doc/usage.md b/doc/usage.md index a0e64ec5d..12eb06dd9 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -110,6 +110,7 @@ each code for callers of the `nvx` library. | `template-ineligible` | 1 | The restore window of a template's generation, which the time contract of its capture sets, excludes the host's time, so no clone may restore it now. | | `template-tombstoned` | 1 | A tombstone quarantines the template's generation, which takes no new clones. | | `exited` | 1 | OpenVMM exited, as when a start fails, whose message names the category and status of OpenVMM's outcome report when it has one. | +| `still-running` | 1 | A stop could not confirm the VM process's exit, so the caller keeps its verified process. | | `exec-refused` | 1 | The guest agent refused the workload, as for a working directory that it cannot use. | | `endpoint-closed` | 1 | OpenVMM closed the control endpoint, as it does when the VM shuts down. | | `protocol` | 1 | The control session failed otherwise, as when the guest breaks the protocol or the session is reset. | diff --git a/nvx-rs/README.md b/nvx-rs/README.md index cd9c6f7ca..3019c6ebc 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -124,7 +124,7 @@ changes only when the module's purpose does. | [`nvx::Hypervisor`](#nvxhypervisor) | KVM, MSHV, or WHP, with its serde form behind the `serde` feature | | [`nvx::exec`](#nvxexec) | The execution engine: the streams, credit, and outcome of an execution's two sides, without I/O, for any framing | | [`nvx::host::runtime`](#nvxhostruntime) | `Runtime`: what every launch shares, the runtime's files and their identity, the hypervisor and its probe, and the launch options | -| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold preparation, verified detached launch, and caller-saved reattachment | +| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold preparation, verified detached launch, confirmed stop, and caller-saved reattachment | | [`nvx::host::snapshot`](#nvxhostsnapshot) | Snapshot compatibility keys, strict manifests, restore-time eligibility, and the opaque input of a future VM restore | | [`nvx::host::config`](#nvxhostconfig) | `VmConfig` and `CaptureConfig`: pure configuration, cold-boot kernel/OpenVMM arguments, portable length checks, and capture identity | | [`nvx::host::restore`](#nvxhostrestore) | A caller-driven restore to Ready without I/O or a clock, and its peer-verified restore-ready listener | @@ -500,6 +500,17 @@ or reading runtime files. `Reattached` holds both the process and authenticated connection. Failures retain their original kinds and sources and leave the saved values unchanged. +`stop(process, outputs, graceful, grace, deadline)` waits through the verified +process, first after the caller's graceful request, then after forced termination +if needed. It returns `Stopped` only after confirmed exit. `self_exited` records +confirmation before a forced attempt, and `forced` records that attempt, not +proof from kill success. An unconfirmed exit is `StillRunning`, and the caller +keeps the process. The callback receives the bounded grace deadline and must +respect it; its boolean result is never exit proof. + +`Stopped::report` retains the outcome-report read's result separately, so report +failure never hides confirmed exit. + `vm::create(&runtime, &config, None, &dir)` creates an inspectable cold `Prepared` launch without starting a process or binding a listener. It requires `config.boot`, refuses a snapshot input, rechecks the directory's privacy, and @@ -976,6 +987,10 @@ On Windows, the duplicate retains the original handle's access rights. Duplication failures have kind `Io` and retain their original I/O source. No watcher thread, future or callback is created by the library. +`Process::wait(deadline)` returns `Some(Exited)` only after verified exit, or +`None` when the deadline passes without confirmation. `Exited` cannot be made +by a caller. + `Process::bytes_read` reports how many bytes the process has read through the operating system's read calls, whether the host's cache served them or not: `rchar` of `/proc//io` on Linux, which it reads only while the process @@ -1422,6 +1437,11 @@ The authenticated control session, which carries NVX's control protocol, `NVXC`, one frame per DATA record of a `HostConnection`; Linux and Windows primitives; and OpenVMM's outcome report. +`report::read` returns `Ok(None)` for a missing report, and an `Io` error for an +unreadable or malformed report, retaining the underlying error. It keeps today's +summary of the outcome category, status code and teardown steps, not free-form +fields. + A `Transport` to an OpenVMM control endpoint reads and writes with deadlines, and tells, through `Transport::unread`, what the endpoint has not read yet of what it wrote, where the platform tells: on Linux, what `SIOCOUTQ` reports of @@ -3055,6 +3075,7 @@ optional cause. | `Unavailable` | A missing or unusable artifact, hypervisor, or release, a guest image without a required control feature, or an unsupported host | | `Unsupported` | A request that NVX refuses: a value that it does not accept, such as a malformed or out-of-range one, options that it cannot combine, or a request that it cannot represent or enforce, such as a policy beyond a limit of OpenVMM or the guest agent | | `Exited` | The VM's OpenVMM process exited | +| `StillRunning` | A stop could not confirm the VM process's exit, so the caller keeps its verified process | | `Io` | A host I/O failure, or state that NVX recorded on the host and cannot read | | `NotFound` | State that an operation names and that does not exist on the host, such as the store of a sandbox that was never created or that a removal deleted | | `Protocol` | A control session that failed, or a request that the guest agent refused | @@ -3087,6 +3108,7 @@ usage errors, which no library error has. | `unavailable` | `Unavailable` | A dependency is missing or unusable, other than as a finer code says: install metadata, the hypervisor, a release, a distro layer or scratch image that the request names, or a host that NVX does not support | | `unsupported` | `Unsupported` | NVX refuses the request: options that it cannot combine, a value that it does not accept, or a policy that it cannot represent or enforce. The message names the option or field | | `exited` | `Exited` | OpenVMM exited, as when a start fails, whose message names the category and status of OpenVMM's outcome report when it has one | +| `still-running` | `StillRunning` | A stop could not confirm the VM process's exit, so the caller keeps its verified process | | `io` | `Io` | A host I/O operation failed, or state that NVX recorded on the host is unreadable or malformed | | `not-found` | `NotFound` | Other state that the operation names does not exist on the host | | `protocol` | `Protocol` | The control session failed otherwise: the peer violated the protocol, reset the session, or failed authentication | diff --git a/nvx-rs/public-api/linux.txt b/nvx-rs/public-api/linux.txt index 6a422c570..2d90a5759 100644 --- a/nvx-rs/public-api/linux.txt +++ b/nvx-rs/public-api/linux.txt @@ -58,6 +58,7 @@ variant nvx::ErrorCode::SandboxNotRunning variant nvx::ErrorCode::SandboxProvisioned variant nvx::ErrorCode::SandboxRunning variant nvx::ErrorCode::ScratchClaimed +variant nvx::ErrorCode::StillRunning variant nvx::ErrorCode::TemplateIneligible variant nvx::ErrorCode::TemplateTombstoned variant nvx::ErrorCode::TimedOut @@ -73,6 +74,7 @@ variant nvx::ErrorKind::NotFound variant nvx::ErrorKind::Other variant nvx::ErrorKind::Protocol variant nvx::ErrorKind::RuntimeMismatch +variant nvx::ErrorKind::StillRunning variant nvx::ErrorKind::TimedOut variant nvx::ErrorKind::Unavailable variant nvx::ErrorKind::Unsupported @@ -2445,6 +2447,7 @@ method nvx::host::process::BootId::as_bytes method nvx::host::process::BootId::current method nvx::host::process::BootId::from_bytes type nvx::host::process::ExitHandle +struct nvx::host::process::Exited struct nvx::host::process::Process method nvx::host::process::Process::bytes_read method nvx::host::process::Process::exit_handle @@ -3768,6 +3771,10 @@ field nvx::host::vm::SavedVm::endpoint field nvx::host::vm::SavedVm::process field nvx::host::vm::SavedVm::runtime field nvx::host::vm::SavedVm::vm_dir +struct nvx::host::vm::Stopped +field nvx::host::vm::Stopped::forced +field nvx::host::vm::Stopped::report +field nvx::host::vm::Stopped::self_exited struct nvx::host::vm::VmDir method nvx::host::vm::VmDir::open method nvx::host::vm::VmDir::path @@ -3780,6 +3787,7 @@ fn nvx::host::vm::create fn nvx::host::vm::create_private_dir fn nvx::host::vm::reattach fn nvx::host::vm::spawn +fn nvx::host::vm::stop fn nvx::host::vm::verify mod nvx::net struct nvx::net::Cidr diff --git a/nvx-rs/public-api/windows.txt b/nvx-rs/public-api/windows.txt index 9aa7bf0f8..c1e2bc8cf 100644 --- a/nvx-rs/public-api/windows.txt +++ b/nvx-rs/public-api/windows.txt @@ -58,6 +58,7 @@ variant nvx::ErrorCode::SandboxNotRunning variant nvx::ErrorCode::SandboxProvisioned variant nvx::ErrorCode::SandboxRunning variant nvx::ErrorCode::ScratchClaimed +variant nvx::ErrorCode::StillRunning variant nvx::ErrorCode::TemplateIneligible variant nvx::ErrorCode::TemplateTombstoned variant nvx::ErrorCode::TimedOut @@ -73,6 +74,7 @@ variant nvx::ErrorKind::NotFound variant nvx::ErrorKind::Other variant nvx::ErrorKind::Protocol variant nvx::ErrorKind::RuntimeMismatch +variant nvx::ErrorKind::StillRunning variant nvx::ErrorKind::TimedOut variant nvx::ErrorKind::Unavailable variant nvx::ErrorKind::Unsupported @@ -1031,6 +1033,7 @@ method nvx::host::process::BootId::as_bytes method nvx::host::process::BootId::current method nvx::host::process::BootId::from_bytes type nvx::host::process::ExitHandle +struct nvx::host::process::Exited struct nvx::host::process::Process method nvx::host::process::Process::bytes_read method nvx::host::process::Process::exit_handle @@ -2353,6 +2356,10 @@ field nvx::host::vm::SavedVm::endpoint field nvx::host::vm::SavedVm::process field nvx::host::vm::SavedVm::runtime field nvx::host::vm::SavedVm::vm_dir +struct nvx::host::vm::Stopped +field nvx::host::vm::Stopped::forced +field nvx::host::vm::Stopped::report +field nvx::host::vm::Stopped::self_exited struct nvx::host::vm::VmDir method nvx::host::vm::VmDir::open method nvx::host::vm::VmDir::path @@ -2365,6 +2372,7 @@ fn nvx::host::vm::create fn nvx::host::vm::create_private_dir fn nvx::host::vm::reattach fn nvx::host::vm::spawn +fn nvx::host::vm::stop fn nvx::host::vm::verify mod nvx::net struct nvx::net::Cidr diff --git a/nvx-rs/src/error.rs b/nvx-rs/src/error.rs index 989538bab..e98ae8830 100644 --- a/nvx-rs/src/error.rs +++ b/nvx-rs/src/error.rs @@ -19,6 +19,8 @@ pub enum ErrorKind { Unsupported, /// The OpenVMM process of a VM exited. Exited, + /// The exit of a VM's process could not be confirmed; it may still run. + StillRunning, /// A host I/O operation failed, or state that NVX recorded on the host is unreadable or /// malformed. Io, @@ -44,6 +46,7 @@ impl ErrorKind { Self::Unavailable => "unavailable", Self::Unsupported => "unsupported", Self::Exited => "exited", + Self::StillRunning => "still-running", Self::Io => "io", Self::NotFound => "not-found", Self::Protocol => "protocol", @@ -88,6 +91,8 @@ pub enum ErrorCode { Unsupported, /// `exited`: the OpenVMM process of a VM exited, as when a start fails. Exited, + /// `still-running`: the exit of a VM's process could not be confirmed. + StillRunning, /// `io`: a host I/O operation failed, or state that NVX recorded on the host is unreadable or /// malformed. Io, @@ -170,6 +175,7 @@ impl ErrorCode { Self::Unavailable, Self::Unsupported, Self::Exited, + Self::StillRunning, Self::Io, Self::NotFound, Self::Protocol, @@ -205,6 +211,7 @@ impl ErrorCode { Self::Unavailable => ErrorKind::Unavailable.as_str(), Self::Unsupported => ErrorKind::Unsupported.as_str(), Self::Exited => ErrorKind::Exited.as_str(), + Self::StillRunning => ErrorKind::StillRunning.as_str(), Self::Io => ErrorKind::Io.as_str(), Self::NotFound => ErrorKind::NotFound.as_str(), Self::Protocol => ErrorKind::Protocol.as_str(), @@ -248,6 +255,7 @@ impl ErrorCode { | Self::ArgumentMissing | Self::ArgumentInvalid => ErrorKind::Unsupported, Self::Exited | Self::SandboxExited => ErrorKind::Exited, + Self::StillRunning => ErrorKind::StillRunning, Self::Io | Self::FormatUnsupported => ErrorKind::Io, Self::NotFound | Self::SandboxNotProvisioned => ErrorKind::NotFound, Self::Protocol | Self::ExecRefused | Self::EndpointClosed => ErrorKind::Protocol, @@ -278,6 +286,7 @@ impl From for ErrorCode { ErrorKind::Unavailable => Self::Unavailable, ErrorKind::Unsupported => Self::Unsupported, ErrorKind::Exited => Self::Exited, + ErrorKind::StillRunning => Self::StillRunning, ErrorKind::Io => Self::Io, ErrorKind::NotFound => Self::NotFound, ErrorKind::Protocol => Self::Protocol, diff --git a/nvx-rs/src/error/tests.rs b/nvx-rs/src/error/tests.rs index eb87d7dc0..a893e72b7 100644 --- a/nvx-rs/src/error/tests.rs +++ b/nvx-rs/src/error/tests.rs @@ -52,6 +52,7 @@ fn every_kind_has_a_code_of_its_own_spelling() { ErrorKind::Unavailable, ErrorKind::Unsupported, ErrorKind::Exited, + ErrorKind::StillRunning, ErrorKind::Io, ErrorKind::NotFound, ErrorKind::Protocol, diff --git a/nvx-rs/src/host/instance/locked.rs b/nvx-rs/src/host/instance/locked.rs index fca2a638b..2cbee7f48 100644 --- a/nvx-rs/src/host/instance/locked.rs +++ b/nvx-rs/src/host/instance/locked.rs @@ -216,7 +216,8 @@ impl LockedInstance { }; Ok(Stopped { forced, - report: report::read(&self.instance.report_path()), + // This legacy wrapper keeps its best-effort report policy. + report: report::read(&self.instance.report_path()).ok().flatten(), }) } diff --git a/nvx-rs/src/host/platform/windows/tests.rs b/nvx-rs/src/host/platform/windows/tests.rs index 44dd7359a..7da84f781 100644 --- a/nvx-rs/src/host/platform/windows/tests.rs +++ b/nvx-rs/src/host/platform/windows/tests.rs @@ -16,7 +16,8 @@ use windows_sys::Win32::Foundation::{ use windows_sys::Win32::Security::SECURITY_ATTRIBUTES; use windows_sys::Win32::System::JobObjects::CreateJobObjectW; use windows_sys::Win32::System::Threading::{ - CreateEventW, GetCurrentProcess, INFINITE, OpenProcess, PROCESS_DUP_HANDLE, TerminateProcess, + CreateEventW, GetCurrentProcess, INFINITE, OpenProcess, PROCESS_DUP_HANDLE, + PROCESS_QUERY_LIMITED_INFORMATION, PROCESS_SYNCHRONIZE, TerminateProcess, }; pub(crate) use super::endpoint::tests::{accept_control, control_server}; @@ -38,6 +39,16 @@ pub(crate) fn wait_exit_handle(process: OwnedHandle, deadline: Instant) -> io::R .map(|exited| exited.expect("Windows process handles always provide an exit event")) } +impl ProcessHandle { + pub(crate) fn wait_only(pid: u32) -> Self { + Self { + process: open_handle(pid, PROCESS_QUERY_LIMITED_INFORMATION | PROCESS_SYNCHRONIZE) + .unwrap(), + terminate: false, + } + } +} + thread_local! { /// What a spawn of this thread runs in its window, once its handles are inheritable and /// before it creates its process. @@ -1084,7 +1095,7 @@ fn a_launch_whose_vmm_exited_ends_while_a_process_spawned_in_its_window_holds_it let vmm = launch.record_process(&launched).unwrap(); launched.detach(); assert!( - vmm.wait(Instant::now() + WINDOW_TIMEOUT).unwrap(), + vmm.wait(Instant::now() + WINDOW_TIMEOUT).unwrap().is_some(), "the stand-in did not exit" ); // The VMM has exited, so the launch ends, whatever the other process holds. diff --git a/nvx-rs/src/host/process.rs b/nvx-rs/src/host/process.rs index 32047f8d6..100abe93d 100644 --- a/nvx-rs/src/host/process.rs +++ b/nvx-rs/src/host/process.rs @@ -188,6 +188,12 @@ pub struct Process { handle: platform::ProcessHandle, } +/// Proof that a verified process exited, produced only by [`Process::wait`]. +/// +/// It proves exit, not the process's exit status, and cannot be constructed by a caller. +#[derive(Debug)] +pub struct Exited(()); + impl Process { /// Finds the process that `reference` names, and returns `None` if it has exited. /// @@ -241,13 +247,17 @@ impl Process { }) } - /// Waits until the process exits or `deadline` passes, and returns whether it exited. + /// Waits until the process exits or `deadline` passes, returning exit proof or `None`. /// /// A process exits with its last thread. The wait sleeps until the exit event of the /// process, or with [`WaitMethod::Polling`], checks its identity at a short interval. An /// error means that the wait cannot determine whether the process exited. - pub fn wait(&self, deadline: Instant) -> io::Result { - self.wait_bool(deadline) + pub fn wait(&self, deadline: Instant) -> io::Result> { + Ok(if self.wait_bool(deadline)? { + Some(Exited(())) + } else { + None + }) } /// Keeps the boolean wait result for the legacy lifecycle callers. @@ -262,7 +272,7 @@ impl Process { /// Returns whether the process has exited, without waiting. pub fn has_exited(&self) -> io::Result { - self.wait(Instant::now()) + self.wait(Instant::now()).map(|exited| exited.is_some()) } /// Ends the process at once, unless it has exited or is exiting: with `SIGKILL` on Linux, and diff --git a/nvx-rs/src/host/process/tests.rs b/nvx-rs/src/host/process/tests.rs index ebc305ffa..a90c83102 100644 --- a/nvx-rs/src/host/process/tests.rs +++ b/nvx-rs/src/host/process/tests.rs @@ -275,7 +275,7 @@ fn hosts_without_pidfds_poll_and_never_signal() { let still_running = process.has_exited(); child.kill().unwrap(); child.wait().unwrap(); - assert!(!running.unwrap()); + assert!(running.unwrap().is_none()); assert_eq!(refused.unwrap_err().kind(), io::ErrorKind::Unsupported); assert!(!still_running.unwrap()); // Once the child is reaped, its ID names no process, or another one. @@ -283,6 +283,7 @@ fn hosts_without_pidfds_poll_and_never_signal() { process .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); } diff --git a/nvx-rs/src/host/process/tests/linux.rs b/nvx-rs/src/host/process/tests/linux.rs index 4c9177b53..a2a0cf72f 100644 --- a/nvx-rs/src/host/process/tests/linux.rs +++ b/nvx-rs/src/host/process/tests/linux.rs @@ -152,13 +152,14 @@ fn a_process_runs_until_its_last_thread_exits() { .unwrap() .expect("a process whose main thread exited still runs"); assert!( - !process + process .wait(Instant::now() + Duration::from_millis(100)) .unwrap() + .is_none() ); child.0.kill().unwrap(); child.0.wait().unwrap(); - assert!(process.wait(Instant::now()).unwrap()); + assert!(process.wait(Instant::now()).unwrap().is_some()); } #[test] diff --git a/nvx-rs/src/host/report.rs b/nvx-rs/src/host/report.rs index 48f86fec4..f07bf2ee4 100644 --- a/nvx-rs/src/host/report.rs +++ b/nvx-rs/src/host/report.rs @@ -9,6 +9,8 @@ use std::path::Path; use serde_json::Value; +use crate::{Error, Result}; + /// Summary of an outcome report. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct Report { @@ -20,9 +22,24 @@ pub struct Report { pub teardown_complete: Option, } -/// Reads the report at `path`, or returns `None` if it is missing or not JSON. -pub fn read(path: &Path) -> Option { - let report: Value = serde_json::from_slice(&fs::read(path).ok()?).ok()?; +/// Reads the report at `path`, or returns `Ok(None)` if it is missing. +/// +/// An unreadable or malformed report fails with [`ErrorKind::Io`](crate::ErrorKind::Io), +/// retaining the underlying filesystem or JSON error. +pub fn read(path: &Path) -> Result> { + let bytes = match fs::read(path) { + Ok(bytes) => bytes, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => { + return Err( + Error::io(format!("cannot read outcome report {}", path.display())) + .with_source(error), + ); + } + }; + let report: Value = serde_json::from_slice(&bytes).map_err(|error| { + Error::io(format!("cannot parse outcome report {}", path.display())).with_source(error) + })?; let mut summary = Report::default(); if let Some(outcome) = report.get("outcome") { summary.category = outcome @@ -35,7 +52,7 @@ pub fn read(path: &Path) -> Option { let complete = teardown.values().all(|step| step.as_bool() == Some(true)); summary.teardown_complete = Some(complete); } - Some(summary) + Ok(Some(summary)) } #[cfg(test)] diff --git a/nvx-rs/src/host/report/tests.rs b/nvx-rs/src/host/report/tests.rs index 127fd5b2e..961377aa4 100644 --- a/nvx-rs/src/host/report/tests.rs +++ b/nvx-rs/src/host/report/tests.rs @@ -11,7 +11,7 @@ fn reports_are_summarized_without_free_form_fields() { ) .unwrap(); assert_eq!( - read(&path).unwrap(), + read(&path).unwrap().unwrap(), Report { category: Some("success".to_owned()), status_code: Some(0), @@ -20,13 +20,60 @@ fn reports_are_summarized_without_free_form_fields() { ); fs::write(&path, r#"{"teardown":{"workers":true,"memory":false}}"#).unwrap(); assert_eq!( - read(&path).unwrap(), + read(&path).unwrap().unwrap(), Report { teardown_complete: Some(false), ..Report::default() } ); - assert!(read(&directory.path().join("missing.json")).is_none()); +} + +#[test] +fn missing_reports_are_distinct_from_malformed_or_unreadable_reports() { + let directory = tempfile::tempdir().unwrap(); + assert!( + read(&directory.path().join("missing.json")) + .unwrap() + .is_none() + ); + let path = directory.path().join("outcome.json"); fs::write(&path, b"{ truncated").unwrap(); - assert!(read(&path).is_none()); + let malformed = read(&path).unwrap_err(); + assert_eq!(malformed.kind(), crate::ErrorKind::Io); + assert!( + std::error::Error::source(&malformed) + .unwrap() + .is::() + ); + let unreadable = read(directory.path()).unwrap_err(); + assert_eq!(unreadable.kind(), crate::ErrorKind::Io); + assert!( + std::error::Error::source(&unreadable) + .unwrap() + .is::() + ); +} + +#[cfg(windows)] +#[test] +fn a_report_opened_without_read_sharing_returns_its_io_error() { + use std::os::windows::fs::OpenOptionsExt; + + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("outcome.json"); + fs::write(&path, b"{}").unwrap(); + let held = fs::OpenOptions::new() + .read(true) + .share_mode(0) + .open(&path) + .unwrap(); + let error = read(&path).unwrap_err(); + assert_eq!(error.kind(), crate::ErrorKind::Io); + assert!( + std::error::Error::source(&error) + .unwrap() + .is::() + ); + drop(held); + assert_eq!(read(&path).unwrap(), Some(Report::default())); } diff --git a/nvx-rs/src/host/template/clone/records.rs b/nvx-rs/src/host/template/clone/records.rs index ce9752576..dc5fbc696 100644 --- a/nvx-rs/src/host/template/clone/records.rs +++ b/nvx-rs/src/host/template/clone/records.rs @@ -296,7 +296,7 @@ impl Recorder { /// report only once the clone's VM ended, or its teardown did. pub(super) fn observed(&self) -> Option<&Report> { self.observed - .get_or_init(|| report::read(&self.report)) + .get_or_init(|| report::read(&self.report).ok().flatten()) .as_ref() } @@ -1119,7 +1119,7 @@ impl CloneRecords { // The launch removed the report of an earlier VM of the sandbox before it wrote the first // of the clone's records, which a progress past the plan shows. let report = (progress > Progress::Planned) - .then(|| report::read(&sandbox.outcome_path())) + .then(|| report::read(&sandbox.outcome_path()).ok().flatten()) .flatten(); Ok(Some(Self { lease, diff --git a/nvx-rs/src/host/template/clone/tests/launch.rs b/nvx-rs/src/host/template/clone/tests/launch.rs index c58043a4c..7ec87eec6 100644 --- a/nvx-rs/src/host/template/clone/tests/launch.rs +++ b/nvx-rs/src/host/template/clone/tests/launch.rs @@ -365,14 +365,14 @@ impl VmmDir { .unwrap() .expect("the VMM runs until the test ends it"); self.end(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); } /// Waits until the VMM, which exits on its own, as its [plan](Self::plan) asks, has exited. pub(super) fn wait_exit(&self) { let reference = self.reference.as_ref().expect("the test knows the VMM"); if let Some(process) = Process::open(reference).unwrap() { - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); } } @@ -413,7 +413,9 @@ impl Drop for VmmDir { let _ = fs::write(self.path().join(EXIT_MARKER), b""); if let Some(reference) = &self.reference && let Ok(Some(process)) = Process::open(reference) - && !process.wait(Instant::now() + TIMEOUT).unwrap_or(false) + && !process + .wait(Instant::now() + TIMEOUT) + .is_ok_and(|exited| exited.is_some()) { let _ = process.kill(); let _ = process.wait(Instant::now() + TIMEOUT); @@ -1567,7 +1569,7 @@ fn a_clone_whose_launch_failed_before_it_began_launches_no_more_until_its_releas // clone's: an inspection takes none, and the launch removes it before it records anything. let report = acquired.locked.outcome_path(); fs::write(&report, guest_exit(195)).unwrap(); - assert!(crate::host::report::read(&report).is_some()); + assert!(crate::host::report::read(&report).unwrap().is_some()); let records = CloneRecords::read(&acquired.locked).unwrap().unwrap(); assert_eq!( (records.progress, records.report), @@ -2323,7 +2325,7 @@ fn a_launch_whose_caller_crashes_once_its_vmm_started_ends_the_vmm_as_reattach_r } => assert_eq!(terminated, reference), other => panic!("unexpected reattach: {other:?}"), } - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); assert!(!exists(&crashed.socket())); crashed.releases(&mut locked); } @@ -2356,7 +2358,10 @@ fn a_restore_whose_caller_crashes_before_its_commit_ends_its_vmm_as_reattach_rec } => assert_eq!(terminated, reference, "{step}"), other => panic!("unexpected reattach at {step}: {other:?}"), } - assert!(process.wait(Instant::now() + TIMEOUT).unwrap(), "{step}"); + assert!( + process.wait(Instant::now() + TIMEOUT).unwrap().is_some(), + "{step}" + ); assert!(!exists(&crashed.socket()), "{step}"); crashed.releases(&mut locked); } diff --git a/nvx-rs/src/host/template/clone/tests/restore.rs b/nvx-rs/src/host/template/clone/tests/restore.rs index 0a33cb1dd..61282c561 100644 --- a/nvx-rs/src/host/template/clone/tests/restore.rs +++ b/nvx-rs/src/host/template/clone/tests/restore.rs @@ -422,7 +422,7 @@ fn vmm_with(plan: &str) -> VmmDir { fn assert_vmm_ended(vmm: &VmmDir) { let reference = vmm.reference.as_ref().unwrap(); if let Some(process) = Process::open(reference).unwrap() { - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); } } diff --git a/nvx-rs/src/host/template/store/leases/tests.rs b/nvx-rs/src/host/template/store/leases/tests.rs index 1205e4ba3..eaeed151c 100644 --- a/nvx-rs/src/host/template/store/leases/tests.rs +++ b/nvx-rs/src/host/template/store/leases/tests.rs @@ -2460,7 +2460,9 @@ impl Drop for VmmDir { let _ = fs::write(self.path().join(EXIT_MARKER), b""); if let Some(reference) = &self.reference && let Ok(Some(process)) = Process::open(reference) - && !process.wait(Instant::now() + TIMEOUT).unwrap_or(false) + && !process + .wait(Instant::now() + TIMEOUT) + .is_ok_and(|exited| exited.is_some()) { let _ = process.kill(); let _ = process.wait(Instant::now() + TIMEOUT); @@ -2639,7 +2641,7 @@ fn a_clone_that_crashes_after_its_spawn_keeps_its_lease_until_its_unrecorded_vmm crashed.refuses_release(&mut locked); // Once the VMM exits, the claim that it released proves it. crashed.vmm.end(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); match reattach_until_determinate(&mut locked) { Reattached::Exited { record: None, @@ -2669,7 +2671,7 @@ fn a_clone_that_crashes_after_recording_its_vmm_keeps_its_lease_until_reattach_e } => assert_eq!(terminated, reference), other => panic!("unexpected reattach: {other:?}"), } - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); crashed.releases(&mut locked); } diff --git a/nvx-rs/src/host/vm.rs b/nvx-rs/src/host/vm.rs index 1429a7a26..72e724aa6 100644 --- a/nvx-rs/src/host/vm.rs +++ b/nvx-rs/src/host/vm.rs @@ -16,6 +16,7 @@ use std::fmt; use std::fs::{self, File, OpenOptions}; use std::io; use std::path::{Path, PathBuf}; +use std::time::Instant; use crate::host::config::{ConsoleAttachment, RootConfig, UpperConfig, VmConfig}; use crate::host::filesystem; @@ -25,6 +26,7 @@ use crate::host::platform; use crate::host::platform::{check_store_dir as check_dir, create_store_dir as create_dir}; #[cfg(windows)] use crate::host::platform::{check_vm_dir as check_dir, create_vm_dir as create_dir}; +use crate::host::report::{self, Report}; use crate::host::runtime::{Runtime, RuntimeIdentity}; use crate::host::snapshot::Snapshot; use crate::host::spawn::{self, VmmCommand}; @@ -522,6 +524,78 @@ pub fn spawn(prepared: Prepared) -> Result { spawn_verified(prepared, spawn::Launched::process) } +/// A confirmed VM exit, with the outcome-report read kept separate from exit proof. +#[derive(Debug)] +pub struct Stopped { + /// Exit was confirmed before any forced termination was attempted. + pub self_exited: bool, + /// Forced termination through the verified process was attempted. + pub forced: bool, + /// The outcome summary, missing report, or report-read error after confirmed exit. + pub report: Result>, +} + +/// Stops a VM, returning only after its verified process has exited. +/// +/// Unless exit is already confirmed, calls `graceful` with `grace`, bounded by `deadline`, +/// then waits until that grace deadline. The callback must respect the supplied deadline; +/// its boolean request result is not exit proof. If exit is still unconfirmed, attempts +/// forced termination and waits until `deadline`. A failed termination is harmless only +/// when that final wait confirms exit. +/// +/// An unconfirmed exit fails with [`ErrorKind::StillRunning`](crate::ErrorKind::StillRunning), +/// retaining host errors. The caller keeps the process in that case. +/// Report-read errors have kind [`ErrorKind::Io`](crate::ErrorKind::Io) +/// inside [`Stopped::report`], never hiding confirmed exit. +pub fn stop( + process: &Process, + outputs: &VmOutputs, + graceful: impl FnOnce(Instant) -> bool, + grace: Instant, + deadline: Instant, +) -> Result { + let failure = |kill_error: Option<&io::Error>| { + let mut message = format!( + "cannot confirm exit of VM process {}", + process.reference().pid + ); + if let Some(error) = kill_error { + message.push_str(&format!("; forced termination failed: {error}")); + } + Error::new(crate::ErrorKind::StillRunning, message) + }; + let wait = |until| { + process + .wait(until) + .map_err(|error| failure(None).with_source(error)) + }; + let self_exited = if wait(Instant::now())?.is_some() { + true + } else { + let grace = grace.min(deadline); + graceful(grace); + wait(grace)?.is_some() + }; + if !self_exited { + let kill_error = process.kill().err(); + let exited = process + .wait(deadline) + .map_err(|error| failure(kill_error.as_ref()).with_source(error))?; + if exited.is_none() { + let error = failure(kill_error.as_ref()); + return Err(match kill_error { + Some(source) => error.with_source(source), + None => error, + }); + } + } + Ok(Stopped { + self_exited, + forced: !self_exited, + report: report::read(&outputs.outcome), + }) +} + /// Keeps the child owned until its process identity is verified, or its exit is confirmed. fn spawn_verified( prepared: Prepared, diff --git a/nvx-rs/src/host/vm/tests.rs b/nvx-rs/src/host/vm/tests.rs index 05af49bd0..b637a7532 100644 --- a/nvx-rs/src/host/vm/tests.rs +++ b/nvx-rs/src/host/vm/tests.rs @@ -1098,6 +1098,7 @@ fn spawn_caller_helper() { .process .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); panic!("cannot record the helper's child: {error}"); } @@ -1181,6 +1182,7 @@ fn an_unrelated_process_started_in_the_library_spawn_window_gets_no_vm_claim() { .process .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); assert!( platform::tests::claim_released_within(&log, identity, Duration::from_secs(10)), @@ -1282,10 +1284,193 @@ fn current_cold_spawn_boots_a_guest_accepts_shell_input_and_exits_cleanly() { ) .unwrap(); read_until(&mut *console, b"NVX-SPAWN-COLD-OK"); - assert!(launched.0.process.wait(deadline).unwrap()); + assert!(launched.0.process.wait(deadline).unwrap().is_some()); assert!(reopened.has_exited().unwrap()); - let outcome = crate::host::report::read(&launched.0.outputs.outcome).unwrap(); + let outcome = crate::host::report::read(&launched.0.outputs.outcome) + .unwrap() + .unwrap(); assert_eq!(outcome.category.as_deref(), Some("success")); assert_eq!(outcome.status_code, Some(0)); println!("NVX-LIBRARY-COLD-SPAWN-EXIT-OK: {}", reference.pid); } + +#[cfg(any(target_os = "linux", windows))] +struct StopChild { + child: std::process::Child, + process: Process, +} + +#[cfg(any(target_os = "linux", windows))] +impl StopChild { + fn start() -> Self { + use std::io::{BufRead, BufReader}; + + #[cfg(windows)] + let mut command = { + let mut command = Command::new( + PathBuf::from(std::env::var_os("SystemRoot").unwrap()).join("System32\\cmd.exe"), + ); + command.args(["/d", "/q", "/c", "echo ready & set /p wait="]); + command + }; + #[cfg(target_os = "linux")] + let mut command = { + let mut command = Command::new("sh"); + command.args(["-c", "printf 'ready\\n'; read -r wait"]); + command + }; + command + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()); + let mut child = crate::host::tests::spawn_locked(&mut command).unwrap(); + let mut ready = String::new(); + BufReader::new(child.stdout.take().unwrap()) + .read_line(&mut ready) + .unwrap(); + assert_eq!(ready.trim_end(), "ready"); + let reference = crate::host::process::ProcessRef { + pid: child.id(), + start_time: platform::process_start_time(child.id()).unwrap().unwrap(), + boot: crate::host::process::BootId::current().unwrap(), + }; + let process = Process::open(&reference).unwrap().unwrap(); + Self { child, process } + } +} + +#[cfg(any(target_os = "linux", windows))] +impl Drop for StopChild { + fn drop(&mut self) { + if self.child.try_wait().unwrap().is_none() { + self.child.kill().unwrap(); + } + self.child.wait().unwrap(); + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn stop_preserves_report_results_after_a_real_self_exit() { + for report in ["missing", "valid", "malformed", "unreadable"] { + let (_directory, vm) = vm_dir(); + let outputs = vm.outputs(false).unwrap(); + match report { + "missing" => {} + "valid" => fs::write( + &outputs.outcome, + br#"{"outcome":{"category":"success","status_code":0},"teardown":{"workers":true}}"#, + ).unwrap(), + "malformed" => fs::write(&outputs.outcome, b"{ truncated").unwrap(), + "unreadable" => fs::create_dir(&outputs.outcome).unwrap(), + _ => unreachable!(), + } + let mut child = StopChild::start(); + let deadline = Instant::now() + Duration::from_secs(10); + let stopped = stop( + &child.process, + &outputs, + |supplied| { + assert_eq!(supplied, deadline); + drop(child.child.stdin.take()); + false + }, + deadline, + deadline, + ) + .unwrap(); + assert!(stopped.self_exited); + assert!(!stopped.forced); + assert!(child.process.wait(Instant::now()).unwrap().is_some()); + assert!(child.child.try_wait().unwrap().is_some()); + let already_exited = stop( + &child.process, + &outputs, + |_| panic!("an exited process needs no graceful request"), + Instant::now(), + Instant::now(), + ) + .unwrap(); + assert!(already_exited.self_exited); + assert!(!already_exited.forced); + match report { + "missing" => assert!(stopped.report.unwrap().is_none()), + "valid" => assert_eq!( + stopped.report.unwrap(), + Some(Report { + category: Some("success".to_owned()), + status_code: Some(0), + teardown_complete: Some(true), + }) + ), + _ => { + let error = stopped.report.unwrap_err(); + assert_eq!(error.kind(), ErrorKind::Io); + assert!(std::error::Error::source(&error).is_some()); + } + } + } +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn stop_forces_only_its_verified_process_after_grace() { + let (_directory, vm) = vm_dir(); + let outputs = vm.outputs(false).unwrap(); + let mut child = StopChild::start(); + let mut other = StopChild::start(); + let grace = Instant::now() + Duration::from_millis(80); + let deadline = grace + Duration::from_secs(10); + let stopped = stop( + &child.process, + &outputs, + |supplied| { + assert_eq!(supplied, grace); + true + }, + grace, + deadline, + ) + .unwrap(); + assert!(!stopped.self_exited); + assert!(stopped.forced); + assert!(stopped.report.unwrap().is_none()); + assert!(child.process.wait(Instant::now()).unwrap().is_some()); + assert!(child.child.try_wait().unwrap().is_some()); + assert!(other.process.wait(Instant::now()).unwrap().is_none()); + assert!(other.child.try_wait().unwrap().is_none()); +} + +#[cfg(any(target_os = "linux", windows))] +#[test] +fn stop_returns_still_running_without_exit_confirmation_at_its_deadline() { + let (_directory, vm) = vm_dir(); + let outputs = vm.outputs(false).unwrap(); + fs::write(&outputs.outcome, b"{ truncated").unwrap(); + let mut child = StopChild::start(); + #[cfg(target_os = "linux")] + let handle = platform::ProcessHandle::without_pidfd(); + #[cfg(windows)] + let handle = platform::ProcessHandle::wait_only(child.child.id()); + let process = Process::held(child.process.reference().clone(), handle); + let grace = Instant::now(); + let deadline = grace + Duration::from_millis(80); + let error = stop(&process, &outputs, |_| false, grace, deadline).unwrap_err(); + assert!(Instant::now() >= deadline); + assert_eq!(error.kind(), ErrorKind::StillRunning); + assert_eq!(error.code().as_str(), "still-running"); + let cause = std::error::Error::source(&error) + .unwrap() + .downcast_ref::() + .unwrap(); + assert_eq!( + cause.kind(), + if cfg!(windows) { + io::ErrorKind::PermissionDenied + } else { + io::ErrorKind::Unsupported + } + ); + assert!(process.wait(Instant::now()).unwrap().is_none()); + assert!(child.child.try_wait().unwrap().is_none()); +} diff --git a/nvx-rs/src/host/vm/tests/reattach.rs b/nvx-rs/src/host/vm/tests/reattach.rs index 88a95d005..0611675b6 100644 --- a/nvx-rs/src/host/vm/tests/reattach.rs +++ b/nvx-rs/src/host/vm/tests/reattach.rs @@ -244,7 +244,7 @@ fn reattaching_returns_the_held_owned_child_and_authenticated_connection() { b"response" ); owner.wait(); - assert!(attached.process.wait(deadline).unwrap()); + assert!(attached.process.wait(deadline).unwrap().is_some()); } #[test] diff --git a/nvx-rs/tests/checked_handoff.rs b/nvx-rs/tests/checked_handoff.rs index b07a9e340..8f1c7b07b 100644 --- a/nvx-rs/tests/checked_handoff.rs +++ b/nvx-rs/tests/checked_handoff.rs @@ -703,6 +703,7 @@ fn a_prepared_launch_hands_the_vmm_its_checked_images_alone() { .unwrap() .wait(Instant::now() + TIMEOUT) .unwrap() + .is_some() ); launched.kill_and_wait(); let scratch = fs::read_to_string(staged.scratch()).unwrap(); @@ -788,6 +789,7 @@ fn a_vmm_keeps_the_scratch_images_claim_once_its_launcher_is_gone() { .unwrap() .wait(Instant::now() + TIMEOUT) .unwrap() + .is_some() ); launched.kill_and_wait(); drop(check_written().unwrap()); diff --git a/nvx-rs/tests/cli.rs b/nvx-rs/tests/cli.rs index f3b64de5b..01191d000 100644 --- a/nvx-rs/tests/cli.rs +++ b/nvx-rs/tests/cli.rs @@ -1007,9 +1007,10 @@ fn a_stop_under_another_identity_never_ends_a_process_that_its_record_does_not_n ); assert_eq!(runtime_record(&root, "state"), None); assert!( - !process + process .wait(Instant::now() + Duration::from_millis(200)) - .unwrap(), + .unwrap() + .is_none(), "{field}" ); process.kill().unwrap(); @@ -1017,6 +1018,7 @@ fn a_stop_under_another_identity_never_ends_a_process_that_its_record_does_not_n process .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); } assert_eq!(transition(&root, "deprovision", "state", &[]), succeeded()); @@ -1153,6 +1155,7 @@ fn stop_reports_its_timeout_and_leaves_the_vm_running() { process .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); assert_eq!( transition(&root, "deprovision", "unanswered", &[]), @@ -1173,6 +1176,7 @@ fn stop_reports_its_timeout_and_leaves_the_vm_running() { process .wait(Instant::now() + Duration::from_secs(30)) .unwrap() + .is_some() ); assert_eq!( transition(&root, "stop", "slow", &[]), @@ -1194,6 +1198,7 @@ fn a_sandbox_whose_openvmm_exited_reports_it_once_and_starts_again() { process .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); }; crash(); diff --git a/nvx-rs/tests/cli/download.rs b/nvx-rs/tests/cli/download.rs index be959f532..f9d758cea 100644 --- a/nvx-rs/tests/cli/download.rs +++ b/nvx-rs/tests/cli/download.rs @@ -171,6 +171,22 @@ fn provision_and_deprovision_need_no_artifact_root() { let executable = elsewhere .path() .join(Path::new(NVX_RS).file_name().unwrap()); + #[cfg(unix)] + { + // Concurrent forks of this test process must inherit no writable executable. + let copied = Command::new("cp") + .arg("-p") + .arg(NVX_RS) + .arg(&executable) + .output() + .unwrap(); + assert!( + copied.status.success(), + "cannot copy the executable: {}", + String::from_utf8_lossy(&copied.stderr) + ); + } + #[cfg(windows)] fs::copy(NVX_RS, &executable).unwrap(); let nvx_rs = |arguments: &[String]| { let output = Command::new(&executable) diff --git a/nvx-rs/tests/cli/lifecycle.rs b/nvx-rs/tests/cli/lifecycle.rs index 7078aba9e..d82a0f548 100644 --- a/nvx-rs/tests/cli/lifecycle.rs +++ b/nvx-rs/tests/cli/lifecycle.rs @@ -604,7 +604,8 @@ fn check_lifecycle(case: &Configuration) { assert!( process .wait(Instant::now() + Duration::from_secs(10)) - .unwrap(), + .unwrap() + .is_some(), "{context}: OpenVMM runs on" ); assert_eq!(entries(&state), STOPPED, "{context}"); @@ -694,7 +695,8 @@ fn lifecycle_of_a_sandbox_whose_upper_is_in_ram() { assert!( process .wait(Instant::now() + Duration::from_secs(10)) - .unwrap(), + .unwrap() + .is_some(), "{context}: OpenVMM runs on" ); assert_eq!(entries(&state), STOPPED, "{context}"); @@ -737,7 +739,8 @@ fn stops_keep_an_outcome_report_that_the_caller_reads_whatever_its_umask() { assert!( process .wait(Instant::now() + Duration::from_secs(10)) - .unwrap(), + .unwrap() + .is_some(), "{context}: OpenVMM runs on" ); assert_eq!(entries(&state), STOPPED, "{context}"); @@ -1437,9 +1440,10 @@ fn recycled_process_id(root: &Root) { assert_refused(root, &outcome, "sandbox-exited", &["stale", "not running"]); assert!(!path.exists()); assert!( - !openvmm + openvmm .wait(Instant::now() + Duration::from_millis(200)) - .unwrap(), + .unwrap() + .is_none(), "the stale record's process was signaled" ); openvmm.kill().unwrap(); @@ -1447,6 +1451,7 @@ fn recycled_process_id(root: &Root) { openvmm .wait(Instant::now() + Duration::from_secs(10)) .unwrap() + .is_some() ); let outcome = operate(root, "stop", "state", &[]); assert_refused(root, &outcome, "sandbox-not-running", &["not running"]); diff --git a/nvx-rs/tests/fake_openvmm.rs b/nvx-rs/tests/fake_openvmm.rs index c667dbe84..b112f951b 100644 --- a/nvx-rs/tests/fake_openvmm.rs +++ b/nvx-rs/tests/fake_openvmm.rs @@ -478,7 +478,7 @@ fn outcome_reports_of_another_runtime_are_never_read() { .unwrap(); if exits_first { process.kill().unwrap(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); } let stopped = locked.terminate(mismatched, TIMEOUT).unwrap(); assert_eq!( @@ -528,11 +528,12 @@ fn records_that_name_another_process_never_end_it() { drop(locked); assert!(records_nothing(&instance)); assert!( - !process + process .wait(Instant::now() + Duration::from_millis(200)) .unwrap() + .is_none() ); process.kill().unwrap(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); } } diff --git a/nvx-rs/tests/store.rs b/nvx-rs/tests/store.rs index 39d55d2e4..9602cce68 100644 --- a/nvx-rs/tests/store.rs +++ b/nvx-rs/tests/store.rs @@ -183,7 +183,9 @@ fn kill(pid: u32, start_time: u64) -> io::Result { return Ok(true); }; process.kill()?; - process.wait(Instant::now() + RECOVERY_TIMEOUT) + process + .wait(Instant::now() + RECOVERY_TIMEOUT) + .map(|exited| exited.is_some()) } /// A VMM that a crashed orchestrator left behind, which is killed when the test ends if it still @@ -533,7 +535,12 @@ fn a_committed_launch_names_its_vm_for_another_build() { .expect("the recorded VMM runs"); assert!(!reattached.has_exited().unwrap()); drop(vmm); - assert!(reattached.wait(Instant::now() + RECOVERY_TIMEOUT).unwrap()); + assert!( + reattached + .wait(Instant::now() + RECOVERY_TIMEOUT) + .unwrap() + .is_some() + ); other.lock(None).unwrap().clear_runtime().unwrap(); assert!(other.runtime().unwrap().is_none()); } diff --git a/nvx-rs/tests/supervisor.rs b/nvx-rs/tests/supervisor.rs index be2fb01a0..45b2058ab 100644 --- a/nvx-rs/tests/supervisor.rs +++ b/nvx-rs/tests/supervisor.rs @@ -450,7 +450,9 @@ impl Drop for VmmDirectory { let _ = File::create(self.path().join(EXIT_MARKER)); if let Some(reference) = &self.reference && let Ok(Some(process)) = Process::open(reference) - && !process.wait(Instant::now() + TIMEOUT).unwrap_or(false) + && !process + .wait(Instant::now() + TIMEOUT) + .is_ok_and(|exited| exited.is_some()) { let _ = process.kill(); let _ = process.wait(Instant::now() + TIMEOUT); @@ -613,7 +615,13 @@ fn long_stores_launch_with_a_short_working_directory() { let mut vmm = VmmDirectory::new(); let mut locked = store.lock(None).unwrap(); let running = launch_vmm(&mut locked, &mut vmm); - assert!(!running.process().wait(Instant::now() + SHORT_WAIT).unwrap()); + assert!( + running + .process() + .wait(Instant::now() + SHORT_WAIT) + .unwrap() + .is_none() + ); let stopped = running .stop(&mut locked, Instant::now() + TIMEOUT, None, |_, _| { vmm.end(); @@ -671,15 +679,15 @@ fn a_child_that_exits_wakes_its_waiter() { assert_eq!(process.reference().boot, BootId::current().unwrap()); assert_eq!(process.wait_method(), WaitMethod::Event); assert!(!process.has_exited().unwrap()); - assert!(!process.wait(Instant::now() + SHORT_WAIT).unwrap()); + assert!(process.wait(Instant::now() + SHORT_WAIT).unwrap().is_none()); // The reference finds the same process through a handle of its own. let found = open_running(process.reference()); assert_eq!(found.reference(), process.reference()); let started = Instant::now(); vmm.end(); - assert!(process.wait(started + TIMEOUT).unwrap()); - assert!(found.wait(started + TIMEOUT).unwrap()); + assert!(process.wait(started + TIMEOUT).unwrap().is_some()); + assert!(found.wait(started + TIMEOUT).unwrap().is_some()); assert!(started.elapsed() < TIMEOUT); assert_ended(&process); // Ending a process that exited does nothing. @@ -698,12 +706,12 @@ fn a_process_that_is_not_a_child_wakes_its_waiters() { launcher.finish(); let process = open_running(&reference); assert_eq!(process.wait_method(), WaitMethod::Event); - assert!(!process.wait(Instant::now() + SHORT_WAIT).unwrap()); + assert!(process.wait(Instant::now() + SHORT_WAIT).unwrap().is_none()); let started = Instant::now(); let waiters: Vec = thread::scope(|scope| { let waiters: Vec<_> = (0..2) - .map(|_| scope.spawn(|| process.wait(started + TIMEOUT).unwrap())) + .map(|_| scope.spawn(|| process.wait(started + TIMEOUT).unwrap().is_some())) .collect(); vmm.end(); waiters @@ -740,10 +748,10 @@ fn a_restarted_launcher_finds_its_vmm_again_and_ends_it() { vmm.record(&reference); let process = open_running(&reference); assert_eq!(process.wait_method(), WaitMethod::Event); - assert!(!process.wait(Instant::now() + SHORT_WAIT).unwrap()); + assert!(process.wait(Instant::now() + SHORT_WAIT).unwrap().is_none()); let started = Instant::now(); process.kill().unwrap(); - assert!(process.wait(started + TIMEOUT).unwrap()); + assert!(process.wait(started + TIMEOUT).unwrap().is_some()); assert_ended(&process); } @@ -783,7 +791,7 @@ fn a_reused_process_id_is_never_signaled() { assert!(!process.has_exited().unwrap()); open_running(&reference); vmm.end(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); assert!(launched.kill_and_wait()); } @@ -814,7 +822,7 @@ fn references_that_name_no_process_are_never_opened() { let launched = spawn_vmm(vmm.path(), false); let process = launched.process().unwrap(); vmm.end(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); assert!(launched.kill_and_wait()); assert!(Process::open(process.reference()).unwrap().is_none()); assert!(process.has_exited().unwrap()); @@ -877,7 +885,7 @@ fn an_unrecorded_vmm_that_serves_no_endpoint_is_never_signaled() { assert!(store.journal().unwrap().is_some()); // Once the VMM exits, the claim that it released proves it. vmm.end(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); match reattach_until_determinate(&mut locked) { Reattached::Exited { record: None, @@ -1264,7 +1272,7 @@ fn check_vmm(reader: PipeReader, writer: &PipeWriter, others: &Mutex> } } vmm.end(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); assert!(launched.kill_and_wait()); } @@ -1493,7 +1501,7 @@ fn a_vmm_in_its_orchestrators_job_ends_with_it() { // The VMM did not break away, so closing the job ends it with its orchestrator. drop(job); orchestrator.wait(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); let mut locked = store.lock(Some(Instant::now() + TIMEOUT)).unwrap(); match supervisor::reattach(&mut locked, &runtime()).unwrap() { Reattached::Exited { @@ -1517,10 +1525,10 @@ fn a_job_that_ends_its_vmm_wakes_the_waiters() { vmm.record(&reference); // The VMM did not break away, so the job holds it too, and closing the job ends it. let process = open_running(&reference); - assert!(!process.wait(Instant::now() + SHORT_WAIT).unwrap()); + assert!(process.wait(Instant::now() + SHORT_WAIT).unwrap().is_none()); let started = Instant::now(); drop(job); - assert!(process.wait(started + TIMEOUT).unwrap()); + assert!(process.wait(started + TIMEOUT).unwrap().is_some()); assert_ended(&process); launcher.wait(); } @@ -1587,7 +1595,7 @@ fn a_process_that_this_user_may_not_terminate_is_waited_for_but_never_signaled() // Its creator ends it, which wakes the waiter, and ending a process that exited does nothing. child.0.kill().unwrap(); - assert!(process.wait(Instant::now() + TIMEOUT).unwrap()); + assert!(process.wait(Instant::now() + TIMEOUT).unwrap().is_some()); process.kill().unwrap(); assert_ended(&process); } diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 449ceb1f0..844c9a4f0 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -2053,6 +2053,7 @@ fn launch_failed( let exited = starting .process() .wait(Instant::now() + STOP_TIMEOUT) + .map(|exited| exited.is_some()) .map_err(|error| format!("cannot wait for the VM to exit: {error}"))?; starting.abandon().map_err(|error| describe(&error))?; println!("stopped={}", if exited { "graceful" } else { "forced" }); @@ -2078,6 +2079,7 @@ fn table_failed( Ok(_) => starting .process() .wait(Instant::now() + STOP_TIMEOUT) + .map(|exited| exited.is_some()) .map_err(|error| format!("cannot wait for the VM to exit: {error}"))?, Err(_) => false, }; @@ -3625,12 +3627,14 @@ fn hold(locked: &mut LockedStore, runtime: &Runtime) -> Result Result<(), String> { - let outcome = report::read(&store.outcome_path()).ok_or_else(|| { - format!( - "the VM left no outcome report {}", - store.outcome_path().display() - ) - })?; + let outcome = report::read(&store.outcome_path()) + .map_err(|error| error.to_string())? + .ok_or_else(|| { + format!( + "the VM left no outcome report {}", + store.outcome_path().display() + ) + })?; print_outcome(&outcome); if ( outcome.category.as_deref(), diff --git a/tests/runtime-adapter/tests/common/mod.rs b/tests/runtime-adapter/tests/common/mod.rs index 3c3540923..dc5e0f914 100644 --- a/tests/runtime-adapter/tests/common/mod.rs +++ b/tests/runtime-adapter/tests/common/mod.rs @@ -154,11 +154,14 @@ impl Sandbox { pub(crate) fn new(name: &str) -> Self { let pid = std::process::id(); let root = (0..) - .map(|attempt| env::temp_dir().join(format!("runtime-adapter-{name}-{pid}-{attempt}"))) + .map(|attempt| env::temp_dir().join(format!("ra-{pid}-{attempt}"))) .find(|root| match fs::create_dir(root) { Ok(()) => true, Err(error) if error.kind() == io::ErrorKind::AlreadyExists => false, - Err(error) => panic!("cannot create {}: {error}", root.display()), + Err(error) => panic!( + "cannot create the {name} sandbox {}: {error}", + root.display() + ), }) .expect("a directory name is free"); Self { diff --git a/tests/runtime-adapter/tests/vm_clone.rs b/tests/runtime-adapter/tests/vm_clone.rs index 654fad486..cbc20e665 100644 --- a/tests/runtime-adapter/tests/vm_clone.rs +++ b/tests/runtime-adapter/tests/vm_clone.rs @@ -2321,7 +2321,7 @@ impl CloneControl for Early { .send(&self.early, Some(deadline)) .map_err(io::Error::other)?; let answer = session.receive(Some(deadline)); - self.exited = self.vmm.wait(deadline)?; + self.exited = self.vmm.wait(deadline)?.is_some(); Err(match answer { Ok(message) => io::Error::other(format!("the clone answered {message:?}")), Err(error) => io::Error::other(error), @@ -3910,7 +3910,9 @@ fn a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identit assert!(!ended.contains_key("epoch"), "{ended:?}"); assert!(sandbox.stderr().contains("differs in initramfs")); assert!( - vmm.wait(Instant::now() + Duration::from_secs(10)).unwrap(), + vmm.wait(Instant::now() + Duration::from_secs(10)) + .unwrap() + .is_some(), "the clone's VMM still runs" ); assert!(sandbox.record().is_none()); @@ -3948,7 +3950,9 @@ fn a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_release assert_eq!(sandbox.paused(&mut stopping, &pause), "found"); vmm.kill().unwrap(); assert!( - vmm.wait(Instant::now() + Duration::from_secs(10)).unwrap(), + vmm.wait(Instant::now() + Duration::from_secs(10)) + .unwrap() + .is_some(), "the clone's VMM still runs" ); fs::remove_file(&pause).unwrap(); @@ -5313,7 +5317,9 @@ fn a_run_whose_clone_s_vm_exits_during_its_execution_ends_with_the_clone_s_failu let vmm = sandbox.vmm().expect("the clone's VMM runs"); vmm.kill().unwrap(); assert!( - vmm.wait(Instant::now() + Duration::from_secs(10)).unwrap(), + vmm.wait(Instant::now() + Duration::from_secs(10)) + .unwrap() + .is_some(), "the clone's VMM still runs" );