host: stop VMs only after verified exit

doc/usage.md

@@ -110,6 +110,7 @@ each code for callers of the `nvx` library.
 | `template-ineligible` | 1 | The restore window of a template's generation, which the time contract of its capture sets, excludes the host's time, so no clone may restore it now. |
 | `template-tombstoned` | 1 | A tombstone quarantines the template's generation, which takes no new clones. |
 | `exited` | 1 | OpenVMM exited, as when a start fails, whose message names the category and status of OpenVMM's outcome report when it has one. |
+| `still-running` | 1 | A stop could not confirm the VM process's exit, so the caller keeps its verified process. |
 | `exec-refused` | 1 | The guest agent refused the workload, as for a working directory that it cannot use. |
 | `endpoint-closed` | 1 | OpenVMM closed the control endpoint, as it does when the VM shuts down. |
 | `protocol` | 1 | The control session failed otherwise, as when the guest breaks the protocol or the session is reset. |

nvx-rs/README.md

@@ -124,7 +124,7 @@ changes only when the module's purpose does.
 | [`nvx::Hypervisor`](#nvxhypervisor) | KVM, MSHV, or WHP, with its serde form behind the `serde` feature |
 | [`nvx::exec`](#nvxexec) | The execution engine: the streams, credit, and outcome of an execution's two sides, without I/O, for any framing |
 | [`nvx::host::runtime`](#nvxhostruntime) | `Runtime`: what every launch shares, the runtime's files and their identity, the hypervisor and its probe, and the launch options |
-| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold preparation, verified detached launch, and caller-saved reattachment |
+| [`nvx::host::vm`](#nvxhostvm) | A private VM directory, cold preparation, verified detached launch, confirmed stop, and caller-saved reattachment |
 | [`nvx::host::snapshot`](#nvxhostsnapshot) | Snapshot compatibility keys, strict manifests, restore-time eligibility, and the opaque input of a future VM restore |
 | [`nvx::host::config`](#nvxhostconfig) | `VmConfig` and `CaptureConfig`: pure configuration, cold-boot kernel/OpenVMM arguments, portable length checks, and capture identity |
 | [`nvx::host::restore`](#nvxhostrestore) | A caller-driven restore to Ready without I/O or a clock, and its peer-verified restore-ready listener |
@@ -500,6 +500,17 @@ or reading runtime files.
 `Reattached` holds both the process and authenticated connection. Failures
 retain their original kinds and sources and leave the saved values unchanged.
 
+`stop(process, outputs, graceful, grace, deadline)` waits through the verified
+process, first after the caller's graceful request, then after forced termination
+if needed. It returns `Stopped` only after confirmed exit. `self_exited` records
+confirmation before a forced attempt, and `forced` records that attempt, not
+proof from kill success. An unconfirmed exit is `StillRunning`, and the caller
+keeps the process. The callback receives the bounded grace deadline and must
+respect it; its boolean result is never exit proof.
+
+`Stopped::report` retains the outcome-report read's result separately, so report
+failure never hides confirmed exit.
+
 `vm::create(&runtime, &config, None, &dir)` creates an inspectable cold
 `Prepared` launch without starting a process or binding a listener. It requires
 `config.boot`, refuses a snapshot input, rechecks the directory's privacy, and
@@ -976,6 +987,10 @@ On Windows, the duplicate retains the original handle's access rights.
 Duplication failures have kind `Io` and retain their original I/O source.
 No watcher thread, future or callback is created by the library.
 
+`Process::wait(deadline)` returns `Some(Exited)` only after verified exit, or
+`None` when the deadline passes without confirmation. `Exited` cannot be made
+by a caller.
+
 `Process::bytes_read` reports how many bytes the process has read through the
 operating system's read calls, whether the host's cache served them or not:
 `rchar` of `/proc/<pid>/io` on Linux, which it reads only while the process
@@ -1422,6 +1437,11 @@ The authenticated control session, which carries NVX's control protocol, `NVXC`,
 one frame per DATA record of a `HostConnection`; Linux and Windows primitives;
 and OpenVMM's outcome report.
 
+`report::read` returns `Ok(None)` for a missing report, and an `Io` error for an
+unreadable or malformed report, retaining the underlying error. It keeps today's
+summary of the outcome category, status code and teardown steps, not free-form
+fields.
+
 A `Transport` to an OpenVMM control endpoint reads and writes with deadlines,
 and tells, through `Transport::unread`, what the endpoint has not read yet of
 what it wrote, where the platform tells: on Linux, what `SIOCOUTQ` reports of
@@ -3055,6 +3075,7 @@ optional cause.
 | `Unavailable` | A missing or unusable artifact, hypervisor, or release, a guest image without a required control feature, or an unsupported host |
 | `Unsupported` | A request that NVX refuses: a value that it does not accept, such as a malformed or out-of-range one, options that it cannot combine, or a request that it cannot represent or enforce, such as a policy beyond a limit of OpenVMM or the guest agent |
 | `Exited` | The VM's OpenVMM process exited |
+| `StillRunning` | A stop could not confirm the VM process's exit, so the caller keeps its verified process |
 | `Io` | A host I/O failure, or state that NVX recorded on the host and cannot read |
 | `NotFound` | State that an operation names and that does not exist on the host, such as the store of a sandbox that was never created or that a removal deleted |
 | `Protocol` | A control session that failed, or a request that the guest agent refused |
@@ -3087,6 +3108,7 @@ usage errors, which no library error has.
 | `unavailable` | `Unavailable` | A dependency is missing or unusable, other than as a finer code says: install metadata, the hypervisor, a release, a distro layer or scratch image that the request names, or a host that NVX does not support |
 | `unsupported` | `Unsupported` | NVX refuses the request: options that it cannot combine, a value that it does not accept, or a policy that it cannot represent or enforce. The message names the option or field |
 | `exited` | `Exited` | OpenVMM exited, as when a start fails, whose message names the category and status of OpenVMM's outcome report when it has one |
+| `still-running` | `StillRunning` | A stop could not confirm the VM process's exit, so the caller keeps its verified process |
 | `io` | `Io` | A host I/O operation failed, or state that NVX recorded on the host is unreadable or malformed |
 | `not-found` | `NotFound` | Other state that the operation names does not exist on the host |
 | `protocol` | `Protocol` | The control session failed otherwise: the peer violated the protocol, reset the session, or failed authentication |

nvx-rs/public-api/linux.txt

@@ -58,6 +58,7 @@ variant nvx::ErrorCode::SandboxNotRunning
 variant nvx::ErrorCode::SandboxProvisioned
 variant nvx::ErrorCode::SandboxRunning
 variant nvx::ErrorCode::ScratchClaimed
+variant nvx::ErrorCode::StillRunning
 variant nvx::ErrorCode::TemplateIneligible
 variant nvx::ErrorCode::TemplateTombstoned
 variant nvx::ErrorCode::TimedOut
@@ -73,6 +74,7 @@ variant nvx::ErrorKind::NotFound
 variant nvx::ErrorKind::Other
 variant nvx::ErrorKind::Protocol
 variant nvx::ErrorKind::RuntimeMismatch
+variant nvx::ErrorKind::StillRunning
 variant nvx::ErrorKind::TimedOut
 variant nvx::ErrorKind::Unavailable
 variant nvx::ErrorKind::Unsupported
@@ -2445,6 +2447,7 @@ method nvx::host::process::BootId::as_bytes
 method nvx::host::process::BootId::current
 method nvx::host::process::BootId::from_bytes
 type nvx::host::process::ExitHandle
+struct nvx::host::process::Exited
 struct nvx::host::process::Process
 method nvx::host::process::Process::bytes_read
 method nvx::host::process::Process::exit_handle
@@ -3737,6 +3740,10 @@ field nvx::host::vm::SavedVm::endpoint
 field nvx::host::vm::SavedVm::process
 field nvx::host::vm::SavedVm::runtime
 field nvx::host::vm::SavedVm::vm_dir
+struct nvx::host::vm::Stopped
+field nvx::host::vm::Stopped::forced
+field nvx::host::vm::Stopped::report
+field nvx::host::vm::Stopped::self_exited
 struct nvx::host::vm::VmDir
 method nvx::host::vm::VmDir::open
 method nvx::host::vm::VmDir::path
@@ -3749,6 +3756,7 @@ fn nvx::host::vm::create
 fn nvx::host::vm::create_private_dir
 fn nvx::host::vm::reattach
 fn nvx::host::vm::spawn
+fn nvx::host::vm::stop
 fn nvx::host::vm::verify
 mod nvx::net
 struct nvx::net::Cidr

nvx-rs/public-api/windows.txt

@@ -58,6 +58,7 @@ variant nvx::ErrorCode::SandboxNotRunning
 variant nvx::ErrorCode::SandboxProvisioned
 variant nvx::ErrorCode::SandboxRunning
 variant nvx::ErrorCode::ScratchClaimed
+variant nvx::ErrorCode::StillRunning
 variant nvx::ErrorCode::TemplateIneligible
 variant nvx::ErrorCode::TemplateTombstoned
 variant nvx::ErrorCode::TimedOut
@@ -73,6 +74,7 @@ variant nvx::ErrorKind::NotFound
 variant nvx::ErrorKind::Other
 variant nvx::ErrorKind::Protocol
 variant nvx::ErrorKind::RuntimeMismatch
+variant nvx::ErrorKind::StillRunning
 variant nvx::ErrorKind::TimedOut
 variant nvx::ErrorKind::Unavailable
 variant nvx::ErrorKind::Unsupported
@@ -1031,6 +1033,7 @@ method nvx::host::process::BootId::as_bytes
 method nvx::host::process::BootId::current
 method nvx::host::process::BootId::from_bytes
 type nvx::host::process::ExitHandle
+struct nvx::host::process::Exited
 struct nvx::host::process::Process
 method nvx::host::process::Process::bytes_read
 method nvx::host::process::Process::exit_handle
@@ -2322,6 +2325,10 @@ field nvx::host::vm::SavedVm::endpoint
 field nvx::host::vm::SavedVm::process
 field nvx::host::vm::SavedVm::runtime
 field nvx::host::vm::SavedVm::vm_dir
+struct nvx::host::vm::Stopped
+field nvx::host::vm::Stopped::forced
+field nvx::host::vm::Stopped::report
+field nvx::host::vm::Stopped::self_exited
 struct nvx::host::vm::VmDir
 method nvx::host::vm::VmDir::open
 method nvx::host::vm::VmDir::path
@@ -2334,6 +2341,7 @@ fn nvx::host::vm::create
 fn nvx::host::vm::create_private_dir
 fn nvx::host::vm::reattach
 fn nvx::host::vm::spawn
+fn nvx::host::vm::stop
 fn nvx::host::vm::verify
 mod nvx::net
 struct nvx::net::Cidr

nvx-rs/src/error.rs

@@ -19,6 +19,8 @@ pub enum ErrorKind {
     Unsupported,
     /// The OpenVMM process of a VM exited.
     Exited,
+    /// The exit of a VM's process could not be confirmed; it may still run.
+    StillRunning,
     /// A host I/O operation failed, or state that NVX recorded on the host is unreadable or
     /// malformed.
     Io,
@@ -44,6 +46,7 @@ impl ErrorKind {
             Self::Unavailable => "unavailable",
             Self::Unsupported => "unsupported",
             Self::Exited => "exited",
+            Self::StillRunning => "still-running",
             Self::Io => "io",
             Self::NotFound => "not-found",
             Self::Protocol => "protocol",
@@ -88,6 +91,8 @@ pub enum ErrorCode {
     Unsupported,
     /// `exited`: the OpenVMM process of a VM exited, as when a start fails.
     Exited,
+    /// `still-running`: the exit of a VM's process could not be confirmed.
+    StillRunning,
     /// `io`: a host I/O operation failed, or state that NVX recorded on the host is unreadable or
     /// malformed.
     Io,
@@ -170,6 +175,7 @@ impl ErrorCode {
         Self::Unavailable,
         Self::Unsupported,
         Self::Exited,
+        Self::StillRunning,
         Self::Io,
         Self::NotFound,
         Self::Protocol,
@@ -205,6 +211,7 @@ impl ErrorCode {
             Self::Unavailable => ErrorKind::Unavailable.as_str(),
             Self::Unsupported => ErrorKind::Unsupported.as_str(),
             Self::Exited => ErrorKind::Exited.as_str(),
+            Self::StillRunning => ErrorKind::StillRunning.as_str(),
             Self::Io => ErrorKind::Io.as_str(),
             Self::NotFound => ErrorKind::NotFound.as_str(),
             Self::Protocol => ErrorKind::Protocol.as_str(),
@@ -248,6 +255,7 @@ impl ErrorCode {
             | Self::ArgumentMissing
             | Self::ArgumentInvalid => ErrorKind::Unsupported,
             Self::Exited | Self::SandboxExited => ErrorKind::Exited,
+            Self::StillRunning => ErrorKind::StillRunning,
             Self::Io | Self::FormatUnsupported => ErrorKind::Io,
             Self::NotFound | Self::SandboxNotProvisioned => ErrorKind::NotFound,
             Self::Protocol | Self::ExecRefused | Self::EndpointClosed => ErrorKind::Protocol,
@@ -278,6 +286,7 @@ impl From<ErrorKind> for ErrorCode {
             ErrorKind::Unavailable => Self::Unavailable,
             ErrorKind::Unsupported => Self::Unsupported,
             ErrorKind::Exited => Self::Exited,
+            ErrorKind::StillRunning => Self::StillRunning,
             ErrorKind::Io => Self::Io,
             ErrorKind::NotFound => Self::NotFound,
             ErrorKind::Protocol => Self::Protocol,

nvx-rs/src/error/tests.rs

@@ -52,6 +52,7 @@ fn every_kind_has_a_code_of_its_own_spelling() {
         ErrorKind::Unavailable,
         ErrorKind::Unsupported,
         ErrorKind::Exited,
+        ErrorKind::StillRunning,
         ErrorKind::Io,
         ErrorKind::NotFound,
         ErrorKind::Protocol,

nvx-rs/src/host/instance/locked.rs

@@ -216,7 +216,8 @@ impl LockedInstance {
         };
         Ok(Stopped {
             forced,
-            report: report::read(&self.instance.report_path()),
+            // This legacy wrapper keeps its best-effort report policy.
+            report: report::read(&self.instance.report_path()).ok().flatten(),
         })
     }
 

nvx-rs/src/host/platform/windows/tests.rs

@@ -16,7 +16,8 @@ use windows_sys::Win32::Foundation::{
 use windows_sys::Win32::Security::SECURITY_ATTRIBUTES;
 use windows_sys::Win32::System::JobObjects::CreateJobObjectW;
 use windows_sys::Win32::System::Threading::{
-    CreateEventW, GetCurrentProcess, INFINITE, OpenProcess, PROCESS_DUP_HANDLE, TerminateProcess,
+    CreateEventW, GetCurrentProcess, INFINITE, OpenProcess, PROCESS_DUP_HANDLE,
+    PROCESS_QUERY_LIMITED_INFORMATION, PROCESS_SYNCHRONIZE, TerminateProcess,
 };
 
 pub(crate) use super::endpoint::tests::{accept_control, control_server};
@@ -38,6 +39,16 @@ pub(crate) fn wait_exit_handle(process: OwnedHandle, deadline: Instant) -> io::R
     .map(|exited| exited.expect("Windows process handles always provide an exit event"))
 }
 
+impl ProcessHandle {
+    pub(crate) fn wait_only(pid: u32) -> Self {
+        Self {
+            process: open_handle(pid, PROCESS_QUERY_LIMITED_INFORMATION | PROCESS_SYNCHRONIZE)
+                .unwrap(),
+            terminate: false,
+        }
+    }
+}
+
 thread_local! {
     /// What a spawn of this thread runs in its window, once its handles are inheritable and
     /// before it creates its process.
@@ -1084,7 +1095,7 @@ fn a_launch_whose_vmm_exited_ends_while_a_process_spawned_in_its_window_holds_it
     let vmm = launch.record_process(&launched).unwrap();
     launched.detach();
     assert!(
-        vmm.wait(Instant::now() + WINDOW_TIMEOUT).unwrap(),
+        vmm.wait(Instant::now() + WINDOW_TIMEOUT).unwrap().is_some(),
         "the stand-in did not exit"
     );
     // The VMM has exited, so the launch ends, whatever the other process holds.

nvx-rs/src/host/process.rs

@@ -188,6 +188,12 @@ pub struct Process {
     handle: platform::ProcessHandle,
 }
 
+/// Proof that a verified process exited, produced only by [`Process::wait`].
+///
+/// It proves exit, not the process's exit status, and cannot be constructed by a caller.
+#[derive(Debug)]
+pub struct Exited(());
+
 impl Process {
     /// Finds the process that `reference` names, and returns `None` if it has exited.
     ///
@@ -241,13 +247,17 @@ impl Process {
         })
     }
 
-    /// Waits until the process exits or `deadline` passes, and returns whether it exited.
+    /// Waits until the process exits or `deadline` passes, returning exit proof or `None`.
     ///
     /// A process exits with its last thread. The wait sleeps until the exit event of the
     /// process, or with [`WaitMethod::Polling`], checks its identity at a short interval. An
     /// error means that the wait cannot determine whether the process exited.
-    pub fn wait(&self, deadline: Instant) -> io::Result<bool> {
-        self.wait_bool(deadline)
+    pub fn wait(&self, deadline: Instant) -> io::Result<Option<Exited>> {
+        Ok(if self.wait_bool(deadline)? {
+            Some(Exited(()))
+        } else {
+            None
+        })
     }
 
     /// Keeps the boolean wait result for the legacy lifecycle callers.
@@ -262,7 +272,7 @@ impl Process {
 
     /// Returns whether the process has exited, without waiting.
     pub fn has_exited(&self) -> io::Result<bool> {
-        self.wait(Instant::now())
+        self.wait(Instant::now()).map(|exited| exited.is_some())
     }
 
     /// Ends the process at once, unless it has exited or is exiting: with `SIGKILL` on Linux, and

nvx-rs/src/host/process/tests.rs

@@ -275,14 +275,15 @@ fn hosts_without_pidfds_poll_and_never_signal() {
     let still_running = process.has_exited();
     child.kill().unwrap();
     child.wait().unwrap();
-    assert!(!running.unwrap());
+    assert!(running.unwrap().is_none());
     assert_eq!(refused.unwrap_err().kind(), io::ErrorKind::Unsupported);
     assert!(!still_running.unwrap());
     // Once the child is reaped, its ID names no process, or another one.
     assert!(
         process
             .wait(Instant::now() + Duration::from_secs(10))
             .unwrap()
+            .is_some()
     );
 }
 

nvx-rs/src/host/process/tests/linux.rs

@@ -152,13 +152,14 @@ fn a_process_runs_until_its_last_thread_exits() {
     .unwrap()
     .expect("a process whose main thread exited still runs");
     assert!(
-        !process
+        process
             .wait(Instant::now() + Duration::from_millis(100))
             .unwrap()
+            .is_none()
     );
     child.0.kill().unwrap();
     child.0.wait().unwrap();
-    assert!(process.wait(Instant::now()).unwrap());
+    assert!(process.wait(Instant::now()).unwrap().is_some());
 }
 
 #[test]

nvx-rs/src/host/report.rs

@@ -9,6 +9,8 @@ use std::path::Path;
 
 use serde_json::Value;
 
+use crate::{Error, Result};
+
 /// Summary of an outcome report.
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
 pub struct Report {
@@ -20,9 +22,24 @@ pub struct Report {
     pub teardown_complete: Option<bool>,
 }
 
-/// Reads the report at `path`, or returns `None` if it is missing or not JSON.
-pub fn read(path: &Path) -> Option<Report> {
-    let report: Value = serde_json::from_slice(&fs::read(path).ok()?).ok()?;
+/// Reads the report at `path`, or returns `Ok(None)` if it is missing.
+///
+/// An unreadable or malformed report fails with [`ErrorKind::Io`](crate::ErrorKind::Io),
+/// retaining the underlying filesystem or JSON error.
+pub fn read(path: &Path) -> Result<Option<Report>> {
+    let bytes = match fs::read(path) {
+        Ok(bytes) => bytes,
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
+        Err(error) => {
+            return Err(
+                Error::io(format!("cannot read outcome report {}", path.display()))
+                    .with_source(error),
+            );
+        }
+    };
+    let report: Value = serde_json::from_slice(&bytes).map_err(|error| {
+        Error::io(format!("cannot parse outcome report {}", path.display())).with_source(error)
+    })?;
     let mut summary = Report::default();
     if let Some(outcome) = report.get("outcome") {
         summary.category = outcome
@@ -35,7 +52,7 @@ pub fn read(path: &Path) -> Option<Report> {
         let complete = teardown.values().all(|step| step.as_bool() == Some(true));
         summary.teardown_complete = Some(complete);
     }
-    Some(summary)
+    Ok(Some(summary))
 }
 
 #[cfg(test)]