diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index e19623a27..512b4fd0c 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -30,7 +30,7 @@ its own in turn. | [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 | | [MicroVM scenarios](#microvm-scenarios) | 52 | 49 | 3 | 0 | | [CI](#ci) | 69 | 43 | 3 | 23 | -| [Docs](#docs) | 34 | 33 | 1 | 0 | +| [Docs](#docs) | 36 | 35 | 1 | 0 | ## The adapter's scenarios @@ -412,17 +412,18 @@ their own. The tests that a step runs have rows of their own where they concern ## Docs Rustdoc checks the code of the crate's documentation, which these rows name as rustdoc does, by -file and line where this matrix was added: it compiles each, runs those that aren't `no_run`, and -checks that those marked `compile_fail` don't compile. The examples compile with the crate's -other targets; `host-workflows` and `guest-workflows`, which came with this matrix, are new, and -have no row. The README's own code is no doctest. +file and line: it compiles each, runs those that aren't `no_run`, and checks that those marked +`compile_fail` don't compile. The examples compile with the crate's other targets; +`host-workflows` and `guest-workflows`, which came with this matrix, are new, and have no row. +The code of `guest` and `guest::snapshot`, which the guest-agent guidelines quote, came after +this matrix. The README's own code is no doctest. | Code | Fate | Note | | --- | --- | --- | -| `nvx-rs/src/activation/mod.rs - activation (line 123)` | kept | | +| `nvx-rs/src/activation/mod.rs - activation (line 130)` | kept | | | `nvx-rs/src/exec/mod.rs - exec (line 113)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/gpt/mod.rs - gpt (line 43)` | kept | | -| `nvx-rs/src/guest/activation.rs - guest::activation (line 189)` | kept | Its capture preparation and admission fence leave (#1087), whose PR rewrites it on `guest::snapshot::capture` | +| `nvx-rs/src/guest/activation.rs - guest::activation (line 195)` | kept | Its capture preparation and admission fence leave (#1087), whose PR rewrites it on `guest::snapshot::capture` | | `nvx-rs/src/guest/activation/state.rs - guest::activation::state::Activation (line 56)` | kept | | | `nvx-rs/src/guest/cgroup.rs - guest::cgroup (line 36)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/entry.rs - guest::entry (line 90)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | @@ -431,11 +432,13 @@ have no row. The README's own code is no doctest. | `nvx-rs/src/guest/gpt.rs - guest::gpt (line 18)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/mappings/mapper.rs - guest::mappings::mapper (line 107)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/mappings/table.rs - guest::mappings::table::MappingTable (line 47)` | kept | | +| `nvx-rs/src/guest/mod.rs - guest (line 41)` | kept | The guest-agent guidelines quote it | | `nvx-rs/src/guest/nvxs.rs - guest::nvxs (line 22)` | kept | | | `nvx-rs/src/guest/process.rs - guest::process (line 12)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/process.rs - guest::process (line 39)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/reseed.rs - guest::reseed (line 17)` | kept | | | `nvx-rs/src/guest/root.rs - guest::root (line 130)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | +| `nvx-rs/src/guest/snapshot.rs - guest::snapshot (line 25)` | kept | The guest-agent guidelines quote it | | `nvx-rs/src/guest/spawn.rs - guest::spawn (line 41)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/spawn.rs - guest::spawn::LaunchStage::from_code (line 201)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/terminal.rs - guest::terminal (line 10)` | kept | | diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index cff382e37..cb2754024 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -3,84 +3,70 @@ [Architecture documentation](README.md) > [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) -As Figure 4.2 shows, the guest agents are the guest's long-lived lifecycle -authority. Their responsibilities are: +The guest's init agent is its long-lived lifecycle authority: it owns the +guest's space, its helpers, its control session and its executions. `nvx` +parses the contracts, captures the guest and drives a restore's activation, +but builds no space, runs no executions and enforces none of the init's +obligations; the one exception is the time daemon, which +`TimeAbi::start_daemon` forks. The +[guest-agent guidelines](../guest-agent-guidelines.md) give the init's +contract, with nvx-init as the worked example. -1. Initialize runtime filesystems and cgroup v2. -2. Discover and validate the fixed virtual devices. -3. Assemble the workload filesystem. -4. Apply workload identity, isolation, and resource policy. -5. Start the workload behind a placement barrier. -6. Supervise the workload and reap descendants. -7. Announce the guest's listeners when the host publishes them. -8. Participate in capture and post-restore repair. -9. Unmount resources and report the final workload status. +A layered root's guest runs one of two inits. nvx-init, NVX's PID 1 for +layered roots, runs the new launches, and the restores of platform captures, +in one VM-wide space, as Figure 4.2 shows; it also serves the Fresh boot that +a platform capture takes, which has no root yet. A layered launch that still +carries OpenVMM's older lifecycle token, `nvx_lifecycle=`, which NVX's CLI no +longer adds, runs the older init agent, whose sandbox agent gives each +execution private namespaces and a cgroup in the +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). +The agent of the +[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) +is PID 1 in the guest's initramfs root, and runs its executions with +nvx-init's plain-process core. ```mermaid flowchart TB Kernel["Linux kernel"] - Agents["Guest agents
outside workload namespaces
in the layered root"] - Root["Assembled workload root"] - Isolation["Identity, capabilities,
and cgroups, plus namespaces
in the layered root"] - Workload["Workload process tree"] + Init["nvx-init, PID 1
space, helpers and control session"] + Space["One VM-wide space
root, view and mappings"] + Workload["Executions
a process group each"] - Kernel --> Agents - Agents --> Root - Agents --> Isolation - Root --> Workload - Isolation --> Workload - Agents -->|"supervise and reap"| Workload + Kernel --> Init + Init --> Space + Space --> Workload + Init -->|"supervise and reap"| Workload ``` -**Figure 4.2.** Relationship between the guest agents and the workload. +**Figure 4.2.** nvx-init, its space and its executions. -In a one-shot lifecycle, the guest agents start one workload, configured by -non-secret kernel command-line tokens, and shut down the VM when it exits. In -a managed lifecycle, the guest agents stay resident and run a sequence of +The workload is always a child of a guest agent, which never replaces itself +with it: supervision, orphan reaping and the control service depend on the +agent remaining alive. In a one-shot lifecycle, the guest agents start one +workload, configured by non-secret kernel command-line tokens, and shut down +the VM when it exits. In a managed lifecycle, they run a sequence of workloads, one at a time, on request over the authenticated control channel. -Both modes preserve the same separation: the workload is always a child of a -guest agent. The guest agents never replace themselves with the workload, -because supervision, orphan reaping, and the control service all depend on -them remaining alive. +An init that runs concurrent executions coordinates them for a capture +itself: `nvx` provides no workspace. ## 4.2.1 Execution Launch -For each managed execution, the guest agent's launch enters the requested -working directory, or `/`, with the workload's identity, and sets `PWD` unless -the request replaces the environment. The working directory is entered once -and kept, so the workload starts in the directory that was checked even if its -path is renamed, replaced, or retargeted in the meantime. If the workload -cannot enter it, the execution fails rather than running elsewhere. - -In the blockless managed root, the guest agent uses the init agent's -plain-process builder and driver. The builder prepares the environment, and -the child applies its identity, enters its working directory, and executes the -workload without a launch helper. In the layered sandbox root, the guest -agent's [execution entry](4.3-isolation-and-resource-model.md#432-namespaces-and-mounts) -applies the identity itself, and the guest agent builds the environment. +Launching an execution belongs to the init, not to the library's capture or +activation. nvx-init's executions, and those of the blockless managed root, +are plain processes of their agent: each child takes its credentials and +enters its working directory itself, in its agent's space, without a root of +its own or a placement barrier. The older layered sandbox root keeps its +execution entry, which enters private namespaces and the workload root, and +its placement barrier. +[Run the executions](../guest-agent-guidelines.md#run-the-executions) gives +nvx-init's steps. ## 4.2.2 Containment -In the blockless managed root, each execution runs in a process group pinned -by the driver. When the workload's first process exits, the guest agent kills -the processes remaining in that group and, as the guest's init process, reaps -the orphans it inherits. It drains the workload's output before reporting the -outcome. A descendant that leaves the process group is not contained by this -plain-process execution and may keep an output pipe open after its leader -exits. A cancellation or timeout after the leader's status is known or lost -does not change that status while the output drains. This path has no execution -cgroup or workload namespace. - -A cancellation that names the running execution kills the workload, and a -cancellation for an execution that already finished is ignored. A session -reset means that the host session that could observe the workload is gone: -the guest agent kills the workload, sends nothing more for it, and -acknowledges the new epoch, and drops the old output descriptors rather than -waiting for them to close, so the next session never waits for an abandoned -workload. - -When the blockless agent ends a running workload for cancellation, timeout, or -the output cap, it collects the leader and drops the remaining output -descriptors rather than waiting for an escaped descendant to close them. -Cancellation and timeout first forward any available output up to the cap. -The layered sandbox continues to use its existing namespace and cgroup cleanup. +For nvx-init and the blockless managed root, the VM is the isolation and +resource boundary. An execution's process group, which its end, +cancellation, timeout and reset kill, is a cleanup scope, not a namespace or +a cgroup: a process that leaves the group escapes it. The init owns +termination, capture admission and cleanup; no workspace or per-execution +enforcement of `nvx` supplies them. The older layered sandbox root keeps its +namespace and cgroup cleanup. \ No newline at end of file diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 07e0eb35a..888c618fa 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -5,85 +5,80 @@ ## 4.3.1 Identity and Credentials -The workload runs under a host-selected, non-root numeric UID and GID fixed at -initial boot. The guest agents require an exact user and primary-group match -in the workload root and reject an unavailable identity before starting the -workload. In the layered sandbox root, each managed execution's entry checks -the account again, reading the root's user and group databases beneath its -directory descriptor, with bounds, before the workload runs. When the boot -configuration requests it, the guest agents of the blockless managed root -instead create an account for a host-selected identity that the image lacks, -and replace an image account that uses the UID with another primary group or -without a usable home. This lets a host run workloads under its own user's IDs. -The layered sandbox root creates no accounts, so it does not advertise +The workload runs under a host-selected, non-root numeric UID and GID, which +the boot fixes for the VM, or, for a restored nvx-init, its activation. The +guest agents require an exact user and primary-group match in the workload +root, and reject an unavailable identity before starting the workload: +nvx-init resolves the account once, before it switches into its root, and the +older layered sandbox root's entry checks it again for each execution. When +the boot configuration requests it, nvx-init and the guest agents of the +blockless managed root instead create an account for a host-selected identity +that the image lacks, and replace an image account that uses the UID with +another primary group or without a usable home. The older layered sandbox +root creates no accounts, so it does not advertise [account creation](4.4-host-guest-control-interface.md#442-feature-negotiation). -Before the workload starts, the guest agents clear supplementary groups and -every capability set and enable `no_new_privs`. With every capability cleared, -the workload cannot perform port I/O, so only the guest agents can request a -capture or acknowledge restore repair. +Before the workload runs, its child clears supplementary groups and every +capability set and enables `no_new_privs`, so the workload keeps none of the +privileges of the guest agents and their helpers. With every capability +cleared, the workload cannot perform port I/O, so only the guest agents can +request a capture or acknowledge restore repair. `nvx` enforces none of this: +[Run the executions](../guest-agent-guidelines.md#run-the-executions) gives +nvx-init's steps. ## 4.3.2 Namespaces and Mounts -In the layered sandbox root, the workload receives private mount, PID, and UTS -namespaces, in which every mount is private, and: - -- A private `/proc` and a read-only `/sys`. -- A restricted `/dev`, devpts, and shared memory. -- The workload root. -- Only the live shares that the host declared. -- The workload's machine ID, bound read-only from a runtime file so that the - guest agents can refresh it while scratch is frozen. -- The guest time status, bound read-only. - -A managed execution enters through the guest library's execution entry, as the -first process of its PID namespace. The entry moves the workload root over `/` -in the execution's mount namespace rather than entering it with `chroot` -alone, gives `/dev` only its allowlisted device nodes, and makes every other -mount `nodev`. The managed agent binds the machine ID and the time status into -the workload root once, in a mount namespace of its own, of which each -execution's starts as a copy, so those binds end with the agent, before the -init agent unmounts the root. A one-shot workload's entry helper enters the -root with `chroot`. - -The guest agents never pivot away from the initramfs, so they keep their own -root and stay outside the workload's namespaces, where they can supervise and -tear down the workload. The control device is never exposed inside the -workload's namespaces. - -The blockless managed root has no such isolation. Each execution runs in the -guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS -namespaces as the guest agents, so it sees the guest's own `/proc`, `/sys`, and -`/dev` and the host mappings bound into that root. The plain-process builder -applies its [credentials](#431-identity-and-credentials), but it has no -execution cgroup or private mount, process, hostname, or device view. +nvx-init's space is VM-wide: one root, with one set of runtime file systems, +one hostname, machine ID and resolver file, and the host's mappings, into +which PID 1 switches once. Its executions share that root and PID 1's mount, +PID and UTS namespaces: no execution has a namespace or a root of its own. +The root's `/dev` holds no control device: PID 1 opens it before the switch +and keeps its descriptor, and the helpers, which run from the initramfs, +start before the switch too. Once switched, nothing unmounts the root, and +the VM ends by powering off. +[Build the space](../guest-agent-guidelines.md#build-the-space) gives the +space's rules. + +The older layered sandbox root gives each workload private mount, PID, and +UTS namespaces, in which every mount is private, with a private `/proc`, a +read-only `/sys`, a restricted `/dev`, the workload root, only the live shares +that the host declared, and the workload's machine ID and the guest's time +status, bound read-only. Its guest agents never pivot away from the +initramfs, so they stay outside the workload's namespaces, where they +supervise and tear down the workload, and the control device is never exposed +inside them. The blockless managed root runs each execution in the guest's +initramfs root, in the guest agents' own namespaces, without a private mount, +process, hostname, or device view. ## 4.3.3 Cgroups and Resources -In the layered sandbox root, the guest agents and workload occupy sibling -cgroups. The workload is placed in its cgroup behind a barrier before it is -released. The guest agents' cgroup carries a default memory protection of -16 MiB, a soft reservation that shields the supervisor from reclaim under -workload memory pressure but does not guarantee its survival. The workload -cgroup carries optional memory and process-count limits and is the unit of -capture freeze, termination, and descendant cleanup. When the overlay's upper -is in guest memory, the files that the workload writes into its root are -charged to the workload cgroup, so its memory limit also bounds them; see +For nvx-init and the blockless managed root, the VM is the resource limit: +they configure no cgroups and no per-execution limits. nvx-init runs PID 1 and +its helpers ahead of the executions, whose children give up that priority, +instead of reserving memory for them in a sibling cgroup. Its one limit +within the VM is the RAM upper's cap; the scratch and the host's devices keep +their own storage boundaries. + +The older layered sandbox root keeps sibling guest-agent and workload cgroups. +The workload is placed in its cgroup behind a barrier before it is released. +The guest agents' cgroup carries a default memory protection of 16 MiB, a +soft reservation that shields the supervisor from reclaim under workload +memory pressure but does not guarantee its survival. The workload cgroup +carries optional memory and process-count limits and is the unit of capture +freeze, termination, and descendant cleanup. When the overlay's upper is in +guest memory, the files that the workload writes into its root are charged to +the workload cgroup, so its memory limit also bounds them; see [Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). -The blockless managed root does not use execution cgroups. Its plain-process -driver terminates the workload's process group. This path does not configure a -sibling guest-agent cgroup, the 16-MiB memory protection, or the layered root's -optional memory and process-count limits. - ## 4.3.4 Enforcement Layers Isolation has two enforcement layers: -- Guest policy constructs the workload's credentials, mounts, capabilities, - and cgroups and, in the layered sandbox root, its namespaces. +- Guest policy, which the init applies and `nvx` does not enforce: the + workload's credentials and capabilities, the mounts of its space, and, in + the older layered sandbox root, its namespaces and cgroups. - Host devices independently enforce block mutability, network policy, and host-filesystem access. The guest cannot widen a host policy by remounting a filesystem, changing a -route, creating a link, or issuing a different device request. +route, creating a link, or issuing a different device request. \ No newline at end of file diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md new file mode 100644 index 000000000..189296142 --- /dev/null +++ b/doc/guest-agent-guidelines.md @@ -0,0 +1,449 @@ +# Guest-agent guidelines + +`nvx` parses the contracts, captures the guest and drives a restore's +activation, but builds no space and runs no executions: each init agent does +both, and `nvx` enforces none of the obligations below. The one exception is +the time daemon, which `TimeAbi::start_daemon` forks. + +These guidelines give an init agent's contract: how it builds its guest's +space, runs its executions, coordinates a capture with its host and completes +a restore. `nvx-init`, NVX's PID 1 for layered roots, is the worked example +throughout. The design's +[Guest Agents and Workload Model](design/4.2-guest-agents-and-workload-model.md) +and [Isolation and Resource Model](design/4.3-isolation-and-resource-model.md) +give the architecture and its boundaries, and the rustdoc of +[`guest`](../nvx-rs/src/guest/mod.rs), +[`guest::snapshot`](../nvx-rs/src/guest/snapshot.rs), +[`guest::activation`](../nvx-rs/src/guest/activation.rs) and +[`activation`](../nvx-rs/src/activation/mod.rs) the public mechanisms. The +[`guest-workflows`](../nvx-rs/examples/guest-workflows.rs) example compiles an +init's calls of them, through the public API alone: its time daemon, its +control session, a capture, a restore's activation and the host's mapping +table. + +## Ownership + +An init reads the public boot contracts at a cold boot, or a restore's +activation record, and builds the VM's one space from them. It owns the root, +with its runtime mounts and view binds; the mapper and the descriptors of the +mappings' source and target roots; its helpers; its control session; and +every execution. It keeps each owner for as long as what it owns is in use: a +reset of the transport is neither a new root nor a new activation. + +The library's calls leave all of this to the init: + +- `guest::snapshot::capture` stops no execution, drives no transport and never + powers the VM off: it returns how the capture ended, or a failure with its + disposition. +- `guest::activation::Activation` holds no workspace or admission fence, makes + no system call and starts no process: the init feeds it what its session + receives, and does what it returns. +- `TimeAbi::start_daemon` forks the time daemon, which the init supervises as + a helper. nvx-init's early boot, the shell `/init`, starts it with + `nvx-time boot`, and the announcer when the network publishes the workload's + listeners, then replaces itself with nvx-init, which inherits both as its + children. + +One other call starts processes, which an init can avoid: a report-only +[`TimeAbi::restore_finish`](../nvx-rs/src/guest/time/api.rs), which +`capture`'s repair calls, runs the restore's deferred time work itself when +no time daemon runs, in a calling process that must have no other thread: it +forks a child, which waits for a grandchild that forces an RCU grace period, +and collects only the child, so a grandchild that outlives the RCU stall +timeout passes to PID 1, or to the nearest subreaper. No call starts the +announcer, which is a program, `/sbin/nvx-ingress`: an init runs it itself, +or has `nvx-ingress start` run it from a socket that it binds for it. Until +the nvx-rs slimming +([microsoft/nvx#1076](https://github.com/microsoft/nvx/issues/1076)) removes +them, the crate also keeps the older execution engine and its process +modules, `guest::{exec, spawn, entry, cgroup, process}`, which NVX's older +guest agents use. nvx-init uses none of these modules, and its time daemon +runs for the VM's life. + +## Build the space + +NVX's space is internal reference code that nvx-init implements, not public +API: the modules that hold it, `guest::root`, `guest::view`, `guest::gpt`, the +mapper of `guest::mappings`, and `guest::activation::apply` with its types, +are hidden from the documentation, unstable, and become private to the crate +once nothing outside NVX uses them. Another init builds its own space, or +adapts this code; it builds on no space module of `nvx`. + +The space's inputs are public contracts: + +- At a cold boot, nvx-init reads them from the kernel command line: the root's + tokens, which `protocol::root::RootLayout` parses; the view's, which + `protocol::view` parses; the host export's, which + `protocol::export::HostExport` parses, or the live shares of NVX's CLI, + whose targets `protocol::share` restricts; the DNS server, which + `protocol::cmdline::dns_server` reads; and the number of host mappings, + which `MappingTable::announced` reads. +- After a restore, the accepted `activation::ActivationRecord` holds the + launch's settings, and the configuration that the snapshot captured, the + same for each restore, holds the assembly's fixed inputs: the runtime + directory, the block devices, the host filesystem device's tag, the NIC's + gateway, and how long the assembly waits for the devices. Never take a + captured launch's view or network in place of the restore's own record. +- The host's mapping table, when it holds any mapping, arrives once the init + is ready, in `MAPS` parts, which `protocol::maps` decodes. + +nvx-init's space keeps these rules: + +- **The root.** The image on the `distro` device, an EROFS file system or a + GPT disk, whose identity the root checks when the contract names one, lies + under an overlay with its upper, mounted `nosuid,nodev` below the runtime + directory, `/run/nvx`. The upper is the ext4 scratch, or a tmpfs whose cap + the contract may set from 1 to 90 percent of the VM's memory, and which + otherwise takes the kernel's default, half of it. A restore's upper is + fresh: the assembly refuses one whose `upper` or `work` directory holds + anything. +- **The runtime file systems.** The root makes every mount of its tree + `nosuid` and `nodev`, then mounts `/proc`, a read-only `/sys`, a `/dev` that + holds only the null, zero, full, random, urandom and tty devices, with + `/dev/pts` and `/dev/shm`, and `/run`, and binds the time status read-only + at `/run/nvx/time`. They come before any live share or mapping, since the + `/run` tmpfs would hide one beneath `/run`: a cold boot's live shares follow + them. +- **The view.** It is VM-wide: the hostname; the image's own machine ID + (`MachineIdMode::Image`), or one that the init generates and binds read-only + over it (`Generated`); and, when the boot contract or this restore's record + names a DNS server, a resolver file that names it, bound over the image's. + The root owns these binds, those made before a later failure included. +- **The mappings.** The host export, from which mappings come, is mounted + outside the root, below the runtime directory, where only the guest's root + user reaches it. The init opens the root's descriptor as the mappings' + target root, adds the runtime's `/run` mount to it, and makes the mapper of + it, which refuses the targets that the runtime reserves; it keeps the + export's source root. A mapping binds beneath these descriptors, never by + path, so both outlive the switch. +- **The helpers.** Every helper runs, and every descriptor that later work + needs is open, before the switch: the root holds neither the initramfs's + programs nor the control device. nvx-init's early boot starts its helpers, + and nvx-init opens its control terminal before it switches; a restore starts + its announcer there too. PID 1 and its helpers outrank every execution, as + [Run the executions](#run-the-executions) describes. +- **The one switch.** The init moves the root onto `/`, then enters it with + `chroot` and `chdir`, once: executions need no root of their own. From then + on nothing unmounts the root, and the VM ends by syncing and powering off. + Before the switch, a step that fails admits no workload: the root tries to + unmount what it mounted, in reverse, and nvx-init ends the VM, whatever the + cleanup leaves mounted. + +### The space's steps, checked + +These excerpts are nvx-init's unit test +[`the_reference_space_steps_compile`](../nvx-rs/src/guest/init/boot/tests.rs), +which compiles them without mounting anything or switching the developer +host's root. They use the space's own names, of `root`, `view`, the mapper and +`activation::{apply, Applied, RootTemplate}`, which are NVX's internal +reference code, not public API. A cold boot's space takes these steps in the +order of [`boot::cold_boot`](../nvx-rs/src/guest/init/boot.rs), which also +mounts the live shares of NVX's CLI after the runtime file systems: + +```rust +use std::error::Error; +use std::net::IpAddr; +use std::path::Path; +use std::time::Instant; + +use crate::activation::{ActivationRecord, View}; +use crate::guest::activation::{self, Applied, RootTemplate}; +use crate::guest::mappings::{Mapper, Reserved, SourceRoot, TargetRoot}; +use crate::guest::root::{self, Assembly, Root}; +use crate::guest::view; +use crate::protocol::export::HostExport; + +// A cold boot, as `boot::cold_boot` builds it. +let _cold = |assembly: &Assembly, + export: &HostExport, + vm_view: &View, + dns: Option| + -> Result<(Root, Mapper, SourceRoot), Box> { + let mut root = root::assemble(assembly)?; + let source = root.mount_export(&export.tag, export.mode.into())?; + root.mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(root.path())?; + target.add_mount(&root.path().join("run"))?; + let mapper = Mapper::new(target, Reserved::container_root()); + view::apply(&mut root, vm_view, dns)?; + // Start the helpers and open the descriptors that later work needs before the switch. + root.switch()?; + Ok((root, mapper, source)) +}; +``` + +A restore's application assembles the record's root, mounts its export and +applies its view, with its DNS server, and starts no process; the init then +mounts the runtime file systems, opens the mappings' handles, starts the +announcer when the record publishes the workload's listeners, and switches: + +```rust +// A restore, as `boot::Restored::apply` builds it from the accepted record. +let _restored = |record: &ActivationRecord, + template: &RootTemplate, + deadline: Option| + -> Result<(Applied, Mapper, Option), Box> { + let mut applied = activation::apply(record, template, deadline)?; + applied + .root_mut() + .mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(applied.root().path())?; + target.add_mount(&applied.root().path().join("run"))?; + let mapper = Mapper::new(target, Reserved::container_root()); + let source = applied.take_export(); + // Start the announcer here when `applied.publication()` publishes the listeners. + applied.root_mut().switch()?; + Ok((applied, mapper, source)) +}; +``` + +### The host's mapping table + +The public [`MappingTable`](../nvx-rs/src/guest/mappings/table.rs) keeps the +account of the host's mapping table, and the init's mapper binds the mappings +that it hands out: one does not replace the other. The host announces how many +mappings its table holds, at a cold boot with `nvx_maps=COUNT`, which it leaves +out for none, and after a restore in the record. Once the init is ready, it +sends a table that holds any mapping in parts of consecutive entries, and an +empty one not at all: + +- **Parts in order.** `MappingTable::apply` refuses a part as a whole, and + changes nothing, when it does not decode, holds no entry, does not start at + the entry that the table expects next, ends past the announced count, or + holds an entry whose source is empty, which names the export's root, or + whose path a `Mapping` refuses. Otherwise it hands the part's mappings, in + order, to the init's mapper. +- **A latch after a failure.** A mapping that fails keeps the mappings before + it and latches the table, which refuses every later part and is never + complete; the init reports the failure and its entry. An init whose protocol + ends the table at a refused part latches it then, with `MappingTable::latch`. +- **A complete table before any workload.** The init starts no workload until + `MappingTable::is_complete`: until it applied every announced mapping and + none failed. A table of no mappings is complete at once. + +nvx-init answers a refused part with `invalid-request`, a failed mapping with +`mapping-failed`, after a diagnostic that names its entry, and refuses every +execution with `mappings-incomplete` until its table is complete. The +guidelines quote this excerpt of the [`guest`](../nvx-rs/src/guest/mod.rs) +rustdoc, where it compiles as a public-API doc test: + +```rust +use nvx::guest::mappings::{Mapping, MappingTable, TableError}; + +fn apply_part( + table: &mut MappingTable, + part: &[u8], + mount: impl FnOnce(&[Mapping]) -> Result<(), (usize, E)>, +) -> Result> { + table.apply(part, mount)?; + Ok(table.is_complete()) +} +``` + +## Run the executions + +An init runs its executions itself. nvx-init runs them one at a time, at the +host's request, as plain processes in the VM's one space, with its own +plain-process core: + +- **Before the fork**, it builds the program's arguments and environment. The + default environment is nvx-init's own, with the workload account's `HOME`, + `USER` and `LOGNAME` and the working directory's `PWD`; the request's + entries replace it, or layer over it when the request asks. The account is + the VM's workload identity, the `nvx_identity=UID:GID` of a cold boot or the + identity of a restore's activate request, which nvx-init resolves in the + root's user and group databases once, before its switch, and creates when + the boot, with `nvx_workload_account=create`, or the activate request asks + for it. +- **In the child**, which leads a process group of its own, with the null + device as its standard input and pipes as its standard output and error, + the execution resets its priority, sets its credentials, changes to its + working directory and execs. It resets its out-of-memory score adjustment + and its nice value to 0, then drops its capability bounding set and takes + the account's user and group IDs, with no supplementary groups, empty + capability sets and `no_new_privs`. It then enters the requested working + directory, `/` by default, with those credentials, resets its signals, and + execs. A step that fails reports its stage and `errno` to the init and exits + with status 125: the execution fails, rather than run as another identity or + in another directory. +- **End, cancellation, timeout and reset** each kill the execution's process + group: nvx-init kills it when the execution's leader ends, normally or not, + when the host cancels the execution, when its timeout passes, and when a + session reset abandons it, whose output it then discards rather than report + it on the new session. PID 1 collects the status of every child, orphans + included. A process group is a cleanup scope, not containment: a process + that leaves its group escapes it, as the capture's sweep allows for. + +This is priority, not limits. PID 1 and its helpers run with an out-of-memory +score adjustment of -1000 and a nice value of -10, ahead of the executions' +0, which each child restores before it drops its privileges, so that no +workload keeps them. There are no cgroups, namespaces, per-execution roots or +per-execution limits: the VM is the limit, and the one limit within it is the +RAM upper's cap. + +## Coordinate a capture + +An init that runs concurrent executions keeps them in workspaces, under a +contract that is the init's own: `nvx` provides no workspace trait and +enforces none of it. + +- `freeze(deadline)` refuses new executions, then stops those that run. A + frozen workspace refuses a start rather than hold it until the thaw. +- A failed freeze says whether it undid its work (`undone`). When it did not, + the VM ends: no work resumes in a partly frozen state. +- Several workspaces freeze in order, all or nothing, and thaw in reverse. + After a failure with `undone`, those already frozen thaw, in reverse; + without it, the VM ends and nothing thaws. +- `thaw()` reverses a freeze, and `after_restore()` renews what the + executions need after a restore, before Ready. + +nvx-init has no workspace: it refuses a capture while an execution runs, or +kills the process groups and sweeps when the host asks, as its capture +request says. Without permission to kill, it refuses the request with `EBUSY` +while any process but PID 1 and its registered helpers runs; with it, it ends +a running execution's process group, then sweeps, killing every process but +PID 1 and its helpers again and again until none remains. The checks, the +sweep and the capture's own preparation share one 30-second deadline. Today +nvx-init captures only the Fresh boot of a platform capture, +`nvx_snapshot_tier=platform`, which assembles no root and admits no +execution: a session with a root, cold-booted or restored, refuses the request +with `ENOTSUP` before it checks or kills anything. + +An init captures before it mounts anything from the devices that a restore +replaces: the `distro` image, the scratch and the host filesystem device, and +any overlay. `capture` refuses while one is mounted, but the decision to +capture, and the cleanup before it, are the init's. + +## Keep the transport silent + +The init answers the host's capture request, then sends nothing on its +transport until `capture` returns: no progress, output or other response. +When the call returns: + +- After `Captured::Continued`, it thaws its workspaces in reverse, where they + exist, and reports on the same session. nvx-init reports + `capture-continued`. +- After a failure whose disposition is `Disposition::Resumed`, it thaws and + reports the failure on the same session. nvx-init writes the failure to the + console, after `NVX-INIT-CAPTURE:`, and reports `capture-failed`. +- After `Captured::Restored`, it takes the session's reset and attaches again: + the restored broker has a fresh instance, restarts at its first epoch with a + reset queued, and ignores the guest's records until the guest acknowledges + it. nvx-init waits for the reset, within the activation's deadline, fails on + anything else, and acknowledges it with `GuestConnection::acknowledge` + before it takes the activation. + +The guidelines quote this excerpt of the +[`guest::snapshot`](../nvx-rs/src/guest/snapshot.rs) rustdoc, where it +compiles as a public-API doc test: + +```rust +use std::time::{Duration, Instant}; + +use nvx::guest::activation::Activation; +use nvx::guest::snapshot::{self, Captured, Devices, Disks, Disposition}; + +/// What the caller does next; it sent nothing on its transport while `capture` ran. +enum Next { + /// Thaw the workspaces in reverse, where they exist, and report on the same session. + Report, + /// Take the session's reset and acknowledge it, then drive this activation to Ready. + Activate(Activation), + /// Start nothing, and power the VM off with this status. + PowerOff(u8), +} + +fn capture(devices: &Devices, deadline: Instant, timeout: Duration) -> Next { + match snapshot::capture(devices, Disks::Fresh, deadline) { + Ok(Captured::Continued) => Next::Report, + Ok(Captured::Restored(repaired)) => { + Next::Activate(Activation::new(&repaired, timeout, Instant::now())) + } + Err(failure) => match failure.disposition { + Disposition::Resumed => Next::Report, + Disposition::MustEnd { status } => Next::PowerOff(status), + // A disposition that a later version adds cannot be known to be safe. + _ => Next::PowerOff(125), + }, + // Nor can an outcome that a later version adds. + Ok(_) => Next::PowerOff(125), + } +} +``` + +## Complete a restore before Ready + +`capture` repairs a restored guest before it returns +`Captured::Restored(repaired)`: it activates the restore's processor and +memory targets, reseeds the kernel's random number generator, and finishes +the restore with its acknowledgement. The init makes one `Activation` of that +`Repaired`, whose deadline counts from the repair's acknowledgement, or from +when the init makes it for a restore that the host did not gate on one, and +never from a session or a retry. A restore then takes these steps, in this +order: + +1. `Activation::new`. nvx-init makes it when `capture` returns, with a + 30-second timeout, takes the session's reset, and accepts the activate + request: the activation decodes its record, and checks its restore section + and the CPUs and memory that the repair reports, before anything has a side + effect. +2. The space: the application of the record, with this restore's own network + policy, then the runtime file systems, the announcer and the switch, as + [the excerpt](#the-spaces-steps-checked) shows. The init applies the + record's DNS server, and, when the record publishes the workload's + listeners, starts the announcer before Ready and owns it as a helper. + nvx-init forks it from PID 1 before its switch, and runs + `/sbin/nvx-ingress` with the captured NIC's gateway, as the early boot runs + it at a cold boot. nvx-init also tells the record's proxy to the executions + that keep their default environment. +3. `after_restore()` on each workspace, where workspaces exist. +4. `Activation::complete`, with the result. +5. The thaw of the workspaces, in reverse, where they exist. +6. Ready. nvx-init answers the activate request, and each retry that joined + it, with Ready, which carries the record's transaction ID. + +nvx-init has no workspace, so it skips the third and fifth steps, not the +order around them. It keeps the applied root, the mapper and its +`MappingTable`, the export's source root, its helpers and the `Activation` +for the VM's life. The host then sends the mapping table, unless it is empty, +and nvx-init records its first execution, once the table is complete, with +`Activation::start`. + +A byte-equal retry joins the activation in progress, or gets its cached +Ready, on a later session too, and never applies or switches again: nvx-init +keeps the activation and its original deadline across sessions, and a reset +on a later session only checks that deadline. A changed request, another +transaction, and an application that failed, wholly or partly, are final. + +The packet with which a platform restore resumes the Fresh boot carries no +network settings, and `capture` refuses one that does: the activation record +gives the restore its network. + +## Failures + +- After `Disposition::Resumed`, the init thaws and reports the failure on the + same session. +- After `Disposition::MustEnd { status }`, it starts nothing, thaws nothing, + and powers the VM off with that status: the library returns the disposition + rather than power the VM off itself. nvx-init writes the failure to the + console, then syncs and powers off with the status. +- A failed activation, wholly or partly applied, never reports Ready: the init + reports the failure and ends the VM. nvx-init writes it to the console, + after `NVX-INIT-ACTIVATION:`, and powers off with status 125. + +## Replay + +Each restore resumes the captured memory, so: + +- An init never captures live external connections, or any other state that + a restored guest cannot reuse safely: it closes them before the capture, or + rebuilds them after the restore. +- It never repeats an irreversible effect because execution resumed from + captured memory. +- The repair reseeds the kernel's random number generator, but not a runtime's + own random state: a runtime repairs its randomness before it accepts work + after a restore, in its workspace's `after_restore()` where it has one. + +nvx-init captures a Fresh boot, before any workload runs, so a restore resumes +no workload's state, and its view generates a machine ID only after the +repair's reseed, when the record asks for one. diff --git a/doc/project-structure.md b/doc/project-structure.md index 6198eb022..591368cd7 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -82,6 +82,7 @@ nvx/ | |-- contribute.md Contribution guidelines | |-- design/ Architecture of the microVM and its guest runtime | |-- distribution.md Packaging and source delivery +| |-- guest-agent-guidelines.md Contract of a guest's init agent, with nvx-init as its example | |-- guests/ Guest userland chapters (Ubuntu and Azure Linux) | |-- modules/ Module diagrams of the nvx crate | |-- openvmm-upstream-roadmap.md Plan for upstreaming the OpenVMM fork's microVM commits diff --git a/nvx-rs/src/activation/mod.rs b/nvx-rs/src/activation/mod.rs index 80dc5fdf3..215233bea 100644 --- a/nvx-rs/src/activation/mod.rs +++ b/nvx-rs/src/activation/mod.rs @@ -17,6 +17,13 @@ //! - [`ReadyActivation`]: what Ready reports about the activation: that it applied, which every //! layered root reports, and the transaction ID, which only a clone reports. //! +//! The [guest-agent guidelines] give the order in which an init completes a restore before Ready: +//! it applies the record, with this restore's own network policy, mounts the runtime's file +//! systems, starts the announcer when the record publishes the workload's listeners, and switches +//! into the root once. +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! [`ActivationRecord::new`] admits a configuration only once [`Configuration::check`] finds that //! it keeps every [limit](#limits), so that every record holds settings that a guest can apply. //! [`ActivationRecord::encode`] writes a record's canonical [encoding](#encoding), and diff --git a/nvx-rs/src/guest/activation.rs b/nvx-rs/src/guest/activation.rs index 7f6436a6b..a617d2ab3 100644 --- a/nvx-rs/src/guest/activation.rs +++ b/nvx-rs/src/guest/activation.rs @@ -145,6 +145,12 @@ //! and its [`ActivationCause`], and which every later call returns again. The activation never //! powers the VM off: the adapter reports the failure, and ends the VM. //! +//! The [guest-agent guidelines] give the order of a restore around the activation, from +//! [`Activation::new`] to Ready, with nvx-init as the worked example, and the obligations of +//! its caller, which the activation enforces none of. +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! The activation's application, `apply` and its types, is NVX's internal reference code of the //! guest's space, hidden from this documentation and unstable: it becomes crate-private once //! nothing outside NVX uses it. diff --git a/nvx-rs/src/guest/init/boot/tests.rs b/nvx-rs/src/guest/init/boot/tests.rs index b1c583acc..d8e75bec0 100644 --- a/nvx-rs/src/guest/init/boot/tests.rs +++ b/nvx-rs/src/guest/init/boot/tests.rs @@ -9,63 +9,56 @@ const CLI: &str = "nvx_identity=1000:1001 nvx_control_tty=hvc2"; const EXPORT: &str = "virtfs_dir=/run/nvx/hostfs/root virtfs_tag=hostfs virtfs_mode=rw virtfs_aggregate=1"; -/// The steps of NVX's reference space, as nvx-init takes them and the guest-agent guidelines -/// quote them, compile against the space's own interface. The test runs none of them: they would -/// mount over and switch the test host's root. +/// Compiles the steps of the private space that `doc/guest-agent-guidelines.md` quotes, without +/// mounting anything or switching the developer host's root. A change here changes the quote too. #[test] fn the_reference_space_steps_compile() { - // A cold boot: the root, an image under its upper, and the host export; the runtime's file - // systems; the view; then the VM's one switch. The mapper keeps the root's descriptors for - // the host's mapping table, which comes after Ready. - let cold = |assembly: &Assembly, - export: Option<&HostExport>, - configuration: &View, - dns: Option| - -> Result<(Root, Mapper, Option)> { - let mut root = root::assemble(assembly).map_err(system_error)?; - let source = export - .map(|export| root.mount_export(&export.tag, export.mode.into())) - .transpose() - .map_err(system_error)?; - root.mount_runtime(Some(Path::new("/run/nvx/time"))) - .map_err(system_error)?; - let mut target = TargetRoot::open(root.path()).map_err(system_error)?; - target - .add_mount(&root.path().join("run")) - .map_err(system_error)?; + use std::error::Error; + use std::net::IpAddr; + use std::path::Path; + use std::time::Instant; + + use crate::activation::{ActivationRecord, View}; + use crate::guest::activation::{self, Applied, RootTemplate}; + use crate::guest::mappings::{Mapper, Reserved, SourceRoot, TargetRoot}; + use crate::guest::root::{self, Assembly, Root}; + use crate::guest::view; + use crate::protocol::export::HostExport; + + // A cold boot, as `boot::cold_boot` builds it. + let _cold = |assembly: &Assembly, + export: &HostExport, + vm_view: &View, + dns: Option| + -> Result<(Root, Mapper, SourceRoot), Box> { + let mut root = root::assemble(assembly)?; + let source = root.mount_export(&export.tag, export.mode.into())?; + root.mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(root.path())?; + target.add_mount(&root.path().join("run"))?; let mapper = Mapper::new(target, Reserved::container_root()); - view::apply(&mut root, configuration, dns)?; - root.switch().map_err(system_error)?; + view::apply(&mut root, vm_view, dns)?; + // Start the helpers and open the descriptors that later work needs before the switch. + root.switch()?; Ok((root, mapper, source)) }; - // The host's mapping table, part by part, beneath the switched root. - let map = |table: &mut MappingTable, mapper: &mut Mapper, source: &SourceRoot, part: &[u8]| { - table.apply(part, |mappings| { - mapper - .apply(source, mappings) - .map_err(|error| (error.entry(), error)) - }) - }; - // A restore: the record's root, export and view; the runtime's file systems; the announcer - // of the restore's listeners, which the init starts and owns; then the VM's one switch. - let restored = |record: &activation::ActivationRecord, - template: &RootTemplate, - deadline: Option, - announce: &mut dyn FnMut(Option) -> Result<()>| - -> Result { - let mut applied = - guest_activation::apply(record, template, deadline).map_err(activation_error)?; + // A restore, as `boot::Restored::apply` builds it from the accepted record. + let _restored = |record: &ActivationRecord, + template: &RootTemplate, + deadline: Option| + -> Result<(Applied, Mapper, Option), Box> { + let mut applied = activation::apply(record, template, deadline)?; applied .root_mut() - .mount_runtime(Some(Path::new("/run/nvx/time"))) - .map_err(activation_error)?; - if applied.publication() == activation::Publication::Published { - announce(applied.gateway())?; - } - applied.root_mut().switch().map_err(activation_error)?; - Ok(applied) + .mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(applied.root().path())?; + target.add_mount(&applied.root().path().join("run"))?; + let mapper = Mapper::new(target, Reserved::container_root()); + let source = applied.take_export(); + // Start the announcer here when `applied.publication()` publishes the listeners. + applied.root_mut().switch()?; + Ok((applied, mapper, source)) }; - let _ = (cold, map, restored); } #[test] diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 9b9fd61dd..7e67ddf1d 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -17,6 +17,11 @@ //! acknowledgement, and then takes its own configuration from the host, in one Activate on a //! restricted control session. //! +//! The [guest-agent guidelines] give an init agent's contract for its space, its executions, a +//! capture, and a restore, with nvx-init as the worked example; the library enforces none of it. +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! The guest's space, the root that an init assembles and what it mounts in it, is each init's //! own. NVX's own space is internal reference code, which NVX's guest programs use: it is hidden //! from this documentation and unstable, and it becomes crate-private once nothing outside NVX @@ -30,6 +35,22 @@ //! stay public: the mapping values and [`MappingTable`](mappings::MappingTable), and the tokens //! and records of [`protocol`](crate::protocol) and [`activation`](crate::activation). //! +//! The guidelines quote this excerpt, in which an init applies one part of the host's mapping +//! table through its own mapper, `mount`, and admits workloads only once the table is complete: +//! +//! ```no_run +//! use nvx::guest::mappings::{Mapping, MappingTable, TableError}; +//! +//! fn apply_part( +//! table: &mut MappingTable, +//! part: &[u8], +//! mount: impl FnOnce(&[Mapping]) -> Result<(), (usize, E)>, +//! ) -> Result> { +//! table.apply(part, mount)?; +//! Ok(table.is_complete()) +//! } +//! ``` +//! //! Each other module holds one program, and the crate's binary of the same name runs its `main`. //! Two of them also offer their steps as typed calls, which any guest init or agent runs in its //! own process and their programs run on: on x86-64, `time`'s `TimeAbi` runs the time ABI's diff --git a/nvx-rs/src/guest/snapshot.rs b/nvx-rs/src/guest/snapshot.rs index 03591d30d..e719dbd09 100644 --- a/nvx-rs/src/guest/snapshot.rs +++ b/nvx-rs/src/guest/snapshot.rs @@ -18,6 +18,46 @@ //! instant. A repair failure is always [`Disposition::MustEnd`], with [`RepairError::status`]. //! Fresh restore network settings are explicitly refused. //! +//! The [guest-agent guidelines] give the caller's obligations around this call: its +//! workspaces, its silent transport, a failure's disposition, and the restore's order. They +//! quote this excerpt of what the caller does once the call returns: +//! +//! ```no_run +//! use std::time::{Duration, Instant}; +//! +//! use nvx::guest::activation::Activation; +//! use nvx::guest::snapshot::{self, Captured, Devices, Disks, Disposition}; +//! +//! /// What the caller does next; it sent nothing on its transport while `capture` ran. +//! enum Next { +//! /// Thaw the workspaces in reverse, where they exist, and report on the same session. +//! Report, +//! /// Take the session's reset and acknowledge it, then drive this activation to Ready. +//! Activate(Activation), +//! /// Start nothing, and power the VM off with this status. +//! PowerOff(u8), +//! } +//! +//! fn capture(devices: &Devices, deadline: Instant, timeout: Duration) -> Next { +//! match snapshot::capture(devices, Disks::Fresh, deadline) { +//! Ok(Captured::Continued) => Next::Report, +//! Ok(Captured::Restored(repaired)) => { +//! Next::Activate(Activation::new(&repaired, timeout, Instant::now())) +//! } +//! Err(failure) => match failure.disposition { +//! Disposition::Resumed => Next::Report, +//! Disposition::MustEnd { status } => Next::PowerOff(status), +//! // A disposition that a later version adds cannot be known to be safe. +//! _ => Next::PowerOff(125), +//! }, +//! // Nor can an outcome that a later version adds. +//! Ok(_) => Next::PowerOff(125), +//! } +//! } +//! ``` +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! Each line of `/proc/self/mountinfo` describes one mount with fields separated by spaces: its //! ID, its parent's ID, the device number of its file system, `MAJ:MIN`, the root of the mount //! in its file system, the mount point, the mount's options, optional fields, a `-` that ends