From 75a6da5ce0218d738ebd4750a47c66d1907f29f5 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 8 Nov 2026 15:48:08 -0800 Subject: [PATCH 1/5] doc: document init-agent lifecycle obligations Centralize the guest-space, execution, capture and restore contract in one guide, with nvx-init as the internal reference example. Keep the architecture pages focused on boundaries and check the quoted public and private calls through existing rustdoc and unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 41345a73-b73f-4332-9ffd-9c513214af7a --- .../4.2-guest-agents-and-workload-model.md | 87 ++---- .../4.3-isolation-and-resource-model.md | 94 ++---- doc/guest-agent-guidelines.md | 294 ++++++++++++++++++ nvx-rs/src/activation/mod.rs | 5 + nvx-rs/src/guest/init/boot/tests.rs | 71 ++--- nvx-rs/src/guest/mod.rs | 21 ++ nvx-rs/src/guest/snapshot.rs | 19 ++ 7 files changed, 409 insertions(+), 182 deletions(-) create mode 100644 doc/guest-agent-guidelines.md diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index cff382e37..5660d5321 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -3,84 +3,47 @@ [Architecture documentation](README.md) > [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) -As Figure 4.2 shows, the guest agents are the guest's long-lived lifecycle -authority. Their responsibilities are: - -1. Initialize runtime filesystems and cgroup v2. -2. Discover and validate the fixed virtual devices. -3. Assemble the workload filesystem. -4. Apply workload identity, isolation, and resource policy. -5. Start the workload behind a placement barrier. -6. Supervise the workload and reap descendants. -7. Announce the guest's listeners when the host publishes them. -8. Participate in capture and post-restore repair. -9. Unmount resources and report the final workload status. +The init agent is the guest's long-lived lifecycle authority. It owns the +VM-wide space, helpers, control session and executions. `nvx` supplies contract +parsers, capture and restore-activation mechanisms, but builds no space, runs +no executions and enforces none of the init's lifecycle obligations. The +explicit exception is `TimeAbi::start_daemon`, which forks the time daemon. ```mermaid flowchart TB Kernel["Linux kernel"] - Agents["Guest agents
outside workload namespaces
in the layered root"] - Root["Assembled workload root"] - Isolation["Identity, capabilities,
and cgroups, plus namespaces
in the layered root"] - Workload["Workload process tree"] + Agents["Init agent
space and lifecycle owner"] + Root["One VM-wide space
root, view and mappings"] + Workload["Execution children
one process group each"] Kernel --> Agents Agents --> Root - Agents --> Isolation Root --> Workload - Isolation --> Workload Agents -->|"supervise and reap"| Workload ``` **Figure 4.2.** Relationship between the guest agents and the workload. -In a one-shot lifecycle, the guest agents start one workload, configured by -non-secret kernel command-line tokens, and shut down the VM when it exits. In -a managed lifecycle, the guest agents stay resident and run a sequence of -workloads, one at a time, on request over the authenticated control channel. -Both modes preserve the same separation: the workload is always a child of a -guest agent. The guest agents never replace themselves with the workload, -because supervision, orphan reaping, and the control service all depend on -them remaining alive. +The workload remains a child; it does not replace the init, whose continued +existence supports supervision, orphan reaping and the control service. +`nvx-init` is the reference implementation, using one shared root and sequential +executions. An init with concurrent executions owns its own workspace +coordination; the library provides no workspace trait. -## 4.2.1 Execution Launch +The detailed construction, execution, capture, transport and restore rules, +with checked excerpts of the reference implementation, live in the +[guest-agent guidelines](../guest-agent-guidelines.md). -For each managed execution, the guest agent's launch enters the requested -working directory, or `/`, with the workload's identity, and sets `PWD` unless -the request replaces the environment. The working directory is entered once -and kept, so the workload starts in the directory that was checked even if its -path is renamed, replaced, or retargeted in the meantime. If the workload -cannot enter it, the execution fails rather than running elsewhere. +## 4.2.1 Execution Launch -In the blockless managed root, the guest agent uses the init agent's -plain-process builder and driver. The builder prepares the environment, and -the child applies its identity, enters its working directory, and executes the -workload without a launch helper. In the layered sandbox root, the guest -agent's [execution entry](4.3-isolation-and-resource-model.md#432-namespaces-and-mounts) -applies the identity itself, and the guest agent builds the environment. +Execution launch belongs to the init, not the library's capture or activation +mechanisms. Children use the VM-wide space and the init's selected credentials +and working directory. There is no per-execution root entry or placement +barrier in this model. ## 4.2.2 Containment -In the blockless managed root, each execution runs in a process group pinned -by the driver. When the workload's first process exits, the guest agent kills -the processes remaining in that group and, as the guest's init process, reaps -the orphans it inherits. It drains the workload's output before reporting the -outcome. A descendant that leaves the process group is not contained by this -plain-process execution and may keep an output pipe open after its leader -exits. A cancellation or timeout after the leader's status is known or lost -does not change that status while the output drains. This path has no execution -cgroup or workload namespace. - -A cancellation that names the running execution kills the workload, and a -cancellation for an execution that already finished is ignored. A session -reset means that the host session that could observe the workload is gone: -the guest agent kills the workload, sends nothing more for it, and -acknowledges the new epoch, and drops the old output descriptors rather than -waiting for them to close, so the next session never waits for an abandoned -workload. - -When the blockless agent ends a running workload for cancellation, timeout, or -the output cap, it collects the leader and drops the remaining output -descriptors rather than waiting for an escaped descendant to close them. -Cancellation and timeout first forward any available output up to the cap. -The layered sandbox continues to use its existing namespace and cgroup cleanup. +The VM is the isolation and resource boundary. Process groups provide an +execution cleanup scope, not a namespace or cgroup containment guarantee. +The init owns termination, capture admission and cleanup; no library workspace +or per-execution enforcement layer supplies them. diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 07e0eb35a..a2c20a2b3 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -5,85 +5,35 @@ ## 4.3.1 Identity and Credentials -The workload runs under a host-selected, non-root numeric UID and GID fixed at -initial boot. The guest agents require an exact user and primary-group match -in the workload root and reject an unavailable identity before starting the -workload. In the layered sandbox root, each managed execution's entry checks -the account again, reading the root's user and group databases beneath its -directory descriptor, with bounds, before the workload runs. When the boot -configuration requests it, the guest agents of the blockless managed root -instead create an account for a host-selected identity that the image lacks, -and replace an image account that uses the UID with another primary group or -without a usable home. This lets a host run workloads under its own user's IDs. -The layered sandbox root creates no accounts, so it does not advertise -[account creation](4.4-host-guest-control-interface.md#442-feature-negotiation). - -Before the workload starts, the guest agents clear supplementary groups and -every capability set and enable `no_new_privs`. With every capability cleared, -the workload cannot perform port I/O, so only the guest agents can request a -capture or acknowledge restore repair. +The init owns execution identity and privilege separation from PID 1 and its +helpers. Workloads use the selected non-root user and primary group, without +the helpers' priority or capability privileges. `nvx` parses the contracts but +does not enforce this execution policy. The credential and launch sequence is +specified in the [guest-agent guidelines](../guest-agent-guidelines.md#own-the-executions). ## 4.3.2 Namespaces and Mounts -In the layered sandbox root, the workload receives private mount, PID, and UTS -namespaces, in which every mount is private, and: - -- A private `/proc` and a read-only `/sys`. -- A restricted `/dev`, devpts, and shared memory. -- The workload root. -- Only the live shares that the host declared. -- The workload's machine ID, bound read-only from a runtime file so that the - guest agents can refresh it while scratch is frozen. -- The guest time status, bound read-only. - -A managed execution enters through the guest library's execution entry, as the -first process of its PID namespace. The entry moves the workload root over `/` -in the execution's mount namespace rather than entering it with `chroot` -alone, gives `/dev` only its allowlisted device nodes, and makes every other -mount `nodev`. The managed agent binds the machine ID and the time status into -the workload root once, in a mount namespace of its own, of which each -execution's starts as a copy, so those binds end with the agent, before the -init agent unmounts the root. A one-shot workload's entry helper enters the -root with `chroot`. - -The guest agents never pivot away from the initramfs, so they keep their own -root and stay outside the workload's namespaces, where they can supervise and -tear down the workload. The control device is never exposed inside the -workload's namespaces. +The space is VM-wide: one root, runtime filesystem set, hostname, machine ID +and resolver view. `nvx-init` implements the internal reference space and +switches into it once; executions need no separate root entry. This model +adds no per-execution mount, PID or UTS namespaces. -The blockless managed root has no such isolation. Each execution runs in the -guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS -namespaces as the guest agents, so it sees the guest's own `/proc`, `/sys`, and -`/dev` and the host mappings bound into that root. The plain-process builder -applies its [credentials](#431-identity-and-credentials), but it has no -execution cgroup or private mount, process, hostname, or device view. +The root owns its mounts and view binds, and the init retains mapping handles +across the switch. After switching, shutdown ends the VM rather than unmounting +the root. The construction and ownership contract is in +[Construct the space](../guest-agent-guidelines.md#construct-the-space). ## 4.3.3 Cgroups and Resources -In the layered sandbox root, the guest agents and workload occupy sibling -cgroups. The workload is placed in its cgroup behind a barrier before it is -released. The guest agents' cgroup carries a default memory protection of -16 MiB, a soft reservation that shields the supervisor from reclaim under -workload memory pressure but does not guarantee its survival. The workload -cgroup carries optional memory and process-count limits and is the unit of -capture freeze, termination, and descendant cleanup. When the overlay's upper -is in guest memory, the files that the workload writes into its root are -charged to the workload cgroup, so its memory limit also bounds them; see -[Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). - -The blockless managed root does not use execution cgroups. Its plain-process -driver terminates the workload's process group. This path does not configure a -sibling guest-agent cgroup, the 16-MiB memory protection, or the layered root's -optional memory and process-count limits. +The VM is the resource limit. The init and its helpers have priority over +workload children, not a sibling cgroup reservation. This model configures no +cgroups or per-execution limits. The sole additional limit is the RAM upper's +cap; scratch and host-device access keep their own storage boundaries. ## 4.3.4 Enforcement Layers -Isolation has two enforcement layers: - -- Guest policy constructs the workload's credentials, mounts, capabilities, - and cgroups and, in the layered sandbox root, its namespaces. -- Host devices independently enforce block mutability, network policy, and - host-filesystem access. - -The guest cannot widen a host policy by remounting a filesystem, changing a -route, creating a link, or issuing a different device request. +The init applies the guest's credentials and space contract; the library does +not enforce its workspace, capture-admission or replay obligations. Host +devices independently enforce block mutability, network policy and +host-filesystem access. Guest mounts and routing cannot widen those host +permissions. diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md new file mode 100644 index 000000000..644829fad --- /dev/null +++ b/doc/guest-agent-guidelines.md @@ -0,0 +1,294 @@ +# Guest-agent guidelines + +`nvx` parses the contracts, captures the guest and drives a restore's activation. +It builds no space, runs no executions and enforces none of the init-agent +obligations below. Each init agent owns its space, executions and lifecycle. +The one exception is `TimeAbi::start_daemon`, which forks the time daemon. +`nvx-init` is the single worked example here, not a public space API. + +The architecture and its boundaries are in +[Guest Agents and Workload Model](design/4.2-guest-agents-and-workload-model.md) +and [Isolation and Resource Model](design/4.3-isolation-and-resource-model.md). +The public mechanisms are documented in +[`guest`](../nvx-rs/src/guest/mod.rs), +[`snapshot`](../nvx-rs/src/guest/snapshot.rs) and +[`activation`](../nvx-rs/src/activation/mod.rs). + +## Ownership + +An init parses the public boot contracts, or receives the restore's activation +record, and builds the VM-wide space from them. It owns the root, its runtime +mounts and view binds, the source and target descriptors of mappings, the +mapper, the helpers, the control session and every execution. Keep these owners +alive for as long as their resources are used; a reset of the transport is not +a new root or a new activation. + +The library's capture call does not freeze executions or drive the transport. +The activation state machine holds no workspace fence, makes no system call +and starts no process. The internal application of a record builds the space +without starting helpers; its caller starts and owns the announcer. Starting +the time daemon through `TimeAbi::start_daemon` is the explicit process-starting +exception, not permission for the library to run workloads. + +## Construct the space + +NVX's space is **internal reference code implemented by `nvx-init`**, not stable +public library API. Hidden compatibility exports do not make it a supported +space interface. Another init may implement its own space or adapt the +reference implementation; it must not depend on a new public space module. + +The inputs are the public root and view tokens (`protocol::root` and +`protocol::view`), the export tokens (`protocol::export`), the reserved share +targets (`protocol::share`), mapping payloads (`protocol::maps`) and the +activation record (`activation`). At cold boot they come from the checked +command line; after restore the record supplies the launch-specific settings. +The captured template supplies fixed assembly inputs, such as the runtime +directory, device windows, filesystem tag and device-wait timeout. Do not +substitute a captured launch's view or network for the restore's own record. + +The reference space keeps these rules: + +- Assemble the checked EROFS image or GPT disk on the `distro` device, checking + its expected identity, with its upper through overlayfs at a fixed runtime + directory. The upper is scratch or a tmpfs capped at 1 to 90 percent of VM + memory, with 50 percent the default. A restored upper is Fresh, not an old + tenant's writable tree. +- Make the tree `nosuid,nodev` first. Mount `/proc`, read-only `/sys`, `/dev`, + `/dev/pts`, `/dev/shm` and `/run`; bind the time status read-only at + `/run/nvx/time`. If a step fails, undo the mounts already made and report any + cleanup that failed. Do not hide a partial space behind success. +- Apply the VM-wide hostname, machine ID and resolver file. `MachineIdMode::Image` + keeps the image's `/etc/machine-id`; `Generated` replaces it. The DNS server + is the one named by the boot contract or this restore's record. The root owns + the machine-ID and resolver binds, including binds made before a later failure. +- Open the `TargetRoot` and the export's `SourceRoot` before switching, and + keep them for later mapping requests. Bind beneath the target root, never + over a reserved target. Account for the root's runtime mounts, such as `/run`, + before constructing its mapper. +- Start and retain all required helpers, and open every descriptor that later + work needs, before the one switch. The time daemon and any announcer are + PID 1's children to supervise, not workload children. +- Switch once: move the root onto `/`, `chroot` and `chdir`. Executions need no + root entry of their own. Before the switch, teardown closes dependent handles + and undoes mounts in reverse. After it, nothing unmounts the root; the VM ends + by syncing and powering off. + +### Checked private-space excerpts + +These excerpts are quoted from +[`the_reference_space_steps_compile`](../nvx-rs/src/guest/init/boot/tests.rs). +That private unit test type-checks the calls without mounting or switching the +developer host's root. The names below are crate-internal, **not public-API +examples**. + +First prepare the root and retain the mapping handles: + +```rust +let _space = |assembly: &root::Assembly, + configuration: &crate::activation::View, + dns: Option, + export: &crate::protocol::export::HostExport, + time_status: Option<&std::path::Path>| + -> Result< + (root::Root, mappings::Mapper, mappings::SourceRoot), + Box, +> { + let mut root = root::assemble(assembly)?; + root.mount_runtime(time_status)?; + view::apply(&mut root, configuration, dns)?; + let source = root.mount_export(&export.tag, export.mode.into())?; + let mut target = mappings::TargetRoot::open(root.path())?; + target.add_mount(&root.path().join("run"))?; + let mapper = mappings::Mapper::new(target, mappings::Reserved::container_root()); + Ok((root, mapper, source)) +}; +``` + +Start and own the helpers between preparation and the final switch. The caller +keeps the returned root, mapper and source afterwards: + +```rust +let _switch = |root: &mut root::Root| root.switch(); +``` + +For a restore, the internal application takes the accepted record and the +captured assembly inputs. It applies the VM-wide view and exposes the +publication and gateway to the init, but starts no announcer: + +```rust +let _restored = |record: &crate::activation::ActivationRecord, + template: &crate::guest::activation::RootTemplate, + deadline: Option| { + crate::guest::activation::apply(record, template, deadline) +}; +``` + +### Account for the host mapping table + +The public `MappingTable` checks and accounts for payloads; the internal mapper +performs mounts. They are different owners and neither replaces the other. +The host announces the total count, then sends consecutive parts in order, +after Ready. `MappingTable::apply` hands a checked part's mappings to the init's +mapper in that order. + +A malformed, empty, out-of-order or excess part is refused as a whole without +changing the table. A mapping that fails after earlier mappings succeeded +leaves those mappings in place and **latches** the table: all later parts are +refused and the table cannot become complete. Report both the failure and its +entry. If the init's protocol treats a refused part as terminal, it explicitly +latches the table too. A workload may start only when `is_complete()` is true; +a table of zero mappings is complete immediately. + +This public-API excerpt is also a +[`guest` rustdoc test](../nvx-rs/src/guest/mod.rs). The mount callback belongs to +the init; the returned `false` means admission must remain closed, not success +for a workload: + +```rust +use nvx::guest::mappings::{Mapping, MappingTable, TableError}; + +fn apply_part( + table: &mut MappingTable, + part: &[u8], + mount: impl FnOnce(&[Mapping]) -> Result<(), (usize, E)>, +) -> Result> { + table.apply(part, mount)?; + Ok(table.is_complete()) +} +``` + +## Own the executions + +`nvx-init` runs executions as children in the single VM-wide space, each in its +own process group. Before exec, the child resets `oom_score_adj` and nice to +their normal values, then sets its credentials: the selected user and primary +group IDs and account mode, no supplementary groups, empty capability sets and +`no_new_privs`. It then changes to the requested working directory and execs. +A failed identity or working-directory step fails the execution; it must not +run with another identity or in another directory. + +End, cancellation, timeout and session reset kill the execution's process +group. The init collects child statuses and cleans up descendants and output +owners; a reset abandons the old epoch's output rather than reporting it in +the replacement session. A process group is a cleanup scope, not a namespace +or cgroup containment boundary. The capture sweep below also deals with +processes that escaped their groups. + +This is priority, not per-execution resource policy. PID 1 and its helpers use +`oom_score_adj = -1000` and nice `-10`. Each exec child resets them before +dropping credentials, so the workload cannot reclaim those privileges. There +are no cgroups, namespaces, per-execution roots or per-execution limits in this +model: the VM is the limit. The one additional limit is the RAM upper's cap. + +## Keep executions still for capture + +An init that supports concurrent executions owns a workspace contract; this +is an init obligation, **not a trait provided or enforced by `nvx`**: + +- `freeze(deadline)` first refuses new executions, then stops those running. + While frozen, a start is refused, not blocked until a later thaw. +- A failed freeze undoes its own work and reports whether it did so (`undone`). + If it could not undo, the VM must end; work must not resume in an uncertain + partially frozen state. +- Freeze several workspaces in order, all or nothing. If one fails with + `undone`, thaw the already frozen workspaces in reverse order. Without + `undone`, end the VM and thaw nothing. +- `thaw()` reverses a successful freeze, in reverse workspace order. + `after_restore()` renews what executions need after restore, before Ready. + +`nvx-init` has no workspace. It refuses capture while an execution is running, +unless the host's capture request asks it to kill executions. In that case it +kills their process groups, then sweeps every process except PID 1 and its +registered helpers, such as the time daemon, until none remains. Only then +does it call capture. + +Capture occurs before mounting the replaceable image, scratch and host export. +Keeping executions still is not sufficient if something from those devices +remains mounted. The public Fresh capture call checks this, but the init owns +the decision to capture and the execution cleanup before it. + +## Keep the transport silent + +Answer the host's capture request, then send nothing on the transport until +`capture` returns. Keep the session and its owners, but do not send progress, +workload output or another response while capture is pending. + +- After `Continued`, thaw workspaces in reverse where they exist, and report + on the **same session**. +- After a failure with `Resumed`, do the same: thaw and report the failure on + the **same session**. +- After `Restored(repaired)`, take the session's reset and attach again before + accepting the activation record. The broker has a fresh instance and epoch, + with no old queued records. The NVXS guest driver acknowledges RESET with + `GuestConnection::acknowledge(window)` as part of this attachment. + +The following public calls are also compiled as +[`snapshot` rustdoc tests](../nvx-rs/src/guest/snapshot.rs); they deliberately do +not implement the init's freeze, transport, shutdown or space operations: + +```rust +use std::time::{Duration, Instant}; +use nvx::guest::activation::Activation; +use nvx::guest::snapshot::{self, CaptureFailure, Captured, Devices, Disks, Repaired}; + +fn fresh_capture(devices: &Devices, deadline: Instant) -> Result { + snapshot::capture(devices, Disks::Fresh, deadline) +} + +fn begin_activation(repaired: &Repaired, timeout: Duration, now: Instant) -> Activation { + Activation::new(repaired, timeout, now) +} +``` + +## Complete restore before Ready + +Capture repairs a restored guest before returning `Restored(repaired)`: its +processor and memory targets, kernel randomness and time acknowledgement. +Keep that `Repaired` and one `Activation` for the restore. The activation's +deadline counts from the repair acknowledgement, not from each attachment or +retry. + +For `nvx-init`, restore proceeds in this order: + +1. `Activation::new`, then the reset/reattachment and the accepted activation + record. Validate the record and repaired resources before space side effects. +2. Apply the record to a Fresh space, including the VM-wide view and **this + restore's** network policy. Mount the runtime filesystems and retain mapping + descriptors. Apply its DNS/listener state; start and own its announcer before + Ready when listeners are published. Do not reuse the captured launch's DNS + or listener policy. +3. Start all required helpers, then switch into the root once. +4. Run each workspace's `after_restore()` where workspaces exist, renewing the + resources its executions need. +5. Call `Activation::complete` with the result. +6. Thaw workspaces in reverse where they exist, then report Ready. + +`nvx-init` omits workspace calls, not the ordering around them. Keep the applied +space, mapper, export source, helpers and activation alive. The host sends the +mapping table after Ready; no workload starts until the table is complete. + +An identical activation retry joins the existing activation or receives its +cached result; it must not apply or switch the root twice. `nvx-init` preserves +the activation and its original deadline across replacement control sessions, +checking the deadline rather than creating another bootstrap. Changed bytes +or another transaction are not a new permission to activate. + +The Fresh repair packet itself carries no replacement network settings. The +post-repair activation record supplies the restore's DNS and publication policy +that the init applies and owns. + +## Failures and replay + +After `Resumed`, thaw and report the failure on the same session. After +`MustEnd { status }`, start nothing, thaw nothing and power off with that exact +status; the library returns the disposition rather than powering off for the +init. A failed or partially applied activation likewise reports its failure +and ends the VM, not Ready. + +Never capture live external connections or other state that cannot be reused +safely by a restored guest. Close or reconstruct it under the init's capture +contract. Never replay an irreversible external effect just because execution +resumed from captured memory. Kernel reseeding does not repair a runtime's +private random state: the runtime must repair its own randomness before it +accepts work after restore, during its `after_restore()` where a workspace +exists. No workspace abstraction in the library enforces these replay rules. diff --git a/nvx-rs/src/activation/mod.rs b/nvx-rs/src/activation/mod.rs index 80dc5fdf3..5c9162fd5 100644 --- a/nvx-rs/src/activation/mod.rs +++ b/nvx-rs/src/activation/mod.rs @@ -17,6 +17,11 @@ //! - [`ReadyActivation`]: what Ready reports about the activation: that it applied, which every //! layered root reports, and the transaction ID, which only a clone reports. //! +//! The [guest-agent guidelines] describe the init's ownership and restore order: applying this +//! record, its own DNS and listener policy, runtime filesystems and helpers precedes Ready. +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! [`ActivationRecord::new`] admits a configuration only once [`Configuration::check`] finds that //! it keeps every [limit](#limits), so that every record holds settings that a guest can apply. //! [`ActivationRecord::encode`] writes a record's canonical [encoding](#encoding), and diff --git a/nvx-rs/src/guest/init/boot/tests.rs b/nvx-rs/src/guest/init/boot/tests.rs index b1c583acc..619313668 100644 --- a/nvx-rs/src/guest/init/boot/tests.rs +++ b/nvx-rs/src/guest/init/boot/tests.rs @@ -14,58 +14,33 @@ const EXPORT: &str = /// mount over and switch the test host's root. #[test] fn the_reference_space_steps_compile() { - // A cold boot: the root, an image under its upper, and the host export; the runtime's file - // systems; the view; then the VM's one switch. The mapper keeps the root's descriptors for - // the host's mapping table, which comes after Ready. - let cold = |assembly: &Assembly, - export: Option<&HostExport>, - configuration: &View, - dns: Option| - -> Result<(Root, Mapper, Option)> { - let mut root = root::assemble(assembly).map_err(system_error)?; - let source = export - .map(|export| root.mount_export(&export.tag, export.mode.into())) - .transpose() - .map_err(system_error)?; - root.mount_runtime(Some(Path::new("/run/nvx/time"))) - .map_err(system_error)?; - let mut target = TargetRoot::open(root.path()).map_err(system_error)?; - target - .add_mount(&root.path().join("run")) - .map_err(system_error)?; - let mapper = Mapper::new(target, Reserved::container_root()); + use crate::guest::{mappings, root, view}; + + // Compile the real private workflow without mounting or switching the developer host's root. + let _space = |assembly: &root::Assembly, + configuration: &crate::activation::View, + dns: Option, + export: &crate::protocol::export::HostExport, + time_status: Option<&std::path::Path>| + -> Result< + (root::Root, mappings::Mapper, mappings::SourceRoot), + Box, + > { + let mut root = root::assemble(assembly)?; + root.mount_runtime(time_status)?; view::apply(&mut root, configuration, dns)?; - root.switch().map_err(system_error)?; + let source = root.mount_export(&export.tag, export.mode.into())?; + let mut target = mappings::TargetRoot::open(root.path())?; + target.add_mount(&root.path().join("run"))?; + let mapper = mappings::Mapper::new(target, mappings::Reserved::container_root()); Ok((root, mapper, source)) }; - // The host's mapping table, part by part, beneath the switched root. - let map = |table: &mut MappingTable, mapper: &mut Mapper, source: &SourceRoot, part: &[u8]| { - table.apply(part, |mappings| { - mapper - .apply(source, mappings) - .map_err(|error| (error.entry(), error)) - }) - }; - // A restore: the record's root, export and view; the runtime's file systems; the announcer - // of the restore's listeners, which the init starts and owns; then the VM's one switch. - let restored = |record: &activation::ActivationRecord, - template: &RootTemplate, - deadline: Option, - announce: &mut dyn FnMut(Option) -> Result<()>| - -> Result { - let mut applied = - guest_activation::apply(record, template, deadline).map_err(activation_error)?; - applied - .root_mut() - .mount_runtime(Some(Path::new("/run/nvx/time"))) - .map_err(activation_error)?; - if applied.publication() == activation::Publication::Published { - announce(applied.gateway())?; - } - applied.root_mut().switch().map_err(activation_error)?; - Ok(applied) + let _switch = |root: &mut root::Root| root.switch(); + let _restored = |record: &crate::activation::ActivationRecord, + template: &crate::guest::activation::RootTemplate, + deadline: Option| { + crate::guest::activation::apply(record, template, deadline) }; - let _ = (cold, map, restored); } #[test] diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 9b9fd61dd..1898b24fa 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -17,6 +17,11 @@ //! acknowledgement, and then takes its own configuration from the host, in one Activate on a //! restricted control session. //! +//! The [guest-agent guidelines] give the init-owned space, execution, capture and restore +//! contract, with nvx-init as the worked example. The library enforces none of those obligations. +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! The guest's space, the root that an init assembles and what it mounts in it, is each init's //! own. NVX's own space is internal reference code, which NVX's guest programs use: it is hidden //! from this documentation and unstable, and it becomes crate-private once nothing outside NVX @@ -30,6 +35,22 @@ //! stay public: the mapping values and [`MappingTable`](mappings::MappingTable), and the tokens //! and records of [`protocol`](crate::protocol) and [`activation`](crate::activation). //! +//! The guidelines' public mapping excerpt forwards mount failures and keeps admission closed +//! until the table is complete; the mount callback belongs to the init: +//! +//! ```no_run +//! use nvx::guest::mappings::{Mapping, MappingTable, TableError}; +//! +//! fn apply_part( +//! table: &mut MappingTable, +//! part: &[u8], +//! mount: impl FnOnce(&[Mapping]) -> Result<(), (usize, E)>, +//! ) -> Result> { +//! table.apply(part, mount)?; +//! Ok(table.is_complete()) +//! } +//! ``` +//! //! Each other module holds one program, and the crate's binary of the same name runs its `main`. //! Two of them also offer their steps as typed calls, which any guest init or agent runs in its //! own process and their programs run on: on x86-64, `time`'s `TimeAbi` runs the time ABI's diff --git a/nvx-rs/src/guest/snapshot.rs b/nvx-rs/src/guest/snapshot.rs index 03591d30d..11511207d 100644 --- a/nvx-rs/src/guest/snapshot.rs +++ b/nvx-rs/src/guest/snapshot.rs @@ -18,6 +18,25 @@ //! instant. A repair failure is always [`Disposition::MustEnd`], with [`RepairError::status`]. //! Fresh restore network settings are explicitly refused. //! +//! The [guest-agent guidelines] specify the caller's workspace, silence, failure and restore +//! obligations. These public excerpts compile without implementing or running those operations: +//! +//! ```no_run +//! use std::time::{Duration, Instant}; +//! use nvx::guest::activation::Activation; +//! use nvx::guest::snapshot::{self, CaptureFailure, Captured, Devices, Disks, Repaired}; +//! +//! fn fresh_capture(devices: &Devices, deadline: Instant) -> Result { +//! snapshot::capture(devices, Disks::Fresh, deadline) +//! } +//! +//! fn begin_activation(repaired: &Repaired, timeout: Duration, now: Instant) -> Activation { +//! Activation::new(repaired, timeout, now) +//! } +//! ``` +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! Each line of `/proc/self/mountinfo` describes one mount with fields separated by spaces: its //! ID, its parent's ID, the device number of its file system, `MAJ:MIN`, the root of the mount //! in its file system, the mount point, the mount's options, optional fields, a `-` that ends From 1131ae8755ff6fbe5c00338e8c6a21006819e8ce Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 8 Nov 2026 16:26:09 -0800 Subject: [PATCH 2/5] doc: retain guest architecture boundaries Scope the VM-wide reference model separately from retained layered and blockless boot paths. Keep control-device and port-I/O boundaries, and distinguish parent-side account validation from child credentials. The daemon/helper compatibility wording remains a separately disclosed contract question. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 41345a73-b73f-4332-9ffd-9c513214af7a --- .../4.2-guest-agents-and-workload-model.md | 27 ++++++++----- .../4.3-isolation-and-resource-model.md | 39 +++++++++++++------ doc/guest-agent-guidelines.md | 15 ++++--- 3 files changed, 55 insertions(+), 26 deletions(-) diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index 5660d5321..c8799f677 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -9,6 +9,12 @@ parsers, capture and restore-activation mechanisms, but builds no space, runs no executions and enforces none of the init's lifecycle obligations. The explicit exception is `TimeAbi::start_daemon`, which forks the time daemon. +The VM-wide reference model below is `nvx-init`'s. The retained +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) +has private execution mount, PID and UTS namespaces and sibling guest-agent +and workload cgroups; the blockless managed root retains its plain-process +model. Those boot paths are separate from the reference space. + ```mermaid flowchart TB Kernel["Linux kernel"] @@ -22,7 +28,7 @@ flowchart TB Agents -->|"supervise and reap"| Workload ``` -**Figure 4.2.** Relationship between the guest agents and the workload. +**Figure 4.2.** Init and workload ownership in nvx-init's VM-wide reference model. The workload remains a child; it does not replace the init, whose continued existence supports supervision, orphan reaping and the control service. @@ -36,14 +42,17 @@ with checked excerpts of the reference implementation, live in the ## 4.2.1 Execution Launch -Execution launch belongs to the init, not the library's capture or activation -mechanisms. Children use the VM-wide space and the init's selected credentials -and working directory. There is no per-execution root entry or placement -barrier in this model. +In nvx-init's reference model, execution launch belongs to the init, not the +library's capture or activation mechanisms. Children use the VM-wide space and +the init's selected credentials and working directory. This reference model +has no per-execution root entry or placement barrier; the retained layered +sandbox root keeps its separate execution entry and placement boundary. ## 4.2.2 Containment -The VM is the isolation and resource boundary. Process groups provide an -execution cleanup scope, not a namespace or cgroup containment guarantee. -The init owns termination, capture admission and cleanup; no library workspace -or per-execution enforcement layer supplies them. +For nvx-init's reference model, the VM is the isolation and resource boundary. +Process groups provide an execution cleanup scope, not a namespace or cgroup +containment guarantee. The init owns termination, capture admission and cleanup; +no library workspace or per-execution enforcement layer supplies them. This +does not remove the retained layered sandbox root's namespace and cgroup +boundaries. diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index a2c20a2b3..d9f28731e 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -7,28 +7,45 @@ The init owns execution identity and privilege separation from PID 1 and its helpers. Workloads use the selected non-root user and primary group, without -the helpers' priority or capability privileges. `nvx` parses the contracts but -does not enforce this execution policy. The credential and launch sequence is -specified in the [guest-agent guidelines](../guest-agent-guidelines.md#own-the-executions). +the helpers' priority or capability privileges. Empty capability sets confer +no port I/O: capture and restore repair stay on the privileged init side, not +with workloads. `nvx` parses the contracts but does not enforce the reference +init's execution policy. Its credential and launch sequence is specified in +the [guest-agent guidelines](../guest-agent-guidelines.md#own-the-executions). ## 4.3.2 Namespaces and Mounts -The space is VM-wide: one root, runtime filesystem set, hostname, machine ID -and resolver view. `nvx-init` implements the internal reference space and -switches into it once; executions need no separate root entry. This model -adds no per-execution mount, PID or UTS namespaces. +In nvx-init's internal reference space, the view is VM-wide: one root, runtime +filesystem set, hostname, machine ID and resolver view. It switches into that +root once; executions need no separate root entry. This reference model adds +no per-execution mount, PID or UTS namespaces. Its root's `/dev` omits the +control device; PID 1 opens that device before the switch and retains its +descriptor. The root owns its mounts and view binds, and the init retains mapping handles across the switch. After switching, shutdown ends the VM rather than unmounting the root. The construction and ownership contract is in [Construct the space](../guest-agent-guidelines.md#construct-the-space). +The retained +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) +still has private execution mount, PID and UTS namespaces, with the control +device outside the workload's view. The blockless managed root instead uses +the guest's initramfs view without those execution namespaces. + ## 4.3.3 Cgroups and Resources -The VM is the resource limit. The init and its helpers have priority over -workload children, not a sibling cgroup reservation. This model configures no -cgroups or per-execution limits. The sole additional limit is the RAM upper's -cap; scratch and host-device access keep their own storage boundaries. +For nvx-init's reference model, the VM is the resource limit. The init and its +helpers have priority over workload children, not a sibling cgroup reservation. +This reference model configures no cgroups or per-execution limits. Its sole +additional limit is the RAM upper's cap; scratch and host-device access keep +their own storage boundaries. + +The retained layered sandbox root keeps sibling guest-agent and workload +cgroups, workload memory/PID accounting and cgroup writeback. Its soft +supervisor memory protection is not a survival guarantee. The blockless +managed root has no execution cgroup; these retained boundaries are not +replaced by the reference model. ## 4.3.4 Enforcement Layers diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md index 644829fad..9e337d410 100644 --- a/doc/guest-agent-guidelines.md +++ b/doc/guest-agent-guidelines.md @@ -160,12 +160,15 @@ fn apply_part( ## Own the executions `nvx-init` runs executions as children in the single VM-wide space, each in its -own process group. Before exec, the child resets `oom_score_adj` and nice to -their normal values, then sets its credentials: the selected user and primary -group IDs and account mode, no supplementary groups, empty capability sets and -`no_new_privs`. It then changes to the requested working directory and execs. -A failed identity or working-directory step fails the execution; it must not -run with another identity or in another directory. +own process group. Before fork, the init validates the request's account mode; +when that mode requires an existing account, it checks the root's user and +group databases. The child resets `oom_score_adj` and nice to their normal +values, then sets its credentials: the selected user and primary group IDs, +no supplementary groups, empty capability sets and `no_new_privs`. It then +changes to the requested working directory and execs. Account mode is a +parent-side check, not credential state that the child sets. A failed identity +or working-directory step fails the execution; it must not run with another +identity or in another directory. End, cancellation, timeout and session reset kill the execution's process group. The init collects child statuses and cleans up descendants and output From 691928831dcf78669ebaabf1c43ab30de5d78db0 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 8 Nov 2026 16:54:25 -0800 Subject: [PATCH 3/5] doc: distinguish retained guest helper call paths Scope the reference lifecycle ownership rule without claiming that every retained API avoids forking. Describe caller-owned announcer startup and the report-only restore-finish child fallback without changing runtime or public API behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 41345a73-b73f-4332-9ffd-9c513214af7a --- .../4.2-guest-agents-and-workload-model.md | 7 +++-- doc/guest-agent-guidelines.md | 27 ++++++++++++++----- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index c8799f677..adca643c7 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -6,8 +6,11 @@ The init agent is the guest's long-lived lifecycle authority. It owns the VM-wide space, helpers, control session and executions. `nvx` supplies contract parsers, capture and restore-activation mechanisms, but builds no space, runs -no executions and enforces none of the init's lifecycle obligations. The -explicit exception is `TimeAbi::start_daemon`, which forks the time daemon. +no executions and enforces none of the reference init's lifecycle obligations. +`TimeAbi::start_daemon` is the reference workflow's named daemon-fork exception, +not a global no-fork guarantee for every retained API. The +[guidelines](../guest-agent-guidelines.md#ownership) record the retained +announcer and report-only restore-finish compatibility paths. The VM-wide reference model below is `nvx-init`'s. The retained [layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md index 9e337d410..fc171e01a 100644 --- a/doc/guest-agent-guidelines.md +++ b/doc/guest-agent-guidelines.md @@ -1,10 +1,13 @@ # Guest-agent guidelines -`nvx` parses the contracts, captures the guest and drives a restore's activation. -It builds no space, runs no executions and enforces none of the init-agent -obligations below. Each init agent owns its space, executions and lifecycle. -The one exception is `TimeAbi::start_daemon`, which forks the time daemon. -`nvx-init` is the single worked example here, not a public space API. +In the capture/activation reference model, `nvx` parses the contracts, captures +the guest and drives a restore's activation. It builds no space, runs no +workload executions and enforces none of the init-agent obligations below. +Each init agent owns its space, executions and lifecycle. The named exception +in this reference workflow is `TimeAbi::start_daemon`, which forks the time +daemon. `nvx-init` is the single worked example here, not a public space API. +This is an ownership contract, not a globally enforced no-fork property of +every retained library API. The architecture and its boundaries are in [Guest Agents and Workload Model](design/4.2-guest-agents-and-workload-model.md) @@ -27,8 +30,18 @@ The library's capture call does not freeze executions or drive the transport. The activation state machine holds no workspace fence, makes no system call and starts no process. The internal application of a record builds the space without starting helpers; its caller starts and owns the announcer. Starting -the time daemon through `TimeAbi::start_daemon` is the explicit process-starting -exception, not permission for the library to run workloads. +the time daemon through the reference workflow's `TimeAbi::start_daemon` is +not permission for the library to run workloads. + +Retained compatibility paths also start helpers. An init may use +[`Announcer::start`](../nvx-rs/src/guest/ingress/announcer.rs), which spawns +`nvx-ingress inherit`; the caller owns that announcer, while `nvx-init` starts +and owns its own announcer. In report-only mode without a running time daemon, +[`TimeAbi::restore_finish`](../nvx-rs/src/guest/time/api.rs) forks a child for +deferred checks and detector release and waits for that child, which coordinates +an RCU-check grandchild. An enforcing or strict finish instead reports the +missing daemon as a time-ABI violation. These are actual retained helper +call paths, not new workload orchestration or workspace enforcement. ## Construct the space From 9ebf9afe980b3824796cadb4b05f5b1399c82a9d Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 8 Nov 2026 17:10:34 -0800 Subject: [PATCH 4/5] doc: clarify restore-finish waiter ownership Distinguish the inline deferred restore checks and detector release from the deadline waiter and grace-period grandchild. Preserve the actual child-reaping boundary without changing helper behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 41345a73-b73f-4332-9ffd-9c513214af7a --- doc/guest-agent-guidelines.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md index fc171e01a..554b0682b 100644 --- a/doc/guest-agent-guidelines.md +++ b/doc/guest-agent-guidelines.md @@ -38,10 +38,14 @@ Retained compatibility paths also start helpers. An init may use `nvx-ingress inherit`; the caller owns that announcer, while `nvx-init` starts and owns its own announcer. In report-only mode without a running time daemon, [`TimeAbi::restore_finish`](../nvx-rs/src/guest/time/api.rs) forks a child for -deferred checks and detector release and waits for that child, which coordinates -an RCU-check grandchild. An enforcing or strict finish instead reports the -missing daemon as a time-ABI violation. These are actual retained helper -call paths, not new workload orchestration or workspace enforcement. +the RCU deadline wait. The deferred restore checks and detector release run in +the calling process, which must have no other thread. The waiter child forks +a grandchild that forces an RCU grace period and waits for it until the RCU +stall timeout. The call reaps only the waiter; a grandchild that outlives the +timeout passes to init or the nearest subreaper. An enforcing or strict finish +instead reports the missing daemon as a time-ABI violation. These are actual +retained helper call paths, not new workload orchestration or workspace +enforcement. ## Construct the space From 137d80296f244df62bb0dd26df30529e6e65fb8c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Tue, 10 Nov 2026 20:47:22 -0800 Subject: [PATCH 5/5] doc: align the guest-agent guidelines with nvx-init The first draft of the guidelines predates the nvx-init that feature/nvx-refactor now holds: #1453 routes the new sandbox boots to it, #1431 serves capture requests, #1447 activates repaired sessions, #1439 keeps a restore's announcer as a helper, and #1490 runs sandbox run through it too. Check each statement against that code, and correct those that did not hold: - an execution takes the VM's one workload identity, which nvx-init resolves, or creates, once before its switch, not an account mode of its own request, which nothing selects yet; - the child also drops its capability bounding set, and nvx-init kills an execution's process group when its leader ends; - nvx-init captures only the Fresh boot of a platform capture, refuses a session with a root with ENOTSUP, and shares one 30-second deadline among its checks, its sweep and the capture; - the restored broker restarts at its first epoch with a reset queued, and nvx-init fails on anything but that reset; - the host sends no MAPS for an empty mapping table, which no nvx_maps token announces; - no launch of NVX's CLI carries OpenVMM's older lifecycle token, which still routes a layered launch to the older init agent; - since #1494, no call of nvx starts the announcer: nvx-init forks it and runs /sbin/nvx-ingress itself, and nvx-ingress start runs it for an agent that cannot hold it; - the crate still keeps older calls that start processes, which nvx-init does not use. Quote, and compile, the space's steps in the order of nvx-init's cold boot and restore, and replace the capture excerpt with one that handles each outcome and disposition. The steps compile in SL5.1a's unit test of nvx-init's boot, the_reference_space_steps_compile, which now holds them as the guidelines quote them, and the guidelines link SL5.1a's guest-workflows example. Organize the page by ownership, the space, the executions, the capture, the transport, the restore, failures and replay. Trim design 4.2 and 4.3 to the architecture and boundaries of nvx-init and of the older layered and blockless roots, which still run. Link the guidelines from the rustdoc of guest::activation too, list the page in the project structure, and give the case matrix the doc tests that the guidelines quote, and the lines to which the links move two others. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/acceptance-case-matrix.md | 17 +- .../4.2-guest-agents-and-workload-model.md | 97 +-- .../4.3-isolation-and-resource-model.md | 102 ++- doc/guest-agent-guidelines.md | 669 +++++++++++------- doc/project-structure.md | 1 + nvx-rs/src/activation/mod.rs | 6 +- nvx-rs/src/guest/activation.rs | 6 + nvx-rs/src/guest/init/boot/tests.rs | 64 +- nvx-rs/src/guest/mod.rs | 8 +- nvx-rs/src/guest/snapshot.rs | 35 +- 10 files changed, 615 insertions(+), 390 deletions(-) diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index e19623a27..512b4fd0c 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -30,7 +30,7 @@ its own in turn. | [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 | | [MicroVM scenarios](#microvm-scenarios) | 52 | 49 | 3 | 0 | | [CI](#ci) | 69 | 43 | 3 | 23 | -| [Docs](#docs) | 34 | 33 | 1 | 0 | +| [Docs](#docs) | 36 | 35 | 1 | 0 | ## The adapter's scenarios @@ -412,17 +412,18 @@ their own. The tests that a step runs have rows of their own where they concern ## Docs Rustdoc checks the code of the crate's documentation, which these rows name as rustdoc does, by -file and line where this matrix was added: it compiles each, runs those that aren't `no_run`, and -checks that those marked `compile_fail` don't compile. The examples compile with the crate's -other targets; `host-workflows` and `guest-workflows`, which came with this matrix, are new, and -have no row. The README's own code is no doctest. +file and line: it compiles each, runs those that aren't `no_run`, and checks that those marked +`compile_fail` don't compile. The examples compile with the crate's other targets; +`host-workflows` and `guest-workflows`, which came with this matrix, are new, and have no row. +The code of `guest` and `guest::snapshot`, which the guest-agent guidelines quote, came after +this matrix. The README's own code is no doctest. | Code | Fate | Note | | --- | --- | --- | -| `nvx-rs/src/activation/mod.rs - activation (line 123)` | kept | | +| `nvx-rs/src/activation/mod.rs - activation (line 130)` | kept | | | `nvx-rs/src/exec/mod.rs - exec (line 113)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/gpt/mod.rs - gpt (line 43)` | kept | | -| `nvx-rs/src/guest/activation.rs - guest::activation (line 189)` | kept | Its capture preparation and admission fence leave (#1087), whose PR rewrites it on `guest::snapshot::capture` | +| `nvx-rs/src/guest/activation.rs - guest::activation (line 195)` | kept | Its capture preparation and admission fence leave (#1087), whose PR rewrites it on `guest::snapshot::capture` | | `nvx-rs/src/guest/activation/state.rs - guest::activation::state::Activation (line 56)` | kept | | | `nvx-rs/src/guest/cgroup.rs - guest::cgroup (line 36)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/entry.rs - guest::entry (line 90)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | @@ -431,11 +432,13 @@ have no row. The README's own code is no doctest. | `nvx-rs/src/guest/gpt.rs - guest::gpt (line 18)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/mappings/mapper.rs - guest::mappings::mapper (line 107)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | | `nvx-rs/src/guest/mappings/table.rs - guest::mappings::table::MappingTable (line 47)` | kept | | +| `nvx-rs/src/guest/mod.rs - guest (line 41)` | kept | The guest-agent guidelines quote it | | `nvx-rs/src/guest/nvxs.rs - guest::nvxs (line 22)` | kept | | | `nvx-rs/src/guest/process.rs - guest::process (line 12)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/process.rs - guest::process (line 39)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/reseed.rs - guest::reseed (line 17)` | kept | | | `nvx-rs/src/guest/root.rs - guest::root (line 130)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests | +| `nvx-rs/src/guest/snapshot.rs - guest::snapshot (line 25)` | kept | The guest-agent guidelines quote it | | `nvx-rs/src/guest/spawn.rs - guest::spawn (line 41)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/spawn.rs - guest::spawn::LaunchStage::from_code (line 201)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it | | `nvx-rs/src/guest/terminal.rs - guest::terminal (line 10)` | kept | | diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index adca643c7..cb2754024 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -3,59 +3,70 @@ [Architecture documentation](README.md) > [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) -The init agent is the guest's long-lived lifecycle authority. It owns the -VM-wide space, helpers, control session and executions. `nvx` supplies contract -parsers, capture and restore-activation mechanisms, but builds no space, runs -no executions and enforces none of the reference init's lifecycle obligations. -`TimeAbi::start_daemon` is the reference workflow's named daemon-fork exception, -not a global no-fork guarantee for every retained API. The -[guidelines](../guest-agent-guidelines.md#ownership) record the retained -announcer and report-only restore-finish compatibility paths. - -The VM-wide reference model below is `nvx-init`'s. The retained -[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) -has private execution mount, PID and UTS namespaces and sibling guest-agent -and workload cgroups; the blockless managed root retains its plain-process -model. Those boot paths are separate from the reference space. +The guest's init agent is its long-lived lifecycle authority: it owns the +guest's space, its helpers, its control session and its executions. `nvx` +parses the contracts, captures the guest and drives a restore's activation, +but builds no space, runs no executions and enforces none of the init's +obligations; the one exception is the time daemon, which +`TimeAbi::start_daemon` forks. The +[guest-agent guidelines](../guest-agent-guidelines.md) give the init's +contract, with nvx-init as the worked example. + +A layered root's guest runs one of two inits. nvx-init, NVX's PID 1 for +layered roots, runs the new launches, and the restores of platform captures, +in one VM-wide space, as Figure 4.2 shows; it also serves the Fresh boot that +a platform capture takes, which has no root yet. A layered launch that still +carries OpenVMM's older lifecycle token, `nvx_lifecycle=`, which NVX's CLI no +longer adds, runs the older init agent, whose sandbox agent gives each +execution private namespaces and a cgroup in the +[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). +The agent of the +[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) +is PID 1 in the guest's initramfs root, and runs its executions with +nvx-init's plain-process core. ```mermaid flowchart TB Kernel["Linux kernel"] - Agents["Init agent
space and lifecycle owner"] - Root["One VM-wide space
root, view and mappings"] - Workload["Execution children
one process group each"] - - Kernel --> Agents - Agents --> Root - Root --> Workload - Agents -->|"supervise and reap"| Workload -``` + Init["nvx-init, PID 1
space, helpers and control session"] + Space["One VM-wide space
root, view and mappings"] + Workload["Executions
a process group each"] -**Figure 4.2.** Init and workload ownership in nvx-init's VM-wide reference model. + Kernel --> Init + Init --> Space + Space --> Workload + Init -->|"supervise and reap"| Workload +``` -The workload remains a child; it does not replace the init, whose continued -existence supports supervision, orphan reaping and the control service. -`nvx-init` is the reference implementation, using one shared root and sequential -executions. An init with concurrent executions owns its own workspace -coordination; the library provides no workspace trait. +**Figure 4.2.** nvx-init, its space and its executions. -The detailed construction, execution, capture, transport and restore rules, -with checked excerpts of the reference implementation, live in the -[guest-agent guidelines](../guest-agent-guidelines.md). +The workload is always a child of a guest agent, which never replaces itself +with it: supervision, orphan reaping and the control service depend on the +agent remaining alive. In a one-shot lifecycle, the guest agents start one +workload, configured by non-secret kernel command-line tokens, and shut down +the VM when it exits. In a managed lifecycle, they run a sequence of +workloads, one at a time, on request over the authenticated control channel. +An init that runs concurrent executions coordinates them for a capture +itself: `nvx` provides no workspace. ## 4.2.1 Execution Launch -In nvx-init's reference model, execution launch belongs to the init, not the -library's capture or activation mechanisms. Children use the VM-wide space and -the init's selected credentials and working directory. This reference model -has no per-execution root entry or placement barrier; the retained layered -sandbox root keeps its separate execution entry and placement boundary. +Launching an execution belongs to the init, not to the library's capture or +activation. nvx-init's executions, and those of the blockless managed root, +are plain processes of their agent: each child takes its credentials and +enters its working directory itself, in its agent's space, without a root of +its own or a placement barrier. The older layered sandbox root keeps its +execution entry, which enters private namespaces and the workload root, and +its placement barrier. +[Run the executions](../guest-agent-guidelines.md#run-the-executions) gives +nvx-init's steps. ## 4.2.2 Containment -For nvx-init's reference model, the VM is the isolation and resource boundary. -Process groups provide an execution cleanup scope, not a namespace or cgroup -containment guarantee. The init owns termination, capture admission and cleanup; -no library workspace or per-execution enforcement layer supplies them. This -does not remove the retained layered sandbox root's namespace and cgroup -boundaries. +For nvx-init and the blockless managed root, the VM is the isolation and +resource boundary. An execution's process group, which its end, +cancellation, timeout and reset kill, is a cleanup scope, not a namespace or +a cgroup: a process that leaves the group escapes it. The init owns +termination, capture admission and cleanup; no workspace or per-execution +enforcement of `nvx` supplies them. The older layered sandbox root keeps its +namespace and cgroup cleanup. \ No newline at end of file diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index d9f28731e..888c618fa 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -5,52 +5,80 @@ ## 4.3.1 Identity and Credentials -The init owns execution identity and privilege separation from PID 1 and its -helpers. Workloads use the selected non-root user and primary group, without -the helpers' priority or capability privileges. Empty capability sets confer -no port I/O: capture and restore repair stay on the privileged init side, not -with workloads. `nvx` parses the contracts but does not enforce the reference -init's execution policy. Its credential and launch sequence is specified in -the [guest-agent guidelines](../guest-agent-guidelines.md#own-the-executions). +The workload runs under a host-selected, non-root numeric UID and GID, which +the boot fixes for the VM, or, for a restored nvx-init, its activation. The +guest agents require an exact user and primary-group match in the workload +root, and reject an unavailable identity before starting the workload: +nvx-init resolves the account once, before it switches into its root, and the +older layered sandbox root's entry checks it again for each execution. When +the boot configuration requests it, nvx-init and the guest agents of the +blockless managed root instead create an account for a host-selected identity +that the image lacks, and replace an image account that uses the UID with +another primary group or without a usable home. The older layered sandbox +root creates no accounts, so it does not advertise +[account creation](4.4-host-guest-control-interface.md#442-feature-negotiation). -## 4.3.2 Namespaces and Mounts +Before the workload runs, its child clears supplementary groups and every +capability set and enables `no_new_privs`, so the workload keeps none of the +privileges of the guest agents and their helpers. With every capability +cleared, the workload cannot perform port I/O, so only the guest agents can +request a capture or acknowledge restore repair. `nvx` enforces none of this: +[Run the executions](../guest-agent-guidelines.md#run-the-executions) gives +nvx-init's steps. -In nvx-init's internal reference space, the view is VM-wide: one root, runtime -filesystem set, hostname, machine ID and resolver view. It switches into that -root once; executions need no separate root entry. This reference model adds -no per-execution mount, PID or UTS namespaces. Its root's `/dev` omits the -control device; PID 1 opens that device before the switch and retains its -descriptor. +## 4.3.2 Namespaces and Mounts -The root owns its mounts and view binds, and the init retains mapping handles -across the switch. After switching, shutdown ends the VM rather than unmounting -the root. The construction and ownership contract is in -[Construct the space](../guest-agent-guidelines.md#construct-the-space). +nvx-init's space is VM-wide: one root, with one set of runtime file systems, +one hostname, machine ID and resolver file, and the host's mappings, into +which PID 1 switches once. Its executions share that root and PID 1's mount, +PID and UTS namespaces: no execution has a namespace or a root of its own. +The root's `/dev` holds no control device: PID 1 opens it before the switch +and keeps its descriptor, and the helpers, which run from the initramfs, +start before the switch too. Once switched, nothing unmounts the root, and +the VM ends by powering off. +[Build the space](../guest-agent-guidelines.md#build-the-space) gives the +space's rules. -The retained -[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root) -still has private execution mount, PID and UTS namespaces, with the control -device outside the workload's view. The blockless managed root instead uses -the guest's initramfs view without those execution namespaces. +The older layered sandbox root gives each workload private mount, PID, and +UTS namespaces, in which every mount is private, with a private `/proc`, a +read-only `/sys`, a restricted `/dev`, the workload root, only the live shares +that the host declared, and the workload's machine ID and the guest's time +status, bound read-only. Its guest agents never pivot away from the +initramfs, so they stay outside the workload's namespaces, where they +supervise and tear down the workload, and the control device is never exposed +inside them. The blockless managed root runs each execution in the guest's +initramfs root, in the guest agents' own namespaces, without a private mount, +process, hostname, or device view. ## 4.3.3 Cgroups and Resources -For nvx-init's reference model, the VM is the resource limit. The init and its -helpers have priority over workload children, not a sibling cgroup reservation. -This reference model configures no cgroups or per-execution limits. Its sole -additional limit is the RAM upper's cap; scratch and host-device access keep +For nvx-init and the blockless managed root, the VM is the resource limit: +they configure no cgroups and no per-execution limits. nvx-init runs PID 1 and +its helpers ahead of the executions, whose children give up that priority, +instead of reserving memory for them in a sibling cgroup. Its one limit +within the VM is the RAM upper's cap; the scratch and the host's devices keep their own storage boundaries. -The retained layered sandbox root keeps sibling guest-agent and workload -cgroups, workload memory/PID accounting and cgroup writeback. Its soft -supervisor memory protection is not a survival guarantee. The blockless -managed root has no execution cgroup; these retained boundaries are not -replaced by the reference model. +The older layered sandbox root keeps sibling guest-agent and workload cgroups. +The workload is placed in its cgroup behind a barrier before it is released. +The guest agents' cgroup carries a default memory protection of 16 MiB, a +soft reservation that shields the supervisor from reclaim under workload +memory pressure but does not guarantee its survival. The workload cgroup +carries optional memory and process-count limits and is the unit of capture +freeze, termination, and descendant cleanup. When the overlay's upper is in +guest memory, the files that the workload writes into its root are charged to +the workload cgroup, so its memory limit also bounds them; see +[Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). ## 4.3.4 Enforcement Layers -The init applies the guest's credentials and space contract; the library does -not enforce its workspace, capture-admission or replay obligations. Host -devices independently enforce block mutability, network policy and -host-filesystem access. Guest mounts and routing cannot widen those host -permissions. +Isolation has two enforcement layers: + +- Guest policy, which the init applies and `nvx` does not enforce: the + workload's credentials and capabilities, the mounts of its space, and, in + the older layered sandbox root, its namespaces and cgroups. +- Host devices independently enforce block mutability, network policy, and + host-filesystem access. + +The guest cannot widen a host policy by remounting a filesystem, changing a +route, creating a link, or issuing a different device request. \ No newline at end of file diff --git a/doc/guest-agent-guidelines.md b/doc/guest-agent-guidelines.md index 554b0682b..189296142 100644 --- a/doc/guest-agent-guidelines.md +++ b/doc/guest-agent-guidelines.md @@ -1,165 +1,231 @@ # Guest-agent guidelines -In the capture/activation reference model, `nvx` parses the contracts, captures -the guest and drives a restore's activation. It builds no space, runs no -workload executions and enforces none of the init-agent obligations below. -Each init agent owns its space, executions and lifecycle. The named exception -in this reference workflow is `TimeAbi::start_daemon`, which forks the time -daemon. `nvx-init` is the single worked example here, not a public space API. -This is an ownership contract, not a globally enforced no-fork property of -every retained library API. - -The architecture and its boundaries are in +`nvx` parses the contracts, captures the guest and drives a restore's +activation, but builds no space and runs no executions: each init agent does +both, and `nvx` enforces none of the obligations below. The one exception is +the time daemon, which `TimeAbi::start_daemon` forks. + +These guidelines give an init agent's contract: how it builds its guest's +space, runs its executions, coordinates a capture with its host and completes +a restore. `nvx-init`, NVX's PID 1 for layered roots, is the worked example +throughout. The design's [Guest Agents and Workload Model](design/4.2-guest-agents-and-workload-model.md) -and [Isolation and Resource Model](design/4.3-isolation-and-resource-model.md). -The public mechanisms are documented in +and [Isolation and Resource Model](design/4.3-isolation-and-resource-model.md) +give the architecture and its boundaries, and the rustdoc of [`guest`](../nvx-rs/src/guest/mod.rs), -[`snapshot`](../nvx-rs/src/guest/snapshot.rs) and -[`activation`](../nvx-rs/src/activation/mod.rs). +[`guest::snapshot`](../nvx-rs/src/guest/snapshot.rs), +[`guest::activation`](../nvx-rs/src/guest/activation.rs) and +[`activation`](../nvx-rs/src/activation/mod.rs) the public mechanisms. The +[`guest-workflows`](../nvx-rs/examples/guest-workflows.rs) example compiles an +init's calls of them, through the public API alone: its time daemon, its +control session, a capture, a restore's activation and the host's mapping +table. ## Ownership -An init parses the public boot contracts, or receives the restore's activation -record, and builds the VM-wide space from them. It owns the root, its runtime -mounts and view binds, the source and target descriptors of mappings, the -mapper, the helpers, the control session and every execution. Keep these owners -alive for as long as their resources are used; a reset of the transport is not -a new root or a new activation. - -The library's capture call does not freeze executions or drive the transport. -The activation state machine holds no workspace fence, makes no system call -and starts no process. The internal application of a record builds the space -without starting helpers; its caller starts and owns the announcer. Starting -the time daemon through the reference workflow's `TimeAbi::start_daemon` is -not permission for the library to run workloads. - -Retained compatibility paths also start helpers. An init may use -[`Announcer::start`](../nvx-rs/src/guest/ingress/announcer.rs), which spawns -`nvx-ingress inherit`; the caller owns that announcer, while `nvx-init` starts -and owns its own announcer. In report-only mode without a running time daemon, -[`TimeAbi::restore_finish`](../nvx-rs/src/guest/time/api.rs) forks a child for -the RCU deadline wait. The deferred restore checks and detector release run in -the calling process, which must have no other thread. The waiter child forks -a grandchild that forces an RCU grace period and waits for it until the RCU -stall timeout. The call reaps only the waiter; a grandchild that outlives the -timeout passes to init or the nearest subreaper. An enforcing or strict finish -instead reports the missing daemon as a time-ABI violation. These are actual -retained helper call paths, not new workload orchestration or workspace -enforcement. - -## Construct the space - -NVX's space is **internal reference code implemented by `nvx-init`**, not stable -public library API. Hidden compatibility exports do not make it a supported -space interface. Another init may implement its own space or adapt the -reference implementation; it must not depend on a new public space module. - -The inputs are the public root and view tokens (`protocol::root` and -`protocol::view`), the export tokens (`protocol::export`), the reserved share -targets (`protocol::share`), mapping payloads (`protocol::maps`) and the -activation record (`activation`). At cold boot they come from the checked -command line; after restore the record supplies the launch-specific settings. -The captured template supplies fixed assembly inputs, such as the runtime -directory, device windows, filesystem tag and device-wait timeout. Do not -substitute a captured launch's view or network for the restore's own record. - -The reference space keeps these rules: - -- Assemble the checked EROFS image or GPT disk on the `distro` device, checking - its expected identity, with its upper through overlayfs at a fixed runtime - directory. The upper is scratch or a tmpfs capped at 1 to 90 percent of VM - memory, with 50 percent the default. A restored upper is Fresh, not an old - tenant's writable tree. -- Make the tree `nosuid,nodev` first. Mount `/proc`, read-only `/sys`, `/dev`, - `/dev/pts`, `/dev/shm` and `/run`; bind the time status read-only at - `/run/nvx/time`. If a step fails, undo the mounts already made and report any - cleanup that failed. Do not hide a partial space behind success. -- Apply the VM-wide hostname, machine ID and resolver file. `MachineIdMode::Image` - keeps the image's `/etc/machine-id`; `Generated` replaces it. The DNS server - is the one named by the boot contract or this restore's record. The root owns - the machine-ID and resolver binds, including binds made before a later failure. -- Open the `TargetRoot` and the export's `SourceRoot` before switching, and - keep them for later mapping requests. Bind beneath the target root, never - over a reserved target. Account for the root's runtime mounts, such as `/run`, - before constructing its mapper. -- Start and retain all required helpers, and open every descriptor that later - work needs, before the one switch. The time daemon and any announcer are - PID 1's children to supervise, not workload children. -- Switch once: move the root onto `/`, `chroot` and `chdir`. Executions need no - root entry of their own. Before the switch, teardown closes dependent handles - and undoes mounts in reverse. After it, nothing unmounts the root; the VM ends - by syncing and powering off. - -### Checked private-space excerpts - -These excerpts are quoted from -[`the_reference_space_steps_compile`](../nvx-rs/src/guest/init/boot/tests.rs). -That private unit test type-checks the calls without mounting or switching the -developer host's root. The names below are crate-internal, **not public-API -examples**. - -First prepare the root and retain the mapping handles: +An init reads the public boot contracts at a cold boot, or a restore's +activation record, and builds the VM's one space from them. It owns the root, +with its runtime mounts and view binds; the mapper and the descriptors of the +mappings' source and target roots; its helpers; its control session; and +every execution. It keeps each owner for as long as what it owns is in use: a +reset of the transport is neither a new root nor a new activation. + +The library's calls leave all of this to the init: + +- `guest::snapshot::capture` stops no execution, drives no transport and never + powers the VM off: it returns how the capture ended, or a failure with its + disposition. +- `guest::activation::Activation` holds no workspace or admission fence, makes + no system call and starts no process: the init feeds it what its session + receives, and does what it returns. +- `TimeAbi::start_daemon` forks the time daemon, which the init supervises as + a helper. nvx-init's early boot, the shell `/init`, starts it with + `nvx-time boot`, and the announcer when the network publishes the workload's + listeners, then replaces itself with nvx-init, which inherits both as its + children. + +One other call starts processes, which an init can avoid: a report-only +[`TimeAbi::restore_finish`](../nvx-rs/src/guest/time/api.rs), which +`capture`'s repair calls, runs the restore's deferred time work itself when +no time daemon runs, in a calling process that must have no other thread: it +forks a child, which waits for a grandchild that forces an RCU grace period, +and collects only the child, so a grandchild that outlives the RCU stall +timeout passes to PID 1, or to the nearest subreaper. No call starts the +announcer, which is a program, `/sbin/nvx-ingress`: an init runs it itself, +or has `nvx-ingress start` run it from a socket that it binds for it. Until +the nvx-rs slimming +([microsoft/nvx#1076](https://github.com/microsoft/nvx/issues/1076)) removes +them, the crate also keeps the older execution engine and its process +modules, `guest::{exec, spawn, entry, cgroup, process}`, which NVX's older +guest agents use. nvx-init uses none of these modules, and its time daemon +runs for the VM's life. + +## Build the space + +NVX's space is internal reference code that nvx-init implements, not public +API: the modules that hold it, `guest::root`, `guest::view`, `guest::gpt`, the +mapper of `guest::mappings`, and `guest::activation::apply` with its types, +are hidden from the documentation, unstable, and become private to the crate +once nothing outside NVX uses them. Another init builds its own space, or +adapts this code; it builds on no space module of `nvx`. + +The space's inputs are public contracts: + +- At a cold boot, nvx-init reads them from the kernel command line: the root's + tokens, which `protocol::root::RootLayout` parses; the view's, which + `protocol::view` parses; the host export's, which + `protocol::export::HostExport` parses, or the live shares of NVX's CLI, + whose targets `protocol::share` restricts; the DNS server, which + `protocol::cmdline::dns_server` reads; and the number of host mappings, + which `MappingTable::announced` reads. +- After a restore, the accepted `activation::ActivationRecord` holds the + launch's settings, and the configuration that the snapshot captured, the + same for each restore, holds the assembly's fixed inputs: the runtime + directory, the block devices, the host filesystem device's tag, the NIC's + gateway, and how long the assembly waits for the devices. Never take a + captured launch's view or network in place of the restore's own record. +- The host's mapping table, when it holds any mapping, arrives once the init + is ready, in `MAPS` parts, which `protocol::maps` decodes. + +nvx-init's space keeps these rules: + +- **The root.** The image on the `distro` device, an EROFS file system or a + GPT disk, whose identity the root checks when the contract names one, lies + under an overlay with its upper, mounted `nosuid,nodev` below the runtime + directory, `/run/nvx`. The upper is the ext4 scratch, or a tmpfs whose cap + the contract may set from 1 to 90 percent of the VM's memory, and which + otherwise takes the kernel's default, half of it. A restore's upper is + fresh: the assembly refuses one whose `upper` or `work` directory holds + anything. +- **The runtime file systems.** The root makes every mount of its tree + `nosuid` and `nodev`, then mounts `/proc`, a read-only `/sys`, a `/dev` that + holds only the null, zero, full, random, urandom and tty devices, with + `/dev/pts` and `/dev/shm`, and `/run`, and binds the time status read-only + at `/run/nvx/time`. They come before any live share or mapping, since the + `/run` tmpfs would hide one beneath `/run`: a cold boot's live shares follow + them. +- **The view.** It is VM-wide: the hostname; the image's own machine ID + (`MachineIdMode::Image`), or one that the init generates and binds read-only + over it (`Generated`); and, when the boot contract or this restore's record + names a DNS server, a resolver file that names it, bound over the image's. + The root owns these binds, those made before a later failure included. +- **The mappings.** The host export, from which mappings come, is mounted + outside the root, below the runtime directory, where only the guest's root + user reaches it. The init opens the root's descriptor as the mappings' + target root, adds the runtime's `/run` mount to it, and makes the mapper of + it, which refuses the targets that the runtime reserves; it keeps the + export's source root. A mapping binds beneath these descriptors, never by + path, so both outlive the switch. +- **The helpers.** Every helper runs, and every descriptor that later work + needs is open, before the switch: the root holds neither the initramfs's + programs nor the control device. nvx-init's early boot starts its helpers, + and nvx-init opens its control terminal before it switches; a restore starts + its announcer there too. PID 1 and its helpers outrank every execution, as + [Run the executions](#run-the-executions) describes. +- **The one switch.** The init moves the root onto `/`, then enters it with + `chroot` and `chdir`, once: executions need no root of their own. From then + on nothing unmounts the root, and the VM ends by syncing and powering off. + Before the switch, a step that fails admits no workload: the root tries to + unmount what it mounted, in reverse, and nvx-init ends the VM, whatever the + cleanup leaves mounted. + +### The space's steps, checked + +These excerpts are nvx-init's unit test +[`the_reference_space_steps_compile`](../nvx-rs/src/guest/init/boot/tests.rs), +which compiles them without mounting anything or switching the developer +host's root. They use the space's own names, of `root`, `view`, the mapper and +`activation::{apply, Applied, RootTemplate}`, which are NVX's internal +reference code, not public API. A cold boot's space takes these steps in the +order of [`boot::cold_boot`](../nvx-rs/src/guest/init/boot.rs), which also +mounts the live shares of NVX's CLI after the runtime file systems: ```rust -let _space = |assembly: &root::Assembly, - configuration: &crate::activation::View, - dns: Option, - export: &crate::protocol::export::HostExport, - time_status: Option<&std::path::Path>| - -> Result< - (root::Root, mappings::Mapper, mappings::SourceRoot), - Box, -> { +use std::error::Error; +use std::net::IpAddr; +use std::path::Path; +use std::time::Instant; + +use crate::activation::{ActivationRecord, View}; +use crate::guest::activation::{self, Applied, RootTemplate}; +use crate::guest::mappings::{Mapper, Reserved, SourceRoot, TargetRoot}; +use crate::guest::root::{self, Assembly, Root}; +use crate::guest::view; +use crate::protocol::export::HostExport; + +// A cold boot, as `boot::cold_boot` builds it. +let _cold = |assembly: &Assembly, + export: &HostExport, + vm_view: &View, + dns: Option| + -> Result<(Root, Mapper, SourceRoot), Box> { let mut root = root::assemble(assembly)?; - root.mount_runtime(time_status)?; - view::apply(&mut root, configuration, dns)?; let source = root.mount_export(&export.tag, export.mode.into())?; - let mut target = mappings::TargetRoot::open(root.path())?; + root.mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(root.path())?; target.add_mount(&root.path().join("run"))?; - let mapper = mappings::Mapper::new(target, mappings::Reserved::container_root()); + let mapper = Mapper::new(target, Reserved::container_root()); + view::apply(&mut root, vm_view, dns)?; + // Start the helpers and open the descriptors that later work needs before the switch. + root.switch()?; Ok((root, mapper, source)) }; ``` -Start and own the helpers between preparation and the final switch. The caller -keeps the returned root, mapper and source afterwards: +A restore's application assembles the record's root, mounts its export and +applies its view, with its DNS server, and starts no process; the init then +mounts the runtime file systems, opens the mappings' handles, starts the +announcer when the record publishes the workload's listeners, and switches: ```rust -let _switch = |root: &mut root::Root| root.switch(); -``` - -For a restore, the internal application takes the accepted record and the -captured assembly inputs. It applies the VM-wide view and exposes the -publication and gateway to the init, but starts no announcer: - -```rust -let _restored = |record: &crate::activation::ActivationRecord, - template: &crate::guest::activation::RootTemplate, - deadline: Option| { - crate::guest::activation::apply(record, template, deadline) +// A restore, as `boot::Restored::apply` builds it from the accepted record. +let _restored = |record: &ActivationRecord, + template: &RootTemplate, + deadline: Option| + -> Result<(Applied, Mapper, Option), Box> { + let mut applied = activation::apply(record, template, deadline)?; + applied + .root_mut() + .mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(applied.root().path())?; + target.add_mount(&applied.root().path().join("run"))?; + let mapper = Mapper::new(target, Reserved::container_root()); + let source = applied.take_export(); + // Start the announcer here when `applied.publication()` publishes the listeners. + applied.root_mut().switch()?; + Ok((applied, mapper, source)) }; ``` -### Account for the host mapping table - -The public `MappingTable` checks and accounts for payloads; the internal mapper -performs mounts. They are different owners and neither replaces the other. -The host announces the total count, then sends consecutive parts in order, -after Ready. `MappingTable::apply` hands a checked part's mappings to the init's -mapper in that order. - -A malformed, empty, out-of-order or excess part is refused as a whole without -changing the table. A mapping that fails after earlier mappings succeeded -leaves those mappings in place and **latches** the table: all later parts are -refused and the table cannot become complete. Report both the failure and its -entry. If the init's protocol treats a refused part as terminal, it explicitly -latches the table too. A workload may start only when `is_complete()` is true; -a table of zero mappings is complete immediately. - -This public-API excerpt is also a -[`guest` rustdoc test](../nvx-rs/src/guest/mod.rs). The mount callback belongs to -the init; the returned `false` means admission must remain closed, not success -for a workload: +### The host's mapping table + +The public [`MappingTable`](../nvx-rs/src/guest/mappings/table.rs) keeps the +account of the host's mapping table, and the init's mapper binds the mappings +that it hands out: one does not replace the other. The host announces how many +mappings its table holds, at a cold boot with `nvx_maps=COUNT`, which it leaves +out for none, and after a restore in the record. Once the init is ready, it +sends a table that holds any mapping in parts of consecutive entries, and an +empty one not at all: + +- **Parts in order.** `MappingTable::apply` refuses a part as a whole, and + changes nothing, when it does not decode, holds no entry, does not start at + the entry that the table expects next, ends past the announced count, or + holds an entry whose source is empty, which names the export's root, or + whose path a `Mapping` refuses. Otherwise it hands the part's mappings, in + order, to the init's mapper. +- **A latch after a failure.** A mapping that fails keeps the mappings before + it and latches the table, which refuses every later part and is never + complete; the init reports the failure and its entry. An init whose protocol + ends the table at a refused part latches it then, with `MappingTable::latch`. +- **A complete table before any workload.** The init starts no workload until + `MappingTable::is_complete`: until it applied every announced mapping and + none failed. A table of no mappings is complete at once. + +nvx-init answers a refused part with `invalid-request`, a failed mapping with +`mapping-failed`, after a diagnostic that names its entry, and refuses every +execution with `mappings-incomplete` until its table is complete. The +guidelines quote this excerpt of the [`guest`](../nvx-rs/src/guest/mod.rs) +rustdoc, where it compiles as a public-API doc test: ```rust use nvx::guest::mappings::{Mapping, MappingTable, TableError}; @@ -174,141 +240,210 @@ fn apply_part( } ``` -## Own the executions - -`nvx-init` runs executions as children in the single VM-wide space, each in its -own process group. Before fork, the init validates the request's account mode; -when that mode requires an existing account, it checks the root's user and -group databases. The child resets `oom_score_adj` and nice to their normal -values, then sets its credentials: the selected user and primary group IDs, -no supplementary groups, empty capability sets and `no_new_privs`. It then -changes to the requested working directory and execs. Account mode is a -parent-side check, not credential state that the child sets. A failed identity -or working-directory step fails the execution; it must not run with another -identity or in another directory. - -End, cancellation, timeout and session reset kill the execution's process -group. The init collects child statuses and cleans up descendants and output -owners; a reset abandons the old epoch's output rather than reporting it in -the replacement session. A process group is a cleanup scope, not a namespace -or cgroup containment boundary. The capture sweep below also deals with -processes that escaped their groups. - -This is priority, not per-execution resource policy. PID 1 and its helpers use -`oom_score_adj = -1000` and nice `-10`. Each exec child resets them before -dropping credentials, so the workload cannot reclaim those privileges. There -are no cgroups, namespaces, per-execution roots or per-execution limits in this -model: the VM is the limit. The one additional limit is the RAM upper's cap. - -## Keep executions still for capture - -An init that supports concurrent executions owns a workspace contract; this -is an init obligation, **not a trait provided or enforced by `nvx`**: - -- `freeze(deadline)` first refuses new executions, then stops those running. - While frozen, a start is refused, not blocked until a later thaw. -- A failed freeze undoes its own work and reports whether it did so (`undone`). - If it could not undo, the VM must end; work must not resume in an uncertain - partially frozen state. -- Freeze several workspaces in order, all or nothing. If one fails with - `undone`, thaw the already frozen workspaces in reverse order. Without - `undone`, end the VM and thaw nothing. -- `thaw()` reverses a successful freeze, in reverse workspace order. - `after_restore()` renews what executions need after restore, before Ready. - -`nvx-init` has no workspace. It refuses capture while an execution is running, -unless the host's capture request asks it to kill executions. In that case it -kills their process groups, then sweeps every process except PID 1 and its -registered helpers, such as the time daemon, until none remains. Only then -does it call capture. - -Capture occurs before mounting the replaceable image, scratch and host export. -Keeping executions still is not sufficient if something from those devices -remains mounted. The public Fresh capture call checks this, but the init owns -the decision to capture and the execution cleanup before it. +## Run the executions + +An init runs its executions itself. nvx-init runs them one at a time, at the +host's request, as plain processes in the VM's one space, with its own +plain-process core: + +- **Before the fork**, it builds the program's arguments and environment. The + default environment is nvx-init's own, with the workload account's `HOME`, + `USER` and `LOGNAME` and the working directory's `PWD`; the request's + entries replace it, or layer over it when the request asks. The account is + the VM's workload identity, the `nvx_identity=UID:GID` of a cold boot or the + identity of a restore's activate request, which nvx-init resolves in the + root's user and group databases once, before its switch, and creates when + the boot, with `nvx_workload_account=create`, or the activate request asks + for it. +- **In the child**, which leads a process group of its own, with the null + device as its standard input and pipes as its standard output and error, + the execution resets its priority, sets its credentials, changes to its + working directory and execs. It resets its out-of-memory score adjustment + and its nice value to 0, then drops its capability bounding set and takes + the account's user and group IDs, with no supplementary groups, empty + capability sets and `no_new_privs`. It then enters the requested working + directory, `/` by default, with those credentials, resets its signals, and + execs. A step that fails reports its stage and `errno` to the init and exits + with status 125: the execution fails, rather than run as another identity or + in another directory. +- **End, cancellation, timeout and reset** each kill the execution's process + group: nvx-init kills it when the execution's leader ends, normally or not, + when the host cancels the execution, when its timeout passes, and when a + session reset abandons it, whose output it then discards rather than report + it on the new session. PID 1 collects the status of every child, orphans + included. A process group is a cleanup scope, not containment: a process + that leaves its group escapes it, as the capture's sweep allows for. + +This is priority, not limits. PID 1 and its helpers run with an out-of-memory +score adjustment of -1000 and a nice value of -10, ahead of the executions' +0, which each child restores before it drops its privileges, so that no +workload keeps them. There are no cgroups, namespaces, per-execution roots or +per-execution limits: the VM is the limit, and the one limit within it is the +RAM upper's cap. + +## Coordinate a capture + +An init that runs concurrent executions keeps them in workspaces, under a +contract that is the init's own: `nvx` provides no workspace trait and +enforces none of it. + +- `freeze(deadline)` refuses new executions, then stops those that run. A + frozen workspace refuses a start rather than hold it until the thaw. +- A failed freeze says whether it undid its work (`undone`). When it did not, + the VM ends: no work resumes in a partly frozen state. +- Several workspaces freeze in order, all or nothing, and thaw in reverse. + After a failure with `undone`, those already frozen thaw, in reverse; + without it, the VM ends and nothing thaws. +- `thaw()` reverses a freeze, and `after_restore()` renews what the + executions need after a restore, before Ready. + +nvx-init has no workspace: it refuses a capture while an execution runs, or +kills the process groups and sweeps when the host asks, as its capture +request says. Without permission to kill, it refuses the request with `EBUSY` +while any process but PID 1 and its registered helpers runs; with it, it ends +a running execution's process group, then sweeps, killing every process but +PID 1 and its helpers again and again until none remains. The checks, the +sweep and the capture's own preparation share one 30-second deadline. Today +nvx-init captures only the Fresh boot of a platform capture, +`nvx_snapshot_tier=platform`, which assembles no root and admits no +execution: a session with a root, cold-booted or restored, refuses the request +with `ENOTSUP` before it checks or kills anything. + +An init captures before it mounts anything from the devices that a restore +replaces: the `distro` image, the scratch and the host filesystem device, and +any overlay. `capture` refuses while one is mounted, but the decision to +capture, and the cleanup before it, are the init's. ## Keep the transport silent -Answer the host's capture request, then send nothing on the transport until -`capture` returns. Keep the session and its owners, but do not send progress, -workload output or another response while capture is pending. - -- After `Continued`, thaw workspaces in reverse where they exist, and report - on the **same session**. -- After a failure with `Resumed`, do the same: thaw and report the failure on - the **same session**. -- After `Restored(repaired)`, take the session's reset and attach again before - accepting the activation record. The broker has a fresh instance and epoch, - with no old queued records. The NVXS guest driver acknowledges RESET with - `GuestConnection::acknowledge(window)` as part of this attachment. - -The following public calls are also compiled as -[`snapshot` rustdoc tests](../nvx-rs/src/guest/snapshot.rs); they deliberately do -not implement the init's freeze, transport, shutdown or space operations: +The init answers the host's capture request, then sends nothing on its +transport until `capture` returns: no progress, output or other response. +When the call returns: + +- After `Captured::Continued`, it thaws its workspaces in reverse, where they + exist, and reports on the same session. nvx-init reports + `capture-continued`. +- After a failure whose disposition is `Disposition::Resumed`, it thaws and + reports the failure on the same session. nvx-init writes the failure to the + console, after `NVX-INIT-CAPTURE:`, and reports `capture-failed`. +- After `Captured::Restored`, it takes the session's reset and attaches again: + the restored broker has a fresh instance, restarts at its first epoch with a + reset queued, and ignores the guest's records until the guest acknowledges + it. nvx-init waits for the reset, within the activation's deadline, fails on + anything else, and acknowledges it with `GuestConnection::acknowledge` + before it takes the activation. + +The guidelines quote this excerpt of the +[`guest::snapshot`](../nvx-rs/src/guest/snapshot.rs) rustdoc, where it +compiles as a public-API doc test: ```rust use std::time::{Duration, Instant}; -use nvx::guest::activation::Activation; -use nvx::guest::snapshot::{self, CaptureFailure, Captured, Devices, Disks, Repaired}; -fn fresh_capture(devices: &Devices, deadline: Instant) -> Result { - snapshot::capture(devices, Disks::Fresh, deadline) +use nvx::guest::activation::Activation; +use nvx::guest::snapshot::{self, Captured, Devices, Disks, Disposition}; + +/// What the caller does next; it sent nothing on its transport while `capture` ran. +enum Next { + /// Thaw the workspaces in reverse, where they exist, and report on the same session. + Report, + /// Take the session's reset and acknowledge it, then drive this activation to Ready. + Activate(Activation), + /// Start nothing, and power the VM off with this status. + PowerOff(u8), } -fn begin_activation(repaired: &Repaired, timeout: Duration, now: Instant) -> Activation { - Activation::new(repaired, timeout, now) +fn capture(devices: &Devices, deadline: Instant, timeout: Duration) -> Next { + match snapshot::capture(devices, Disks::Fresh, deadline) { + Ok(Captured::Continued) => Next::Report, + Ok(Captured::Restored(repaired)) => { + Next::Activate(Activation::new(&repaired, timeout, Instant::now())) + } + Err(failure) => match failure.disposition { + Disposition::Resumed => Next::Report, + Disposition::MustEnd { status } => Next::PowerOff(status), + // A disposition that a later version adds cannot be known to be safe. + _ => Next::PowerOff(125), + }, + // Nor can an outcome that a later version adds. + Ok(_) => Next::PowerOff(125), + } } ``` -## Complete restore before Ready - -Capture repairs a restored guest before returning `Restored(repaired)`: its -processor and memory targets, kernel randomness and time acknowledgement. -Keep that `Repaired` and one `Activation` for the restore. The activation's -deadline counts from the repair acknowledgement, not from each attachment or -retry. - -For `nvx-init`, restore proceeds in this order: - -1. `Activation::new`, then the reset/reattachment and the accepted activation - record. Validate the record and repaired resources before space side effects. -2. Apply the record to a Fresh space, including the VM-wide view and **this - restore's** network policy. Mount the runtime filesystems and retain mapping - descriptors. Apply its DNS/listener state; start and own its announcer before - Ready when listeners are published. Do not reuse the captured launch's DNS - or listener policy. -3. Start all required helpers, then switch into the root once. -4. Run each workspace's `after_restore()` where workspaces exist, renewing the - resources its executions need. -5. Call `Activation::complete` with the result. -6. Thaw workspaces in reverse where they exist, then report Ready. - -`nvx-init` omits workspace calls, not the ordering around them. Keep the applied -space, mapper, export source, helpers and activation alive. The host sends the -mapping table after Ready; no workload starts until the table is complete. - -An identical activation retry joins the existing activation or receives its -cached result; it must not apply or switch the root twice. `nvx-init` preserves -the activation and its original deadline across replacement control sessions, -checking the deadline rather than creating another bootstrap. Changed bytes -or another transaction are not a new permission to activate. - -The Fresh repair packet itself carries no replacement network settings. The -post-repair activation record supplies the restore's DNS and publication policy -that the init applies and owns. - -## Failures and replay - -After `Resumed`, thaw and report the failure on the same session. After -`MustEnd { status }`, start nothing, thaw nothing and power off with that exact -status; the library returns the disposition rather than powering off for the -init. A failed or partially applied activation likewise reports its failure -and ends the VM, not Ready. - -Never capture live external connections or other state that cannot be reused -safely by a restored guest. Close or reconstruct it under the init's capture -contract. Never replay an irreversible external effect just because execution -resumed from captured memory. Kernel reseeding does not repair a runtime's -private random state: the runtime must repair its own randomness before it -accepts work after restore, during its `after_restore()` where a workspace -exists. No workspace abstraction in the library enforces these replay rules. +## Complete a restore before Ready + +`capture` repairs a restored guest before it returns +`Captured::Restored(repaired)`: it activates the restore's processor and +memory targets, reseeds the kernel's random number generator, and finishes +the restore with its acknowledgement. The init makes one `Activation` of that +`Repaired`, whose deadline counts from the repair's acknowledgement, or from +when the init makes it for a restore that the host did not gate on one, and +never from a session or a retry. A restore then takes these steps, in this +order: + +1. `Activation::new`. nvx-init makes it when `capture` returns, with a + 30-second timeout, takes the session's reset, and accepts the activate + request: the activation decodes its record, and checks its restore section + and the CPUs and memory that the repair reports, before anything has a side + effect. +2. The space: the application of the record, with this restore's own network + policy, then the runtime file systems, the announcer and the switch, as + [the excerpt](#the-spaces-steps-checked) shows. The init applies the + record's DNS server, and, when the record publishes the workload's + listeners, starts the announcer before Ready and owns it as a helper. + nvx-init forks it from PID 1 before its switch, and runs + `/sbin/nvx-ingress` with the captured NIC's gateway, as the early boot runs + it at a cold boot. nvx-init also tells the record's proxy to the executions + that keep their default environment. +3. `after_restore()` on each workspace, where workspaces exist. +4. `Activation::complete`, with the result. +5. The thaw of the workspaces, in reverse, where they exist. +6. Ready. nvx-init answers the activate request, and each retry that joined + it, with Ready, which carries the record's transaction ID. + +nvx-init has no workspace, so it skips the third and fifth steps, not the +order around them. It keeps the applied root, the mapper and its +`MappingTable`, the export's source root, its helpers and the `Activation` +for the VM's life. The host then sends the mapping table, unless it is empty, +and nvx-init records its first execution, once the table is complete, with +`Activation::start`. + +A byte-equal retry joins the activation in progress, or gets its cached +Ready, on a later session too, and never applies or switches again: nvx-init +keeps the activation and its original deadline across sessions, and a reset +on a later session only checks that deadline. A changed request, another +transaction, and an application that failed, wholly or partly, are final. + +The packet with which a platform restore resumes the Fresh boot carries no +network settings, and `capture` refuses one that does: the activation record +gives the restore its network. + +## Failures + +- After `Disposition::Resumed`, the init thaws and reports the failure on the + same session. +- After `Disposition::MustEnd { status }`, it starts nothing, thaws nothing, + and powers the VM off with that status: the library returns the disposition + rather than power the VM off itself. nvx-init writes the failure to the + console, then syncs and powers off with the status. +- A failed activation, wholly or partly applied, never reports Ready: the init + reports the failure and ends the VM. nvx-init writes it to the console, + after `NVX-INIT-ACTIVATION:`, and powers off with status 125. + +## Replay + +Each restore resumes the captured memory, so: + +- An init never captures live external connections, or any other state that + a restored guest cannot reuse safely: it closes them before the capture, or + rebuilds them after the restore. +- It never repeats an irreversible effect because execution resumed from + captured memory. +- The repair reseeds the kernel's random number generator, but not a runtime's + own random state: a runtime repairs its randomness before it accepts work + after a restore, in its workspace's `after_restore()` where it has one. + +nvx-init captures a Fresh boot, before any workload runs, so a restore resumes +no workload's state, and its view generates a machine ID only after the +repair's reseed, when the record asks for one. diff --git a/doc/project-structure.md b/doc/project-structure.md index 6198eb022..591368cd7 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -82,6 +82,7 @@ nvx/ | |-- contribute.md Contribution guidelines | |-- design/ Architecture of the microVM and its guest runtime | |-- distribution.md Packaging and source delivery +| |-- guest-agent-guidelines.md Contract of a guest's init agent, with nvx-init as its example | |-- guests/ Guest userland chapters (Ubuntu and Azure Linux) | |-- modules/ Module diagrams of the nvx crate | |-- openvmm-upstream-roadmap.md Plan for upstreaming the OpenVMM fork's microVM commits diff --git a/nvx-rs/src/activation/mod.rs b/nvx-rs/src/activation/mod.rs index 5c9162fd5..215233bea 100644 --- a/nvx-rs/src/activation/mod.rs +++ b/nvx-rs/src/activation/mod.rs @@ -17,8 +17,10 @@ //! - [`ReadyActivation`]: what Ready reports about the activation: that it applied, which every //! layered root reports, and the transaction ID, which only a clone reports. //! -//! The [guest-agent guidelines] describe the init's ownership and restore order: applying this -//! record, its own DNS and listener policy, runtime filesystems and helpers precedes Ready. +//! The [guest-agent guidelines] give the order in which an init completes a restore before Ready: +//! it applies the record, with this restore's own network policy, mounts the runtime's file +//! systems, starts the announcer when the record publishes the workload's listeners, and switches +//! into the root once. //! //! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md //! diff --git a/nvx-rs/src/guest/activation.rs b/nvx-rs/src/guest/activation.rs index 7f6436a6b..a617d2ab3 100644 --- a/nvx-rs/src/guest/activation.rs +++ b/nvx-rs/src/guest/activation.rs @@ -145,6 +145,12 @@ //! and its [`ActivationCause`], and which every later call returns again. The activation never //! powers the VM off: the adapter reports the failure, and ends the VM. //! +//! The [guest-agent guidelines] give the order of a restore around the activation, from +//! [`Activation::new`] to Ready, with nvx-init as the worked example, and the obligations of +//! its caller, which the activation enforces none of. +//! +//! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md +//! //! The activation's application, `apply` and its types, is NVX's internal reference code of the //! guest's space, hidden from this documentation and unstable: it becomes crate-private once //! nothing outside NVX uses it. diff --git a/nvx-rs/src/guest/init/boot/tests.rs b/nvx-rs/src/guest/init/boot/tests.rs index 619313668..d8e75bec0 100644 --- a/nvx-rs/src/guest/init/boot/tests.rs +++ b/nvx-rs/src/guest/init/boot/tests.rs @@ -9,37 +9,55 @@ const CLI: &str = "nvx_identity=1000:1001 nvx_control_tty=hvc2"; const EXPORT: &str = "virtfs_dir=/run/nvx/hostfs/root virtfs_tag=hostfs virtfs_mode=rw virtfs_aggregate=1"; -/// The steps of NVX's reference space, as nvx-init takes them and the guest-agent guidelines -/// quote them, compile against the space's own interface. The test runs none of them: they would -/// mount over and switch the test host's root. +/// Compiles the steps of the private space that `doc/guest-agent-guidelines.md` quotes, without +/// mounting anything or switching the developer host's root. A change here changes the quote too. #[test] fn the_reference_space_steps_compile() { - use crate::guest::{mappings, root, view}; + use std::error::Error; + use std::net::IpAddr; + use std::path::Path; + use std::time::Instant; - // Compile the real private workflow without mounting or switching the developer host's root. - let _space = |assembly: &root::Assembly, - configuration: &crate::activation::View, - dns: Option, - export: &crate::protocol::export::HostExport, - time_status: Option<&std::path::Path>| - -> Result< - (root::Root, mappings::Mapper, mappings::SourceRoot), - Box, - > { + use crate::activation::{ActivationRecord, View}; + use crate::guest::activation::{self, Applied, RootTemplate}; + use crate::guest::mappings::{Mapper, Reserved, SourceRoot, TargetRoot}; + use crate::guest::root::{self, Assembly, Root}; + use crate::guest::view; + use crate::protocol::export::HostExport; + + // A cold boot, as `boot::cold_boot` builds it. + let _cold = |assembly: &Assembly, + export: &HostExport, + vm_view: &View, + dns: Option| + -> Result<(Root, Mapper, SourceRoot), Box> { let mut root = root::assemble(assembly)?; - root.mount_runtime(time_status)?; - view::apply(&mut root, configuration, dns)?; let source = root.mount_export(&export.tag, export.mode.into())?; - let mut target = mappings::TargetRoot::open(root.path())?; + root.mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(root.path())?; target.add_mount(&root.path().join("run"))?; - let mapper = mappings::Mapper::new(target, mappings::Reserved::container_root()); + let mapper = Mapper::new(target, Reserved::container_root()); + view::apply(&mut root, vm_view, dns)?; + // Start the helpers and open the descriptors that later work needs before the switch. + root.switch()?; Ok((root, mapper, source)) }; - let _switch = |root: &mut root::Root| root.switch(); - let _restored = |record: &crate::activation::ActivationRecord, - template: &crate::guest::activation::RootTemplate, - deadline: Option| { - crate::guest::activation::apply(record, template, deadline) + // A restore, as `boot::Restored::apply` builds it from the accepted record. + let _restored = |record: &ActivationRecord, + template: &RootTemplate, + deadline: Option| + -> Result<(Applied, Mapper, Option), Box> { + let mut applied = activation::apply(record, template, deadline)?; + applied + .root_mut() + .mount_runtime(Some(Path::new("/run/nvx/time")))?; + let mut target = TargetRoot::open(applied.root().path())?; + target.add_mount(&applied.root().path().join("run"))?; + let mapper = Mapper::new(target, Reserved::container_root()); + let source = applied.take_export(); + // Start the announcer here when `applied.publication()` publishes the listeners. + applied.root_mut().switch()?; + Ok((applied, mapper, source)) }; } diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 1898b24fa..7e67ddf1d 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -17,8 +17,8 @@ //! acknowledgement, and then takes its own configuration from the host, in one Activate on a //! restricted control session. //! -//! The [guest-agent guidelines] give the init-owned space, execution, capture and restore -//! contract, with nvx-init as the worked example. The library enforces none of those obligations. +//! The [guest-agent guidelines] give an init agent's contract for its space, its executions, a +//! capture, and a restore, with nvx-init as the worked example; the library enforces none of it. //! //! [guest-agent guidelines]: https://github.com/microsoft/nvx/blob/dev/doc/guest-agent-guidelines.md //! @@ -35,8 +35,8 @@ //! stay public: the mapping values and [`MappingTable`](mappings::MappingTable), and the tokens //! and records of [`protocol`](crate::protocol) and [`activation`](crate::activation). //! -//! The guidelines' public mapping excerpt forwards mount failures and keeps admission closed -//! until the table is complete; the mount callback belongs to the init: +//! The guidelines quote this excerpt, in which an init applies one part of the host's mapping +//! table through its own mapper, `mount`, and admits workloads only once the table is complete: //! //! ```no_run //! use nvx::guest::mappings::{Mapping, MappingTable, TableError}; diff --git a/nvx-rs/src/guest/snapshot.rs b/nvx-rs/src/guest/snapshot.rs index 11511207d..e719dbd09 100644 --- a/nvx-rs/src/guest/snapshot.rs +++ b/nvx-rs/src/guest/snapshot.rs @@ -18,20 +18,41 @@ //! instant. A repair failure is always [`Disposition::MustEnd`], with [`RepairError::status`]. //! Fresh restore network settings are explicitly refused. //! -//! The [guest-agent guidelines] specify the caller's workspace, silence, failure and restore -//! obligations. These public excerpts compile without implementing or running those operations: +//! The [guest-agent guidelines] give the caller's obligations around this call: its +//! workspaces, its silent transport, a failure's disposition, and the restore's order. They +//! quote this excerpt of what the caller does once the call returns: //! //! ```no_run //! use std::time::{Duration, Instant}; +//! //! use nvx::guest::activation::Activation; -//! use nvx::guest::snapshot::{self, CaptureFailure, Captured, Devices, Disks, Repaired}; +//! use nvx::guest::snapshot::{self, Captured, Devices, Disks, Disposition}; //! -//! fn fresh_capture(devices: &Devices, deadline: Instant) -> Result { -//! snapshot::capture(devices, Disks::Fresh, deadline) +//! /// What the caller does next; it sent nothing on its transport while `capture` ran. +//! enum Next { +//! /// Thaw the workspaces in reverse, where they exist, and report on the same session. +//! Report, +//! /// Take the session's reset and acknowledge it, then drive this activation to Ready. +//! Activate(Activation), +//! /// Start nothing, and power the VM off with this status. +//! PowerOff(u8), //! } //! -//! fn begin_activation(repaired: &Repaired, timeout: Duration, now: Instant) -> Activation { -//! Activation::new(repaired, timeout, now) +//! fn capture(devices: &Devices, deadline: Instant, timeout: Duration) -> Next { +//! match snapshot::capture(devices, Disks::Fresh, deadline) { +//! Ok(Captured::Continued) => Next::Report, +//! Ok(Captured::Restored(repaired)) => { +//! Next::Activate(Activation::new(&repaired, timeout, Instant::now())) +//! } +//! Err(failure) => match failure.disposition { +//! Disposition::Resumed => Next::Report, +//! Disposition::MustEnd { status } => Next::PowerOff(status), +//! // A disposition that a later version adds cannot be known to be safe. +//! _ => Next::PowerOff(125), +//! }, +//! // Nor can an outcome that a later version adds. +//! Ok(_) => Next::PowerOff(125), +//! } //! } //! ``` //!