diff --git a/doc/design/1.3-system-components-and-responsibilities.md b/doc/design/1.3-system-components-and-responsibilities.md index 5d0d295d7..83e5b4fec 100644 --- a/doc/design/1.3-system-components-and-responsibilities.md +++ b/doc/design/1.3-system-components-and-responsibilities.md @@ -4,20 +4,22 @@ [1. Architecture Overview](1-architecture-overview.md) NVX spans three layers (Figure 1.2), and their responsibilities follow the -boundary between policy and mechanism. Host software owns policy, -orchestration, placement, identity, and attachment authorization. The OpenVMM -virtual machine monitor (VMM) owns the virtual platform and transition safety. -Guest software owns workload construction, supervision, and guest-visible -repair, while the workload owns only its execution inside the resulting -sandbox. +boundary between policy and mechanism. In the host, the `nvx` library holds +the mechanisms that every user of a microVM needs, and its callers, such as an +orchestrator or NVX's own command-line interface, hold the policy and the +state: placement, identity, storage, attachment authorization, and recovery. +The OpenVMM virtual machine monitor (VMM) owns the virtual platform and +transition safety. Guest software owns the guest's space, its executions, and +guest-visible repair, while the workload owns only its execution inside the +resulting sandbox. ```mermaid flowchart TB subgraph HostLayer["Host"] direction LR - Launcher["NVX launcher"] - ControlLibrary["Host control library"] - Storage["Snapshot storage"] + Callers["Callers
an orchestrator or NVX's CLI"] + Library["nvx library"] + Storage["Caller-owned VM and
snapshot directories"] Attachments["External attachments"] end @@ -35,14 +37,14 @@ flowchart TB subgraph GuestLayer["Guest"] direction LR Kernel["Linux kernel"] - Agents["Guest agents"] + Agents["Guest init and helpers,
with the nvx guest library"] Workload["Workload"] end - Launcher --> Frontends - ControlLibrary --> Frontends - Launcher -. control session .-> Virtio - ControlLibrary -. control session .-> Virtio + Callers --> Library + Callers --> Storage + Library --> Frontends + Library -. control session .-> Virtio Frontends --> Profile Frontends --> Worker Profile --> Worker @@ -62,16 +64,37 @@ flowchart TB ## 1.3.1 Host Components -- **NVX launcher** runs one-shot microVMs and the managed sandbox lifecycle, - keeps per-sandbox state, prechecks host shares and places them in the guest, - and compiles structured egress-policy files into VMM egress rules. -- **Host control library** implements the host side of the control protocol, - artifact discovery, blockless managed launch, VMM process supervision, - host-path export with its access policy, delivery of the host mapping table, - and egress-rule translation. -- **Snapshot storage** retains immutable machine state, memory, and optional - paired scratch together with the identities needed to reconstruct external - resources. +- **Callers**, such as an orchestrator or NVX's command-line interface (CLI), + `nvx-rs`, own the policy and the state that the library leaves to them: the + runtime files and their labels, each VM's configuration, the private + directories of their VMs and snapshots, with their names and retention, the + values with which they find a running VM again, their per-snapshot locks, + and the protocol that they speak with their guest over the control session, + with its timing, retries, and recovery. NVX's CLI is one such caller, which + the `nvx` crate builds with its `cli` feature, outside its documented API: + it keeps its own store of sandboxes and named snapshots, its locks, the path + rules of its exports, with the identities of the host objects that it pins + in them, the staging of its images, and the compilation of its egress-policy + files. +- **The `nvx` library** holds the mechanisms and value types that every caller + needs. It checks a VM's configuration without reading the host, and lowers + it into OpenVMM's arguments and the guest's boot tokens. It creates a cold + boot, a capture, or a restore in a private VM directory that its caller + owns, starts OpenVMM detached, holds the VM's process by a verified + reference, and verifies, reattaches to, and stops it. It verifies the + control endpoint's server and plays the host's role of the control session, + which carries the caller's protocol. It finishes a capture with the key and + description that make it a strict snapshot directory, opens and checks such + a directory, checks a VM's compatibility with it, and orders a restore's + activation to Ready without I/O. It keeps no durable state, takes no lock + but the one that serializes its own spawns on Windows, and starts no process + but OpenVMM. +- **Caller-owned directories** hold what the library and OpenVMM write for a + caller: each VM's private directory, with its log, outcome report, + endpoints, and snapshot files, and the strict snapshot directories in which + the caller keeps its captures, each with its immutable machine state and + memory, its key, and its description. The caller names, places, protects, + and removes them. - **External attachments** supply process-local resources such as console endpoints, network endpoints, host filesystem exports, and block files. @@ -109,19 +132,30 @@ flowchart TB ## 1.3.3 Guest Components - **Linux kernel** boots from the fixed Linux-direct handoff, drives the - allowlisted virtual devices, and provides the namespaces, filesystems, - cgroups, and process primitives used by the runtime. -- **Guest agents** mount runtime filesystems, apply boot configuration, mount - live host shares, and assemble the workload filesystem. They construct the - workload's root and credentials, and in the layered sandbox root its - namespaces and device view. In the blockless managed root, they create - workload accounts when requested. They supervise one-shot or managed - workloads, prepare snapshots, and repair - processors, memory, entropy, and identity after restore. They also announce - the guest's listeners when the host publishes them, check time ABI - conformance, watch for clock violations, discipline the wall clock, and - perform the snapshot time steps. + allowlisted virtual devices, and provides the filesystems, mounts, and + process primitives that the guest's software uses. +- **Guest agents**, the guest's init, PID 1, and its helpers, build the VM's + one space: they mount runtime filesystems, apply boot configuration, mount + live host shares, and, in a layered root, assemble the root from its image + and upper and switch into it once. They run the workload's executions as + plain processes with the VM's workload credentials, without a namespace or + cgroup of their own, and create workload accounts when requested. They + supervise one-shot or managed workloads, prepare snapshots, repair + processors, memory, entropy, and identity after restore, and activate a + restored layered root before Ready. They also announce the guest's listeners + when the host publishes them, check time ABI conformance, watch for clock + violations, discipline the wall clock, and perform the snapshot time steps. + The space and the executions are the init's own code: nvx-init, NVX's init + for layered roots, implements NVX's reference space, which is private to + `nvx`. The `nvx` guest library gives any init the contracts that it reads, + the guest's side of a capture, which also repairs a restore, the activation + of a restore, and the account of the host's mapping table, `MappingTable`; + it starts no helper or workload but the time daemon, which + `TimeAbi::start_daemon` forks, and the + [guest-agent guidelines](../guest-agent-guidelines.md#ownership) give the + one other call that may start a process, and the init's obligations, which + `nvx` does not enforce. - **Workload** is the microVM's single tenant: it owns the microVM's lifetime, resource envelope, and network identity, and it may run any number of - processes, all inside the isolation boundary that the guest agents - construct. It has no direct authority over host resources or VMM policy. + processes, all inside the VM, its isolation boundary. It has no direct + authority over host resources or VMM policy. diff --git a/doc/design/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md index 372af6da6..bc7057471 100644 --- a/doc/design/1.4-trust-boundaries.md +++ b/doc/design/1.4-trust-boundaries.md @@ -3,25 +3,31 @@ [Architecture documentation](README.md) > [1. Architecture Overview](1-architecture-overview.md) -As Figure 1.3 shows, the host and its orchestrator are trusted with guest -memory, image contents, and attachment policy, while the workload is -untrusted. +As Figure 1.3 shows, the host's control plane, a caller and the `nvx` library +that it uses, is trusted with guest memory, image contents, and attachment +policy, while the workload is untrusted. Inside the control plane, the library +checks what its caller gives it, but trusts the caller with what the caller +owns, as [The Library and Its Callers](#146-the-library-and-its-callers) +describes. ```mermaid flowchart LR - Control["Trusted host control plane"] + Caller["Trusted caller
policy, paths, and state"] + Library["nvx library
checks and mechanisms"] Vmm["VMM trust boundary"] Agent["Guest agent boundary"] Workload["Untrusted workload"] Resources["Host resources"] - Control -->|"policy and capabilities"| Vmm + Caller -->|"configuration, paths, and labels"| Library + Library -->|"checked launch and capability"| Vmm Resources -->|"validated attachments"| Vmm Vmm <-->|"bounded device protocols"| Agent - Agent -->|"isolation, identity,
resources, and supervision"| Workload + Agent -->|"space, credentials,
and supervision"| Workload ``` -**Figure 1.3.** Trust boundaries of the host, VMM, guest agents, and workload. +**Figure 1.3.** Trust boundaries of the caller, the library, the VMM, the guest +agents, and the workload. Because each microVM has a single tenant, the virtual machine boundary is also the boundary between workloads. A workload may use ordinary Linux interfaces @@ -38,6 +44,10 @@ validation boundaries. Authority to provide an input does not make its structure valid. OpenVMM therefore validates the machine configuration before opening host resources, validates it again at worker construction, and validates all persisted state before any restored virtual processor runs. +`nvx` checks a caller's configuration without reading the host, checks a +launch completely before it starts OpenVMM, and checks a snapshot's directory, +key, and manifest before it restores from them, in addition to OpenVMM's +checks, never in their place. ## 1.4.1 Defense in Depth @@ -62,12 +72,19 @@ of resuming with uncertain state. A host control client reaches the guest only through the control-console broker. The broker admits one local peer whose operating-system identity is -the VMM's user, requires a launcher-provided capability before any host data -reaches the guest, and bounds every record, queue, and credit window in both -directions. Malformed, stale, or out-of-order records fail closed. -Capabilities are process-local authority: they never appear in arguments, -environment variables, logs, endpoint names, snapshots, or attachment -identities. +the VMM's user, requires the capability that the VM's launch provided before +any host data reaches the guest, and bounds every record, queue, and credit +window in both directions. Malformed, stale, or out-of-order records fail +closed. Capabilities are process-local authority: they never appear in +arguments, environment variables, logs, endpoint names, snapshots, or +attachment identities. + +`nvx` generates a fresh capability for each launch, passes it to OpenVMM only +on its standard input, never writes it to a file, and hides it from its +debugging output: a caller that must reconnect after a restart keeps it +itself. Before a caller presents the capability, `nvx` checks that the control +endpoint's server is the VM's held process: through its socket's original peer +on Linux, and through the named pipe's server process on Windows. ## 1.4.3 Device-Enforced Policy @@ -81,24 +98,29 @@ The guest's announcements choose which of its ports are published, never the host addresses or the clients. The host filesystem server applies each export's access mode and path policy on every request, whichever guest mount or link reaches it. Guest mount flags, routes, and links are therefore never -security boundaries. +security boundaries. `nvx` lowers its caller's network policy and export into +these device policies, and enforces none of its own. ## 1.4.4 Images and Snapshot Artifacts Image layers are trusted inputs: the overlay filesystem interprets metadata stored in lower layers, and bytes read from any layer can appear in captured -memory. A snapshot identifies its blocks by a storage generation, which -transfers trust for layer and scratch contents to the storage owner; the VMM -trusts the control plane in the host to authenticate that generation. +memory. A snapshot that binds its blocks identifies each, by default, by the +SHA-256 digest of its file, which OpenVMM computes, or by a storage generation, +which transfers trust for layer and scratch contents to the storage owner; the +VMM trusts the control plane in the host to authenticate that generation. A +platform snapshot, the only kind that `nvx` captures, binds no block: each +restore attaches its caller's own image and upper, which the caller trusts as +it trusts any image. The VMM trusts the host control plane not to give the same writable file, such as a caller-supplied RAM backing or scratch file, to more than one VMM. It -takes no file locks to detect such sharing, and the RAM backing that it -creates itself is a new file for every VM. Snapshot artifacts are shared by -design: every VM restored from a snapshot maps the same immutable memory -artifact with copy-on-write, so each VM's changes stay private and never reach -the artifact. A capture always publishes new files, and a restored VM is not -captured again. +takes no file locks to detect such sharing, nor does `nvx`, and the RAM +backing that the VMM creates itself is a new file for every VM. Snapshot +artifacts are shared by design: every VM restored from a snapshot maps the same +immutable memory artifact with copy-on-write, so each VM's changes stay private +and never reach the artifact. A capture always publishes new files, and a +restored VM is not captured again. Copy-on-write isolates writes, not physical pages. Clean pages of a shared memory artifact may remain shared through the host page cache among the VMs @@ -106,15 +128,80 @@ that map it, and guest kernel, guest agent, and cached workload pages can all originate in captured memory. Snapshot copy-on-write is therefore not, by itself, a tenant authorization or physical-page isolation mechanism. Artifact access control, sharing scope, backing-file placement, retention, and host -memory deduplication remain deployment responsibilities. +memory deduplication remain the responsibilities of the caller and its +deployment. ## 1.4.5 Host-Facing Outputs -Host-facing outputs are bounded. The optional outcome report is written after -teardown to a previously absent local path. It records only a schema version, -an opaque instance ID, the backend category, the operation and outcome -category with a numeric status, whether the network policy was requested and -applied with its mode, rule counts, and host-loopback action, and per-step -teardown flags. It never contains commands, environment values, paths, -network destinations, proxy details, workload output, credentials, or -free-form error text, and the VMM does not upload it. +A launch's host-facing outputs are OpenVMM's log and its outcome report. `nvx` +gives both fixed names in the caller's private VM directory, beside the VM's +endpoints: it creates the log anew, private to the caller, before OpenVMM +starts, and refuses a launch whose outcome report or Linux socket endpoints +already exist rather than replace them. OpenVMM writes the report after its +VM's teardown, so an OpenVMM process that ends before then leaves none. A +creation that fails removes only the entries that it created, and once a +launch's creation succeeds, `nvx` removes none of its outputs: the caller +reads and removes them. + +The outcome report is bounded. It is written after teardown to a previously +absent local path. It records only a schema version, an opaque instance ID, +the backend category, the operation and outcome category with a numeric +status, whether the network policy was requested and applied with its mode, +rule counts, and host-loopback action, and per-step teardown flags. It never +contains commands, environment values, paths, network destinations, proxy +details, workload output, credentials, or free-form error text, and the VMM +does not upload it. + +## 1.4.6 The Library and Its Callers + +`nvx` checks what its caller gives it, but holds nothing for the caller +between calls: it keeps no record, retains no handle of a snapshot, and takes +no lock of a VM or a snapshot. It checks: + +- That each VM directory is private to the current user: on Linux, owned by + the user with mode 0700, which OpenVMM requires of the directory of its + control socket; on Windows, owned by the user, with a protected access + control list that grants access to the user alone. It never changes an + existing directory, since restricting it would not revoke access through + files that others already hold open. +- That a snapshot directory has the strict form that + [Validation and Integrity](5.4-snapshot-artifact-and-attachment-model.md#543-validation-and-integrity) + describes, once, when it opens the snapshot. +- That a VM's process is the one that the caller recorded, by its host boot, + process ID, and start time, and that the peers of its control and + restore-ready endpoints are that process. +- That a saved VM's runtime identity is the current one, before it drives the + VM; see [Runtime Identity](1.5-compatibility-and-versioning-model.md#154-runtime-identity). + +The caller keeps what `nvx` cannot check once and for all: + +- **Paths.** It keeps the runtime files and the block images stable, and its + VM and snapshot directories stable and private, from a launch's creation to + the end of their use, and each snapshot directory immutable while it uses + the snapshot. A restore hard-links the snapshot's files into the VM's + directory, so it keeps both on one filesystem. The `ExportPlan` of an export + names host paths only, never the objects at them, so the caller also keeps + each path naming the object that it means, as the + [export's module](../../nvx-rs/src/host/filesystem/mod.rs) describes: across + the race between its own checks and OpenVMM's opens, through a recheck of a + workload's writable parts before each later launch, and for as long as each + denial is to hold. NVX's CLI pins the identities of the mapped and denied + objects inside those writable parts when it plans an export, and checks + them before each start. +- **Outputs.** It owns each launch's log, outcome report, and endpoints, which + `nvx` never removes, though on Linux OpenVMM removes the control and + boot-console sockets that it bound, at its VM's teardown. It also owns each + capture's files, which it moves into a snapshot directory of its own. It + saves what finds a running VM again: its directory, control endpoint, + capability, process reference, and runtime identity. +- **Per-snapshot locks.** On Linux, adding or removing a restore's links to a + snapshot's files changes their metadata, which OpenVMM checks until it + connects the restore-ready endpoint. The caller therefore holds a lock of + its own for each snapshot from the creation of a restore, which adds the + links, through its peer-verified restore-ready event or its VM's verified + exit, and around the removal of a restore's view of the snapshot. Windows + needs none. +- **Spawns.** On Windows, `nvx` serializes its own spawns of OpenVMM, but a + process that the caller starts while a spawn passes handles to OpenVMM may + inherit them, so the caller keeps its other process starts out of a VM's + start. diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index 0a8926e9f..3fffb8154 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -3,23 +3,31 @@ [Architecture documentation](README.md) > [1. Architecture Overview](1-architecture-overview.md) -Table 1.1 lists the explicit contracts that define NVX compatibility. Each -contract carries its own version or identity, independent of the others. +Table 1.1 lists the explicit contracts that define NVX compatibility, and the +owner of each: OpenVMM, or the `nvx` library, which the host's callers and the +guest's inits share. Each contract carries its own version or identity, +independent of the others. A caller's own records, such as the store of NVX's +command-line interface (CLI), and any protocol of its own with its guest are +the caller's to version. **Table 1.1.** Compatibility contracts. -| Contract | Governs | -| --- | --- | -| Machine ABI | CPU topology, memory layout, device topology, addresses, interrupts, transport, and guest-visible policy. | -| Boot layout | Placement and format of the Linux-direct boot structures: MP tables, bootstrap GDT and page tables, Linux boot-parameter block (zero page), and kernel command line. | -| CPU profile | Guest-visible CPU surface of one vendor and CPU generation, identified by vendor, generation, and revision, and by the digest of its canonical encoding. | -| Time ABI | Hypervisor identity, frequency interfaces, clocksources, timers, downtime semantics, and restore-time repair. The guest reads its version from the CPUID functions reserved for hypervisor information (hypervisor identity leaves). | -| Device-private schemas | Saved queue and device state that must be understood before a device can resume. | -| Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | -| RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | -| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, and fresh entropy. | -| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, and lifecycle epochs. | -| Management RPC machine profile | Identity of the microVM machine profile in the management RPC. | +| Contract | Owner | Governs | +| --- | --- | --- | +| Machine ABI | OpenVMM | CPU topology, memory layout, device topology, addresses, interrupts, transport, and guest-visible policy. | +| Boot layout | OpenVMM | Placement and format of the Linux-direct boot structures: MP tables, bootstrap GDT and page tables, Linux boot-parameter block (zero page), and kernel command line. | +| CPU profile | OpenVMM | Guest-visible CPU surface of one vendor and CPU generation, identified by vendor, generation, and revision, and by the digest of its canonical encoding. | +| Time ABI | OpenVMM | Hypervisor identity, frequency interfaces, clocksources, timers, downtime semantics, and restore-time repair. The guest reads its version from the CPUID functions reserved for hypervisor information (hypervisor identity leaves). | +| Device-private schemas | OpenVMM | Saved queue and device state that must be understood before a device can resume. | +| Snapshot format | OpenVMM | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. | +| RAM expansion capability | OpenVMM | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. | +| Restore packet | OpenVMM | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, fresh entropy, and the network settings of a policy that the restore installs on a snapshot without a tier. | +| Control session | OpenVMM | The authenticated session between the host and the guest that the control-console broker frames: its records, credits, authentication, and lifecycle epochs. | +| Management RPC machine profile | OpenVMM | Identity of the microVM machine profile in the management RPC. | +| Boot contract | `nvx` | The microVM ABI on which NVX's guests boot, the kernel command-line tokens through which the host configures them, such as a layered root's image, upper, and view, and the root that the guest assembles from them. | +| Activation record | `nvx` | The settings that a restored layered root applies in place of its captured configuration, and what Ready reports about the activation. | +| Snapshot key | `nvx` | What a VM restored from a strict snapshot directory must match, in one canonical encoding whose digest is the snapshot's compatibility identity. | +| Control protocol | `nvx` | NVX's own requests and responses between its CLI and its guest agents, which the control session carries; private to `nvx`. | A change to a guest-visible address, interrupt, command-line token, feature mask, queue shape, time rule, or device behavior requires a new contract @@ -52,34 +60,55 @@ A snapshot restores only: - With a VMM build that computes the same effective CPUID for that CPU profile and topology. - With external resources that reproduce the recorded attachment identities. +- Through `nvx`, under the runtime identity that the snapshot's key records; + see [Runtime Identity](#154-runtime-identity). Released CPU-profile revisions are immutable and never deleted, so existing snapshots stay restorable; a normative change creates a new revision. -## 1.5.2 Template Compatibility - -A reusable template represents a compatibility class, not an arbitrary fleet -image. The class includes the backend, the CPU profile and generation, the -effective CPUID, the guest kernel and guest agent build, the effective command -line, device roles and geometry, network identity and policy, and consumed -layer identities. Restoring guest memory restores the captured guest agent -build, not a newly deployed initramfs, so deployments rebuild or recertify -templates on relevant rollouts. +## 1.5.2 Snapshot Compatibility + +A snapshot of `nvx` represents a compatibility class, not an arbitrary fleet +image. When `nvx` finishes a capture, it derives the class from the capture's +context and OpenVMM's manifest alone, without reading a binary, an image, or +the memory, and records it in the snapshot's key: the runtime identity; the +versions of the snapshot format, saved device state, time ABI, restore packet, +and activation record; the backend, the CPU profile, the effective CPUID, the +topology, and the declared rates; the processors and RAM at boot, their +capacities, and the expansion ranges; the effective command line; the +geometries of the placeholder image and the dormant scratch; the devices; the +NIC, if any; and the caller's invariants, up to 64 named settings that `nvx` +keeps without interpreting them. The key's identity, a `KeyIdentity`, digests +all of these, but not the capture's own time fields: its anchors, host clock, +host and boot identities, lineage, and cut-to-anchor interval. Captures of one +class at different times therefore share it, while the declared rates remain +part of it. Restoring guest memory restores the captured guest build, not a +newly deployed initramfs, so a deployment that changes a runtime label +captures its snapshots again. Processor capacity and RAM geometry are immutable, but restore-time activation -lets one template serve several online processor counts and RAM sizes. -Templates are therefore keyed by capacities, base state, and supported -activation ranges. Layer-role presence and device geometry stay fixed: a -platform template accepts a substitute layer only with the same recorded -logical length and block sizes. - -A template's compatibility key also records its NIC, if it has one: the -guest's address and the identity of the complete deny-all policy under which -the capture attached it. Every restore of the template replaces that policy -with its clone's own, so a template serves the clones of every network policy -at its guest's address, and its key records none of them. The key's encoding, -version 1, has not been released, so no key that an earlier build encoded is -supported. There is no migration. +lets one snapshot serve several online processor counts and RAM sizes, from +those at boot to the capacities. Layer-role presence and device geometry stay +fixed: a restore attaches an image with the placeholder's logical and physical +block sizes and at most its length, and a scratch with the dormant scratch's +exact geometry, or, for an upper in RAM, declares the scratch absent. Nothing +of a restore is part of the key: its image, upper, export, view, network +policy, workload, or activation targets. + +The key also records the snapshot's NIC, if it has one: the guest's address +and the identity of the complete deny-all policy under which the capture +attached it. Every restore of the snapshot replaces that policy with its own, +so a snapshot serves the restores of every network policy at its guest's +address, and its key records none of them. The key's encoding, version 1, has +not been released, so no key that an earlier build encoded is supported. There +is no migration. + +Before it restores a snapshot, `nvx` checks the requested VM against the +snapshot's key and manifest with `Snapshot::accepts`: a layered root; +processor and memory targets within the snapshot's limits; the image and +scratch geometries; the NIC's presence and address; the boot console's captured +attachment; the backend, CPU profile, and runtime identity; and the time +contract's restore window at the host's current time. ## 1.5.3 Persisted State @@ -87,3 +116,33 @@ The snapshot manifest is authoritative for a restored process. Native file descriptors, sockets, named-pipe handles, and hypervisor objects are never part of the portable state. They are recreated from stable identities and declared reconstruction policies. + +`nvx` persists no state of its own. Besides the files of a launch in its +caller's VM directory, it writes only a finished capture's key and +description. A caller that must find a running VM again after a restart saves +its own values: the VM's directory, its control endpoint, its capability, its +process reference, and its runtime identity. NVX's CLI keeps them in its own +store, beside records of its own, in formats of its own. + +## 1.5.4 Runtime Identity + +A VM's runtime identity names the runtime that launched it: a label for each of +its three runtime files, OpenVMM, the guest kernel, and the guest initramfs, +and the versions of the boot contract, the control contract, and the control +session that `nvx` speaks. A label is an opaque token that the caller takes +from the files' install or staging metadata, such as `sha256:` and the digest +that a release's `SHA256SUMS` lists for the file: `nvx` compares labels and +never reads or digests a runtime file, which would cost time in proportion to +its size at every launch. NVX's CLI takes them from a release's `SHA256SUMS`, +or from the build provenance that a repository checkout keeps beside its +artifacts. + +The runtime identity decides whether a host keeps driving a VM that it finds +recorded, as after the caller restarts or upgrades its own code. Under the same +identity, `vm::verify` returns the VM's held process, and `vm::reattach` +authenticates again with the saved capability. Under another, `nvx` refuses to +drive the VM through artifacts or contracts that did not start it, with an +error of kind `RuntimeMismatch` that names the parts that differ, and only a +stop through the VM's verified process applies. A snapshot's key records the +runtime identity of its capture, so a snapshot restores only under that +identity. diff --git a/doc/design/2.1-machine-model-and-configuration.md b/doc/design/2.1-machine-model-and-configuration.md index 33c957e01..edd78512f 100644 --- a/doc/design/2.1-machine-model-and-configuration.md +++ b/doc/design/2.1-machine-model-and-configuration.md @@ -7,7 +7,9 @@ The machine profile is selected explicitly and owns all guest-visible configuration. Machine identity is never inferred from a kernel, device, or hypervisor choice: every frontend, including the command line, the management RPC, and snapshot configuration, carries it to the VM worker. Validation occurs -before host resources are opened and again at the worker boundary. +before host resources are opened and again at the worker boundary. `nvx` names +the microVM machine profile in every launch that it renders, and checks its +caller's configuration before OpenVMM runs, in addition to that validation. The current machine profile's architectural invariants are: @@ -53,24 +55,29 @@ lifetime is one of: | Configuration | Owner | Lifetime | | --- | --- | --- | | Boot layout, device placement, interrupts, and feature masks | Machine profile | Contract version | -| Processor capacity, base RAM and optional RAM capacity, and CPU profile | Host, validated by the machine profile | VM | -| Effective kernel command line | Machine profile, from host policy and caller arguments | VM | -| Workload identity and lifecycle | Host | VM | -| Console, network, block, and host filesystem devices, with their identities and policies | Host | VM | -| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Host, matching the recorded identities | VM process | -| Online processor and RAM activation targets | Host, within snapshot limits | Restore | - -The host fixes the workload policy of a fresh boot. NVX's launchers name a -fixed workload identity, a nonzero numeric UID and GID, and a one-shot or -managed lifecycle, which selects whether the guest agents run one workload or -stay resident for sequential requests over the control console, with -command-line tokens of their own (see +| Processor capacity, base RAM and optional RAM capacity, and CPU profile | Caller, checked by `nvx` and validated by the machine profile | VM | +| Effective kernel command line | Machine profile, from the boot tokens of `nvx` and the caller's | VM | +| Workload identity and lifecycle | Caller, in its guest's own tokens or requests | VM, or each restore of a platform snapshot | +| Console, network, block, and host filesystem devices, with their identities and policies | Caller, checked and lowered by `nvx` | VM | +| Host attachments: console endpoints, the network endpoint, block files, and exported host directories | Caller, matching the recorded identities | VM process | +| A platform snapshot's image and scratch, a dormant filesystem slot's export, and a NIC template's network policy | Caller, within snapshot limits | Restore | +| Online processor and RAM activation targets | Caller, within snapshot limits | Restore | + +The caller fixes the workload policy of a fresh boot, and `nvx` names none: its +configuration describes the machine, and the caller names its guest's workload +in tokens or requests of its own. NVX's command-line interface (CLI) names a +fixed workload identity, a nonzero numeric UID and GID, and, for a blockless +root, a one-shot or managed lifecycle, which selects whether the guest agents +run one workload or stay resident for sequential requests over the control +console, with command-line tokens of its guests (see [Effective Command Line](5.1-boot-protocol.md#512-effective-command-line)). A +layered root is always managed, and its guest takes a restore's workload +identity from the request that carries the restore's activation record. A managed lifecycle requires a fixed workload identity and a live, authenticated control console. The machine profile also takes an identity and a lifecycle as -tokens of its own, which caller arguments cannot supply. NVX's guests select -neither from them, and fail closed when the profile names an identity that the -launcher's token does not. +tokens of its own, which caller arguments cannot supply and `nvx` never asks +for. NVX's guests select neither from them, and fail closed when the profile +names an identity that the CLI's token does not. ## 2.1.3 Storage Configuration @@ -91,10 +98,19 @@ writable scratch, both backed by cached, regular raw files with nonzero, 512-byte-aligned geometry. Blockless machines are also snapshot-capable and carry no sandbox tier metadata. +`nvx` captures one storage configuration: a blockless boot, whose guest +assembles no root, to which it attaches a sparse placeholder image as the +`distro` layer and a dormant scratch, private files that it creates in the +caller's VM directory at the geometries that the caller names, at the platform +tier (see [Tiers](5.4-snapshot-artifact-and-attachment-model.md#544-tiers)). +Each restore then attaches the caller's own layered root: its image, and its +scratch, or, for an upper in RAM, a scratch device without a file, unlike the +RAM-backed overlay layout, which has no scratch device. + Host filesystem sharing uses the fixed filesystem slot rather than adding devices. The slot exports one host directory, or an aggregate of named directory or regular-file children, each with its own access mode, beneath a -read-only synthetic root. +read-only synthetic root. `nvx` always exports an aggregate. ## 2.1.4 Restore-Time Configuration @@ -103,8 +119,15 @@ the snapshot's machine contract. Restore-time input may only: - Select the backend kind that the snapshot records. - Supply required attachments. -- Bind a new directory export to a captured dormant filesystem slot, when the - snapshot records that capability. +- For a platform snapshot, supply its own image, with the captured block sizes + and at most the captured length, and its own scratch, with the captured + geometry, or declare the scratch absent. +- Bind a new directory export to a captured dormant filesystem slot, or, for a + platform snapshot, a new aggregate export, when the snapshot records that + capability. +- Install a network policy of its own on a NIC template: a NIC that a platform + snapshot, or a snapshot without a tier, captured under the complete deny-all + policy. - Select processor and RAM activation targets that the contract allows. These process-local inputs never change processor capacity, RAM capacity, the @@ -112,7 +135,10 @@ command line, device placement, feature masks, or filesystem and network identity. Restore always uses the snapshot's CPU profile and never falls back. Because the captured command line fixes network addressing, workload identity, and lifecycle, restore rejects inputs that would change them, and a supplied -processor count may only repeat the captured capacity. +processor count may only repeat the captured capacity. A restore of `nvx` +changes none of them: it renders the devices at the snapshot's processor +capacity, and its guest takes its root, view, and workload from the restore's +activation record and the request that carries it. ## 2.1.5 Execution Backends diff --git a/doc/design/3.4-block-storage.md b/doc/design/3.4-block-storage.md index a261ee360..fea5f5362 100644 --- a/doc/design/3.4-block-storage.md +++ b/doc/design/3.4-block-storage.md @@ -75,18 +75,23 @@ writable role. A snapshot destination or a restore makes every block snapshot-capable. Each block must then be a nonempty, 512-byte-aligned raw regular file that uses -cached I/O and is not reached through a final symbolic link. The VMM retains -that opened file for flushing and paired-scratch publication. +cached I/O and is not reached through a final symbolic link, except the +scratch that a platform restore declares absent, which has no file. The VMM +retains that opened file for flushing and paired-scratch publication. ## 3.4.5 Snapshot Contract and Quiesce The snapshot contract records each block's role, read-only flag, logical -length, logical and physical block sizes, and identity. Every consumed block -shares one identity, a caller-authenticated immutable storage generation; see +length, logical and physical block sizes, and identity. A bound block's +identity is, by default, the SHA-256 digest of its file, which capture records +and restore verifies by reading the file, or, when its storage owner opts in, +a caller-authenticated immutable storage generation that every bound block +shares, which neither reads a block to check; see [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). Platform-tier layers are recorded as unbound, because image binding has not -been consumed, so a restore may supply different layers of the same geometry. -Writable scratch follows one of two policies: +been consumed, so a restore may supply a different image with the captured +block sizes and at most the captured length, past whose end reads return +zeros. Writable scratch follows one of two policies: - **Paired.** Capture publishes the exact scratch image with its length and identity. Restore rejects a caller-supplied scratch block and materializes @@ -94,11 +99,23 @@ Writable scratch follows one of two policies: single-use claimed continuation. - **Fresh.** Capture happens before scratch is mounted and publishes no scratch artifact. Restore requires a new writable file with matching - geometry. + geometry, or, for a platform snapshot, may declare the scratch absent: the + device keeps the captured geometry and stays writable, but its reads return + zeros and its writes fail, for a guest that never writes it, such as one + that keeps its upper in RAM. + +`nvx` captures only the platform tier's configuration: an unbound placeholder +image and a fresh scratch, which it creates as sparse private files in the +caller's VM directory, at the geometries of the caller's capture +configuration, for a guest that captures before it mounts either. Before each +restore, `Snapshot::accepts` checks the caller's image against the +placeholder's length and block sizes, and its scratch against the dormant +scratch's geometry, and a restore whose upper is in RAM declares the scratch +absent. Transport and queue progress are saved with the other device state. Stopping a queue prevents new descriptor intake and drains every accepted I/O to one used-ring completion. The drain has no deadline of its own: a backend that -never completes an I/O stalls the capture until its launcher ends the VMM; see +never completes an I/O stalls the capture until its caller ends the VMM; see [Capture Transaction](5.3-snapshot-protocol.md#533-capture-transaction). The controller flushes and copies scratch only after that drain. diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index 81fed733f..5f39cb3b6 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -41,8 +41,9 @@ interrupts on IOAPIC input 16 on every backend portable network profile is mandatory and selects the same in-process data plane on every backend. -Table 3.12 shows an example of the static network identity that the host owns. -The identity consists of: +Table 3.12 shows an example of the static network identity that the caller +chooses: `nvx` gives a cold boot `10.0.0.2/24` when its caller names no +address, and a restore keeps its snapshot's. The identity consists of: - An IPv4 prefix from `/1` through `/30`, whose first usable address is the gateway. @@ -118,7 +119,9 @@ independent of rule order: A rule matches only destinations of its own address family. The canonical policy also supports IPv4-only host allow lists, host block lists, and exact TCP endpoint allow lists; these deny all IPv6 traffic and do not combine with -generic rules. +generic rules. `nvx` lowers its caller's rules into generic rules exactly, at +most 256 for each action, and NVX's command-line interface compiles its +egress-policy files through the same lowering. Filtering is inactive only when the policy allows everything. Otherwise, packet checks enforce the bound guest MAC and IP identity and validate the @@ -151,7 +154,7 @@ Denying host loopback disables the gateway-to-loopback mappings, blocks loopback, unspecified, and link-local destinations, and disallows every forward. One exact IPv4 gateway TCP proxy endpoint may remain available. It is bound into the policy digest, admitted before the policy mode is evaluated, -and translated to the same port on host `127.0.0.1`, where the launcher +and translated to the same port on host `127.0.0.1`, where the caller supplies the proxy service. An outbound connection never makes guest services reachable from the host or @@ -264,13 +267,21 @@ connectivity that it never had. Native sockets, DNS requests, and flows are never serialized. Published guest listeners and explicit host forwards are process-local attachments, so capture rejects private-network ingress and explicit host-loopback access, which create them, and so does every restore -that keeps the captured policy. +that keeps the captured policy. `nvx` captures every NIC as a template, under +the complete deny-all policy, whatever its caller's policy, and installs each +restore's own policy with `--restore-network-policy`, at the captured address. A restored guest keeps its captured command line, which under a template's deny-all names no DNS server and asks for no listener announcements. A platform -clone takes both from its activation. The guest of a snapshot without a tier -takes them from the network settings of the restore packet, which carry the new -policy's DNS server and whether the guest announces its listeners ([Restore +clone takes both from its activation record, which `nvx` builds from the +restore's own policy, and whose network settings its init applies after the +restore's repair and before it reports Ready: it names the record's DNS server +in the guest's resolver and, when the record publishes the guest's listeners, +starts the announcer, which it owns from then on; see +[Activation and Ready](5.5-restore-protocol.md#559-activation-and-ready). The +guest of a snapshot without a tier takes them from the network settings of the +restore packet, which carry the new policy's DNS server and whether the guest +announces its listeners ([Restore Packet](5.5-restore-protocol.md#555-restore-packet)), and its repair applies them before it acknowledges the restore, which then gates host input ([Guest Repair](5.5-restore-protocol.md#557-guest-repair)). diff --git a/doc/design/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md index 4686ae5a2..5c945a00f 100644 --- a/doc/design/5.1-boot-protocol.md +++ b/doc/design/5.1-boot-protocol.md @@ -3,10 +3,11 @@ [Architecture documentation](README.md) > [5. VM Lifecycle and State](5-vm-lifecycle-and-state.md) -Cold boot consumes a host-selected kernel, optional initramfs, initial RAM -size, processor capacity, device attachments, and workload policy. The -machine profile validates these inputs and constructs the complete effective -configuration before opening guest execution. +Cold boot consumes a caller-selected kernel, optional initramfs, initial RAM +size, processor capacity, device attachments, and workload policy, which `nvx` +renders into OpenVMM's command line from the caller's runtime and VM +configuration. The machine profile validates these inputs and constructs the +complete effective configuration before opening guest execution. ## 5.1.1 Linux Direct Loader @@ -81,23 +82,39 @@ line carries no clock parameter: the guest reads its TSC and LAPIC rates from the [time ABI](3.2-clock-and-time.md#324-rates) MSRs, and `tsc_early_khz=` or `lapic_timer_hz=`, from the caller or a snapshot, is rejected. -Launchers add runtime tokens of their own: the +Two kinds of user tokens follow the base tokens. First come those of the boot +contract of `nvx`, which it renders from the caller's configuration: the [boot tokens of a layered root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), -one share token per host-share child, the number of host mappings for a -blockless managed root, and a request to create the workload account. A -launcher names the workload identity and the lifecycle of a blockless guest -with tokens of its own, `nvx_identity=UID:GID` and `nvx_mode=one-shot` or -`nvx_mode=managed`, which the guest's `/init` reads. It ignores the lifecycle -token that OpenVMM appends, and fails closed on OpenVMM's identity tokens: a -boot that has them stops with `NVX-WORKLOAD-ERROR` and status 125 before it -starts anything, unless `nvx_identity=` names the same UID and GID. In the -one-shot mode, `/init` runs the workload that `nvx_entrypoint=` and its -`nvx_arg=` tokens name once, as that identity, after the network and the -shares, and powers the VM off with the workload's status. Each `nvx_arg=` -token carries one argument, which cannot contain a space, since `/init` splits -the command line on spaces. A caller may also declare the RAM-backed overlay -layout, whose overlay upper the guest agents keep on a tmpfs, with exactly one -`nvx_overlay_upper=ramfs` token; any other form of that token is rejected. +the view's hostname and machine-ID mode, `nvx_hostname=` and +`nvx_machine_id=`, and `nvx_maps=`, the number of host mappings, when the VM +exports host paths. `nvx` reserves their names, and every `virtfs_` name, +which the caller's own tokens, appended after them unchanged, may not use. A +capture's command line holds none of them, since every restore keeps it +verbatim: there `nvx` admits only the invariant tokens of OpenVMM's platform +allowlist, with exactly one `nvx_sandbox=1`, which hands the guest to its init +for layered roots, and adds `maxcpus=` when fewer processors than the capacity +boot online. + +The caller's own tokens configure its guest. NVX's command-line interface +(CLI) passes `nvx_sandbox=1` for a layered root, and a share token for each +mapping of a blockless root's export, beside the tokens that its own user +gives it, such as a request to create the workload account, +`nvx_workload_account=create`. It names the workload identity and the +lifecycle with tokens of its own, `nvx_identity=UID:GID` and +`nvx_mode=one-shot` or `nvx_mode=managed`, which the guest's `/init` reads; +`nvx` asks OpenVMM for neither of the machine profile's identity and +lifecycle tokens. `/init` ignores the lifecycle token that OpenVMM appends, +and fails closed on OpenVMM's identity tokens: a boot that has them stops with +`NVX-WORKLOAD-ERROR` and status 125 before it starts anything, unless +`nvx_identity=` names the same UID and GID. In the one-shot mode, `/init` runs +the workload that `nvx_entrypoint=` and its `nvx_arg=` tokens name once, as +that identity, after the network and the shares, and powers the VM off with +the workload's status. Each `nvx_arg=` token carries one argument, which +cannot contain a space, since `/init` splits the command line on spaces. A +caller may also declare the RAM-backed overlay layout, whose overlay upper the +guest agents keep on a tmpfs, with exactly one `nvx_overlay_upper=ramfs` +token, which `nvx` renders for a layered root whose upper is in RAM; any other +form of that token is rejected. Embedded NULs are rejected, and the complete NUL-terminated command line must fit in the x86 guest kernel's `COMMAND_LINE_SIZE` of 2,048 bytes, which leaves @@ -127,5 +144,5 @@ A cold boot: A cold boot writes no TSC: every backend starts the processors' TSCs together. The kernel then discovers the fixed virtio-mmio devices and starts the guest agents, which step the guest clock to host UTC before readiness, -assemble the workload root, apply isolation, and start either the one-shot -workload or the managed control service. +build the VM's one space, and start either the one-shot workload or the +managed control service. diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index da35dd8b5..e04df7641 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -6,7 +6,8 @@ Readiness is an explicit lifecycle boundary, not the observation of arbitrary console text. A cold boot is usable when the guest agents have initialized the required devices and either prepared the one-shot workload or established the -managed control path. +managed control path, and a managed guest's init says so with +[Ready](#524-ready). ## 5.2.1 Control Frontends @@ -16,11 +17,12 @@ validation, snapshot boundaries, and management exclusion to both. Either frontend creates, captures, and restores machines, attaches their host resources, publishes readiness, and reports guest exit. Generic host-initiated save and pulse save-and-restore operations are unavailable for the microVM: -capture is always guest-requested. +capture is always guest-requested. `nvx` drives the command-line frontend: it +renders each launch's whole command line and starts OpenVMM with it. ## 5.2.2 Readiness Endpoint -An orchestrator may supply a single-use readiness endpoint for a restore: an +A caller may supply a single-use readiness endpoint for a restore: an existing Unix socket on Linux or named pipe on Windows. The VMM connects to it before the worker launches. When the restored VM first starts, the worker writes one fixed readiness record after every state unit has started and @@ -29,6 +31,14 @@ still gated at that point, and guest repair and its acknowledgement follow the event. A connection, write, or flush failure aborts startup, and the endpoint is never saved in a snapshot. +`nvx` gives every restore that it creates such an endpoint, which it binds +before OpenVMM starts: a socket private to the caller in the VM's directory on +Linux, or a fresh named pipe with an owner-only access control list on +Windows. It accepts the record only from the VM's held process, which it +verifies through the socket's original peer on Linux and the pipe client's +process ID on Windows, and only byte for byte as OpenVMM writes it: +`OPENVMM_RESTORE_READY_V1` and a newline. + ## 5.2.3 Management Exclusion While the worker holds the snapshot stop guard, its central dispatcher @@ -38,35 +48,92 @@ rather than queueing them. Snapshot quiesce, rollback, boundary release, and memory reads remain available. The exclusion applies at the worker boundary that every management frontend shares. -## 5.2.4 Managed Sandbox Lifecycle - -As Table 5.2 shows, host launchers present a persistent sandbox with five -operations. - -**Table 5.2.** Managed sandbox operations. +## 5.2.4 Ready + +A managed guest's init reports Ready on the control session once it can take +work: after a cold boot, once it has built its space, and after a restore, +once its activation has applied the restore's activation record, in answer to +the request that carries the record. Ready belongs to the protocol that the +caller speaks with its guest inside the control session, not to OpenVMM, whose +readiness record only says that the restored VM started. The host sends the +mapping table, and starts a workload, only after Ready. + +What a layered root's Ready reports about its activation is a public contract +of `nvx`, `ReadyActivation`, with a canonical encoding: nothing more for a cold +boot, and for a restore, the transaction ID of its activation record, which +the caller picks for each restore and keeps across the restore's retries, so +that Ready ties to the record of that restore. A restore that `nvx`'s +`RestoreMachine` orders ends at a Ready that echoes its transaction, and fails +at any other. NVX's own control protocol, NVXC, private to `nvx`, carries Ready +with its protocol version, which NVX's host checks. + +## 5.2.5 The Host's Mapping Table + +When a VM exports host paths, its host hands the guest a mapping table: the +ordered binds of the export, each with a source relative to the guest's mount +of the export, an absolute target in the guest, and its access, read-only or +read-write. `nvx` derives the table from the caller's export, in the order of +its binds, and encodes it in `MAPS` parts. The host announces how many +mappings the table holds before the guest is ready: at a cold boot with the +`nvx_maps=` boot token that `nvx` renders, and after a restore in the +activation record. Once the guest is ready, the host sends the table in parts +of consecutive entries, in the caller's protocol, and sends none of an empty +table. + +The guest keeps its account of the table in the public `MappingTable` of +`nvx`, which any init uses. It refuses as a whole, and changes nothing for, a +part that does not decode, holds no entry, does not continue the table, ends +past the announced count, or names the export's root itself. It hands the init +each other part's mappings in order. A mapping that fails latches it: the +mappings before it stay, and the table refuses every later part. It is +complete once every announced mapping is applied and none failed, and an init +starts no workload before then. Applying a mapping is the init's own: in NVX's +reference space, which nvx-init implements and which is private to `nvx`, a +mapper binds each mapping beneath the descriptors of the root's mounts, never +by path. The +[guest-agent guidelines](../guest-agent-guidelines.md#the-hosts-mapping-table) +give an init's part of the table's contract. + +## 5.2.6 Lifecycle Operations + +As Table 5.2 shows, `nvx` gives its caller five operations on a VM's +lifecycle, in the caller's private VM directory. + +**Table 5.2.** A VM's lifecycle operations in `nvx`. | Operation | Effect | | --- | --- | -| Provision | Records the sandbox's configuration, including its host shares | -| Start | Launches the VMM with a freshly generated control capability and the recorded shares' export, waits for guest readiness, and delivers the shares' mapping table | -| Exec | Runs one workload over the control channel | -| Stop | Shuts down the VM and releases its runtime resources | -| Deprovision | Removes the sandbox's state | - -Lifecycle transitions fail closed and take effect one at a time per sandbox, -while executions run beside them. The host identifies the running VMM process -of each sandbox, reconciles VMM processes with recorded state, and recovers -interrupted launches. Teardown leaves no VMM process, control endpoint, -capability, or runtime record, while supplied layers and scratch survive. A -sandbox's writes to its root therefore persist from one start to the next on -scratch, but not when its overlay's upper is in guest memory, which each start -begins empty. - -## 5.2.5 Authority +| Create | Checks a cold boot, a capture, or a restore, and creates the launch's log; for a restore, checks the snapshot, links its files, and binds the restore-ready endpoint. It starts no process | +| Spawn | Starts OpenVMM detached, with the capability on its standard input, and returns its verified process | +| Verify and reattach | Checks a saved VM's runtime identity and process, then connects to its verified control endpoint and authenticates with its saved capability | +| Stop | Waits for the VM to end after the caller's own request, then terminates only its verified process, and reports an exit only once it is confirmed | +| Finish a capture | Checks the outcome of an exited capture, and writes its snapshot's key and description | + +A caller may also take a created launch's complete command, with its opened +log and inherited resources, and start OpenVMM with a launcher of its own. +Everything around these operations is the caller's: where its VM directories +live and for how long, what it saves to find a VM again, how it serializes its +own operations on one VM, and its protocol with the guest, which also ends the +VM gracefully. + +NVX's command-line interface (CLI) is one such caller. Its `sandbox` command +keeps a persistent sandbox in a store of its own, with five operations of its +own, provision, start, exec, stop, and deprovision, which fail closed and take +effect one at a time per sandbox, under its store's lock, while executions run +beside them. It journals each launch before its spawn, reconciles VMM +processes with its records, and recovers interrupted launches. Teardown leaves +no VMM process, control endpoint, capability, or runtime record, while +supplied layers and scratch survive. A sandbox's writes to its root therefore +persist from one start to the next on scratch, but not when its overlay's +upper is in guest memory, which each start begins empty. + +## 5.2.7 Authority Runtime control is divided by authority: -- The host controller may request supported VM lifecycle transitions. +- The caller requests VM lifecycle transitions through `nvx`, which reaches + only the VM's verified process and keeps no lifecycle lock, so the caller + serializes its own transitions of one VM. - The VMM serializes transitions that affect machine state. - The guest agents own workload start, stop, supervision, and capture preparation. diff --git a/doc/design/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md index 53f2cc16d..8dc17f887 100644 --- a/doc/design/5.3-snapshot-protocol.md +++ b/doc/design/5.3-snapshot-protocol.md @@ -17,24 +17,23 @@ lists. This section describes the settings that configure a capture, the launch settings that prevent one, and the launch choices that limit later restores. -A capture requires up to three settings: +A capture requires up to two settings: - **Destination directory.** Required for every capture. The snapshot artifacts are saved in this directory, which must not already exist. - **Snapshot tier.** Required for a capture with sandbox blocks. The [snapshot tier](5.4-snapshot-artifact-and-attachment-model.md#544-tiers) - identifies the point in the sandbox's lifecycle at which the snapshot is - taken, from a platform snapshot that the whole fleet can clone to a - single-use checkpoint of one instance. It determines whether the snapshot is - cloned or resumed and whether it includes the scratch disk, and a capture - whose settings contradict it is rejected. -- **Storage generation.** Required for a capture whose snapshot tier binds - blocks. The storage owner supplies it to identify the exact block files, and - a restore must supply the same one; see - [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). + names the configuration of the snapshot's blocks: whether their identities + are bound, whether its scratch is fresh or paired, and whether its restores + clone it or resume it once. A capture whose settings contradict it is + rejected. -A capture also accepts three optional settings: +A capture also accepts four optional settings: +- **Block identity.** How a capture whose tier binds blocks identifies them: + by default, by the SHA-256 digest of each file, or, at the storage owner's + choice, by an immutable storage generation that the owner supplies; see + [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). - **Memory backing file.** The file that backs guest RAM. When none is supplied, the VMM creates one at launch in the parent of the destination directory and, where the filesystem allows, hard-links it into the @@ -53,12 +52,26 @@ A capture also accepts three optional settings: All of these settings are given when the virtual machine is created; see [Control Frontends](5.2-readiness-and-runtime-control.md#521-control-frontends). +`nvx` captures one configuration, from the capture settings of a cold boot's +configuration: the platform tier, with a sparse placeholder image and a +dormant scratch, of the geometries that the caller names, attached to a +blockless boot without an export; the processor capacity and the processors +online at boot, and the RAM at boot and its capacity, which the caller names; +the `openvmm/` directory in the caller's VM directory as the destination; and +the VM's NIC, if it has one, under the complete deny-all policy, whatever the +VM's own. It names no block identity or scratch restore mode, which unbound +layers and fresh scratch do not use, and no memory backing file, so OpenVMM +creates the RAM backing in the VM directory and, where the filesystem allows, +links it into the snapshot. The capture's context, which `vm::create` +prepares, is the caller's to keep until it finishes the capture; see +[Finishing a Capture](#538-finishing-a-capture). + The parent of the destination directory must already exist, because the VMM publishes the snapshot by creating a private staging directory there and renaming it. That filesystem needs room for the VM's RAM and any paired scratch. The snapshot contains guest memory, which can include secrets, so -access to the destination and any sharing of the snapshot are deployment -responsibilities; see +access to the destination and any sharing of the snapshot are the +responsibilities of the caller and its deployment; see [Images and Snapshot Artifacts](1.4-trust-boundaries.md#144-images-and-snapshot-artifacts). The following launch settings prevent a capture: @@ -78,9 +91,10 @@ The following launch choices limit what restores of the snapshot can do: capacity, only if fewer were online at capture; see [Processor Activation](5.5-restore-protocol.md#553-processor-activation). - **RAM.** A restore can add memory only up to a declared RAM capacity. -- **Host filesystem.** A restore can attach a new directory only if the slot - had no export at capture and the guest had not mounted it. Otherwise, every - restore needs the same export; see +- **Host filesystem.** A restore can attach a new directory, or, to a platform + snapshot, a new aggregate, only if the slot had no export at capture and the + guest had not mounted it. Otherwise, every restore needs the same export; + see [Dormant Slot and Saved-State Versions](3.6-host-filesystem.md#3610-dormant-slot-and-saved-state-versions). - **Network.** A restore cannot add a network device, so a snapshot captured without one restores without one. @@ -147,6 +161,22 @@ source, they remove their barriers and restore the detector settings, and a failure to do so powers off the guest. A rejected capture likewise thaws every guest-owned barrier and then releases the stall-detector suppression. +An init that captures through `nvx` calls `guest::snapshot::capture`, which +takes steps 1, 2, and 5 for the platform tier's fresh scratch, once it has +found nothing mounted from the `distro`, scratch, and host filesystem devices: +scratch is unmounted, as step 3 requires, and no host share has pages to write +back. One deadline of its caller's bounds every step before the request, whose +own wait has none while OpenVMM saves the VM. It never inspects or freezes the +init's executions, which the init keeps still itself: nvx-init refuses a +capture while an execution runs, or, when the host asks, kills the +executions' process groups and sweeps the guest's other processes. Nor does +`capture` power the VM off: it returns whether the guest continues or was +restored, once its repair completed, or a failure whose disposition says +whether the guest resumed or must end, with the status with which the init +then powers it off. NVX's guest snapshot tool, `nvx-snapshot`, takes all five +steps for the other captures of NVX's guests, a paired capture's freeze +included. + ## 5.3.3 Capture Transaction Figure 5.2 shows the capture transaction from the guest's request to @@ -231,10 +261,10 @@ The transaction has no deadline of its own. Gating host input in step 1 and the quiesce in step 4 wait for every device, and delivering the console output in step 3 waits for its endpoint to read it. A device that never finishes, or an endpoint that stops reading, stalls the capture, with nothing published and -the source never resumed, until the process that launched the VMM ends it; the +the source never resumed, until the caller that launched the VMM ends it; the VMM logs each state unit that has not answered after five seconds, so that the -log of such a capture names it. Every launcher therefore bounds a capture with -a deadline that ends the VMM. +log of such a capture names it. Every caller therefore bounds a capture with a +deadline that ends the VMM: `nvx` sets none. ## 5.3.4 State-Unit Ordering @@ -307,18 +337,39 @@ As Table 5.3 shows, a published snapshot never coexists with a running source. | Missing destination, rejected preflight, or failed input gate | Continues | None | | Failure after the quiesce, before the commit point, that rolls back | Continues | None | | Failure before the commit point that cannot roll back | Terminated | None | -| Input gate, console delivery, or quiesce that never completes | Ended by its launcher | None | +| Input gate, console delivery, or quiesce that never completes | Ended by its caller | None | After a commit, execution continues only in virtual machines restored from the snapshot, either as clones or as one resumed instance; see -[Snapshot Tiers](5.4-snapshot-artifact-and-attachment-model.md#544-tiers). +[Snapshot Tiers](5.4-snapshot-artifact-and-attachment-model.md#544-tiers). A +snapshot of `nvx` restores only as clones. ## 5.3.7 Guest and Host Barriers -Guest and host barriers have separate jobs. The workload cgroup is frozen in -captured guest state, the VMM gates external input, and a guest agent with -worker threads must park them at a capture-safe point. None of these -substitutes for the others. Device workers and the guest kernel must run -during a gated restore so that processors and memory can be repaired, so the -gate does not imply that interrupts stay disabled. After repair, the guest -agents acknowledge the VMM gate before thawing scratch and then the workload. +Guest and host barriers have separate jobs. The workload cgroup of a paired +capture is frozen in captured guest state, while an init that captures +through `nvx` keeps its executions still itself; the VMM gates external input; +and a guest agent with worker threads must park them at a capture-safe point. +None of these substitutes for the others. Device workers and the guest kernel +must run during a gated restore so that processors and memory can be +repaired, so the gate does not imply that interrupts stay disabled. After +repair, the guest agents acknowledge the VMM gate before thawing scratch and +then the workload. + +## 5.3.8 Finishing a Capture + +A published OpenVMM snapshot is not yet a snapshot of `nvx`. Once the caller +holds the proof that the capture's VM exited, `finish_capture` requires the +VM's outcome report to record a successful capture, checks OpenVMM's snapshot +strictly against the context with which `vm::create` prepared the capture, +derives the snapshot's key, and writes the key, `key.bin`, and an +informational description, `snapshot.json`, beside `openvmm/` in the VM +directory. The caller then moves those three entries into a private directory +of its own, the strict snapshot directory that +[Artifact Layout and Publication](5.4-snapshot-artifact-and-attachment-model.md#541-artifact-layout-and-publication) +describes, which it names, keeps immutable, and removes; the log, the outcome +report, and the VM's other files stay in the VM directory. A failure may leave +either file written, but moves nothing, so the caller can correct it and +finish again while the VM directory and the context are intact. NVX's +command-line interface stages the entries in a private directory and +publishes them under the snapshot's name with one non-replacing rename. diff --git a/doc/design/5.4-snapshot-artifact-and-attachment-model.md b/doc/design/5.4-snapshot-artifact-and-attachment-model.md index 5bc960e1d..9d54bc5e8 100644 --- a/doc/design/5.4-snapshot-artifact-and-attachment-model.md +++ b/doc/design/5.4-snapshot-artifact-and-attachment-model.md @@ -5,18 +5,22 @@ ## 5.4.1 Artifact Layout and Publication -Figure 5.4 shows the files of a published snapshot, and Figure 5.5 shows how -they are published. +Figure 5.4 shows the files of a snapshot that OpenVMM publishes, and the strict +snapshot directory in which `nvx` keeps OpenVMM's platform snapshot, and +Figure 5.5 shows how OpenVMM publishes its files. ```text -snapshot/ -|-- manifest.bin -|-- state.bin -|-- memory.bin -`-- scratch.img # paired scratch only +OpenVMM's snapshot The strict snapshot directory of nvx +snapshot/ / +|-- manifest.bin |-- key.bin # the snapshot key +|-- state.bin |-- snapshot.json # never read +|-- memory.bin `-- openvmm/ +`-- scratch.img # paired |-- manifest.bin + |-- memory.bin + `-- state.bin ``` -**Figure 5.4.** Snapshot directory layout. +**Figure 5.4.** Snapshot directory layouts. ```mermaid flowchart LR @@ -65,6 +69,23 @@ directory has been removed and its parent flushed; otherwise, the VMM terminates the source rather than resume it with a writable alias of the artifact. +The strict snapshot directory of `nvx` wraps OpenVMM's platform snapshot in an +envelope of two files: the snapshot's key, `key.bin`, which `nvx` writes when +it finishes the capture and checks against the manifest whenever it opens the +snapshot (see +[Snapshot Compatibility](1.5-compatibility-and-versioning-model.md#152-snapshot-compatibility)), +and an informational description, `snapshot.json`, which must exist but whose +contents `nvx` never reads, parses, or trusts. The directory holds exactly +these three entries, and `openvmm/` exactly OpenVMM's `manifest.bin`, +`memory.bin`, and `state.bin`, without paired scratch. `nvx` publishes no +snapshot directory itself: the caller names and places it, and moves a +finished capture's entries into it, as +[Finishing a Capture](5.3-snapshot-protocol.md#538-finishing-a-capture) +describes. A restore hard-links the three OpenVMM files into a new `openvmm/` +view in the restore's VM directory, from which OpenVMM restores, so the +snapshot directory and the VM directory share a filesystem; nothing is copied. +The view, like the rest of the VM directory, is the caller's to remove. + ## 5.4.2 Manifest The manifest is authoritative for: @@ -95,13 +116,14 @@ The manifest is authoritative for: Snapshot paths and repeated fields are bounded. Restore rejects truncated, oversized, malformed, wrong-type, path-escaping, symlinked, incompatible, missing, extra, or reordered state before guest execution. The manifest stores -no checksums, and the VMM never reads an artifact or a block to verify its -contents, so a same-length change to any of them is outside the validation -contract: integrity relies on host access control and on the storage owner -keeping each block's storage generation immutable, and authenticated export -or transport belongs outside the local artifact format. Paired scratch must -match the captured filesystem state, so restore checks it structurally and by -exact size and storage generation. +no checksums of the memory and state artifacts, and the VMM never reads them +to verify their contents, so a same-length change to either is outside the +validation contract: integrity relies on host access control, and +authenticated export or transport belongs outside the local artifact format. +A bound block and paired scratch are checked structurally and by exact size, +and by their identity: restore reads a file of SHA-256 identity to verify its +digest, and relies on the storage owner to keep a storage generation +immutable. Paired scratch must match the captured filesystem state. Restore opens the snapshot directory once and resolves every artifact, including the resume claim, relative to that directory handle. It holds the @@ -113,18 +135,33 @@ descriptors, opened without following symbolic links, and detect metadata changes before worker handoff, so pathname replacement has no effect. Content immutability beyond that depends on host access control. +`nvx` checks a strict snapshot directory once, when it opens the snapshot, +reading only the key and the manifest: both directories are private to the +current user and hold exactly their entries; every file is a regular file, +neither a link nor a reparse point, and private to its owner; the manifest, at +most `MAX_MANIFEST_LEN` bytes, is of OpenVMM's snapshot format version 2 and +of a platform capture of the configuration that `nvx` captures; the +artifacts' lengths are the manifest's; and the key holds its one canonical +encoding and agrees with the manifest. It then holds no handle or lock of the +snapshot, so its caller keeps the directory and its entries private and +immutable while it uses the snapshot, and, on Linux, serializes the restores +of one snapshot with a lock of its own; see +[The Library and Its Callers](1.4-trust-boundaries.md#146-the-library-and-its-callers). + ## 5.4.4 Tiers -As Table 5.4 shows, snapshots with sandbox blocks declare one of three -lifecycle tiers. +A snapshot's tier is a configuration of its devices, which OpenVMM records and +enforces: whether the identities of its blocks are bound, whether its scratch +is fresh or paired, and whether its restores clone it or resume it once. As +Table 5.4 shows, a capture with sandbox blocks declares one of three tiers. **Table 5.4.** Snapshot tiers. -| Tier | Restore policy | Scratch | Consumed configuration | Sharing scope | +| Tier | `distro` identity | Scratch | Restore policy | Consumed configuration | | --- | --- | --- | --- | --- | -| `platform` | Clone | Fresh | Invariants only | Trusted platform build | -| `workload-start` | Clone | Paired | Invariants, image binding, and sandbox | Tenant-scoped warm workload or runtime | -| `instance-checkpoint` | Resume | Paired | Invariants, image binding, and sandbox | One live instance with one claimed continuation | +| `platform` | Unbound | Fresh | Clone | Invariants only | +| `workload-start` | Bound | Paired | Clone | Invariants, image binding, and sandbox | +| `instance-checkpoint` | Bound | Paired | Resume | Invariants, image binding, and sandbox | Tier metadata requires sandbox blocks; blockless snapshots do not declare a tier. The saved host-owned tier token must agree with the manifest. Platform @@ -132,33 +169,47 @@ lower layers have unbound identities, whereas later tiers bind every consumed layer. The consumed-section set records which configuration the guest had already consumed at capture. -The platform point removes kernel and guest agent initialization from later -launches. A workload-start snapshot also removes runtime initialization, but -it contains workload memory and often cached image data and secrets. -Read-only layer files remain external attachments, yet bytes read from them -can appear in captured memory. Paired scratch is an artifact in the snapshot -directory, not a reference to the caller's original writable file. - -Sharing scope is a deployment requirement, not a tenant-authorization feature -of the VMM. The snapshot controller enforces tier combinations, layer -binding, scratch pairing, and the single-use resume claim. Host artifact -access control, template provenance, and a trustworthy pre-capture guest -determine whether cross-tenant publication is permissible. +A platform capture comes before its guest mounts its `distro` or its scratch: +each restore attaches an image of its own, with the captured block sizes and +at most the captured length, and a scratch of its own, with the captured +geometry, or declares the scratch absent. A capture of a later tier comes +after its guest mounted both, so its memory holds what the guest and its +workload read and wrote, often cached image data and secrets. Read-only layer +files remain external attachments, yet bytes read from them can appear in +captured memory. Paired scratch is an artifact in the snapshot directory, not +a reference to the caller's original writable file. + +`nvx` uses one tier, as the configuration of its captures' devices: the +platform tier, whose placeholder image and dormant scratch let each restore +bring its own layered root. It has no type for a tier, and neither captures +nor restores a snapshot of the paired tiers, which OpenVMM alone provides. + +What a snapshot's memory holds decides with whom it may be shared, and that +sharing scope is a requirement of the caller's deployment, not a +tenant-authorization feature of the VMM or of `nvx`. The snapshot controller +enforces tier combinations, layer binding, scratch pairing, and the +single-use resume claim. Host artifact access control, snapshot provenance, +and a trustworthy pre-capture guest determine whether cross-tenant +publication is permissible. ## 5.4.5 Snapshot Identity A snapshot binds each consumed block by role, access mode, exact length, and -block geometry, plus one caller-supplied, nonzero storage generation that -every bound block shares. Neither capture nor restore reads a block to -identify it: capture records the generation, and restore requires it and -checks the structural fields. The storage owner therefore vouches for the -bytes and must keep the generation immutable and bound to the exact files. -The generation is not a byte digest, and the VMM does not authenticate it. +block geometry, plus an identity: by default, the SHA-256 digest of its file, +which capture records and restore verifies by reading the file, or, when the +storage owner opts in, one caller-supplied, nonzero storage generation that +every bound block shares, which neither capture nor restore reads a block to +check. With a generation, the storage owner vouches for the bytes and must +keep the generation immutable and bound to the exact files: the generation is +not a byte digest, and the VMM does not authenticate it. A platform snapshot +binds no block: a restore's blocks need only fit its recorded geometry. A sealed snapshot is immutable. A clone creates a separate writable descendant, while a resume atomically claims the snapshot and transfers its continuation into a live instance, after which it is no longer an available -snapshot. There is no mutable-snapshot state. +snapshot. There is no mutable-snapshot state. The key of a snapshot of `nvx` +identifies its compatibility class, not its bytes; see +[Snapshot Compatibility](1.5-compatibility-and-versioning-model.md#152-snapshot-compatibility). ## 5.4.6 Replay Contract @@ -175,7 +226,10 @@ neither is transparent checkpointing of arbitrary requests. A workload whose cgroup the capture does not freeze, such as one that requests a blockless capture itself, must likewise hold every process that writes to a shared mapping of a host share from its request until the capture returns, because -the guest agents write back only the pages dirtied before the request. +the guest agents write back only the pages dirtied before the request. The +[guest-agent guidelines](../guest-agent-guidelines.md#replay) give an init's +part of the contract: nvx-init captures before it runs any workload, so a +restore of its snapshots resumes no workload's state. ## 5.4.7 Host Attachments @@ -190,20 +244,22 @@ declarative reconstruction policy. | Byte channel | Pending RX and TX bytes | Host endpoint, a fresh generation ID, and an optional restore packet | | Boot console | Queue progress, staged RX, partial TX, endpoint identity, and reconnect policy | An eligible listener at its captured endpoint or an approved fresh endpoint of the same kind, a required client connection, or a supplied inherited terminal | | Control console | Attachment identity, broker policy, and bounded broker state without queued records, host output, or receive credit; never the capability or peer identity | An approved same-user local endpoint, possibly at a fresh location, with a fresh capability; the broker restarts with a fresh instance and epoch | -| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy; published guest listeners and host forwards are rejected | -| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export | -| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry | +| Network | Static identity, queue and packet progress, network profile, and policy digest | A fresh in-process endpoint with the matching policy, which rejects published guest listeners and host forwards, or, on a NIC template, the restore's own policy, with the listeners and forwards that it publishes | +| Host filesystem | FUSE namespace, handles, cookies, root and object identity, access mode, and path policy, or explicit dormant state | A fresh export with the same canonical path, target, mode, and policy; a dormant slot may stay unattached or bind a new directory export, or, in a platform snapshot, a new aggregate export | +| Sandbox block | Queue and device state, roles, access, geometry, layer identities, scratch policy, and paired-scratch materialization | Matching read-only layers, or, for a platform snapshot's unbound `distro`, an image with its block sizes and at most its length; plus private-copy or copy-on-write scratch for clones, direct scratch after an exclusive resume claim, or a new scratch file of the same geometry, which a platform restore may instead declare absent | | Readiness endpoint | Nothing | An optional single-use endpoint for one restore | Attachment resolution happens before any processor starts. A supplied attachment must reproduce the saved identity, except that an eligible listener may replace only its endpoint while keeping its stable ID, attachment and backend kinds, reconnect policy, required flag, length, and timeout. -Missing privileges, endpoint binding failures, a changed egress policy, +Missing privileges, endpoint binding failures, an egress policy that differs +from the captured one, unless a NIC template's restore installs its own, replaced filesystem objects, or a wrong attachment kind fail restore -explicitly. Orchestrators of reusable clones supply fresh private boot-console -and control endpoints, so that independent or concurrent restores never -collide with the terminated source. +explicitly. Callers of reusable clones supply fresh private boot-console and +control endpoints, so that independent or concurrent restores never collide +with the terminated source: `nvx` gives each restore the endpoints of its own +VM directory on Linux, and fresh named pipes on Windows. ## 5.4.8 Memory Sharing diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index 7e4046405..7d779713d 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -7,7 +7,9 @@ Restore treats the manifest as the source of machine configuration. A failure of validation or reconstruction rejects the restore before a restored processor runs. Guest repair runs afterward on the restored processors, and a failure there powers the guest off, as -[Guest Repair](#557-guest-repair) describes. +[Guest Repair](#557-guest-repair) describes. OpenVMM performs the restore; +`nvx` checks it beforehand, and orders the guest's activation afterward, as +[Activation and Ready](#559-activation-and-ready) describes. ## 5.5.1 Restore Sequence @@ -122,6 +124,11 @@ checkpoint reusable. The single-use rule prevents accidental forks; it does not replace artifact confidentiality, retention policy, or protection against host-side copying. +A platform snapshot, the only kind that `nvx` restores, has no paired +scratch: each restore attaches a new scratch file of the captured geometry, +or, for a guest that keeps its upper in RAM, declares the scratch absent, and +OpenVMM attaches the scratch device without a file. + ## 5.5.3 Processor Activation Restore-time processor activation separates the immutable processor capacity @@ -308,6 +315,17 @@ Until the acknowledgement, a gated restore keeps host input gated, so no host input reaches a guest whose repair failed, and the VMM stops a guest that does not acknowledge within the restore's gate timeout. +For a platform restore, `guest::snapshot::capture` performs this repair in +the init's own process before it returns the restored guest: steps 1 through +3, the processor and memory activation of step 4, the reseed of step 5, and +the acknowledgement of step 6, after which the time daemon finishes step 7. It +refuses a packet that carries network settings, which a platform restore never +sends: the restore's activation record carries its network instead. It powers +nothing off either: it returns a repair failure as a disposition that the +guest must end, with the status of the failure's class, from 40 through 45, or +the time ABI's, and the init powers the guest off with that status; see +[Final Output and Exit Status](5.6-shutdown-and-failure-semantics.md#561-final-output-and-exit-status). + Validation and resource reconstruction may create process-local objects, but they never expose a partially restored guest: no virtual processor runs after either fails. @@ -329,8 +347,11 @@ To prevent this, every restore delivers fresh entropy, which a clone of a 3. For a clone, the guest agents credit all 64 bytes to the kernel's entropy pool and force the kernel's random-number generator to reseed, so all later kernel randomness depends on the fresh bytes. -4. They refresh the workload's machine ID, which they derive from the fresh - bytes, and its UTS namespace. +4. They refresh the guest's identity: NVX's guest snapshot tool derives a new + machine ID from the fresh bytes and refreshes the hostname, and the + workload's UTS namespace, while an init that activates a platform clone + applies its activation record's view, a hostname and, when the record asks + for one, a machine ID that it generates only after the reseed. 5. The workload runtime's post-restore hook resets the runtime's own random-number state, which a kernel reseed cannot reach. 6. The guest agents acknowledge the restore and only then thaw scratch and @@ -363,3 +384,54 @@ run, and gating host input does not hold it back. The snapshot's caller must therefore provide the execution barrier, for example by capturing before any workload starts, and, where the guest does not reseed, repair the kernel's and the runtime's random state with the entropy handed to it. + +## 5.5.9 Activation and Ready + +A restore of `nvx` resumes the captured guest of a platform snapshot before +the guest has a root: its init has built no space and run no workload, and +the restore's activation record gives it the settings of this launch. The +caller, `nvx`, OpenVMM, and the guest's init take these steps, in this order: + +1. The caller opens the snapshot, and `vm::create` checks the restore with + `Snapshot::accepts` at the host's current time, links the snapshot's + OpenVMM files into a new `openvmm/` view in the restore's VM directory, + renders the restore's own network policy, with `--restore-network-policy`, + at the captured NIC's address, and binds the restore-ready endpoint, before + anything starts. From the same configuration and a transaction ID that the + caller picks, `nvx` builds the restore's activation record: its image, + upper, export, view, network settings, and the processors and memory that + the guest must have. +2. OpenVMM restores the machine, as + [Restore Sequence](#551-restore-sequence) describes. It installs the + restore's network policy, with its forwards, before any vCPU runs, and + writes its readiness record, which `nvx` accepts only from the VM's held + process. The packet of a platform restore carries no network settings. +3. The guest's init repairs the restore with `guest::snapshot::capture`, as + [Guest Repair](#557-guest-repair) describes, takes the control session's + reset, and acknowledges the session's new instance. Meanwhile, the caller + connects a control session to the VM's verified endpoint. +4. The caller sends the activation record, the same canonical bytes on every + session, in its own protocol. The init's `Activation` accepts only a + restore's record that expects the processors and memory that the repair + reports, before any side effect, and the init applies it with this + restore's own network policy: its space, whose view names the record's DNS + server; the runtime file systems; the announcer of the guest's listeners, + which it starts when the record publishes them and owns from then on; and + its one switch into the root. +5. The init reports Ready with the record's transaction ID, and the caller's + `RestoreMachine` ends the restore at that Ready. +6. Only then does the caller send the host's mapping table and start a + workload, and the init starts none before its `MappingTable` is complete. + +`RestoreMachine` orders these steps without I/O, a clock, or a retry budget: +the caller performs each step's I/O, chooses its deadlines, and reports what +it observes, including a lost session, after which the machine has it connect +again and send the same record. A failure, a `RestoreError`, stops no process +and writes no record; see +[Failure Classes](5.6-shutdown-and-failure-semantics.md#562-failure-classes). +NVX's command-line interface drives the machine with a blocking driver, +private to `nvx`, which allows a restore 60 seconds to Ready over at most +three sessions, then 30 seconds more for the mapping table and the workload's +start. The +[guest-agent guidelines](../guest-agent-guidelines.md#complete-a-restore-before-ready) +give the init's part of this order. diff --git a/doc/design/5.6-shutdown-and-failure-semantics.md b/doc/design/5.6-shutdown-and-failure-semantics.md index 19a497d67..7372b59b6 100644 --- a/doc/design/5.6-shutdown-and-failure-semantics.md +++ b/doc/design/5.6-shutdown-and-failure-semantics.md @@ -21,7 +21,10 @@ streams, not every console endpoint. Under the management RPC, a zero guest status completes a pending wait and stops the server cleanly, while a nonzero status or a failed drain fails the wait and ends the server with an error. After teardown, the VMM can write the bounded -[outcome report](1.4-trust-boundaries.md#145-host-facing-outputs). +[outcome report](1.4-trust-boundaries.md#145-host-facing-outputs), which `nvx` +reads for its caller: `vm::stop` returns that read beside its confirmation of +the VM's exit, so that a report that cannot be read never hides a confirmed +exit. As Table 5.9 shows, the guest reports its own fail-fast conditions through exit statuses, each after a violation event or an error line that names the @@ -35,18 +38,28 @@ cause. | 194 | The violation watcher detected a runtime clock violation | | 195 | The time ABI's part of post-restore repair failed | | 125 | Sandbox setup, such as share placement, failed before the workload started | -| 1 | Another snapshot or restore step of the guest agents failed, such as processor activation or the identity refresh | +| 1 | Another snapshot or restore step of NVX's guest snapshot tool failed, such as processor activation or the identity refresh | +| 40 to 45 | Another step of a platform clone's repair failed, one status for each class: the restore itself, the processor target, the memory target, the memory's activation, the reseed, and the restore's finish | +| 50 to 58 | A platform clone's activation failed, one status for each class: its deadline, the host's session, the record, the resources that the record expects, the root, the workload, the host export, the network settings, and the init's own order | A status alone does not identify the cause. In a one-shot lifecycle, the guest's exit status is otherwise the workload's, which can take any 8-bit value, and status 1 is also the guest's generic failure status. A host therefore identifies a fail-fast condition by its status together with the violation event or error line that precedes it. Statuses in the control -protocol's exit reports and errors, such as 125 for an exceeded output limit, -travel with their own categories instead, as +protocol's exit reports and errors, such as 124 for an execution that timed +out, travel with their own categories instead, as [Host-Guest Control Interface](4.4-host-guest-control-interface.md#441-operations) describes. +The statuses of a clone's repair and activation, and the time ABI's, are +public contracts of `nvx`, in its `protocol::status`. `guest::snapshot::capture` +returns a repair's as the status with which the guest must end, and an init +ends a failed activation with the status of its class. When a restore ends +before Ready, `RestoreMachine` attributes it by the status in the outcome +report that its caller reads: to the guest's repair or time ABI, to the +restore's request, or to nothing that the evidence establishes. + ## 5.6.2 Failure Classes Failures are classified by whether execution can safely continue: @@ -68,3 +81,14 @@ This fail-closed rule prevents success-shaped recovery from uncertain state. The architecture prefers terminating one process generation over resuming a guest whose device ownership, external input, persisted artifacts, or execution boundary can no longer be established. + +`nvx` fails closed the same way, and reports each refusal or failure with an +error whose kind its caller acts on, `ErrorKind`: for example, `Unsupported` +for a configuration or a restore that it refuses, `Unavailable` for a +snapshot whose time contract the host cannot meet now, `Exited` for a VM that +ended, `StillRunning` for an exit that it could not confirm, and +`RuntimeMismatch` for a VM of another runtime identity. A failed restore, a +`RestoreError`, keeps its phase, its evidence, what that evidence attributes +it to, and whether guest code may have run, and neither stops the VM nor +records anything: the caller decides whether to stop the VM, retry, or set the +snapshot aside.