diff --git a/doc/design/4-guest-runtime-architecture.md b/doc/design/4-guest-runtime-architecture.md index 9e741c1b1..7a272347b 100644 --- a/doc/design/4-guest-runtime-architecture.md +++ b/doc/design/4-guest-runtime-architecture.md @@ -2,25 +2,39 @@ [Architecture documentation](README.md) -The NVX guest runtime turns one microVM into one workload sandbox. The guest -agents supervise the workload for the life of the VM. In the +The NVX guest runtime turns one microVM into one workload sandbox, whose +isolation and resource boundary is the VM itself. For the life of the VM, the +guest's init, PID 1, runs the workload's executions as plain processes in one +root that the whole VM shares. In the [layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), -they remain outside the workload's namespaces; the -[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) -creates none and contains the workload with its credentials and a cgroup. The +nvx-init assembles that root, sets one VM-wide hostname, and switches into the +root once; in the +[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root), +the managed agent runs the executions in the guest's initramfs root. Neither +creates a namespace or a cgroup for an execution: each execution takes the +VM's workload credentials and runs in a process group of its own. The architecture does not introduce pod infrastructure, a pause container, or a guest-side service for creating arbitrary additional containers. +nvx-init's space is NVX's internal reference code that nvx-init implements, +not public `nvx` API. `nvx` keeps the contracts that a space reads, such as +the public `MappingTable`, the account of the host's mapping table, whose +ordered mappings the reference space's mapper binds. The +[guest-agent guidelines](../guest-agent-guidelines.md) give an init's +contract, with nvx-init as the worked example. + This chapter describes how the guest turns the virtual machine into an isolated workload sandbox. ## Table of Contents - [4.1 Guest Filesystem and Image Model](4.1-guest-filesystem-and-image-model.md): - Explains image layers, the writable upper, overlay assembly, and live shares. + Explains the image, the writable upper, the root's assembly, live shares, + and the host mapping table. - [4.2 Guest Agents and Workload Model](4.2-guest-agents-and-workload-model.md): - Defines the relationship between the guest agents and the workload. + Defines the init, its executions, and its helpers. - [4.3 Isolation and Resource Model](4.3-isolation-and-resource-model.md): - Describes identities, namespaces, capabilities, devices, and cgroups. + Describes credentials, the VM-wide space, priority, and the one limit, the + RAM upper's cap. - [4.4 Host-Guest Control Interface](4.4-host-guest-control-interface.md): Defines authenticated runtime operations and their session semantics. diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index 7d2b67e58..b178dc440 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -3,47 +3,56 @@ [Architecture documentation](README.md) > [4. Guest Runtime Architecture](4-guest-runtime-architecture.md) -As Table 4.1 shows, the guest agents run workloads in one of two roots. +As Table 4.1 shows, the guest runs its workloads in one of two roots, one for +the whole VM. **Table 4.1.** Workload roots. | Root | Construction | Host content | | --- | --- | --- | -| Layered sandbox root | An overlay of one read-only image layer and a private writable upper, on scratch or in guest memory | Live host shares bound at declared targets | -| Blockless managed root | The guest's own initramfs root, without block devices | Host directories bound by a host mapping table | +| Layered sandbox root | nvx-init's overlay of the read-only layers of one image, an EROFS file system or a GPT disk, under a private writable upper, on scratch or in guest memory | Live host shares that the boot names, or the mappings of a host mapping table | +| Blockless managed root | The guest's own initramfs root, without block devices | Live host shares that the boot binds, or the mappings of a host mapping table | -The guest kernel provides the facilities both roots need: virtio-blk, -virtio-fs, EROFS with compression and extended attributes, tmpfs with -extended attributes, overlayfs, cgroup v2 with the memory and process -controllers, namespaces, BPF system calls, and cgroup BPF programs. +The guest kernel provides the facilities that the roots need: virtio-blk with +GPT partitions, virtio-fs, EROFS with compression and extended attributes, +ext4, tmpfs with extended attributes, and overlayfs. In neither root does an +execution get a cgroup or a namespace of its own; see +[Isolation and Resource Model](4.3-isolation-and-resource-model.md). ## 4.1.1 Layered Sandbox Root -As Figure 4.1 shows, the layered root combines one immutable EROFS `distro` -layer with one private writable upper: either an ext4 scratch device or a -tmpfs in guest memory. +A layered root's guest runs nvx-init, which builds the root as the VM's one +space. The space is NVX's internal reference code that nvx-init implements, +not public `nvx` API; the guest-agent guidelines' +[Build the space](../guest-agent-guidelines.md#build-the-space) gives its +rules. As Figure 4.1 shows, the root combines the read-only layers of one +immutable `distro` image with one private writable upper: either an ext4 +scratch device or a tmpfs in guest memory. ```text -/run/nvx/layers/distro (EROFS) ----------> lower directory -/run/nvx/scratch/upper (ext4) -----------> upper directory +/run/nvx/layers/distro (EROFS image) -------> lower directory + or /run/nvx/layers/pN (ext4 layer in partition N of a GPT disk, + from partition 2 on) --> lower directories +/run/nvx/scratch/upper (ext4) --------------> upper directory or /run/nvx/tmpfs/upper (tmpfs) -/run/nvx/scratch/work (same ext4) ------> work directory +/run/nvx/scratch/work (same ext4) ---------> work directory or /run/nvx/tmpfs/work (same tmpfs) - overlay --> /run/nvx/rootfs -one-shot: virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET - or an aggregate ---> /run/nvx/shares, each child bound at - /run/nvx/rootfs/TARGET -managed: the aggregate export ---> /run/nvx/hostfs/root, each mapping of the - table mounted at /run/nvx/rootfs/TARGET + overlay --> /run/nvx/rootfs, moved onto / at the switch +live share: one directory ---> /run/nvx/rootfs/TARGET + or an aggregate ---> /run/nvx/shares, each child bound at + /run/nvx/rootfs/TARGET +host export: an aggregate ---> /run/nvx/hostfs/root, each mapping of the + table mounted at TARGET in the root ``` **Figure 4.1.** Layered sandbox root assembly. -The `distro` layer is required and holds a curated base operating system. One -immutable layer file can serve many sandboxes, and compressed EROFS reduces -stored and cached image bytes, because the guest kernel decompresses blocks as -it reads them. No FUSE daemon or in-guest fetch service is involved in the root -filesystem. +The `distro` image is required and holds a curated base operating system: an +EROFS file system, the overlay's one lower layer, or a GPT disk, whose +partitions after the first each hold an ext4 layer. One immutable image file +can serve many sandboxes, and compressed EROFS reduces stored and cached image +bytes, because the guest kernel decompresses blocks as it reads them. No FUSE +daemon or in-guest fetch service is involved in the root filesystem. The launcher describes the root to the guest in two boot tokens, and in up to four more for its settings, whose grammar one contract module defines, with the @@ -70,62 +79,93 @@ Guests refuse a token without a value, a repeated token, both uppers, an image without an upper or an upper or a setting without an image, a setting of another upper or of another format, a RAM upper beside a `scratch` block device, and every other form, including the earlier `nvx_layer` form without a -format, rather than probe the devices. The guest agents described here -assemble only an EROFS image, over either upper, for trusted lowers, and don't -check a scratch for a journal: they refuse a GPT image, untrusted lowers, and -a forbidden journal with a message that says so. - -The guest agents assemble the root in this order: - -1. Mount runtime tmpfs and cgroup v2, create sibling guest-agent and - container cgroups, and move the guest agents into the guest-agent cgroup. -2. Read the root's boot tokens, and resolve each role's MMIO address through - the kernel's platform-device resources rather than assuming a device name - or enumeration order. For a RAM upper, refuse any device at the `scratch` - role's address. -3. Check any expected sector count, flush cached block-device buffers, mount - the `distro` layer read-only as EROFS, and check its filesystem UUID when - the token names one. -4. Mount the upper's file system, and create the overlay's upper and work - directories on it: resolve and flush scratch and mount it as ext4 with - `rw,nosuid,nodev`, or mount a tmpfs with `mode=0755,nosuid,nodev`, and +format, rather than probe the devices. nvx-init assembles every form that the +tokens name: either image format, over either upper, with either overlay mode, +and it checks a scratch for a journal when the tokens forbid one. + +nvx-init builds the root in this order: + +1. The early boot, the shell `/init`, mounts the kernel's file systems and a + tmpfs at `/run`, starts the time daemon, configures the network, and starts + the announcer when the network publishes the workload's listeners. It then + replaces itself with nvx-init, which takes the helpers over as PID 1's + children. +2. nvx-init reads the boot tokens of the root, of the view, and of its own + settings, such as the workload identity and the control terminal, and + refuses an incomplete or conflicting set before it mounts anything. +3. It finds each role's block device by its MMIO address, through the + kernel's platform-device resources rather than a device name or an + enumeration order, checks that the `distro` device is read-only and the + scratch writable, and flushes their cached buffers. It checks the image + before it mounts anything: an EROFS image's filesystem UUID, when the token + names one, or a GPT disk's partition table, its disk GUID, when the token + names one, and each layer's ext4 superblock, which must not need a journal + recovery. When the tokens forbid one, it also checks that the scratch's + ext4 file system has no journal. +4. It mounts the image's layers read-only, then the upper, on which it creates + the overlay's upper and work directories: the scratch's ext4 file system + with `rw,nosuid,nodev`, or a tmpfs with `mode=0755,nosuid,nodev`, and `size=N%` when `nvx_ram_upper_percent=N` caps it, or else at the kernel's default size, half of the guest's RAM. -5. Mount the overlay with the `distro` layer as its lower directory, - metadata-only copy-up, and extended inode numbers. -6. Verify the workload identity in the assembled root, mount each live share - of a one-shot sandbox at its target inside that root, or the host's export - of a managed sandbox, and write the workload's machine ID to a runtime - file. -7. Start the workload behind a barrier and place it in the workload cgroup - before releasing it; a managed sandbox's agent first maps the shares from - the export, as their mapping table orders. -8. Wait for the workload, unmount the shares in reverse order and then the - overlay, the `distro` layer, and the upper, and report the workload's - status. +5. It mounts the overlay with `nosuid,nodev`, the layers as its lower + directories, and the option set of its mode: metadata-only copy-up and + extended inode numbers for trusted lowers, or, for untrusted lowers, a set + under which the overlay neither creates nor follows metacopy and redirect + attributes. It then checks that the kernel applied each option, rather than + accept an overlay that the kernel mounted read-only or without one. +6. It mounts the host export, when the boot names one, out of the root, below + the runtime directory, where only the guest's root user reaches it. +7. It makes every mount of the root's tree `nosuid` and `nodev`, and mounts + the root's runtime file systems: `/proc`; a read-only `/sys`; a `/dev` that + holds only the null, zero, full, random, urandom, and tty devices, with + `/dev/pts` and `/dev/shm`; and `/run`, with the guest's time status bound + read-only at `/run/nvx/time`. The boot's live shares follow, as 4.1.2 + describes, since the `/run` tmpfs would hide one beneath `/run`. +8. It opens the descriptors of the root and of its `/run` mount, beneath which + the mapper binds the host's mappings, resolves the workload's account in + the root's user and group databases, creating it when the boot asks, opens + its control terminal, and applies the VM-wide view: it sets the hostname, + keeps the image's machine ID or binds a generated one over it, and binds a + resolver file that names the boot's DNS server, if any, over the image's. +9. It switches into the root, once: it moves the root's mount onto `/`, then + enters it with `chroot` and `chdir`. From then on, nothing unmounts the + root, and the VM ends by powering off. + +Until the switch, a step that fails starts no workload: nvx-init ends the VM, +whatever the root's attempt to unmount what it mounted leaves mounted. After +the switch, nvx-init serves its control session: it applies the host's mapping +table as the host sends it, and runs executions once the table is complete. + +A restore of a platform capture builds the same root before Ready, from its +activation record rather than from boot tokens: nvx-init assembles the +record's image over a fresh upper, refusing an `upper` or `work` directory +that holds anything, mounts the host export, if any, applies the record's +view, mounts the runtime file systems, and switches; see +[Complete a restore before Ready](../guest-agent-guidelines.md#complete-a-restore-before-ready). The UUID check detects attachment mistakes; it is not a content-integrity -proof, and neither the guest nor the VMM verifies layer contents. -Metadata-only copy-up interprets overlay metadata stored in the `distro` -layer, so that layer is a trusted input that must not carry untrusted overlay -attributes. A metadata-only change avoids copying file contents, but a data -write still copies the whole lower file into the upper, so the upper's -capacity must account for copy-up. The two uppers spend different resources: +proof, and neither the guest nor the VMM verifies layer contents. With +trusted lowers, metadata-only copy-up interprets overlay metadata stored in +the image's layers, so they are trusted inputs that must not carry untrusted +overlay attributes; untrusted lowers copy a file's data up with its metadata +instead. A metadata-only change avoids copying file contents, but a data write +still copies the whole lower file into the upper, so the upper's capacity must +account for copy-up. The two uppers spend different resources: - Scratch consumes host storage rather than guest RAM, can run out of space independently of guest memory pressure, and keeps a managed sandbox's changes from one start to the next. -- A RAM upper consumes guest memory. The memory cgroup of the process that - writes a page, normally the workload's, is charged for it, so the - workload's memory limit also bounds its files, and since the guest has no - swap, those pages stay resident until their files are removed. A write - beyond the tmpfs size fails with `ENOSPC`, and nothing the workload writes - outlives the virtual machine. - -The initramfs remains the guest agents' root; it is not another workload -layer. A RAM upper uses the machine's RAM-backed overlay layout, which has -exactly one read-only `distro` block and no scratch, and which supports -neither capture nor restore; see +- A RAM upper consumes guest memory, and its cap, the one limit that nvx-init + sets within the VM, bounds the files that the upper holds, as + [Priority and Resources](4.3-isolation-and-resource-model.md#433-priority-and-resources) + describes. The guest has no swap, so those pages stay resident until their + files are removed. A write beyond the cap fails with `ENOSPC`, and nothing + that the upper holds outlives the virtual machine. + +The initramfs is not a layer of the root: it stays the root of the helpers, +which run from it, while nvx-init leaves it at its switch. A cold boot with a +RAM upper takes the machine's RAM-backed overlay layout, which has exactly one +read-only `distro` block and no scratch; see [Storage Configuration](2.1-machine-model-and-configuration.md#213-storage-configuration). ## 4.1.2 Live Host Shares @@ -135,54 +175,55 @@ cache, or a single host file, such as a settings file, to the workload without copying it into an image: edits are visible on both sides without staging or copy-back. -One shared directory attaches as a directory export. Several shares, or any -shared file, attach as an aggregate whose children are independent volumes -with their own access modes and policies; see -[Host Filesystem](3.6-host-filesystem.md). Each child is named -after its index and a digest of its guest target, so a snapshot pins every -target, and a restore that requests another target fails before the guest -runs. Before launch, the launcher rejects policy paths outside their share or -on a shared file, guest targets or host paths that equal or contain one -another, and one-shot shares whose boot tokens exceed the command-line budget. - -A one-shot sandbox receives its shares at boot. The machine profile announces -the export through its host filesystem tokens, and the launcher adds one share -token per child that names the child, its guest target, its mode, and whether -it is a file. The guest agents then: - -- Parse every share token and create every target before mounting anything, - so a malformed bootstrap mounts nothing. -- Create each target inside the workload root one component at a time and - refuse a path that crosses a symbolic link, so an image layer cannot - redirect a share outside the root. -- Create a regular file as the last component of a file target and a - directory for any other target. -- Refuse repeated children, overlapping targets, and targets that the - runtime later binds over, such as the machine ID file. -- Mount a single share at its target with its mode and `nosuid,nodev`. -- Mount an aggregate at a root-only location outside the workload root and - bind each child at its target with its mode and `nosuid,nodev`. - -The workload's mount namespace inherits these mounts, and teardown unmounts -the binds in reverse order before the aggregate. Any refusal or mount failure -aborts the sandbox instead of starting the workload without its shares. +One shared directory can attach to OpenVMM's host filesystem device as a +directory export. Several shares, or any shared file, attach as an aggregate +whose children are independent volumes with their own access modes and +policies; see [Host Filesystem](3.6-host-filesystem.md). NVX's CLI attaches +its shares through one aggregate, the host export, whose children, named by +their index, are the outermost shared directories and the shared files +outside them. Before launch, its `run` and `sandbox` commands refuse guest +targets that overlap one another or that the guest's root reserves. + +A boot that names its shares in its tokens, with one share token per bind, +which names its source, a child or, in a blockless root, a path in one, its +guest target, its mode, and whether it is a file, has them in place before any +workload runs: + +- In a blockless root, the guest's `/init` mounts the device where the host + filesystem's tokens name it, with their mode, and binds each token's source + at its target with its mode, `nosuid`, and `nodev`, creating a missing + directory, or a regular file for a file, whose target must not be a symbolic + link. +- In a layered root, nvx-init mounts them once the root's runtime file systems + are mounted. Before it mounts anything for them, the root checks every + target: an absolute, canonical path, none that the root's runtime reserves, + such as `/proc`, `/etc/machine-id`, or `/run/nvx/time`, and none that + equals, holds, or lies inside another. It mounts the device out of the root, + below the runtime directory, and its mapper then moves a directory export + onto its target, or binds each child of an aggregate at its own, beneath the + overlay's descriptor, through no symbolic link, creating what a target lacks + only on the root's own mounts, with the share's mode, `nosuid`, and `nodev` + set before the mount is visible. + +Any refusal or mount failure ends the boot instead of starting a workload +without its shares. A managed sandbox records its shares when it is provisioned, and on every start receives them as the blockless root receives its host directories (see 4.1.3): the host exports every share as a numbered child of one aggregate, -which the guest agents mount at the same root-only location, and delivers the -mapping table once the managed agent is ready. The agent maps each entry -beneath the assembled root, whose own mount is the overlay, and refuses the -targets that the container runtime keeps, as the boot tokens' targets are -refused. A mapping that fails ends the start, after the guest shuts down as -stop has it do: it unmounts the mappings in reverse order before the export -and the overlay. +which nvx-init mounts out of its root, and delivers the mapping table once +nvx-init is ready. nvx-init maps each entry beneath its root, whose own mount +is the overlay, and refuses the targets that the root's runtime reserves, as +it refuses those of a boot's live shares. A mapping that fails ends the start: +the host stops the VM, which nvx-init powers off without unmounting its root. ## 4.1.3 Blockless Managed Root -Without image layers, each workload runs as a plain process directly in the -guest's initramfs root. Host directories reach it through a host mapping table -rather than an image: +Without image layers, the managed agent, PID 1, runs each execution as a plain +process directly in the guest's initramfs root, which it never leaves. Host +directories and files reach it as the live shares that its boot binds, or +through a host mapping table, as they reach a managed sandbox, rather than +through an image: - The host exports the outermost mapped directories, and each mapped file outside them on its own, as numbered children of one aggregate, so a file @@ -194,14 +235,21 @@ rather than an image: need more children than an aggregate holds is refused rather than exported through the directories of its files, which would expose more than the files. -- The kernel command line announces the number of mappings, and the host - delivers the table over the - [control interface](4.4-host-guest-control-interface.md). The guest agents accept - entries only in order and up to the announced count, and validate each request - before mounting anything. They walk to each target beneath the descriptor of - their root, a mount of a whole file system rather than a bind of a directory, - through no symbolic or magic link, create what it lacks only on the root's own - mounts, never inside another mapping, and attach a copy of the source's mount - there with `nosuid,nodev`, read-only when requested, set before the mapping is - visible. They refuse to run workloads until every announced mapping is - mounted, and a failed mapping keeps them refusing. +- The kernel command line announces the number of mappings, or a restored + nvx-init's activation record does, and the host delivers the table in parts + over the [control interface](4.4-host-guest-control-interface.md). Each + guest agent, the managed agent here and nvx-init in a layered root, keeps + its account of the table in the public `MappingTable`, which accepts parts + only in order and up to the announced count, refuses a part as a whole + before anything of it is mounted, and hands each part's mappings, in order, + to the agent's mapper, part of NVX's internal reference space. The mapper + walks to each target beneath the descriptor of its root, a mount of a whole + file system rather than a bind of a directory, through no symbolic or magic + link, creates what the target lacks only on the root's own mounts, never + inside another mapping, and attaches a copy of the source's mount there + with `nosuid,nodev`, read-only when requested, set before the mapping is + visible. The agents refuse to run workloads until every announced mapping + is mounted, and a failed mapping latches the table, which keeps them + refusing. + [The host's mapping table](../guest-agent-guidelines.md#the-hosts-mapping-table) + gives the table's contract. diff --git a/doc/design/4.2-guest-agents-and-workload-model.md b/doc/design/4.2-guest-agents-and-workload-model.md index 667e4a538..31112f73c 100644 --- a/doc/design/4.2-guest-agents-and-workload-model.md +++ b/doc/design/4.2-guest-agents-and-workload-model.md @@ -15,7 +15,9 @@ contract, with nvx-init as the worked example. A layered root's guest runs nvx-init, NVX's PID 1 for layered roots, which runs its launches, and the restores of platform captures, in one VM-wide space, as Figure 4.2 shows; it also serves the Fresh boot that a platform -capture takes, which has no root yet. The agent of the +capture takes, which has no root yet. That space, the root that nvx-init +assembles and what it mounts in it, is NVX's internal reference code that +nvx-init implements, not public `nvx` API. The agent of the [blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root) is PID 1 in the guest's initramfs root, and runs its executions with nvx-init's plain-process core. @@ -23,42 +25,68 @@ nvx-init's plain-process core. ```mermaid flowchart TB Kernel["Linux kernel"] + Boot["Early boot, the shell /init
in the initramfs"] + Helpers["Helpers
time daemon and announcer"] Init["nvx-init, PID 1
space, helpers and control session"] Space["One VM-wide space
root, view and mappings"] Workload["Executions
a process group each"] - Kernel --> Init + Kernel --> Boot + Boot -->|"start before the switch"| Helpers + Boot -->|"replaced by"| Init + Init -->|"supervise and reap"| Helpers Init --> Space Space --> Workload Init -->|"supervise and reap"| Workload ``` -**Figure 4.2.** nvx-init, its space and its executions. +**Figure 4.2.** nvx-init, its helpers, its space and its executions. The workload is always a child of a guest agent, which never replaces itself with it: supervision, orphan reaping and the control service depend on the -agent remaining alive. In a one-shot lifecycle, the guest agents start one -workload, configured by non-secret kernel command-line tokens, and shut down -the VM when it exits. In a managed lifecycle, they run a sequence of -workloads, one at a time, on request over the authenticated control channel. -An init that runs concurrent executions coordinates them for a capture -itself: `nvx` provides no workspace. +agent remaining alive. A layered root is always managed: nvx-init runs a +sequence of executions, one at a time, on request over the authenticated +control channel. A blockless root takes the lifecycle that its boot names: in +the one-shot lifecycle, the guest's `/init` runs once the workload that +non-secret kernel command-line tokens name, and powers the VM off with its +status, as the [Boot Protocol](5.1-boot-protocol.md) describes; in the managed +lifecycle, the managed agent runs executions as nvx-init does. An init that +runs concurrent executions coordinates them for a capture itself: `nvx` +provides no workspace. ## 4.2.1 Execution Launch Launching an execution belongs to the init, not to the library's capture or activation. nvx-init's executions, and those of the blockless managed root, -are plain processes of their agent: each child takes its credentials and -enters its working directory itself, in its agent's space, without a root of -its own or a placement barrier. +are plain processes of their agent: each child leads a process group of its +own, resets its priority, takes the VM's workload credentials, and enters its +working directory itself, in its agent's space, without a root of its own or +a placement barrier, before it runs its program. [Run the executions](../guest-agent-guidelines.md#run-the-executions) gives nvx-init's steps. ## 4.2.2 Containment For nvx-init and the blockless managed root, the VM is the isolation and -resource boundary. An execution's process group, which its end, -cancellation, timeout and reset kill, is a cleanup scope, not a namespace or -a cgroup: a process that leaves the group escapes it. The init owns -termination, capture admission and cleanup; no workspace or per-execution -enforcement of `nvx` supplies them. \ No newline at end of file +resource boundary. An execution's process group is a cleanup scope, not a +namespace or a cgroup: the agent kills the group when the execution's leader +ends, normally or not, when the host cancels the execution, when its timeout +passes, and when a reset of the control session abandons it, and a process +that leaves the group escapes it. The init owns termination, capture admission +and cleanup; no workspace or per-execution enforcement of `nvx` supplies them. + +## 4.2.3 Helpers + +nvx-init's helpers are the time daemon, which disciplines the guest's clock +and watches it, and the announcer, which tells the gateway the workload's +listeners when the network publishes them. Both start before nvx-init switches +into its root, and run from the initramfs, which holds their programs: the +early boot, the shell `/init`, starts the time daemon, and the announcer when +its network publishes the listeners, then replaces itself with nvx-init, which +takes them over as its children, and a restore whose record publishes the +listeners forks its announcer from PID 1 before its switch. nvx-init registers +exactly these helpers, runs them with its own priority, ahead of the +executions, as +[Priority and Resources](4.3-isolation-and-resource-model.md#433-priority-and-resources) +describes, and ends the VM when either ends, since the VM cannot go on without +them. \ No newline at end of file diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 026cf4fcd..2f10b7d37 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -5,23 +5,26 @@ ## 4.3.1 Identity and Credentials -The workload runs under a host-selected, non-root numeric UID and GID, which -the boot fixes for the VM, or, for a restored nvx-init, its activation. The -guest agents require an exact user and primary-group match in the workload -root, and reject an unavailable identity before starting the workload: -nvx-init resolves the account once, before it switches into its root, and the -older layered sandbox root's entry checks it again for each execution. When -the boot configuration requests it, nvx-init and the guest agents of the -blockless managed root instead create an account for a host-selected identity -that the image lacks, and replace an image account that uses the UID with -another primary group or without a usable home. The older layered sandbox -root creates no accounts. +The workload runs under a host-selected, non-root numeric UID and GID, one +identity for the VM, which the boot fixes, or, for a restored nvx-init, its +activation. The guest agents resolve its account once, before any execution: +nvx-init in its root's user and group databases, before it switches into the +root, and the guest's `/init` of a blockless root in the initramfs's, before +it starts the workload or the managed agent. They require exactly one user +entry with the UID, whose primary group is the GID, a group entry with the +GID, and an absolute home that is a directory, and end the VM rather than run +a workload under an identity that does not resolve. When the boot, or a +restored nvx-init's activation, requests it, they instead create an account +for an identity that the image lacks, and replace an image account that uses +the UID with another primary group or without a usable home. -Before the workload runs, its child clears supplementary groups and every -capability set and enables `no_new_privs`, so the workload keeps none of the -privileges of the guest agents and their helpers. With every capability -cleared, the workload cannot perform port I/O, so only the guest agents can -request a capture or acknowledge restore repair. `nvx` enforces none of this: +Each execution takes these credentials itself, in its own child: before the +workload runs, the child clears supplementary groups and every capability +set, takes the UID and GID, and enables `no_new_privs`, so the workload keeps +none of the privileges of the guest agents and their helpers. With every +capability cleared, the workload cannot perform port I/O, so only the guest +agents can request a capture or acknowledge restore repair. `nvx` enforces +none of this: [Run the executions](../guest-agent-guidelines.md#run-the-executions) gives nvx-init's steps. @@ -30,54 +33,41 @@ nvx-init's steps. nvx-init's space is VM-wide: one root, with one set of runtime file systems, one hostname, machine ID and resolver file, and the host's mappings, into which PID 1 switches once. Its executions share that root and PID 1's mount, -PID and UTS namespaces: no execution has a namespace or a root of its own. -The root's `/dev` holds no control device: PID 1 opens it before the switch -and keeps its descriptor, and the helpers, which run from the initramfs, -start before the switch too. Once switched, nothing unmounts the root, and -the VM ends by powering off. +PID and UTS namespaces: no execution has a namespace, a root, or a hostname of +its own. The root's `/dev` holds no control device: PID 1 opens it before the +switch and keeps its descriptor, and the helpers, which run from the +initramfs, start before the switch too. Once switched, nothing unmounts the +root, and the VM ends by powering off. [Build the space](../guest-agent-guidelines.md#build-the-space) gives the space's rules. -The older layered sandbox root gives each workload private mount, PID, and -UTS namespaces, in which every mount is private, with a private `/proc`, a -read-only `/sys`, a restricted `/dev`, the workload root, only the live shares -that the host declared, and the workload's machine ID and the guest's time -status, bound read-only. Its guest agents never pivot away from the -initramfs, so they stay outside the workload's namespaces, where they -supervise and tear down the workload, and the control device is never exposed -inside them. The blockless managed root runs each execution in the guest's -initramfs root, in the guest agents' own namespaces, without a private mount, -process, hostname, or device view. +The blockless managed root runs each execution in the guest's initramfs root, +in the guest agents' own namespaces, without a private mount, process, +hostname, or device view. -## 4.3.3 Cgroups and Resources +## 4.3.3 Priority and Resources For nvx-init and the blockless managed root, the VM is the resource limit: -they configure no cgroups and no per-execution limits. nvx-init runs PID 1 and -its helpers ahead of the executions, whose children give up that priority, -instead of reserving memory for them in a sibling cgroup. Its one limit -within the VM is the RAM upper's cap; the scratch and the host's devices keep -their own storage boundaries. - -The older layered sandbox root keeps sibling guest-agent and workload cgroups. -The workload is placed in its cgroup behind a barrier before it is released. -The guest agents' cgroup carries a default memory protection of 16 MiB, a -soft reservation that shields the supervisor from reclaim under workload -memory pressure but does not guarantee its survival. The workload cgroup -carries optional memory and process-count limits and is the unit of capture -freeze, termination, and descendant cleanup. When the overlay's upper is in -guest memory, the files that the workload writes into its root are charged to -the workload cgroup, so its memory limit also bounds them; see -[Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). +they configure no cgroups and no per-execution resource limits, such as +memory or process-count limits. nvx-init runs PID 1 and its helpers ahead of +the executions instead, with an out-of-memory score adjustment of -1000, which +exempts them from the out-of-memory killer, and a nice value of -10. Each +execution's child resets both to 0 before it takes its credentials, so that no +workload keeps them. The one limit that nvx-init sets within the VM is the +RAM upper's cap, which bounds the files that the root's upper holds in guest +memory, but not those in the runtime's own tmpfs mounts, such as `/dev/shm`, +which keep the kernel's default size; the scratch and the host's devices keep +their own storage boundaries. An execution's timeout, which its request +carries, ends the execution rather than limit its resources. ## 4.3.4 Enforcement Layers Isolation has two enforcement layers: - Guest policy, which the init applies and `nvx` does not enforce: the - workload's credentials and capabilities, the mounts of its space, and, in - the older layered sandbox root, its namespaces and cgroups. + workload's credentials and capabilities, and the mounts of its space. - Host devices independently enforce block mutability, network policy, and host-filesystem access. The guest cannot widen a host policy by remounting a filesystem, changing a -route, creating a link, or issuing a different device request. \ No newline at end of file +route, creating a link, or issuing a different device request. diff --git a/doc/design/4.4-host-guest-control-interface.md b/doc/design/4.4-host-guest-control-interface.md index 85c712e81..517890501 100644 --- a/doc/design/4.4-host-guest-control-interface.md +++ b/doc/design/4.4-host-guest-control-interface.md @@ -14,7 +14,13 @@ unstructured byte stream with them. ## 4.4.1 Operations -Table 4.2 lists the operations of the control protocol. +Table 4.2 lists the operations of the control protocol. nvx-init also takes a +platform capture's request and a restore's activation, as the guest-agent +guidelines' +[Coordinate a capture](../guest-agent-guidelines.md#coordinate-a-capture) +and +[Complete a restore before Ready](../guest-agent-guidelines.md#complete-a-restore-before-ready) +describe. **Table 4.2.** Control operations. @@ -23,7 +29,7 @@ Table 4.2 lists the operations of the control protocol. | `READY` | Reports readiness, and is the success reply to requests that return no other data | | `MAPS` | Delivers a contiguous range of host mapping-table entries | | `EXEC` | Runs one workload with a timeout, arguments, and an optional working directory and environment | -| `CANCEL` | Kills the running execution that the request names | +| `CANCEL` | Kills the process group of the running execution that the request names | | Shutdown | Shuts the VM down gracefully, ending a running execution first | A mapping request carries a contiguous range of the host mapping table's @@ -32,7 +38,8 @@ host's export, and an absolute target in the guest's root. An empty source names the export's root itself, as the table of a host whose export is one directory maps the whole directory. NVX's hosts, whose exports are aggregates of host directories and files, whose roots list them, send none, and NVX's -managed agent refuses one. +guest agents refuse one: the `MappingTable` that keeps their account of the +table refuses an empty source. An execution request carries a 32-bit timeout and length-prefixed arguments, and optionally an absolute working directory of at most 4,096 bytes and up to @@ -57,11 +64,11 @@ an outcome category and a status. | `cancelled` | A cancellation killed the workload; the status is 137 | | `failed` | The guest agents could not supervise the workload; the status is 125 | -Otherwise, the guest agents reply with an error, which carries one of the -categories in Table 4.4 and a status. An error refuses a request that the -guest agents cannot carry out, or takes the place of the exit report of an -execution that could not start or whose processes the guest agents could not -account for. +Otherwise, the guest agents answer a request of Table 4.2 with an error, which +carries a status and one of the categories in Table 4.4, or, while an +execution runs, with `BUSY`, which carries neither. An error refuses a request +that the guest agents cannot carry out, or takes the place of the exit report +of an execution that could not start. **Table 4.4.** Error categories. @@ -69,10 +76,8 @@ account for. | --- | --- | | `launch-failed` | The guest agents could not prepare the execution | | `cwd-failed` | The workload could not enter its working directory; the status is the error number | -| `containment-failed` | The guest agents could not prove that every workload process exited | | `mapping-failed` | A host mapping could not be mounted | | `mappings-incomplete` | Not every announced host mapping is mounted yet | -| `busy` | Another execution is running | | `invalid-request` | The request is malformed or invalid in the current configuration | | `unsupported-operation` | The guest does not implement the operation | @@ -81,20 +86,25 @@ account for. The protocol negotiates nothing: every frame names its version, the guest's `READY` names the version that its agents speak, and NVX's host requires version 2, whose guest agents provide every behavior that the host depends on. -A peer of another version is refused: the host refuses a `READY` that names -another version, and the guest agents refuse a frame of another version as a -malformed one, which ends the session. Version 1, whose `FEATURES` request -reported the control behaviors that a guest image provided, is gone. +nvx-init and the managed agent serve the control session through one +dispatcher, which gives both the operations of Table 4.2. A peer of another +version is refused: the host refuses a `READY` that names another version, and +the guest agents refuse a frame of another version as a malformed one, which +ends the session. Version 1, whose `FEATURES` request reported the control +behaviors that a guest image provided, is gone. ## 4.4.3 Session Semantics Requests are typed, length-bounded, ordered, and associated with one VM instance and epoch. Credit limits bound queued data in both directions. Malformed, unauthenticated, stale, duplicated, or out-of-order records fail -closed. Only one execution runs at a time; any request other than a -cancellation, the execution's standard input, or a shutdown, that arrives -during an execution is refused as `busy`. A shutdown ends the running -execution first. +closed. Only one execution runs at a time: while it runs, the guest agents +take its standard input, a cancellation and a shutdown, and refuse any other +request of Table 4.2, such as another execution or a mapping request, with +`BUSY`. A shutdown cancels the running execution, whose exit report precedes +the shutdown's acknowledgement. A reset of the session ends a running +execution: the guest agents kill its process group, and report none of its +output on the new session. The control session is process-local even when guest parser state is snapshotted. Restore establishes a fresh host capability, peer identity,