Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions doc/acceptance-case-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ its own in turn.
| [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 |
| [MicroVM scenarios](#microvm-scenarios) | 52 | 49 | 3 | 0 |
| [CI](#ci) | 69 | 43 | 3 | 23 |
| [Docs](#docs) | 34 | 33 | 1 | 0 |
| [Docs](#docs) | 36 | 35 | 1 | 0 |

## The adapter's scenarios

Expand Down Expand Up @@ -412,17 +412,18 @@ their own. The tests that a step runs have rows of their own where they concern
## Docs

Rustdoc checks the code of the crate's documentation, which these rows name as rustdoc does, by
file and line where this matrix was added: it compiles each, runs those that aren't `no_run`, and
checks that those marked `compile_fail` don't compile. The examples compile with the crate's
other targets; `host-workflows` and `guest-workflows`, which came with this matrix, are new, and
have no row. The README's own code is no doctest.
file and line: it compiles each, runs those that aren't `no_run`, and checks that those marked
`compile_fail` don't compile. The examples compile with the crate's other targets;
`host-workflows` and `guest-workflows`, which came with this matrix, are new, and have no row.
The code of `guest` and `guest::snapshot`, which the guest-agent guidelines quote, came after
this matrix. The README's own code is no doctest.

| Code | Fate | Note |
| --- | --- | --- |
| `nvx-rs/src/activation/mod.rs - activation (line 123)` | kept | |
| `nvx-rs/src/activation/mod.rs - activation (line 130)` | kept | |
| `nvx-rs/src/exec/mod.rs - exec (line 113)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
| `nvx-rs/src/gpt/mod.rs - gpt (line 43)` | kept | |
| `nvx-rs/src/guest/activation.rs - guest::activation (line 189)` | kept | Its capture preparation and admission fence leave (#1087), whose PR rewrites it on `guest::snapshot::capture` |
| `nvx-rs/src/guest/activation.rs - guest::activation (line 195)` | kept | Its capture preparation and admission fence leave (#1087), whose PR rewrites it on `guest::snapshot::capture` |
| `nvx-rs/src/guest/activation/state.rs - guest::activation::state::Activation (line 56)` | kept | |
| `nvx-rs/src/guest/cgroup.rs - guest::cgroup (line 36)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
| `nvx-rs/src/guest/entry.rs - guest::entry (line 90)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
Expand All @@ -431,11 +432,13 @@ have no row. The README's own code is no doctest.
| `nvx-rs/src/guest/gpt.rs - guest::gpt (line 18)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests |
| `nvx-rs/src/guest/mappings/mapper.rs - guest::mappings::mapper (line 107)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests |
| `nvx-rs/src/guest/mappings/table.rs - guest::mappings::table::MappingTable (line 47)` | kept | |
| `nvx-rs/src/guest/mod.rs - guest (line 41)` | kept | The guest-agent guidelines quote it |
| `nvx-rs/src/guest/nvxs.rs - guest::nvxs (line 22)` | kept | |
| `nvx-rs/src/guest/process.rs - guest::process (line 12)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
| `nvx-rs/src/guest/process.rs - guest::process (line 39)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
| `nvx-rs/src/guest/reseed.rs - guest::reseed (line 17)` | kept | |
| `nvx-rs/src/guest/root.rs - guest::root (line 130)` | kept | The guest's space becomes private (#1127), whose PR moves it to the space's unit tests |
| `nvx-rs/src/guest/snapshot.rs - guest::snapshot (line 25)` | kept | The guest-agent guidelines quote it |
| `nvx-rs/src/guest/spawn.rs - guest::spawn (line 41)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
| `nvx-rs/src/guest/spawn.rs - guest::spawn::LaunchStage::from_code (line 201)` | kept | Its module leaves with the exec engine (#1087), whose PR retires it |
| `nvx-rs/src/guest/terminal.rs - guest::terminal (line 10)` | kept | |
Expand Down
116 changes: 51 additions & 65 deletions doc/design/4.2-guest-agents-and-workload-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,84 +3,70 @@
[Architecture documentation](README.md) >
[4. Guest Runtime Architecture](4-guest-runtime-architecture.md)

As Figure 4.2 shows, the guest agents are the guest's long-lived lifecycle
authority. Their responsibilities are:
The guest's init agent is its long-lived lifecycle authority: it owns the
guest's space, its helpers, its control session and its executions. `nvx`
parses the contracts, captures the guest and drives a restore's activation,
but builds no space, runs no executions and enforces none of the init's
obligations; the one exception is the time daemon, which
`TimeAbi::start_daemon` forks. The
[guest-agent guidelines](../guest-agent-guidelines.md) give the init's
contract, with nvx-init as the worked example.

1. Initialize runtime filesystems and cgroup v2.
2. Discover and validate the fixed virtual devices.
3. Assemble the workload filesystem.
4. Apply workload identity, isolation, and resource policy.
5. Start the workload behind a placement barrier.
6. Supervise the workload and reap descendants.
7. Announce the guest's listeners when the host publishes them.
8. Participate in capture and post-restore repair.
9. Unmount resources and report the final workload status.
A layered root's guest runs one of two inits. nvx-init, NVX's PID 1 for
layered roots, runs the new launches, and the restores of platform captures,
in one VM-wide space, as Figure 4.2 shows; it also serves the Fresh boot that
a platform capture takes, which has no root yet. A layered launch that still
carries OpenVMM's older lifecycle token, `nvx_lifecycle=`, which NVX's CLI no
longer adds, runs the older init agent, whose sandbox agent gives each
execution private namespaces and a cgroup in the
[layered sandbox root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root).
The agent of the
[blockless managed root](4.1-guest-filesystem-and-image-model.md#413-blockless-managed-root)
is PID 1 in the guest's initramfs root, and runs its executions with
nvx-init's plain-process core.

```mermaid
flowchart TB
Kernel["Linux kernel"]
Agents["Guest agents<br/>outside workload namespaces<br/>in the layered root"]
Root["Assembled workload root"]
Isolation["Identity, capabilities,<br/>and cgroups, plus namespaces<br/>in the layered root"]
Workload["Workload process tree"]
Init["nvx-init, PID 1<br/>space, helpers and control session"]
Space["One VM-wide space<br/>root, view and mappings"]
Workload["Executions<br/>a process group each"]

Kernel --> Agents
Agents --> Root
Agents --> Isolation
Root --> Workload
Isolation --> Workload
Agents -->|"supervise and reap"| Workload
Kernel --> Init
Init --> Space
Space --> Workload
Init -->|"supervise and reap"| Workload
```

**Figure 4.2.** Relationship between the guest agents and the workload.
**Figure 4.2.** nvx-init, its space and its executions.

In a one-shot lifecycle, the guest agents start one workload, configured by
non-secret kernel command-line tokens, and shut down the VM when it exits. In
a managed lifecycle, the guest agents stay resident and run a sequence of
The workload is always a child of a guest agent, which never replaces itself
with it: supervision, orphan reaping and the control service depend on the
agent remaining alive. In a one-shot lifecycle, the guest agents start one
workload, configured by non-secret kernel command-line tokens, and shut down
the VM when it exits. In a managed lifecycle, they run a sequence of
workloads, one at a time, on request over the authenticated control channel.
Both modes preserve the same separation: the workload is always a child of a
guest agent. The guest agents never replace themselves with the workload,
because supervision, orphan reaping, and the control service all depend on
them remaining alive.
An init that runs concurrent executions coordinates them for a capture
itself: `nvx` provides no workspace.

## 4.2.1 Execution Launch

For each managed execution, the guest agent's launch enters the requested
working directory, or `/`, with the workload's identity, and sets `PWD` unless
the request replaces the environment. The working directory is entered once
and kept, so the workload starts in the directory that was checked even if its
path is renamed, replaced, or retargeted in the meantime. If the workload
cannot enter it, the execution fails rather than running elsewhere.

In the blockless managed root, the guest agent uses the init agent's
plain-process builder and driver. The builder prepares the environment, and
the child applies its identity, enters its working directory, and executes the
workload without a launch helper. In the layered sandbox root, the guest
agent's [execution entry](4.3-isolation-and-resource-model.md#432-namespaces-and-mounts)
applies the identity itself, and the guest agent builds the environment.
Launching an execution belongs to the init, not to the library's capture or
activation. nvx-init's executions, and those of the blockless managed root,
are plain processes of their agent: each child takes its credentials and
enters its working directory itself, in its agent's space, without a root of
its own or a placement barrier. The older layered sandbox root keeps its
execution entry, which enters private namespaces and the workload root, and
its placement barrier.
[Run the executions](../guest-agent-guidelines.md#run-the-executions) gives
nvx-init's steps.

## 4.2.2 Containment

In the blockless managed root, each execution runs in a process group pinned
by the driver. When the workload's first process exits, the guest agent kills
the processes remaining in that group and, as the guest's init process, reaps
the orphans it inherits. It drains the workload's output before reporting the
outcome. A descendant that leaves the process group is not contained by this
plain-process execution and may keep an output pipe open after its leader
exits. A cancellation or timeout after the leader's status is known or lost
does not change that status while the output drains. This path has no execution
cgroup or workload namespace.

A cancellation that names the running execution kills the workload, and a
cancellation for an execution that already finished is ignored. A session
reset means that the host session that could observe the workload is gone:
the guest agent kills the workload, sends nothing more for it, and
acknowledges the new epoch, and drops the old output descriptors rather than
waiting for them to close, so the next session never waits for an abandoned
workload.

When the blockless agent ends a running workload for cancellation, timeout, or
the output cap, it collects the leader and drops the remaining output
descriptors rather than waiting for an escaped descendant to close them.
Cancellation and timeout first forward any available output up to the cap.
The layered sandbox continues to use its existing namespace and cgroup cleanup.
For nvx-init and the blockless managed root, the VM is the isolation and
resource boundary. An execution's process group, which its end,
cancellation, timeout and reset kill, is a cleanup scope, not a namespace or
a cgroup: a process that leaves the group escapes it. The init owns
termination, capture admission and cleanup; no workspace or per-execution
enforcement of `nvx` supplies them. The older layered sandbox root keeps its
namespace and cgroup cleanup.
123 changes: 59 additions & 64 deletions doc/design/4.3-isolation-and-resource-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,85 +5,80 @@

## 4.3.1 Identity and Credentials

The workload runs under a host-selected, non-root numeric UID and GID fixed at
initial boot. The guest agents require an exact user and primary-group match
in the workload root and reject an unavailable identity before starting the
workload. In the layered sandbox root, each managed execution's entry checks
the account again, reading the root's user and group databases beneath its
directory descriptor, with bounds, before the workload runs. When the boot
configuration requests it, the guest agents of the blockless managed root
instead create an account for a host-selected identity that the image lacks,
and replace an image account that uses the UID with another primary group or
without a usable home. This lets a host run workloads under its own user's IDs.
The layered sandbox root creates no accounts, so it does not advertise
The workload runs under a host-selected, non-root numeric UID and GID, which
the boot fixes for the VM, or, for a restored nvx-init, its activation. The
guest agents require an exact user and primary-group match in the workload
root, and reject an unavailable identity before starting the workload:
nvx-init resolves the account once, before it switches into its root, and the
older layered sandbox root's entry checks it again for each execution. When
the boot configuration requests it, nvx-init and the guest agents of the
blockless managed root instead create an account for a host-selected identity
that the image lacks, and replace an image account that uses the UID with
another primary group or without a usable home. The older layered sandbox
root creates no accounts, so it does not advertise
[account creation](4.4-host-guest-control-interface.md#442-feature-negotiation).

Before the workload starts, the guest agents clear supplementary groups and
every capability set and enable `no_new_privs`. With every capability cleared,
the workload cannot perform port I/O, so only the guest agents can request a
capture or acknowledge restore repair.
Before the workload runs, its child clears supplementary groups and every
capability set and enables `no_new_privs`, so the workload keeps none of the
privileges of the guest agents and their helpers. With every capability
cleared, the workload cannot perform port I/O, so only the guest agents can
request a capture or acknowledge restore repair. `nvx` enforces none of this:
[Run the executions](../guest-agent-guidelines.md#run-the-executions) gives
nvx-init's steps.

## 4.3.2 Namespaces and Mounts

In the layered sandbox root, the workload receives private mount, PID, and UTS
namespaces, in which every mount is private, and:

- A private `/proc` and a read-only `/sys`.
- A restricted `/dev`, devpts, and shared memory.
- The workload root.
- Only the live shares that the host declared.
- The workload's machine ID, bound read-only from a runtime file so that the
guest agents can refresh it while scratch is frozen.
- The guest time status, bound read-only.

A managed execution enters through the guest library's execution entry, as the
first process of its PID namespace. The entry moves the workload root over `/`
in the execution's mount namespace rather than entering it with `chroot`
alone, gives `/dev` only its allowlisted device nodes, and makes every other
mount `nodev`. The managed agent binds the machine ID and the time status into
the workload root once, in a mount namespace of its own, of which each
execution's starts as a copy, so those binds end with the agent, before the
init agent unmounts the root. A one-shot workload's entry helper enters the
root with `chroot`.

The guest agents never pivot away from the initramfs, so they keep their own
root and stay outside the workload's namespaces, where they can supervise and
tear down the workload. The control device is never exposed inside the
workload's namespaces.

The blockless managed root has no such isolation. Each execution runs in the
guest's initramfs root, without a `chroot`, in the same mount, PID, and UTS
namespaces as the guest agents, so it sees the guest's own `/proc`, `/sys`, and
`/dev` and the host mappings bound into that root. The plain-process builder
applies its [credentials](#431-identity-and-credentials), but it has no
execution cgroup or private mount, process, hostname, or device view.
nvx-init's space is VM-wide: one root, with one set of runtime file systems,
one hostname, machine ID and resolver file, and the host's mappings, into
which PID 1 switches once. Its executions share that root and PID 1's mount,
PID and UTS namespaces: no execution has a namespace or a root of its own.
The root's `/dev` holds no control device: PID 1 opens it before the switch
and keeps its descriptor, and the helpers, which run from the initramfs,
start before the switch too. Once switched, nothing unmounts the root, and
the VM ends by powering off.
[Build the space](../guest-agent-guidelines.md#build-the-space) gives the
space's rules.

The older layered sandbox root gives each workload private mount, PID, and
UTS namespaces, in which every mount is private, with a private `/proc`, a
read-only `/sys`, a restricted `/dev`, the workload root, only the live shares
that the host declared, and the workload's machine ID and the guest's time
status, bound read-only. Its guest agents never pivot away from the
initramfs, so they stay outside the workload's namespaces, where they
supervise and tear down the workload, and the control device is never exposed
inside them. The blockless managed root runs each execution in the guest's
initramfs root, in the guest agents' own namespaces, without a private mount,
process, hostname, or device view.

## 4.3.3 Cgroups and Resources

In the layered sandbox root, the guest agents and workload occupy sibling
cgroups. The workload is placed in its cgroup behind a barrier before it is
released. The guest agents' cgroup carries a default memory protection of
16 MiB, a soft reservation that shields the supervisor from reclaim under
workload memory pressure but does not guarantee its survival. The workload
cgroup carries optional memory and process-count limits and is the unit of
capture freeze, termination, and descendant cleanup. When the overlay's upper
is in guest memory, the files that the workload writes into its root are
charged to the workload cgroup, so its memory limit also bounds them; see
For nvx-init and the blockless managed root, the VM is the resource limit:
they configure no cgroups and no per-execution limits. nvx-init runs PID 1 and
its helpers ahead of the executions, whose children give up that priority,
instead of reserving memory for them in a sibling cgroup. Its one limit
within the VM is the RAM upper's cap; the scratch and the host's devices keep
their own storage boundaries.

The older layered sandbox root keeps sibling guest-agent and workload cgroups.
The workload is placed in its cgroup behind a barrier before it is released.
The guest agents' cgroup carries a default memory protection of 16 MiB, a
soft reservation that shields the supervisor from reclaim under workload
memory pressure but does not guarantee its survival. The workload cgroup
carries optional memory and process-count limits and is the unit of capture
freeze, termination, and descendant cleanup. When the overlay's upper is in
guest memory, the files that the workload writes into its root are charged to
the workload cgroup, so its memory limit also bounds them; see
[Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root).

The blockless managed root does not use execution cgroups. Its plain-process
driver terminates the workload's process group. This path does not configure a
sibling guest-agent cgroup, the 16-MiB memory protection, or the layered root's
optional memory and process-count limits.

## 4.3.4 Enforcement Layers

Isolation has two enforcement layers:

- Guest policy constructs the workload's credentials, mounts, capabilities,
and cgroups and, in the layered sandbox root, its namespaces.
- Guest policy, which the init applies and `nvx` does not enforce: the
workload's credentials and capabilities, the mounts of its space, and, in
the older layered sandbox root, its namespaces and cgroups.
- Host devices independently enforce block mutability, network policy, and
host-filesystem access.

The guest cannot widen a host policy by remounting a filesystem, changing a
route, creating a link, or issuing a different device request.
route, creating a link, or issuing a different device request.
Loading
Loading