diff --git a/doc/run.md b/doc/run.md index f6d2b6d4a..9bc3920f8 100644 --- a/doc/run.md +++ b/doc/run.md @@ -58,8 +58,9 @@ nvx-rs/target/release/nvx-rs run \ Networking requires the explicit `portable` capability profile. It uses the same in-process data plane on KVM, MSHV, and WHP; a cold boot that omits either `--net` or `--network-profile portable` is rejected before OpenVMM -starts. A restore takes its addresses from the snapshot instead; see -[Networked snapshots](#networked-snapshots). +starts. A restore takes its NIC, with its addresses, from the snapshot +instead, and installs a network policy of its own on it; see +[`snapshot restore`](usage.md#snapshot-restore). Select an explicit processor count after building the matching specialized guest kernel: @@ -359,26 +360,27 @@ preserves the existing guest-to-host mapping without publishing guest ports. Guest-originated traffic remains subject to egress policy. Most options of a cold boot of `nvx-rs run` pass through unchanged: -`--machine`, `--processors`, `--cpu-profile`, `--mount`, `--net`, -`--network-profile`, and `--cmdline`. The `nvx-rs run` performs these -translations and additions: +`--machine`, `--processors`, `--cpu-profile`, `--net`, and +`--network-profile`. The `nvx-rs run` performs these translations and +additions: | `nvx-rs run` | Direct OpenVMM option | | --- | --- | -| `--guest alpine` | `--initrd .../initramfs.cpio.gz` on a fresh boot | -| `--guest ubuntu` | `--initrd .../initramfs-ubuntu.cpio.gz` and a 512 MiB default on a fresh boot | -| `--guest azurelinux` | `--initrd .../initramfs-azurelinux.cpio.gz` and a 512 MiB default on a fresh boot | +| `--guest alpine` | `--initrd .../initramfs.cpio.gz` | +| `--guest ubuntu` | `--initrd .../initramfs-ubuntu.cpio.gz` and a 512 MiB default | +| `--guest azurelinux` | `--initrd .../initramfs-azurelinux.cpio.gz` and a 512 MiB default | | `--hypervisor auto` | `--hypervisor kvm` on Linux or `--hypervisor whp` on Windows | | `--memory-mib N` | `--memory NM` | -| `--memory-capacity-mib N` | `--memory-capacity NM` on a fresh boot | -| A fresh boot | `--virtio-console console`, which attaches the guest's console, the boot console `hvc1`, to the terminal | +| `--memory-capacity-mib N` | `--memory-capacity NM` | +| `--mount` | `--mount-aggregate /run/nvx/hostfs/root`, with one `--mount-child` for each outermost mapped directory and each mapped file outside them, and `--mount-write` for a read-write `--mount` inside a read-only one; see [virtio-fs host mapping](#virtio-fs-host-mapping) | +| `--cmdline TEXT` | `--cmdline` with `TEXT` among the tokens of the guest's view, mode, and shares, which NVX renders | +| Every cold boot | `--virtio-console listen=ENDPOINT`, whose client, `nvx-rs run`, connects the guest's console, the boot console `hvc1`, to the terminal, and the `--microvm-control-console`, `--microvm-control-auth-stdin`, `--microvm-report`, and `--guest-shutdown-action exit` that every launch of NVX passes | | `--dry-run` | No equivalent; this only prints the generated command | `nvx-rs run --restore-snapshot NAME --store-dir PATH --state-dir PATH` restores -a named snapshot that `nvx-rs snapshot capture` published once, as -`nvx-rs snapshot restore` does, and no longer a raw OpenVMM snapshot directory, -which it refuses; see [`run`](usage.md#run). Restore a raw snapshot with -OpenVMM directly, as below. +only a named snapshot that `nvx-rs snapshot capture` published, once, as +`nvx-rs snapshot restore` does, and refuses a raw OpenVMM snapshot directory; +see [`run`](usage.md#run). OpenVMM itself restores a raw snapshot, as below. Always include `--single-process`. Include `--virtio-console console` to type into the guest's shell on a fresh boot: without a boot console, the guest's @@ -389,10 +391,8 @@ to restore a snapshot of a boot that passed it, and refuses `--virtio-console` for a snapshot without a boot console. When restoring, omit `--memory`, `--kernel`, `--initrd`, and `--net`, whose addresses the snapshot keeps; see [Networked snapshots](#networked-snapshots). Every restore gives the guest -fresh entropy through the time ABI's restore packet. Do not pass -`--guest ubuntu` during restore; the captured RAM and machine contract already -identify the restored guest. For example, for a snapshot of a boot with -`--virtio-console console`: +fresh entropy through the time ABI's restore packet. For example, for a +snapshot of a boot with `--virtio-console console`: ```bash ./bin/openvmm \ @@ -598,11 +598,12 @@ grants it connectivity that it never had. OpenVMM rejects the option for a cold boot, for a snapshot without a NIC or whose NIC was captured with any other policy, and for a `workload-start` or `instance-checkpoint` snapshot. A [platform NIC template](#platform-nic-templates) takes the option too, and so -does each restore of a named snapshot whose NIC `nvx-rs snapshot capture --net` -captured under the complete deny-all policy, through -`nvx-rs run --restore-snapshot` or `nvx-rs snapshot restore`, which installs the -policy of its network options; see -[`snapshot capture`](usage.md#snapshot-capture) and [`run`](usage.md#run). +does each restore of a named snapshot, a platform snapshot whose NIC +`nvx-rs snapshot capture --net` captured under the complete deny-all policy, +through `nvx-rs run --restore-snapshot` or `nvx-rs snapshot restore`, which +install the policy of their network options, and whose guest learns its DNS +server and whether to announce its listeners from its activation, as +[`snapshot restore`](usage.md#snapshot-restore) describes. ### Platform NIC templates @@ -677,10 +678,9 @@ export again. OpenVMM's v4 microVM restarted the versions of its contracts: microVM ABI 1, boot layout 1, time ABI 1, restore packet 1, and snapshot manifest 2, whose version 1 is upstream OpenVMM's. It reads no snapshot of an earlier OpenVMM, -whose manifest it refuses by its version with `E_SNAPSHOT_VERSION`, and -`nvx-rs` refuses a template of an earlier OpenVMM by its snapshot format. -Capture them again; to restore them as they are, use NVX commit -`304f8e1686b4fba188ad2d9c8d932498a6ca2980` with the OpenVMM that it pins. +whose manifest it refuses by its version with `E_SNAPSHOT_VERSION`: capture +such a snapshot again, as [Upgrades](#upgrades) says of every snapshot of +other artifacts. The machine is the one that the earlier OpenVMM called microVM ABI 3: no virtio device offers `VIRTIO_F_ACCESS_PLATFORM`, no console offers its size @@ -707,22 +707,25 @@ host-readiness endpoint: --restore-ready-path /run/nvx/restore-ready.sock ``` -`nvx-rs run --restore-snapshot` and `nvx-rs snapshot restore` own the endpoint -of each of their restores, whose event they wait for, and restore the -processors that their snapshot had online. +`nvx-rs run --restore-snapshot`, `nvx-rs snapshot restore`, and each +`nvx-rs sandbox start` of a restored sandbox own the endpoint of their +restore, whose event they accept only from the VM's own OpenVMM process, and +restore the processors that their snapshot had online. The endpoint must already be listening. OpenVMM connects to a Unix domain socket on Linux or a `//./pipe/...` named pipe on Windows and writes exactly -`OPENVMM_RESTORE_READY_V1\n` after snapshot verification, attachment -resolution, worker startup, gated guest repair, and host-input re-enable -complete while the restored vCPU remains stopped. With +`OPENVMM_RESTORE_READY_V1\n` once it has verified the snapshot, resolved its +attachments, and started the VM's workers, and before it releases a restored +vCPU, so the event precedes the guest's repair. A restore that requires the +guest's acknowledgement gates host input from before the event until the +guest's repair acknowledges the restore. With `--restore-processors`, the snapshot keeps its immutable eight-vCPU capacity; the source must have captured a canonical `maxcpus` boot-online prefix, and the -guest onlines exactly the requested prefix before readiness and host input -release. Targets are limited to 1, 2, 4, or 8 and cannot be below the captured -boot-online count or above capacity. Platform snapshots opt in the same way, so -each clone of one platform capture can bring its own prefix online. Legacy -snapshots reject the option. The +guest's repair onlines exactly the requested prefix while host input stays +gated, before it acknowledges the restore. Targets are limited to 1, 2, 4, or 8 +and cannot be below the captured boot-online count or above capacity. Platform +snapshots opt in the same way, so each clone of one platform capture can bring +its own prefix online. Legacy snapshots reject the option. The peer must accept and read while startup is in progress; Windows flush completion waits for the named-pipe peer to consume the frame. Failure to write the complete event aborts and tears down the restore. @@ -731,7 +734,8 @@ For restore-time memory expansion, capture a fresh snapshot with `--memory-mib 512 --memory-capacity-mib 2048`, then select a target with `--restore-memory 512M`, `1024M`, or `2048M`. The snapshot's `memory.bin` remains exactly 512 MiB; selected expansion ranges receive fresh per-launch -backing and are onlined before restore readiness. +backing before any restored vCPU runs, and the guest's repair onlines them +while host input stays gated, before it acknowledges the restore. ## virtio-fs host mapping @@ -787,46 +791,50 @@ nvx-rs/target/release/nvx-rs run \ --mount "/results/output.txt,/srv/app/results/output.txt,rw" ``` -With several mappings, or a shared file, NVX attaches the slot as an -aggregate: OpenVMM's `--mount-aggregate /run/nvx/shares` with one -`--mount-child NAME,HOST_PATH,MODE[,file]` per mapping, in order, where -OpenVMM's `file` flag marks a shared file, so OpenVMM refuses a directory that -replaced the file rather than export all of it. The initramfs mounts -the aggregate at `/run/nvx/shares`, whose root lists one entry per mapping and -only root can enter, and binds each entry at its target with its mode and -`nosuid,nodev`, as an `nvx_share=NAME,TARGET,MODE[,file]` token on the kernel -command line describes. The `file` suffix marks a shared file, which the -initramfs binds at a regular file that it creates at the target, along with -the target's missing directories; an existing target must already be a -regular file. A -mapping's name is its index, a hyphen, and the first 32 hexadecimal digits of -the SHA-256 digest of its target. A snapshot records every name, so it pins -each target: a restore that requests another target fails before the guest -runs, rather than resume binds at the snapshot's targets. -OpenVMM enforces each mapping's access mode on the host, so a read-only mapping -rejects writes with `EROFS` even if guest root remounts its bind read-write or -writes through `/run/nvx/shares`. A rename between mappings fails with -`EXDEV`, and so does a hard link, unless its destination is read-only, which -fails with `EROFS` first. OpenVMM serves a shared file alone: the guest cannot -list, inspect, or change the file's directory or reach its siblings, a lookup -below the file fails with `ENOTDIR`, and a read-write file can be rewritten in -place but not replaced, renamed, or removed. OpenVMM follows the file's name, -so the guest sees the file that host software, such as an editor, saves over -it, and requests fail with `EACCES` while anything but a regular file holds the -name. Guest targets that equal or contain one another or `/run/nvx/shares`, -and host paths that equal or contain one another, such as a shared file inside -a shared directory or two hard links to one file, are rejected before boot. +NVX attaches the slot as an aggregate, even for one mapping: OpenVMM's +`--mount-aggregate /run/nvx/hostfs/root` with one +`--mount-child INDEX,HOST_PATH,MODE[,file]` per child, numbered from 0. The +children are the outermost mapped directories and the mapped files outside +them, so a `--mount` of a directory again, or of a path inside a mapped +directory, maps a part of that directory's child rather than a child of its +own. OpenVMM's `file` flag marks a child that is a shared file, so OpenVMM +refuses a directory that replaced the file rather than export all of it. A +child is read-write only if it holds a read-write mapping, and OpenVMM then +limits its writes to its read-write mappings, which NVX passes as +`--mount-write` paths when they lie inside a read-only one. A read-only +mapping inside a writable part of its child, such as inside a read-write +mapping without `--mount-write` paths, is rejected before boot, since OpenVMM +cannot keep a path read-only there. The initramfs mounts the +aggregate at `/run/nvx/hostfs/root`, whose root lists one entry per child and +only root can enter, and binds each mapping at its target with its mode and +`nosuid,nodev`, as an `nvx_share=SOURCE,TARGET,MODE[,file]` token on the +kernel command line describes: `SOURCE` is the index of the mapping's child, +followed by the mapping's path inside it, if any, such as `0/single.txt`. The +`file` suffix marks a shared file, which the initramfs binds at a regular file +that it creates at the target, along with the target's missing directories; an +existing target must already be a regular file. +OpenVMM enforces each mapping's access on the host, so a read-only mapping +rejects writes with `EROFS`, except inside a read-write mapping that it +contains, even if guest root remounts its bind read-write or writes through +`/run/nvx/hostfs/root`. A rename between children fails with `EXDEV`, and so +does a hard link, unless its destination is read-only, which fails with +`EROFS` first. OpenVMM serves a file child alone: the guest cannot list, +inspect, or change the file's directory or reach its siblings, a lookup below +the file fails with `ENOTDIR`, and a read-write file can be rewritten in place +but not replaced, renamed, or removed. OpenVMM follows the file's name, so the +guest sees the file that host software, such as an editor, saves over it, and +requests fail with `EACCES` while anything but a regular file holds the name. +A mapped file inside a mapped directory is part of that directory's child +instead, whose mapping reaches the file's siblings. Guest targets that equal or +contain one another are rejected before boot. The number of mappings is bounded only by the 1024-byte budget that their tokens and `--cmdline` share on the kernel command line. -A snapshot captured with mappings requires the same mappings, in the same -order, with the same canonical host paths, kinds, targets, modes, and -filesystem identities, so a restore refuses a shared file that was replaced -since the capture. A repeatable `--mount-deny HOST_PATH` hides an existing file or directory inside a mapped root. With several mappings, or a shared file, each `--mount-deny`, `--mount-allow`, and `--mount-write` path must be absolute, and -it applies to the mapping whose root contains it. A shared file takes none of -them, because its mode applies to the whole file. Denied +it applies to the child whose root contains it, whichever of the child's +mappings reaches it. A file child takes none of them, because its mode applies +to the whole file. Denied names are omitted from directory listings and remain inaccessible through `..`, a symlink/junction, or another mount of the same virtio-fs device. Names may contain spaces, but not begin or end with one. @@ -836,8 +844,7 @@ duplicate, overlapping, and nested-mount rules are rejected before boot. A repeatable `--mount-allow HOST_PATH` exposes a path inside a denied path again, and a repeatable `--mount-write HOST_PATH` makes a path one of the only writable parts of an -`rw` mapping; see [Access policy](#access-policy). A snapshot captured with -these paths requires the same denied, allowed, and writable paths. +`rw` mapping; see [Access policy](#access-policy). In an `rw` mapping, the guest can create symbolic links, and OpenVMM stores each target exactly as given. The guest resolves links in its own namespace; OpenVMM never follows a link while resolving a host path, so a link to an @@ -847,29 +854,19 @@ are WSL-style reparse points, which Windows path resolution never follows. Treat links in a writable share as untrusted when host software later reads the directory. On a Linux host, the guest can also swap two names in an `rw` mapping -atomically with `renameat2(RENAME_EXCHANGE)`, as `exch` does, and a snapshot -captured right after the swap keeps each name with its object. A Windows host +atomically with `renameat2(RENAME_EXCHANGE)`, as `exch` does. A Windows host rejects rename flags with `EINVAL` and changes neither name. By default, OpenVMM performs every guest operation on the mapping as its own user. On a Linux host, `--mount-owner caller` instead performs each one as the guest caller's UID and GID and squashes guest root to the owner of the shared host directory or file, so files that guest root creates are owned by that -owner rather than by root or OpenVMM; see [File ownership](#file-ownership). A -snapshot captured with a mapping also requires the same `--mount-owner` mode. -Capture and restore inspect and reopen the guest's open files as OpenVMM's -user, so unless OpenVMM runs as root, they fail while the guest holds a file -that only its caller can reach or reopen, such as one open for writing. -A snapshot captured without a mapping may restore with one new `--mount` of a -directory, but not with several or with a file, because `nvx-rs run` restores -only snapshots without sandbox blocks, whose dormant slot takes no aggregate; -OpenVMM binds an aggregate to the dormant slot of a -[platform snapshot](#platform-filesystem-exports). After resume, mount it -explicitly inside the guest because the initramfs hook has already completed: +owner rather than by root or OpenVMM; see [File ownership](#file-ownership). -```sh -mkdir -p /mnt/host -mount -t virtiofs microvm /mnt/host -``` +`nvx-rs` snapshots no VM with mappings: `snapshot capture` captures a VM +without them, and `run --restore-snapshot` refuses the share options. A +restored sandbox maps the shares of its own `--mount` options, which +[`snapshot restore`](usage.md#snapshot-restore) takes, once its guest is +ready, as [Live host shares](#live-host-shares) describes. ## Experimental single-workload sandbox @@ -1013,8 +1010,9 @@ artifact name, SHA-256 digest, and package metadata still bind it to the layer. A run is the cold-filesystem bootstrap described in [the sandbox design](design/4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), which nvx-init -manages. It does not expose sandbox snapshot capture or restore, the -configuration region, or runtime RPC. Guest share targets cannot contain +manages. It does not expose snapshot capture or restore, which +[named snapshots](#named-snapshots) provide, the configuration region, or +runtime RPC. Guest share targets cannot contain whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace, including `0xa0`, while the workload's entrypoint and arguments, which nvx-init receives over the control session, may. @@ -1061,11 +1059,17 @@ nvx-rs/target/release/nvx-rs sandbox \ `--upper ram` refuses `--scratch`, and only `run` and `provision` take `--upper`: a provisioned sandbox keeps the upper that its configuration -records. OpenVMM admits a RAM upper only with exactly one read-only `distro` -block device, and refuses it for snapshot capture and restore, and the guest -refuses a RAM upper beside a scratch block device. With either upper, the -overlay keeps `metacopy=on` and `xino=on`, which `/sbin/nvx-sandbox-smoke` -checks. +records. A restore, of [`snapshot restore`](usage.md#snapshot-restore) or +`run --restore-snapshot`, takes `--upper` and `--ram-upper-percent` as +`provision` does, and hands its guest the RAM upper's cap in its activation, +since the snapshot keeps the kernel command line of its capture. OpenVMM +admits the RAM upper of a cold boot, `nvx_overlay_upper=ramfs`, only with +exactly one read-only `distro` block device, and refuses it for a snapshot +capture or restore: a restore over a RAM upper attaches the snapshot's dormant +scratch without a file instead, with OpenVMM's `--restore-absent-scratch`. The +guest of a cold boot refuses a RAM upper beside a scratch block device. With +either upper, the overlay keeps `metacopy=on` and `xino=on`, which +`/sbin/nvx-sandbox-smoke` checks. The `sandbox-ram-upper` scenario of `test-microvm --backend BACKEND` is the acceptance for the RAM upper. Like the @@ -1120,15 +1124,16 @@ cells, which the [case matrix](acceptance-case-matrix.md#cold-boots) lists. ### Named snapshots `nvx-rs snapshot capture` captures a VM that nvx-init runs, before any -workload, with a placeholder image, a dormant scratch, and no NIC, into a named -snapshot of a store, which `nvx-rs snapshot restore` and -`nvx-rs run --restore-snapshot` restore under the snapshot's lock, with an -image and an upper of their own; see -[`snapshot capture`](usage.md#snapshot-capture) and +workload, with a placeholder image, a dormant scratch, and a NIC only if +`--net` gives it one, into a named snapshot of a store, which +`nvx-rs snapshot restore` and `nvx-rs run --restore-snapshot` restore under the +snapshot's lock, with an image, an upper, and a network policy of their own; +see [`snapshot capture`](usage.md#snapshot-capture) and [`snapshot restore`](usage.md#snapshot-restore). The public `snapshot-core` and `snapshot-tiers` scenarios of `test-microvm --backend BACKEND` are their acceptance, with the Ubuntu layer and the scratch template that -`nvx-rs download` installs. The workload of each restore is a probe, a +`nvx-rs download` installs, and `untiered-network-policy` that of a restore's +network policy. The workload of each restore of the first two is a probe, a script that `/bin/sh -c` runs, which reports the guest's clock and uptime, fresh random values, its machine ID, host name, and identity, the lower and upper directories of its root's overlay, its block devices and network @@ -1195,10 +1200,25 @@ with a `snapshot-core-` or `snapshot-tiers-` prefix, holds the bounded command observations, the output of each command, and the outcome reports of the managed sandboxes, and that of `snapshot-tiers` the console of the guest that wrote its GPT disk. CI runs them on KVM, MSHV, and WHP with the public sandbox -scenarios. A snapshot that `snapshot capture` captures has no NIC, so its -restores take no network option, and the raw `untiered-network-policy` -scenario keeps the coverage of a restore's DNS, egress, forwards, and listener -publication until a capture can keep a NIC. +scenarios. + +The public `untiered-network-policy` scenario captures a snapshot with `--net`, +whose NIC has the complete deny-all policy, and restores it four times with +the Ubuntu layer over a RAM upper, each under a network policy of its own: the +capture's deny-all, and private-network ingress with allowed egress, through +`snapshot restore`, whose sandboxes it then stops and deprovisions; and egress +rules that admit DNS over IPv4 and one host service, and explicit +host-loopback access with a forward, under egress rules that admit DNS over +IPv6 alone, through `run --restore-snapshot`. Each restored guest must name +the DNS server of its policy in its resolver, and no gateway without one, run +the announcer of its listeners exactly when its policy publishes them, and +reach exactly the host services that its policy admits, and the host must +reach the guest's server through the forward, and through the port that the +policy publishes, on host loopback alone or to the host's private networks. +Before it provisions anything, the CLI must refuse a restore without +`--network-profile`, and one that names the NIC's address with `--net`. No +restore may change the snapshot or leave its sandbox or an OpenVMM process. +CI runs it with the other public sandbox scenarios. ### Live host shares @@ -1260,7 +1280,9 @@ A sandbox, of `run` or of `start`, delivers them once its guest agent is ready. OpenVMM exports every share as a child of one aggregate, which the guest mounts at `/run/nvx/hostfs/root`, where only root can enter, and the kernel command line -announces only the number of mappings, as `nvx_maps=COUNT`. Once the guest +announces only the number of mappings, as `nvx_maps=COUNT`, or, for a restore, +whose snapshot keeps the command line of its capture, the guest's activation. +Once the guest agent answers, `start` sends it the mapping table, in order, in as many requests as it takes. The agent mounts each entry at its target with its mode, and creates a missing target only on the container overlay, never inside @@ -1352,9 +1374,9 @@ or link reaches the share: link`), so the link cannot make the file writable. Each path must exist inside the share's host directory, and is attributed to -a share like `--mount-deny`. `sandbox` resolves it once, when it plans the -share, so it may cross a symbolic link or junction to a path inside the same -directory; `run` passes it to OpenVMM, which refuses a path that crosses one. +a share like `--mount-deny`. `sandbox` and `run` resolve it once, when they +plan the shares, so it may cross a symbolic link or junction to a path inside +the same directory. Its names may contain spaces, but not begin or end with one, nor contain other whitespace, colons, or backslashes. Allowed paths follow the share's write policy: without `--mount-write`, an @@ -1516,33 +1538,14 @@ relays the workload's TCP connections to that port whatever they carry, so the proxy decides what passes. The port can't be 53: OpenVMM answers DNS at the gateway's port 53, over TCP too, before it relays a connection to a proxy, so the proxy would receive nothing and the workload would reach the host's -resolver instead. NVX gives the workload no proxy settings: pass the proxy's -URL as the workload expects it, such as +resolver instead. NVX gives the workloads of a cold boot no proxy settings: +pass the proxy's URL as the workload expects it, such as `--inherit-default-environment --environment HTTPS_PROXY=http://10.0.0.1:3128` -on a managed `exec`. The plan of the NVX Rust API reports the URL as the guest -sees it, `SCHEME://GATEWAY:PORT`, or `GATEWAY:PORT` for a proxy without a -scheme, as `--network-proxy` names it. - -Earlier releases passed the network options to OpenVMM. The planner now -refuses, before launch: - -- a proxy with egress allowed, with egress rules, with `--network-ingress - allow`, with `--host-loopback allow` or without `--host-loopback`, with - forwards, or on port 53, which OpenVMM accepted; -- UDP forwards with egress denied by default, or with a deny rule that covers - the guest network, which OpenVMM accepted but whose replies its egress policy - dropped; -- `--host-loopback allow` with a `/30` guest network, which OpenVMM accepted; -- a network option without `--net`, a malformed `--net`, rule, forward, or - proxy, and a proxy that is not at the gateway, which OpenVMM refused as it - started. - -A policy with `--network-egress deny` and `--host-loopback deny`, and no allow -rules, `--network-ingress allow`, or `--network-proxy`, now attaches no network -device, where OpenVMM attached one that carried nothing. `start` refuses a -sandbox that an earlier release provisioned, whose configuration records the -network options rather than the typed policy; deprovision it and provision it -again. +on a managed `exec`. The guest of a restore whose own network policy is +proxy-only gives its workloads the proxy in their default environment instead, +as [`snapshot restore`](usage.md#snapshot-restore) describes. The plan of the +NVX Rust API reports the URL as the guest sees it, `SCHEME://GATEWAY:PORT`, or +`GATEWAY:PORT` for a proxy without a scheme, as `--network-proxy` names it. The `sandbox-proxy-only` scenario of `test-microvm --backend BACKEND` is the acceptance for proxy-only networking. Like the @@ -1601,9 +1604,8 @@ absence, and the distro policy that accepted it. It records the layer at its resolved path, so for a layer given through a link to another name, it also verifies the manifest beside the link's target. `start` checks the binding from metadata alone, without reading the layer again, and refuses a layer or -manifest that was replaced, modified, added, or removed since `provision`, and a -configuration without a binding, as older releases wrote; deprovision such a -sandbox and provision it again. `provision` and `start` refuse a manifest path +manifest that was replaced, modified, added, or removed since `provision`. +`provision` and `start` refuse a manifest path that they cannot look up, as through a link loop, rather than take it for no manifest. Keep provisioned layers and their manifests read-only, and never rewrite them in place. @@ -1631,11 +1633,8 @@ while the VM uses them. Images are attached raw: stage an image in a container format, such as a VHD, as a raw file first. `provision` records the guest memory of `--memory-mib`, which it takes from 1 -to 17592186044415 MiB, as `run` does. `provision` in `nvx.py` and in earlier -releases of `nvx-rs` recorded any integer, such as 0 or -1, whose RAM OpenVMM -then refused at every start. `start` refuses a configuration that records -memory outside that range before it launches OpenVMM; deprovision such a -sandbox and provision it again. +to 17592186044415 MiB, as `run` does, and `start` refuses a configuration that +records memory outside that range before it launches OpenVMM. Lifecycle transitions fail closed: `start` rejects a sandbox that is already running; `exec` and `stop` require a live OpenVMM process; and `deprovision` @@ -1663,13 +1662,9 @@ terminates its OpenVMM without a guest shutdown, waits up to `--timeout` seconds for it to exit, and warns that it did. A `stop` that cannot determine the OpenVMM process, whose terminated OpenVMM does not exit in time, or whose wait Ctrl-C ends, keeps the sandbox's runtime record, and the next `stop` -retries. The store reads -runtime records of its own format only, so -stop every running sandbox with the NVX version that started it before you -switch versions: a version with another format fails every transition of such -a sandbox with a message that names the format of its record. An OpenVMM that -an interrupted `start` of an earlier version left behind has no record, and -that version asked for it to be ended by hand: end it before the switch too. +retries. The store's records belong to the version of NVX that wrote them, so +stop and deprovision each sandbox with the version that started it before you +switch versions, as [Upgrades](#upgrades) says. `provision`, `start`, `stop`, and `deprovision` hold the store's lock, `sandbox.lock` in the state directory, until they return, so the transitions @@ -1679,9 +1674,11 @@ OpenVMM while the other then fails because the sandbox is already running. `exec` holds the lock only while it finds the running OpenVMM, so a long workload does not delay `stop`. One `exec` at a time runs in a sandbox, across processes: an `exec` takes the sandbox's exec lock, `exec.lock` in the state -directory, before it finds the running OpenVMM, and holds it until the workload -ends. An `exec` that finds it held tries again until `--wait` seconds, 60 by -default, have passed, and then fails with `sandbox-busy`. `stop` takes no exec +directory, before it finds the running OpenVMM, and holds it until the command +ends: once its workload has ended, or, after Ctrl-C or once `--timeout` has +passed, without waiting for the guest to end the workload. An `exec` that +finds it held tries again until `--wait` seconds, 60 by default, have passed, +and then fails with `sandbox-busy`. `stop` takes no exec lock, so it proceeds while a workload runs. A provisioned state directory keeps `sandbox.lock`, and `exec.lock` from its first `exec` on, and `deprovision` removes them with NVX's other files. A @@ -1744,8 +1741,10 @@ path is limited to 4096 UTF-8 bytes; the combined execution request must also fit the existing 64 KiB control-payload bound. A managed `exec` streams its standard input to the workload, which sees it end when the command's own input ends, and the workload's standard output and standard error back as -they arrive, without a limit on their size. Ctrl-C cancels the workload, and -`exec` exits with status 130. +they arrive, without a limit on their size; the workload must end within +`--timeout`, which bounds the whole exchange, or `exec` fails with +`timed-out`. Ctrl-C asks the guest to end the workload, and `exec` exits with +status 130 without waiting for it, while the sandbox runs on. The `managed-exec-config` scenario of `test-microvm --backend BACKEND` is the authoritative acceptance for these public options. It invokes @@ -1799,10 +1798,10 @@ template, so it has the same artifact requirements. It checks that: is on scratch; - `/sbin/nvx-sandbox-smoke` passes its security profile checks in a sandbox provisioned with `--hostname`, after earlier requests; -- a request that writes 2 MiB, more than the 1 MiB that version 1 of NVXC - forwarded, streams all of it and exits with status 0 and an `exit` outcome - report, and a request whose `--outcome-report` path exists fails before its - workload runs and leaves the file intact; +- a request that writes 2 MiB streams all of it, with no limit on its output, + and exits with status 0 and an `exit` outcome report, and a request whose + `--outcome-report` path exists fails before its workload runs and leaves the + file intact; - `stop` ends OpenVMM and its control socket or pipe, leaves only `config.json`, `sandbox.lock`, `exec.lock`, `openvmm.log`, and a successful `outcome.json`, and leaves a cleanly unmounted scratch file system, after @@ -1856,3 +1855,22 @@ On Linux, NVX starts OpenVMM with the caller's umask without the owner's bits, so OpenVMM's reports keep their owner's read and write permissions whatever that umask, unless their directory has a default POSIX ACL, which Linux applies instead of the umask. Neither OpenVMM nor NVX uploads these files. + +## Upgrades + +NVX keeps no state from one version to the next, and migrates none: each +version starts with fresh state. A version keeps the state directories of its +sandboxes and its snapshot stores in formats of its own, and may refuse those +that another version wrote, as it refuses a sandbox's configuration of another +format with `format-unsupported`, or fail to read their records at all. A +snapshot restores only under the runtime identity that captured it, which +names the OpenVMM, kernel, and Alpine initramfs of its artifacts and the +contracts of NVX, so a restore refuses it once an upgrade has installed new +artifacts. To upgrade: + +1. With the version that you leave, stop and deprovision each sandbox, which + also ends the VM of a start that was interrupted. +2. Switch to the new version of `nvx-rs` and its artifacts. +3. Capture the snapshots again, into new stores, since a store never replaces + a snapshot's name, and provision or restore the sandboxes again. No command + removes a snapshot: remove an old store once no sandbox restores from it. diff --git a/doc/usage.md b/doc/usage.md index 275e23a42..781d7414e 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -2,9 +2,12 @@ NVX has two command-line interfaces, which run from the repository root: -- [`nvx-rs`](#nvx-rs) runs NVX microVMs and sandboxes: [`run`](#run) boots or - restores a microVM on the terminal, [`sandbox`](#sandbox) runs or manages - workloads over EROFS layers and a private ext4 scratch or tmpfs upper, and +- [`nvx-rs`](#nvx-rs) runs NVX microVMs and sandboxes: [`run`](#run) boots a + microVM on the terminal, or restores a named snapshot once, + [`sandbox`](#sandbox) runs or manages workloads over an EROFS image or a GPT + disk and a private ext4 scratch or tmpfs upper, + [`snapshot capture`](#snapshot-capture) captures a named snapshot, from which + [`snapshot restore`](#snapshot-restore) starts managed sandboxes, and [`download`](#download) installs the release of the checkout, which `run` and `sandbox` also download into a checkout that holds no artifacts. - `scripts/nvx.py`, with Python 3.10 or newer, builds, tests, benchmarks, and @@ -45,6 +48,11 @@ the guest artifacts in `build/` and OpenVMM in `openvmm/target/release/`. The `nvx-rs` of an extracted release package, `bin/nvx-rs`, uses the release's artifacts; see [Artifact root](../nvx-rs/README.md#artifact-root). +`nvx-rs` keeps the state of its sandboxes and snapshots in the directories that +their commands name, `--state-dir` and `--store-dir`, in formats of its own, +which another version of NVX may not read and never migrates: upgrade with +fresh state, as [Upgrades](run.md#upgrades) says. + The [`test-microvm`](#test-microvm) scenarios run `nvx-rs` too. They run, in this order: @@ -99,7 +107,7 @@ each code, which callers of the `nvx` library see in its place. | `sandbox-not-running` | 1 | `exec` or `stop` found the VM of a provisioned sandbox not running. | | `sandbox-running` | 1 | `start` or `deprovision` found the sandbox's VM, or a launch of it, running. | | `sandbox-exited` | 1 | The sandbox's OpenVMM exited without a `stop`: the transition that finds it reports it and clears its runtime record, and `start` launches the sandbox again. | -| `sandbox-busy` | 1 | Another lifecycle transition held the sandbox's lock until `--timeout` passed, or another `exec` held its exec lock until `--wait` passed. | +| `sandbox-busy` | 1 | Another lifecycle transition held the sandbox's lock, or another command the lock of a snapshot or of a store's captures, until `--timeout` passed, or another `exec` held the sandbox's exec lock until `--wait` passed. | | `runtime-mismatch` | 1 | The sandbox's VM runs under another runtime identity than the artifacts', whose differing parts the message names, so only `stop`, which terminates it, applies. | | `vmm-indeterminate` | 1 | Whether the sandbox's OpenVMM runs cannot be determined, so its records remain for a later attempt. | | `format-unsupported` | 1 | A record or report has a format that this build does not read, such as the state of a sandbox that another build provisioned or started. | @@ -167,7 +175,7 @@ nvx-rs/target/release/nvx-rs run | `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot, in the range of `--memory-mib`. A restore refuses it. | | `--processors {1,2,4,8}` | `1` | Select the microVM processor count, written as one of the choices. A restore refuses it: its VM keeps the processors that the snapshot had online. | | `--cpu-profile ID` | `auto` | Select the guest's [CPU profile](#cpu-profiles): `auto` for the built-in profile of the host's CPU, which falls back with a warning to a development profile derived from this host on an Intel or AMD CPU that no built-in profile serves, a built-in profile ID, or `host` for that development profile. A restore refuses it: it uses the snapshot's profile. | -| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose a host directory, or a single regular host file without its directory, at the guest target: an absolute, canonical path other than `/`, without whitespace, control characters, double quotes, backslashes, `=`, or any UTF-8 byte that Linux treats as whitespace, such as the `0xa0` in `à`. Cold boots also protect the system paths listed below. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, as the children of an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets; a shared file always uses the aggregate. Targets and host paths must not overlap, and the shares' kernel command-line tokens must fit its 1024-byte budget. An `rw` mapping accepts guest-created symbolic links, which the host never follows. A restore refuses it and the other share options. | +| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose a host directory, or a single regular host file without its directory, at the guest target: an absolute, canonical path other than `/`, without whitespace, control characters, double quotes, backslashes, `=`, or any UTF-8 byte that Linux treats as whitespace, such as the `0xa0` in `à`. Cold boots also protect the system paths listed below. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, through an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets. Targets must not overlap, but host paths may: a mapped path inside a mapped directory shares that directory's child, unless it is read-only inside a writable part of it, which is refused; see [virtio-fs host mapping](run.md#virtio-fs-host-mapping). The shares' kernel command-line tokens must fit its 1024-byte budget. An `rw` mapping accepts guest-created symbolic links, which the host never follows. A restore refuses it and the other share options. | | `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a mounted host root; repeat to deny multiple paths. It may name the root itself when `--mount-allow` paths inside it remain. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | | `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path again; the hidden directories on the way list only the entries that lead to it. Repeat to allow multiple paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. See [Access policy](run.md#access-policy). | | `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` mapping; the guest's writes anywhere else in it fail with `EROFS`. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | @@ -175,15 +183,15 @@ nvx-rs/target/release/nvx-rs run | `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. A restore takes its NIC, with these addresses, from the snapshot, which [`snapshot capture --net`](#snapshot-capture) gives one, and rejects this option. | | `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net` on a fresh boot, and without it to restore a snapshot with a NIC. | | `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | -| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture rejects `allow`. | +| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). | | `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | | `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | | `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions from JSON. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | -| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; it needs a `--net` prefix of `/29` or shorter, which OpenVMM checks; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture rejects `allow`. | +| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; it needs a `--net` prefix of `/29` or shorter, which OpenVMM checks; see [Host-loopback access](run.md#host-loopback-access). | | `--network-proxy IPV4:TCP-PORT` | none | Allow one explicit host TCP proxy endpoint. | -| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Snapshot capture rejects forwards. | +| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. | | `--outcome-report PATH` | none | Write a bounded local JSON outcome report to a new file in an existing plain directory. A cold boot rejects an existing destination before launch and never replaces a file or follows a destination symlink. A missing report or publication failure is diagnosed without replacing OpenVMM's exit status. A restore refuses it. | -| `--cmdline TEXT` | empty | Append kernel parameters; `nvx_*` and `tsc=` tokens are reserved. Cold boots reject double quotes and a standalone `--` token, which the VM control console cannot represent. A restore refuses it. | +| `--cmdline TEXT` | empty | Append kernel parameters. The tokens that NVX renders for the guest's root, view, and shares, such as `nvx_hostname`, `nvx_machine_id`, and `nvx_maps`, and `virtfs_*` tokens, are refused, and OpenVMM refuses those that it owns; see [Effective Command Line](design/5.1-boot-protocol.md#512-effective-command-line). Cold boots reject double quotes and a standalone `--` token, which the VM control console cannot represent. A restore refuses it. | | `--restore-snapshot NAME` | none | Restore the snapshot `NAME` of `--store-dir`, which [`snapshot capture`](#snapshot-capture) published, in place of a cold boot, as described below. A raw OpenVMM snapshot directory is refused. | | `--store-dir PATH` | none | The named snapshot store of `--restore-snapshot`, which a restore requires. | | `--state-dir PATH` | none | The directory of a restore's sandbox, which a restore requires: one that does not exist, outside the store, on the store's filesystem. The run creates it, and removes it once it ends. A cold boot refuses it. | @@ -200,11 +208,12 @@ require networking. A proxy is the guest's sole egress path, so it cannot be combined with `--network-egress-allow` rules for direct connections. `nvx-rs run` checks its options, prints the OpenVMM command of a cold boot -after `>> `, and unless `--dry-run` is given, runs OpenVMM with the standard -streams. A cold boot attaches the guest's console, the boot console `hvc1`, to -them, so that the terminal's input reaches the guest through it rather than -through the byte channel, `hvc0`, whose output, such as the kernel's early -messages, goes to standard error. It exits +after `>> `, and unless `--dry-run` is given, runs OpenVMM in a private +temporary VM directory, which it removes once OpenVMM has exited, on the +terminal. A cold boot attaches the guest's console, the boot console `hvc1`, +to the standard streams, so that the terminal's input reaches the guest through +it rather than through the byte channel, `hvc0`, whose output, such as the +kernel's early messages, goes to standard error with OpenVMM's log. It exits with status 1 for an error, 2 for a usage error, and 130 after Ctrl-C, and otherwise with the status that OpenVMM returns. Its errors and usage errors carry their [error codes](#error-codes), as `error[CODE]: MESSAGE` and @@ -227,24 +236,25 @@ restores one, but once: it provisions a managed sandbox in `--state-dir`, a new directory outside the store, on the filesystem of the snapshot's files, which the restore links into it, with the image, upper, and workload of its restore options, and starts it from the snapshot under the snapshot's lock. It -prints no OpenVMM command, and forwards the workload's output. Once the -workload has exited with status 0 within `--timeout`, 60 seconds by default, it -stops the VM; it then removes the sandbox and `--state-dir`, also after a -failure or Ctrl-C, except as the next paragraph says, and exits with the -workload's status, or with status 1 for a restore that fails, and 130 after -Ctrl-C. It refuses a raw OpenVMM snapshot directory, which -`--restore-snapshot` named before, without reading it, a `--state-dir` that -exists or lies in the store, a workload that `snapshot restore` refuses, and -the options that the snapshot or its sandbox fixes: `--guest`, -`--memory-capacity-mib`, `--processors`, `--cpu-profile`, `--net`, the share -options, `--outcome-report`, `--cmdline`, and `--dry-run`. A cold boot refuses +prints no OpenVMM command, and forwards the workload's output, as it comes; the +workload reads no standard input. Once the workload has exited with status 0 +within `--timeout`, 60 seconds by default, it stops the VM; it then removes the +sandbox and `--state-dir`, also after a failure or Ctrl-C, except as the next +paragraph says, and exits with the workload's status, or with status 1 for a +restore that fails, and 130 after Ctrl-C. `--restore-snapshot` names only a +snapshot of the store, so `run` refuses a raw OpenVMM snapshot directory +without reading it. It also refuses a `--state-dir` that exists or lies in the +store, a workload that `snapshot restore` refuses, and the options that the +snapshot or its sandbox fixes: `--guest`, `--memory-capacity-mib`, +`--processors`, `--cpu-profile`, `--net`, the share options, +`--outcome-report`, `--cmdline`, and `--dry-run`. A cold boot refuses `--store-dir`, `--state-dir`, and the restore options. A restore's NIC comes from the snapshot: `--net` belongs to the cold boot and to [`snapshot capture`](#snapshot-capture), which gives a snapshot its NIC. The restore of a snapshot without a NIC takes no network options, and that of a -snapshot with a NIC installs the policy of its network options, which need -`--network-profile`, as the policy of the NIC, with its DNS server, egress, -forwards, and published listeners. +snapshot with a NIC installs them, with `--network-profile`, as the restore's +own network policy, with its DNS server, egress, forwards, and published +listeners, as [`snapshot restore`](#snapshot-restore) describes. A `run` that dies before it removes its sandbox, as `kill -9` ends it, leaves `--state-dir` with the sandbox's files and its view of the snapshot, and its @@ -332,9 +342,12 @@ errors and usage errors carry their [error codes](#error-codes), as `error[CODE]: MESSAGE` and `nvx-rs sandbox: error[CODE]: MESSAGE`. An `exec`, and the workload of a `run`, stream the command's standard input to the workload until it ends, and the workload's standard output and error back as -they arrive, without a limit; a command whose input is the null device gives -the workload none. An error of the command's own follows the workload's output -on a new line, and Ctrl-C cancels the workload. Options take their values as +they arrive, without a limit on their size; a command whose input is the null +device gives the workload none. The workload must end within `--timeout`, or +the command fails with `timed-out`. An error of the command's own follows the +workload's output on a new line. Ctrl-C asks the guest to end the workload, +and the command exits with status 130 without waiting for it: an `exec` leaves +its sandbox running, while a `run` ends its VM. Options take their values as those of [`run`](#run) do, and their names cannot be abbreviated either. | Option | Default | Description | @@ -344,15 +357,15 @@ those of [`run`](#run) do, and their names cannot be abbreviated either. | `--layer-format {erofs,gpt}` | `erofs` | Select the image format of `run` or `provision` explicitly, without probing the file or inferring its format from its name. Other sandbox operations reject this option. | | `--scratch PATH` | required for `run` and `provision` with a scratch upper | Attach a preformatted ext4 scratch image as the overlay's writable upper. `--upper ram` refuses it. | | `--upper {scratch,ram}` | `scratch` | Select the overlay's writable upper for `run` or `provision`: `scratch`, the image of `--scratch`, or `ram`, a tmpfs in guest memory with no scratch image, which loses the workload's writes when the VM stops. The default RAM cap is half of guest memory, which `--ram-upper-percent` changes. See [Overlay upper](run.md#overlay-upper). | -| `--ram-upper-percent PERCENT` | `50` | Set the RAM upper's cap of `run` or `provision` to an ASCII decimal integer from 1 through 90 percent of VM memory. Requires `--upper ram`; other sandbox operations reject this option. The typed configuration of `provision` records the cap, image format and path, and upper kind, which each start reads unchanged. Older configuration formats are refused rather than migrated. | +| `--ram-upper-percent PERCENT` | `50` | Set the RAM upper's cap of `run` or `provision` to an ASCII decimal integer from 1 through 90 percent of VM memory. Requires `--upper ram`; other sandbox operations reject this option. The typed configuration of `provision` records the cap, image format and path, and upper kind, which each start reads unchanged. A configuration of another format is refused, not migrated; see [Upgrades](run.md#upgrades). | | `--state-dir PATH` | required for managed operations | Select persistent sandbox state. `run`, which keeps no state, rejects this option. | | `--entrypoint PATH` | `/bin/sh` | Select the absolute entrypoint of the workload of `run` or `exec`, which runs it over the control protocol. `provision`, which runs no workload, refuses one with whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. | | `--arg VALUE` | none | Append one nonempty argument of the workload of `run` or `exec`. Repeat to pass multiple arguments. `provision` refuses one with whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. | | `--hostname NAME` | `nvx-sandbox` | Set the workload UTS hostname. | | `--workload-user UID:GID` | `65534:65534` | Select the fixed non-root workload identity for `run` or `provision`. Both IDs must be ASCII decimal integers from 1 to 4294967294, without signs, whitespace, or underscores. | | `--memory-mib MIB` | `256` | Set guest memory in MiB, an ASCII decimal integer from 1 to 17592186044415, without signs, whitespace, or underscores. | -| `--timeout SECONDS` | `60` | Set the timeout for each managed control request and its response in `run`, `start`, `exec`, and `stop`, the longest time that `run` and `stop` wait for the VM to exit once the guest shuts it down, and the longest time that `provision`, `start`, `stop`, and `deprovision` wait for another lifecycle transition of the sandbox. It is a decimal number, with an optional fraction and exponent, that must be finite and greater than 0. | -| `--wait SECONDS` | `60` | Set the longest time that a managed `exec` waits for the `exec` that another `nvx-rs` process runs in the sandbox to end, since a sandbox runs one `exec` at a time; it then fails with `sandbox-busy`. `stop` never waits for an `exec`. Like `--timeout`, which still bounds each wait for another lifecycle transition, it is a decimal number that must be finite and greater than 0. | +| `--timeout SECONDS` | `60` | Set the timeout for each managed control request and its response in `run`, `start`, `exec`, and `stop`, within which the workload of `run` and `exec`, whose output streams until it ends, must end; the longest time that `run` and `stop` wait for the VM to exit once the guest shuts it down; and the longest time that `provision`, `start`, `exec`, `stop`, and `deprovision` wait for another lifecycle transition of the sandbox. It is a decimal number, with an optional fraction and exponent, that must be finite and greater than 0. | +| `--wait SECONDS` | `60` | Set the longest time that a managed `exec` waits for the `exec` that another `nvx-rs` process runs in the sandbox to end and release the sandbox's exec lock, since a sandbox runs one `exec` at a time; it then fails with `sandbox-busy`. `stop` never waits for an `exec`. Like `--timeout`, which still bounds each wait for another lifecycle transition, it is a decimal number that must be finite and greater than 0. | | `--exec-timeout-ms MILLISECONDS` | `0` | Set the guest's timeout of the workload of `run` or `exec` to an ASCII decimal integer from 0 to 4294967295, without signs, whitespace, or underscores; zero disables the workload deadline. This is separate from the finite host `--timeout` control deadline. | | `--cwd GUEST_PATH` | `/` | Set an absolute working directory inside the workload root for `run` or `exec`. It is resolved after the workload identity and root are applied; missing, inaccessible, or non-directory paths fail the workload launch. Default and layered environments point `PWD` at this directory; an exact replacement environment controls `PWD` itself. | | `--environment KEY=VALUE` | omitted | Set the exact environment of the workload of `run` or `exec`. Repeat for multiple entries. Empty values, spaces, additional equals signs, and UTF-8 are preserved. Inline values are visible in the invoking host process arguments; use `--environment-file` for sensitive values. | @@ -372,7 +385,7 @@ those of [`run`](#run) do, and their names cannot be abbreviated either. | `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Requires explicit `--network-egress`; deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | | `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions for `run` or `provision`. Managed provision persists the compiled rules in its network policy, not this path. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | | `--host-loopback {allow,deny}` | guest to host | Control access between the workload and host loopback services for `run` or `provision`. Without the option, the workload reaches host loopback through its gateway, subject to the egress policy, and no port is published; `deny` closes both directions. Explicit `allow` is bidirectional: the workload reaches host loopback through its gateway, and every TCP and UDP port on which the workload listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; see [Host-loopback access](run.md#host-loopback-access). | -| `--network-proxy IPV4:TCP-PORT` | none | Make the network proxy-only: the workload reaches the HTTP or HTTPS proxy that listens on host `127.0.0.1:TCP-PORT` at `IPV4:TCP-PORT`, where `IPV4` must be the guest gateway, and nothing else. `TCP-PORT` can't be 53, where OpenVMM answers DNS at the gateway. Requires `--network-egress deny` and `--host-loopback deny`, and refuses egress rules, `--network-ingress allow`, and forwards. NVX gives the workload no proxy settings; see [Proxy-only networking](run.md#proxy-only-networking). | +| `--network-proxy IPV4:TCP-PORT` | none | Make the network proxy-only: the workload reaches the HTTP or HTTPS proxy that listens on host `127.0.0.1:TCP-PORT` at `IPV4:TCP-PORT`, where `IPV4` must be the guest gateway, and nothing else. `TCP-PORT` can't be 53, where OpenVMM answers DNS at the gateway. Requires `--network-egress deny` and `--host-loopback deny`, and refuses egress rules, `--network-ingress allow`, and forwards. NVX gives the workloads of a cold boot no proxy settings; see [Proxy-only networking](run.md#proxy-only-networking). | | `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Requires `--host-loopback allow`. A UDP forward also requires egress allowed by default and no deny rule that covers the guest network, which carries its replies; OpenVMM lets the replies on a forwarded TCP connection through whatever the egress policy. | | `--outcome-report PATH` | none | Write a bounded local JSON outcome report: for `run`, OpenVMM's report of its VM, which the guest stopped after the workload; for `exec`, the workload's outcome. | | `--cmdline TEXT` | empty | Append non-sandbox kernel parameters, which `nvx-rs` reads as Linux does: split at whitespace, which includes the `0xa0` in `à`, outside double quotes, with names that match whether they spell a dash or an underscore. Parameters whose names start with `nvx_` or `virtfs_` are reserved and refused, even inside a quoted value, and so are a `--` or unclosed quote that hides the tokens that OpenVMM appends, and double quotes, which the VM control console cannot represent; see [Experimental single-workload sandbox](run.md#experimental-single-workload-sandbox). The guest powers off if its kernel applies a clock parameter that the time ABI forbids, such as `tsc=unstable`. | @@ -382,13 +395,14 @@ those of [`run`](#run) do, and their names cannot be abbreviated either. artifacts before they launch, as `nvx-rs run` does. A `run` keeps no state: it launches the VM that a `provision` and a `start` of its options would launch, in a temporary VM directory that it removes, executes its workload -once, as `exec` does, has the guest shut the VM down, which it terminates if -it does not exit within `--timeout` seconds, and then prints the workload's -standard output and error and exits with its status; see +once, as `exec` does, with its streams, has the guest shut the VM down, which +it terminates if it does not exit within `--timeout` seconds, and exits with +the workload's status; see [Experimental single-workload sandbox](run.md#experimental-single-workload-sandbox). -See [Run](run.md) for artifact preparation, the security boundary, and current -snapshot/configuration limitations. +See [Run](run.md) for artifact preparation, the security boundary, the +[managed lifecycle](run.md#managed-lifecycle), and +[named snapshots](run.md#named-snapshots). ### `snapshot capture` @@ -420,19 +434,21 @@ the NIC another address, and installs a network policy of its own on it, as [`snapshot restore`](#snapshot-restore) and [`run`](#run) describe. Only a successful capture whose own VMM exits within `--timeout` (60 whole -seconds by default) can publish. The immutable named directory contains exactly -`key.bin`, `snapshot.json` and `openvmm/`; an existing name is never replaced. -Capture's own VM directory is then removed. A timed-out VM is stopped but its -capture is never finished. Other control or launch-record failures also stop -the command's verified VM without finishing it. The next capture command using -this store ends and removes an interrupted attempt through ordinary launch -recovery, without resuming or publishing it. Indeterminate recovery retains its -evidence and reports an error. Unknown files or nonempty unjournaled capture-slot -contents are kept for inspection, not deleted; an empty launch lock alone is -harmless. Known host and OpenVMM atomic staging files are recovered only with -a valid launch journal. Captures in one store serialize; snapshot locks remain -separate from their immutable directories. [`snapshot restore`](#snapshot-restore) -starts managed sandboxes from them. +seconds by default) can publish, and prints `Snapshot captured`. The immutable +named directory contains exactly `key.bin`, `snapshot.json` and `openvmm/`; an +existing name is never replaced. Capture's own VM directory is then removed. A +timed-out VM is stopped but its capture is never finished. Other control or +launch-record failures also stop the command's verified VM without finishing +it. The next capture command using this store ends and removes an interrupted +attempt through ordinary launch recovery, without resuming or publishing it. +Indeterminate recovery retains its evidence and reports an error. Unknown files +or nonempty unjournaled capture-slot contents are kept for inspection, not +deleted; an empty launch lock alone is harmless. Known host and OpenVMM atomic +staging files are recovered only with a valid launch journal. Captures in one +store serialize, and one that waits for another longer than `--timeout` fails +with `sandbox-busy`; snapshot locks remain separate from their immutable +directories. [`snapshot restore`](#snapshot-restore) starts managed sandboxes +from them. ### `snapshot restore` @@ -468,30 +484,46 @@ placeholder image and dormant scratch: the image must fit the placeholder, with its block sizes, and a scratch upper must be as long as the dormant scratch. The VM keeps the processors that the snapshot had online, and `--memory-mib` must be a multiple of 128 between the snapshot's boot memory and -its capacity. The sandbox's NIC comes from the snapshot, with its presence and -its address: `--net` belongs to [`snapshot capture`](#snapshot-capture), which -gives a snapshot its NIC, and a restore takes no `--net`. A restore of a -snapshot without a NIC refuses the network options before it provisions -anything. That of a snapshot with a NIC needs `--network-profile`, and installs -the policy of its network options, at the NIC's address, as the restore's own -network policy: OpenVMM enforces its egress, ingress, host-loopback access, and -forwards, and the guest applies its DNS server and announces its listeners, -when it publishes them, as it activates. +its capacity. The snapshot's key records the runtime identity that captured +it: the OpenVMM, kernel, and Alpine initramfs of the artifacts, and the +contracts of NVX. A restore under another one, as after an upgrade that +installs new artifacts, is refused, and the snapshot must be captured again; +see [Upgrades](run.md#upgrades). + +The sandbox's NIC comes from the snapshot, with its presence and its address: +`--net` belongs to [`snapshot capture`](#snapshot-capture), which gives a +snapshot its NIC, and a restore takes no `--net`. A restore of a snapshot +without a NIC refuses the network options before it provisions anything. That +of a snapshot with a NIC needs `--network-profile`, and installs the policy of +its network options, at the NIC's address, as the restore's own network policy, +in place of the capture's deny-all. OpenVMM installs its egress, ingress, +host-loopback access, and forwards, whose host ports it binds, before any vCPU +runs. The guest applies the rest as it activates, before it reports Ready, so +before the host sends it the mapping table of the sandbox's shares and starts +the workload: it names in `/etc/resolv.conf` the DNS server that a cold boot +under the policy would name, if any; it starts announcing its listeners when +the policy publishes them, through private-network ingress or explicit +host-loopback access; and it gives the proxy of a proxy-only policy, as +`--network-proxy` names it, to the workloads of the sandbox as `http_proxy`, +`https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY` in their default environment. A +guest that cannot apply them ends its VM, which fails the restore. Each start +of the sandbox installs the policy again. Each start holds the snapshot's exclusive lock, which it waits for up to -`--timeout`, from before it links the snapshot's files into the sandbox's view -of it, `openvmm/` in the state directory, until the start ends or the VM has -exited, and `sandbox deprovision` holds it while it removes the view, so it -needs the snapshot's store: concurrent restores of one snapshot start one at a -time. Once the guest is ready, it maps the sandbox's shares and runs the -workload, whose output the command forwards, and waits for it to exit with -status 0 within `--timeout`, as the default workload, `/bin/sh`, does at once; -any other end fails the start, which ends the VM. The sandbox then runs as a -started one does, for `sandbox exec` and `sandbox stop`. A start journals its -launch as a cold start does, so a restore whose command dies before the guest -is ready never completes, and the next command of the sandbox ends its VM. -[`run --restore-snapshot`](#run) restores a snapshot once, in a sandbox that it -removes. +`--timeout` before it fails with `sandbox-busy`, from before it links the +snapshot's files into the sandbox's view of it, `openvmm/` in the state +directory, until the start ends or the VM has exited, and `sandbox deprovision` +holds it while it removes the view, so it needs the snapshot's store: +concurrent restores of one snapshot start one at a time. Once the guest is +ready, it maps the sandbox's shares and runs the workload, whose standard input +ends at once and whose output the command forwards as it comes, and waits for +it to exit with status 0 within `--timeout`, as the default workload, `/bin/sh`, +does at once; any other end fails the start, which ends the VM. The sandbox +then runs as a started one does, for `sandbox exec` and `sandbox stop`. A start +journals its launch as a cold start does, so a restore whose command dies +before the guest is ready never completes, and the next command of the sandbox +ends its VM. [`run --restore-snapshot`](#run) restores a snapshot once, in a +sandbox that it removes. A restore that fails before its guest is ready, in `snapshot restore`, a later `sandbox start`, or `run --restore-snapshot`, fails with the restore's evidence