From f4d377b3a8da763d1e1fdeae6b1fce6526974805 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 08:49:27 -0800 Subject: [PATCH 1/2] cli: restore named snapshots once with run (SL3.8b) `run --restore-snapshot PATH` handed a raw OpenVMM snapshot directory to OpenVMM with the options of a cold boot, outside the store, the locks, and the restore driver of the named snapshots that `snapshot capture` captures and `snapshot restore` restores. It now takes the name of a snapshot of `--store-dir` and restores it once, through the path of `snapshot restore`: `sandbox::restore_once` provisions a managed sandbox in `--state-dir`, starts it from the snapshot under the snapshot's lock, through the launch journal and the blocking restore driver, which forwards the workload's output, stops the VM once the workload has exited with status 0, deprovisions the sandbox once it is provisioned, whatever happened, and ends with the workload's status. The restore inputs are options of `snapshot restore`, taken from its own option list and parsed by `sandbox` for both: `--state-dir`, `--layer`, `--layer-format`, `--scratch`, `--upper`, `--ram-upper-percent`, `--entrypoint`, `--arg`, `--hostname`, `--workload-user`, and `--timeout`, so a restore of `run` takes none that `snapshot restore` refuses, such as the exec-only `--wait`; `--memory-mib` takes the sandbox's default for a restore. Both commands prepare a restore through one function, into which `snapshot restore`'s own preparation moves: it resolves the store, refuses a state directory in the store, checks the workload as the exec request that each restored start sends, which may thus hold whitespace and quotes, and opens the snapshot, before anything is provisioned. The restore links the snapshot's files into `--state-dir`, which must therefore be on the store's filesystem, and outside the store, which SL3.8a requires of every restored sandbox, at its provision, start, and deprovision. Since the run removes it, it must not exist, and an empty path, which names the working directory, is refused: a sandbox that it names is never removed. A cold boot refuses `--store-dir`, `--state-dir`, and the restore inputs, and a restore refuses the options of a cold boot that the snapshot or its sandbox fixes: `--guest`, `--memory-capacity-mib`, `--processors`, `--cpu-profile`, `--net`, the share options, `--outcome-report`, `--cmdline`, and `--dry-run`. A raw OpenVMM snapshot is refused before anything is read or launched: `--restore-snapshot` without `--store-dir`, or with a value that is no snapshot name, such as a path, fails with a message that says that the option names a snapshot of the store that `snapshot capture` captured. `--restore-processors`, `--restore-memory-mib`, `--restore-ready-path`, and `--restore-network-policy` only served the raw restore, and go: the named restore takes its processors from the snapshot's key and its memory from `--memory-mib`, owns its restore-ready endpoint, and always installs the restore's own network policy. A restored workload that does not exit with status 0 still fails the start, which ends its VM and launch, but the start that `run` asks for ends with the workload's status, which `run` exits with, as `sandbox run` does: 128 plus the signal's number for a workload that a signal ends. `snapshot restore`, and each later `sandbox start` of a restored sandbox, keep their error, such as "the restored workload ended with Exit status N", and status 1. The snapshot fixes the NIC. The restore of a snapshot with a NIC takes the NIC's address, and lowers the run's DNS server, egress, forwards, and published listeners, which need `--network-profile`, into the restore's own policy, as `sandbox` lowers them; that of a snapshot without one refuses network options before it provisions anything. `snapshot capture` gives its VM no NIC today, so only fake-backed tests cover the policy that reaches the guest of a networked restore. A `run` that dies before it removes its sandbox, as `kill -9` ends it, leaves `--state-dir` and its VM, which keeps running until a command of the sandbox ends it, and the snapshot's lock goes with the `run`. While the workload runs, the start is unfinished, and `sandbox deprovision` ends the VM, through the recovery of the start's launch journal, and removes the directory; once the workload has exited with status 0, `sandbox stop` must end the VM first, as it must after Ctrl-C while the run stops the VM. usage.md says so. The raw restore's code goes with it: the restore body and console reattachment of `run`, the OpenVMM arguments of the network options, the reader of a snapshot manifest's attachments that only it used, and `sandbox::one_shot`, which has run OpenVMM on the terminal for it alone since `sandbox run` runs its workload through the managed VM API, with the Windows `spawn_foreground` and `standard_handle` that only `one_shot` used. The `run-console-restore` microVM scenario, which restored a raw snapshot with its boot console on the terminal, is retired with its shard entries. esaurez's partial implementation on esaurez/private-sl38b-0325-b2 (ef52ba6b5a) restored blockless snapshots through unmerged prerequisites. This keeps its refusal of raw snapshots and of the options that a restore fixes, and restores through the layered sandbox path of SL3.8a instead. Evidence, on nvx-kvm-test (KVM, Azure Linux 3, kernel 6.6) and nvx-mshv-test (MSHV, kernel 6.18), virtual machines, and on WHP on a Windows 11 developer machine, with OpenVMM 14dc4116, the kernel of release v0.1.0-dev.53ad09678bc1, and this tree's Alpine initramfs: `snapshot capture` published `base`, and `run --restore-snapshot base` restored the release's Ubuntu EROFS layer and a GPT image of its tree, over a RAM upper and over a fresh scratch (GPT over a scratch on KVM and MSHV), as nobody and as daemon (`--workload-user 1:1`), forwarded the workload's output and its statuses 0, 7, and 1, and left no directory, VM, or link to the snapshot's files behind, while `snapshot restore` of /usr/bin/false still failed with its error and status 1. The raw snapshot directory, with and without `--store-dir`, network options for the snapshot without a NIC, a missing, existing, or in-store `--state-dir`, and `--wait` were refused before launch. On KVM and MSHV also: `--memory-mib 384` of a snapshot of 512 MiB capacity (MemTotal 372564 kB, against 241492 kB), three concurrent one-offs of one snapshot, SIGINT (status 130, the directory removed), and a state directory of 89 bytes, whose socket fails the VM's creation and which the run removes. A `run` that kill -9, or Stop-Process -Force on Windows, ended while its workload ran left its VM running 120 s later, while another restore of the snapshot succeeded; `sandbox deprovision` alone ended that VM and removed the directory, as `sandbox stop`, which then reports sandbox-not-running, and `deprovision` also did. A restored sandbox whose workload had exited with status 0 refused `deprovision` until `sandbox stop` ended it. Tests: unit tests parse the restore options of `run` as those of `snapshot restore`, give the restore of a snapshot the snapshot's NIC with the policy of the run's options, and refuse network options for a snapshot without a NIC; fake-backed restores install the run's network options as the restore's own policy, with the DNS server and published listeners of the activation record, and OpenVMM's egress, forward, and host-loopback options without `--net`, refuse network options for a snapshot without a NIC before they provision anything, and end with the status of their workload, 3, or 143 for one that SIGTERM ends, once the VM and its launch have ended, where `snapshot restore` keeps its error, and take a workload with whitespace and quotes, as `snapshot restore` does. CLI tests cover the raw refusals, a missing store, the state directory's refusals, a workload that is not a valid exec request, the options that each kind of run refuses, the removed options and `--wait`, and the help. The dry runs and launches of raw restores, the Python test that a raw restore left OpenVMM the terminal, which no `nvx-rs run` gives OpenVMM anymore, the proptest of a restore's policy paths, the tests of the attachment reader, and the test of the handles that `spawn_foreground` passes, with its case in the test of refused batch files, are retired with the code that they tested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/run.md | 65 ++- doc/usage.md | 119 ++-- nvx-rs/README.md | 95 ++-- nvx-rs/src/host/cli/mod.rs | 5 +- nvx-rs/src/host/cli/network.rs | 43 +- nvx-rs/src/host/cli/run.rs | 510 +++++++----------- nvx-rs/src/host/cli/run/tests.rs | 132 +++-- .../src/host/cli/sandbox/lifecycle/start.rs | 28 +- .../sandbox/lifecycle/start/tests/restore.rs | 226 +++++++- nvx-rs/src/host/cli/sandbox/mod.rs | 43 +- nvx-rs/src/host/cli/sandbox/one_shot.rs | 161 ------ nvx-rs/src/host/cli/snapshots.rs | 39 +- nvx-rs/src/host/cli/tests.rs | 2 +- nvx-rs/src/host/platform/windows/mod.rs | 2 - nvx-rs/src/host/platform/windows/spawn.rs | 62 --- nvx-rs/src/host/platform/windows/tests.rs | 34 -- nvx-rs/src/host/snapshot/manifest.rs | 29 +- nvx-rs/src/host/snapshot/manifest/tests.rs | 103 +--- nvx-rs/src/host/template/mod.rs | 2 - nvx-rs/tests/cli.rs | 16 +- nvx-rs/tests/cli/download.rs | 31 +- nvx-rs/tests/cli/dry_run.rs | 313 +---------- nvx-rs/tests/cli/one_shot.rs | 175 +----- nvx-rs/tests/cli/openvmm.rs | 106 +--- nvx-rs/tests/cli/usage.rs | 78 ++- nvx-rs/tests/cli/validation.rs | 252 ++++++--- scripts/nvx_tools/microvm_shards.json | 18 +- .../run-console-restore.sh | 5 - scripts/nvx_tools/microvm_tests.py | 84 --- scripts/test_microvm_tests.py | 123 ----- scripts/test_nvx_tools.py | 71 --- 31 files changed, 1069 insertions(+), 1903 deletions(-) delete mode 100644 nvx-rs/src/host/cli/sandbox/one_shot.rs delete mode 100644 scripts/nvx_tools/microvm_test_scripts/run-console-restore.sh diff --git a/doc/run.md b/doc/run.md index 7ead91862..fa002887e 100644 --- a/doc/run.md +++ b/doc/run.md @@ -358,10 +358,10 @@ loopback in both directions without a port list, and preserves the existing guest-to-host mapping without publishing guest ports. Guest-originated traffic remains subject to egress policy. -Most `nvx-rs run` options pass through unchanged: `--machine`, `--processors`, -`--cpu-profile`, `--mount`, `--net`, `--network-profile`, `--cmdline`, -`--restore-snapshot`, `--restore-processors`, and `--restore-ready-path`. The -`nvx-rs run` performs these translations and additions: +Most options of a cold boot of `nvx-rs run` pass through unchanged: +`--machine`, `--processors`, `--cpu-profile`, `--mount`, `--net`, +`--network-profile`, and `--cmdline`. The `nvx-rs run` performs these +translations and additions: | `nvx-rs run` | Direct OpenVMM option | | --- | --- | @@ -371,23 +371,23 @@ Most `nvx-rs run` options pass through unchanged: `--machine`, `--processors`, | `--hypervisor auto` | `--hypervisor kvm` on Linux or `--hypervisor whp` on Windows | | `--memory-mib N` | `--memory NM` | | `--memory-capacity-mib N` | `--memory-capacity NM` on a fresh boot | -| `--restore-memory-mib N` | `--restore-memory NM` | | A fresh boot | `--virtio-console console`, which attaches the guest's console, the boot console `hvc1`, to the terminal | -| A restore of a snapshot whose boot console was on the terminal | `--virtio-console console`, which attaches that console to the terminal again | | `--dry-run` | No equivalent; this only prints the generated command | +`nvx-rs run --restore-snapshot NAME --store-dir PATH --state-dir PATH` restores +a named snapshot that `nvx-rs snapshot capture` published once, as +`nvx-rs snapshot restore` does, and no longer a raw OpenVMM snapshot directory, +which it refuses; see [`run`](usage.md#run). Restore a raw snapshot with +OpenVMM directly, as below. + Always include `--single-process`. Include `--virtio-console console` to type into the guest's shell on a fresh boot: without a boot console, the guest's console is the byte channel, `hvc0`, which takes no input, so OpenVMM discards the terminal's keystrokes, though Ctrl-Q still opens its REPL. A restore keeps the console of its snapshot. OpenVMM requires `--virtio-console console` again to restore a snapshot of a boot that passed it, and refuses `--virtio-console` -for a snapshot without a boot console. `nvx-rs run` reads the snapshot's -`manifest.bin`, even with `--dry-run`, and passes `--virtio-console console` -to a restore exactly when the snapshot's boot console was on the terminal; it -refuses a snapshot whose manifest it cannot read, which OpenVMM could not -restore either. When restoring, omit `--memory`, `--kernel`, `--initrd`, and -`--net`, whose addresses the snapshot keeps; see +for a snapshot without a boot console. When restoring, omit `--memory`, +`--kernel`, `--initrd`, and `--net`, whose addresses the snapshot keeps; see [Networked snapshots](#networked-snapshots). Every restore gives the guest fresh entropy through the time ABI's restore packet. Do not pass `--guest ubuntu` during restore; the captured RAM and machine contract already @@ -531,11 +531,14 @@ attachments, and so does a restore, unless it installs a new policy on a A snapshot captured with `--net` keeps its NIC's addresses and the digest of its network policy. Restore it with `--network-profile portable`, without `--net`, and with the network policy options of the capture, which OpenVMM -checks against that digest before the guest runs. `nvx-rs run` passes them to -OpenVMM as given, and refuses `--net` on a restore: +checks against that digest before the guest runs: ```bash -nvx-rs/target/release/nvx-rs run \ +./bin/openvmm \ + --single-process \ + --machine microvm \ + --processors 1 \ + --hypervisor kvm \ --restore-snapshot /var/lib/nvx/snapshot \ --network-profile portable \ --network-egress deny \ @@ -562,7 +565,11 @@ options of the new policy, which may include `--network-ingress allow`, explicit `--host-loopback allow`, and forwards, for example: ```bash -nvx-rs/target/release/nvx-rs run \ +./bin/openvmm \ + --single-process \ + --machine microvm \ + --processors 1 \ + --hypervisor kvm \ --restore-snapshot /var/lib/nvx/template \ --restore-network-policy \ --network-profile portable \ @@ -589,9 +596,11 @@ workload that has already run, but under the deny-all that workload had no connectivity, so nothing in the snapshot depends on the policy, and the restore grants it connectivity that it never had. OpenVMM rejects the option for a cold boot, for a snapshot without a NIC or whose NIC was captured with any other -policy, and for a `workload-start` or `instance-checkpoint` snapshot, and -`nvx-rs run` rejects it without `--restore-snapshot`. A [platform NIC -template](#platform-nic-templates) takes the option too. +policy, and for a `workload-start` or `instance-checkpoint` snapshot. A +[platform NIC template](#platform-nic-templates) takes the option too, and so +does each restore of a named snapshot with a NIC through +`nvx-rs run --restore-snapshot` or `nvx-rs snapshot restore`, which installs the +policy of its network options; see [`run`](usage.md#run). ### Platform NIC templates @@ -659,8 +668,7 @@ slot's device and the guest's kernel command line stay as captured, without a its host, mount tag `microvm` itself, and bind each child where it needs it. The dormant slot of a snapshot of another tier, or without a tier, still binds only a `--mount`, and a snapshot captured with an export requires that -export again. `nvx-rs run`, which passes no sandbox blocks, cannot restore a -platform snapshot. +export again. ## Snapshots of an earlier OpenVMM @@ -682,18 +690,25 @@ same on every backend (see ## Snapshot restore readiness -Restore an existing microVM snapshot with an optional host-readiness endpoint: +Restore an existing microVM snapshot with OpenVMM and an optional +host-readiness endpoint: ```bash -nvx-rs/target/release/nvx-rs run \ +./bin/openvmm \ + --single-process \ --machine microvm \ --processors 8 \ + --hypervisor kvm \ --restore-snapshot /var/lib/nvx/snapshot \ --restore-processors 4 \ - --restore-memory-mib 1024 \ + --restore-memory 1024M \ --restore-ready-path /run/nvx/restore-ready.sock ``` +`nvx-rs run --restore-snapshot` and `nvx-rs snapshot restore` own the endpoint +of each of their restores, whose event they wait for, and restore the +processors that their snapshot had online. + The endpoint must already be listening. OpenVMM connects to a Unix domain socket on Linux or a `//./pipe/...` named pipe on Windows and writes exactly `OPENVMM_RESTORE_READY_V1\n` after snapshot verification, attachment @@ -712,7 +727,7 @@ write the complete event aborts and tears down the restore. For restore-time memory expansion, capture a fresh snapshot with `--memory-mib 512 --memory-capacity-mib 2048`, then select a target with -`--restore-memory-mib 512`, `1024`, or `2048`. The snapshot's `memory.bin` +`--restore-memory 512M`, `1024M`, or `2048M`. The snapshot's `memory.bin` remains exactly 512 MiB; selected expansion ranges receive fresh per-launch backing and are onlined before restore readiness. diff --git a/doc/usage.md b/doc/usage.md index 8954e5e8d..7c6bc704b 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -147,73 +147,73 @@ nvx-rs/target/release/nvx-rs run [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]... [--outcome-report PATH] [--cmdline TEXT] - [--restore-snapshot PATH] - [--restore-processors {1,2,4,8}] - [--restore-memory-mib MIB] - [--restore-ready-path PATH] - [--restore-network-policy] + [--restore-snapshot NAME --store-dir PATH --state-dir PATH] + [--layer ROLE,PATH,IMAGE_UUID]... + [--layer-format {erofs,gpt}] + [--scratch PATH] + [--upper {scratch,ram}] + [--ram-upper-percent PERCENT] + [--entrypoint PATH] + [--arg VALUE]... + [--hostname NAME] + [--workload-user UID:GID] + [--timeout SECONDS] [--dry-run] ``` | Option | Default | Description | | --- | --- | --- | -| `--guest {alpine,ubuntu,azurelinux}` | `alpine` | Select Alpine, Ubuntu, or Azure Linux userland with the same NVX kernel. This option is not used for snapshot restore. | +| `--guest {alpine,ubuntu,azurelinux}` | `alpine` | Select Alpine, Ubuntu, or Azure Linux userland with the same NVX kernel. A restore refuses it. | | `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the OpenVMM hypervisor. `auto` chooses WHP on Windows and KVM elsewhere. | | `--machine {microvm}` | `microvm` | Select the fixed-topology microVM with shared-status edge interrupts. | -| `--memory-mib MIB` | guest-specific | Set guest memory in MiB, an ASCII decimal integer from 1 to 17592186044415, without signs, whitespace, or underscores. Defaults to 128 for Alpine, 512 for Ubuntu, and 512 for Azure Linux. | -| `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot, in the range of `--memory-mib`. | -| `--processors {1,2,4,8}` | `1` | Select the microVM processor count, written as one of the choices. | -| `--cpu-profile ID` | `auto` | Select the guest's [CPU profile](#cpu-profiles): `auto` for the built-in profile of the host's CPU, which falls back with a warning to a development profile derived from this host on an Intel or AMD CPU that no built-in profile serves, a built-in profile ID, or `host` for that development profile. A restore uses the snapshot's profile, which `auto` and, for a host profile, `host` also name. | -| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose a host directory, or a single regular host file without its directory, at the guest target: an absolute, canonical path other than `/`, without whitespace, control characters, double quotes, backslashes, `=`, or any UTF-8 byte that Linux treats as whitespace, such as the `0xa0` in `à`. Cold boots also protect the system paths listed below. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, as the children of an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets; a shared file always uses the aggregate. Targets and host paths must not overlap, and the shares' kernel command-line tokens must fit its 1024-byte budget. An `rw` mapping accepts guest-created symbolic links, which the host never follows. Active snapshot restore requires the same mappings in the same order, with the same canonical paths, kinds, targets, modes, and ownership mode; a dormant-slot restore may attach one new directory mapping that the resumed guest mounts explicitly. | +| `--memory-mib MIB` | guest-specific | Set guest memory in MiB, an ASCII decimal integer from 1 to 17592186044415, without signs, whitespace, or underscores. Defaults to 128 for Alpine, 512 for Ubuntu, and 512 for Azure Linux, and for a restore to 256, as for [`snapshot restore`](#snapshot-restore), which must be a multiple of 128 between the snapshot's boot memory and its capacity. | +| `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot, in the range of `--memory-mib`. A restore refuses it. | +| `--processors {1,2,4,8}` | `1` | Select the microVM processor count, written as one of the choices. A restore refuses it: its VM keeps the processors that the snapshot had online. | +| `--cpu-profile ID` | `auto` | Select the guest's [CPU profile](#cpu-profiles): `auto` for the built-in profile of the host's CPU, which falls back with a warning to a development profile derived from this host on an Intel or AMD CPU that no built-in profile serves, a built-in profile ID, or `host` for that development profile. A restore refuses it: it uses the snapshot's profile. | +| `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose a host directory, or a single regular host file without its directory, at the guest target: an absolute, canonical path other than `/`, without whitespace, control characters, double quotes, backslashes, `=`, or any UTF-8 byte that Linux treats as whitespace, such as the `0xa0` in `à`. Cold boots also protect the system paths listed below. The mode follows the last comma, so `HOST_PATH` may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, as the children of an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets; a shared file always uses the aggregate. Targets and host paths must not overlap, and the shares' kernel command-line tokens must fit its 1024-byte budget. An `rw` mapping accepts guest-created symbolic links, which the host never follows. A restore refuses it and the other share options. | | `--mount-deny HOST_PATH` | none | Hide one existing file or directory inside a mounted host root; repeat to deny multiple paths. It may name the root itself when `--mount-allow` paths inside it remain. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | | `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path again; the hidden directories on the way list only the entries that lead to it. Repeat to allow multiple paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. See [Access policy](run.md#access-policy). | -| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` mapping; the guest's writes anywhere else in it fail with `EROFS`. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. Active snapshot restore requires the same denied, allowed, and writable paths. | +| `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` mapping; the guest's writes anywhere else in it fail with `EROFS`. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | | `--mount-owner {vmm,caller}` | `vmm` | Select the host identity of the guest's operations on every `--mount` directory or file. `caller` performs each one as the guest caller's UID and GID and squashes guest root to the owner of the shared directory or file; it requires a Linux host. See [Run](run.md#file-ownership). | | `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. A restore takes the addresses from the snapshot and rejects this option. | -| `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net` on a fresh boot, and without it to restore a snapshot captured with `--net`. | +| `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net` on a fresh boot, and without it to restore a snapshot with a NIC. | | `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | -| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture rejects `allow`, and so does a restore, unless `--restore-network-policy` installs a new policy. | +| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture rejects `allow`. | | `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | | `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | | `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions from JSON. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | -| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; it needs a `--net` prefix of `/29` or shorter, which OpenVMM checks; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture rejects `allow`, and so does a restore, unless `--restore-network-policy` installs a new policy. | +| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; it needs a `--net` prefix of `/29` or shorter, which OpenVMM checks; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture rejects `allow`. | | `--network-proxy IPV4:TCP-PORT` | none | Allow one explicit host TCP proxy endpoint. | -| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Snapshot capture rejects forwards, and so does a restore, unless `--restore-network-policy` installs a new policy. | -| `--outcome-report PATH` | none | Write a bounded local JSON outcome report to a new file in an existing plain directory. A cold boot rejects an existing destination before launch and never replaces a file or follows a destination symlink. A missing report or publication failure is diagnosed without replacing OpenVMM's exit status. | -| `--cmdline TEXT` | empty | Append kernel parameters; `nvx_*` and `tsc=` tokens are reserved. Cold boots reject double quotes and a standalone `--` token, which the VM control console cannot represent; snapshot restore retains its existing command-line handling. | -| `--restore-snapshot PATH` | none | Restore the immutable machine contract and saved state from a snapshot directory. A snapshot captured with `--net` needs `--network-profile portable` and the network policy options of its capture, unless `--restore-network-policy` installs a new policy; see [Networked snapshots](run.md#networked-snapshots). | -| `--restore-processors {1,2,4,8}` | none | Bring this contiguous processor prefix online before restore readiness. Requires an opt-in microVM snapshot and cannot exceed `--processors` capacity. | -| `--restore-memory-mib MIB` | none | Select the 128 MiB-aligned RAM target for an expansion-capable snapshot restore, in the range of `--memory-mib`. | -| `--restore-ready-path PATH` | none | Publish one restore-readiness event to an existing Unix socket or Windows named pipe. | -| `--restore-network-policy` | off | Install the policy of the network options, which may include `--network-ingress allow`, explicit `--host-loopback allow`, and forwards, in place of the complete deny-all policy, `--network-egress deny` and `--host-loopback deny` without egress rules, a proxy, or ingress, that the snapshot's NIC was captured with. The guest applies the new policy's DNS server and listener announcements before the host releases its input; see [Networked snapshots](run.md#networked-snapshots). Requires `--restore-snapshot`. | -| `--dry-run` | off | Print the generated OpenVMM command without running it. | +| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Snapshot capture rejects forwards. | +| `--outcome-report PATH` | none | Write a bounded local JSON outcome report to a new file in an existing plain directory. A cold boot rejects an existing destination before launch and never replaces a file or follows a destination symlink. A missing report or publication failure is diagnosed without replacing OpenVMM's exit status. A restore refuses it. | +| `--cmdline TEXT` | empty | Append kernel parameters; `nvx_*` and `tsc=` tokens are reserved. Cold boots reject double quotes and a standalone `--` token, which the VM control console cannot represent. A restore refuses it. | +| `--restore-snapshot NAME` | none | Restore the snapshot `NAME` of `--store-dir`, which [`snapshot capture`](#snapshot-capture) published, in place of a cold boot, as described below. A raw OpenVMM snapshot directory is refused. | +| `--store-dir PATH` | none | The named snapshot store of `--restore-snapshot`, which a restore requires. | +| `--state-dir PATH` | none | The directory of a restore's sandbox, which a restore requires: one that does not exist, outside the store, on the store's filesystem. The run creates it, and removes it once it ends. A cold boot refuses it. | +| `--layer`, `--layer-format`, `--scratch`, `--upper`, `--ram-upper-percent`, `--entrypoint`, `--arg`, `--hostname`, `--workload-user`, `--timeout` | as for [`sandbox`](#sandbox) | The image, the upper, and the workload of a restore, and the seconds of its waits, as [`snapshot restore`](#snapshot-restore) takes and checks them: the workload runs over the control protocol, so its entrypoint and arguments may contain whitespace and double quotes. A cold boot refuses them. | +| `--dry-run` | off | Print the generated OpenVMM command without running it. A restore refuses it. | Cold `--mount` targets may not cover, equal, or lie below `/bin`, `/boot`, `/dev`, `/etc`, `/lib`, `/proc`, `/root`, `/run`, `/sbin`, `/sys`, `/usr`, `/var`, or `/init`: these paths belong to the guest's blockless system. For example, use `/config/single.txt` rather than `/etc/single.txt` for a -shared configuration file. Snapshot restore keeps its existing mapping rules. +shared configuration file. Cold mount-policy options require a `--mount`, and network-policy options require networking. A proxy is the guest's sole egress path, so it cannot be combined with `--network-egress-allow` rules for direct connections. -`nvx-rs run` checks its options, prints the OpenVMM command after `>> `, and -unless `--dry-run` is given, runs OpenVMM with the standard streams. A cold -boot attaches the guest's console, the boot console `hvc1`, to them, so that -the terminal's input reaches the guest through it rather than through the byte -channel, `hvc0`, whose output, such as the kernel's early messages, goes to -standard error. A restore attaches the boot console to them again when its -snapshot's boot console was on the terminal, as the snapshot's `manifest.bin` -records, since OpenVMM restores such a snapshot only with that console -attached again, and attaches none otherwise. It reads the manifest even with -`--dry-run`, and fails, before it downloads anything or starts OpenVMM, when it -cannot read it. It exits +`nvx-rs run` checks its options, prints the OpenVMM command of a cold boot +after `>> `, and unless `--dry-run` is given, runs OpenVMM with the standard +streams. A cold boot attaches the guest's console, the boot console `hvc1`, to +them, so that the terminal's input reaches the guest through it rather than +through the byte channel, `hvc0`, whose output, such as the kernel's early +messages, goes to standard error. It exits with status 1 for an error, 2 for a usage error, and 130 after Ctrl-C, and otherwise with the status that OpenVMM returns. Its errors and usage errors carry their [error codes](#error-codes), as `error[CODE]: MESSAGE` and -`nvx-rs run: error[CODE]: MESSAGE`. A cold boot requires the OpenVMM release +`nvx-rs run: error[CODE]: MESSAGE`. A run requires the OpenVMM release binary, `build/vmlinux`, and the selected `build/initramfs*.cpio.gz`, and a -restore only OpenVMM. When one of them is missing from a Git checkout that +restore the Alpine one. When one of them is missing from a Git checkout that holds none of the files that a release installs, `run` first installs the release of the checkout, as [`download`](#download) does, reporting on standard error; a checkout that holds some of them gets a warning instead, and the @@ -224,6 +224,42 @@ name cannot be abbreviated. Ubuntu selection never falls back to Alpine. See [Run](run.md) for host setup, guest shutdown, networking, and virtio-fs examples. +With `--restore-snapshot NAME`, `run` restores the named snapshot of +`--store-dir` in place of a cold boot, as [`snapshot restore`](#snapshot-restore) +restores one, but once: it provisions a managed sandbox in `--state-dir`, a +new directory outside the store, on the filesystem of the snapshot's files, +which the restore links into it, with the image, upper, and workload of its +restore options, and starts it from the snapshot under the snapshot's lock. It +prints no OpenVMM command, and forwards the workload's output. Once the +workload has exited with status 0 within `--timeout`, 60 seconds by default, it +stops the VM; it then removes the sandbox and `--state-dir`, also after a +failure or Ctrl-C, except as the next paragraph says, and exits with the +workload's status, or with status 1 for a restore that fails, and 130 after +Ctrl-C. It refuses a raw OpenVMM snapshot directory, which +`--restore-snapshot` named before, without reading it, a `--state-dir` that +exists or lies in the store, a workload that `snapshot restore` refuses, and +the options that the snapshot or its sandbox fixes: `--guest`, +`--memory-capacity-mib`, `--processors`, `--cpu-profile`, `--net`, the share +options, `--outcome-report`, `--cmdline`, and `--dry-run`. A cold boot refuses +`--store-dir`, `--state-dir`, and the restore options. The snapshot fixes the +NIC: the restore of a snapshot without one, as `snapshot capture` captures +them, takes no network options, and that of a snapshot with a NIC installs the +policy of its network options, which need `--network-profile`, as the policy +of the NIC, with its DNS server, egress, forwards, and published listeners. + +A `run` that dies before it removes its sandbox, as `kill -9` ends it, leaves +`--state-dir` with the sandbox's files and its view of the snapshot, and its +VM, whose OpenVMM process keeps running, also once its workload has exited, +until a command of the sandbox ends it; nothing else does. So does Ctrl-C once +the workload has exited with status 0, while the run stops the VM. The +snapshot's lock goes with the `run`, so other restores of the snapshot proceed, +and a later `run` refuses the existing `--state-dir`. While the workload ran, +the start was unfinished, and `nvx-rs sandbox deprovision --state-dir PATH` +ends the VM, through the recovery of the start's launch journal, and removes +the directory. Once the workload had exited with status 0, the VM runs as a +started sandbox does, and `nvx-rs sandbox stop --state-dir PATH` must end it +before `deprovision` removes the directory. + If no built-in CPU profile serves an Intel or AMD host's CPU, OpenVMM falls back to a host profile and warns with `NVX-CPU-PROFILE-FALLBACK:` at every cold boot, as [CPU profiles](#cpu-profiles) describes. After a failed cold @@ -437,6 +473,8 @@ any other end fails the start, which ends the VM. The sandbox then runs as a started one does, for `sandbox exec` and `sandbox stop`. A start journals its launch as a cold start does, so a restore whose command dies before the guest is ready never completes, and the next command of the sandbox ends its VM. +[`run --restore-snapshot`](#run) restores a snapshot once, in a sandbox that it +removes. ### `download` @@ -1246,10 +1284,7 @@ backend and splits the backend's suite into COUNT shards. The test hook, which makes OpenVMM's default CPU profile selection, `auto`, fall back to a host profile on a host that a built-in profile serves, checks the fallback's warning, captures the guest, and restores it on the snapshot's host -profile. The `run-console-restore` scenario captures a cold boot whose boot -console is on OpenVMM's standard streams, restores the snapshot through -[`nvx-rs run`](#run), which must attach that console to the terminal again, and -types a script into the restored guest's shell. The `time-host-correction` +profile. The `time-host-correction` scenario reads the host's own clock correction from a time sample of an unhooked cold boot, then cold boots with OpenVMM's `host-rate-offset-ppm` test hook. The harness composes the host and diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 5acf67c51..56dfc8795 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -1270,10 +1270,7 @@ capture, at most `MAX_MANIFEST_LEN` bytes of a regular file that is not a link or other reparse point, and refuses another snapshot format version, an unknown, mistyped, or repeated field, truncated bytes, and a capture that is not a template's: a platform-tier capture for clones, with a read-only `distro` whose -identity is unbound, a fresh scratch, and the dormant filesystem slot. The -restores of [`run`](#run) read the host attachments of the machine contract of -any microVM snapshot's manifest through the same reader, with every check but -those that make a manifest a template's. +identity is unbound, a fresh scratch, and the dormant filesystem slot. `CaptureClass` holds the invariant inputs of a capture, known before it: the runtime, the guest's build, the backend, the requested CPU profile, the @@ -2781,46 +2778,50 @@ message. `nvx-rs run` ([Usage](../doc/usage.md#run)) has the options, with their names, defaults, and value checks, and the checks, messages, outputs, and exit statuses of the `nvx.py run` that it replaced, except as -[Other differences from nvx.py](#other-differences-from-nvxpy) lists. It -prints `>> COMMAND`, OpenVMM's command line, quoted as `sandbox run` quotes -it, and unless `--dry-run` is given, it runs OpenVMM with the standard +[Other differences from nvx.py](#other-differences-from-nvxpy) lists. A cold +boot prints `>> COMMAND`, OpenVMM's command line, quoted as `sandbox run` +quotes it, and unless `--dry-run` is given, it runs OpenVMM with the standard streams, so that the guest's console and the terminal settings that OpenVMM restores behave as they do for OpenVMM itself, and exits with OpenVMM's -status. +status; a restore does none of these, as the second item says. - A cold boot runs the kernel and the initramfs of `--guest`, `alpine` by default, with `--memory-mib` or else the guest's default, 128 MiB for Alpine and 512 MiB for Ubuntu and Azure Linux, which `Guest::default_memory_mib` - gives. `--restore-snapshot` restores a snapshot instead, without them, and - with the `--restore-*` options, which need it. + gives. +- `--restore-snapshot NAME` restores the named snapshot of `--store-dir` in + place of a cold boot, once, through the path of `snapshot restore`: it + provisions a managed sandbox in `--state-dir`, a new directory outside the + store, on its filesystem, with the image, upper, workload, and timeout of the + options of `snapshot restore` that `sandbox` defines and parses, + `--state-dir`, `--layer`, `--layer-format`, `--scratch`, `--upper`, + `--ram-upper-percent`, `--entrypoint`, `--arg`, `--hostname`, + `--workload-user`, and `--timeout`, starts it from the snapshot, which + forwards the workload's output, stops the VM once the workload has exited + with status 0, removes the sandbox and its directory in any case, and exits + with the workload's status. It refuses a raw OpenVMM snapshot before it + reads anything, a `--state-dir` that exists or lies in the store, a workload + that `snapshot restore` refuses, and the options of a cold boot that the + snapshot or its sandbox fixes; a cold boot refuses `--store-dir`, + `--state-dir`, and the restore options. A `run` that dies before it removes + the sandbox leaves it with its VM running, which `sandbox deprovision` ends, + after `sandbox stop` once the workload has exited with status 0; see + [`run`](../doc/usage.md#run). - A cold boot attaches the guest's console, the boot console `hvc1`, to the terminal with `--virtio-console console`, so that the terminal's input reaches the guest through it, not through the byte channel, `hvc0`, whose - output, such as the kernel's early messages, goes to standard error. A - restore reads its snapshot's `manifest.bin`, dry run or not, and passes - `--virtio-console console` when the manifest's machine contract holds the - attachment of the boot console, `console:microvm-virtio0`, with the reconnect - policy `require-inherited-attachment`, which a capture with - `--virtio-console console` records, and which OpenVMM restores only with that - option again. It passes no `--virtio-console` otherwise, which OpenVMM - requires of a snapshot without a boot console. A manifest that it cannot read - fails the restore before OpenVMM starts, and before anything is downloaded, - as options that it refuses do. + output, such as the kernel's early messages, goes to standard error. - A cold boot takes `--net` and `--network-profile` together. A restore takes - the addressing of its NIC from the snapshot, as OpenVMM does, so it refuses - `--net`, and passes `--network-profile` alone, which OpenVMM requires, with - the network policy that the snapshot was captured with, to restore a - snapshot that has a NIC, or, with `--restore-network-policy`, which needs - `--restore-snapshot`, with a policy of its own, which OpenVMM installs on a - NIC template in place of the complete deny-all policy that the template was - captured with, and whose DNS server and listener announcements the guest - then applies; OpenVMM checks the snapshot and the new policy. + the addressing of its NIC from the snapshot, so it refuses `--net`. Its other + network options are the restore's own policy, at the NIC's address, which + `sandbox` lowers, with its DNS server, egress, forwards, and published + listeners; a snapshot with a NIC needs `--network-profile` with them, and one + without, as `snapshot capture` captures, refuses them. - One directory `--mount` is OpenVMM's `--mount`. Several, or a shared file, are the children of an aggregate at `/run/nvx/shares`, whose guest targets must be absolute, must not overlap, and must survive the kernel command line, and on a cold boot, their `nvx_share=` tokens join `--cmdline` within the - 1024 bytes that OpenVMM leaves to them; a restore adds none, since its guest - keeps the binds that its boot made. `--mount-deny`, `--mount-allow`, and + 1024 bytes that OpenVMM leaves to them. `--mount-deny`, `--mount-allow`, and `--mount-write` name host paths, which OpenVMM attributes to the share that holds them, and which must be absolute with an aggregate, unlike those of `sandbox`, which apply to the `--mount` before them. @@ -3011,11 +3012,10 @@ guest agents of NVX advertise bits 0, 4, and 6 in sandbox mode. `"/state/config.json": Permission denied (os error 13)`, where `nvx.py` printed Python's `[Errno 13] Permission denied: '/state/config.json'`. - Integer options take ASCII decimal digits, in a range of their own: the MiB - of `--memory-mib`, `--memory-capacity-mib`, and `--restore-memory-mib` from 1 - to 17592186044415, the most whose bytes OpenVMM counts in 64 bits, and - `--exec-timeout-ms` from 0 to 4294967295. `--processors` and - `--restore-processors` take one of their choices as written, and `--timeout` - a number as Rust reads an `f64`. `nvx.py` read them with Python's `int` and + of `--memory-mib` and `--memory-capacity-mib` from 1 to 17592186044415, the + most whose bytes OpenVMM counts in 64 bits, and `--exec-timeout-ms` from 0 to + 4294967295. `--processors` takes one of its choices as written, and + `--timeout` a number as Rust reads an `f64`. `nvx.py` read them with Python's `int` and `float`, which also accept signs, underscores, surrounding whitespace, and the digits of other scripts, and left OpenVMM to refuse a negative size, and `exec` to refuse a timeout beyond 32 bits, where `nvx-rs` refuses them as @@ -3103,19 +3103,14 @@ guest agents of NVX advertise bits 0, 4, and 6 in sandbox mode. [`nvx::host::supervisor`](#nvxhostsupervisor). - `start` and `exec` refuse, and `stop` terminates, a VM that another runtime identity started; see [Runtime identity](#runtime-identity). -- `run --restore-snapshot` refuses `--net`, whose addressing a restored NIC - takes from the snapshot, and passes `--network-profile` without it, as - OpenVMM requires to restore a snapshot that has a NIC. `nvx.py run` took the - two only together, so it could not restore such a snapshot: OpenVMM refused - its `--net`, or required the `--network-profile` that it omitted. -- `run --restore-network-policy`, which `nvx.py run` lacked, passes OpenVMM's - `--restore-network-policy`, so that a restore of a NIC template installs the - policy of its network options; see - [Networked snapshots](../doc/run.md#networked-snapshots). -- `run --restore-snapshot` reads the snapshot's `manifest.bin`, dry run or not, - to attach the boot console as the snapshot had it, and fails when it cannot - read it, where `nvx.py run` read nothing of the snapshot and left its checks - to OpenVMM. +- `run --restore-snapshot` names a snapshot of `--store-dir` that + `snapshot capture` published, and restores it once through the path of + `snapshot restore`, where `nvx.py run` handed OpenVMM the directory of a raw + snapshot, which `nvx-rs run` refuses before it reads anything. It refuses + `--net`, whose addressing a restored NIC takes from the snapshot, and takes + none of the options of a raw restore, `--restore-processors`, + `--restore-memory-mib`, and `--restore-ready-path`, whose values the snapshot + and the restore fix. ### Test hooks @@ -3124,8 +3119,8 @@ serves the sandbox profile: its managed lifecycle, with the capability on its standard input, the control console, and an outcome report at stop, and its one-shot runs, which run `/bin/echo`, `/bin/sleep`, `/bin/true`, or `/bin/false`, write an outcome report if asked, and exit with the workload's -status. It also runs the microVMs of `nvx-rs run`, without a guest: a cold -boot, or the restore of a snapshot directory, prints that it booted or restored +status. It also runs microVMs without a guest, a cold boot of `nvx-rs run` or +the restore of a snapshot directory: it prints that it booted or restored the microVM, writes an outcome report if asked, and exits with status 0, and a restore whose snapshot directory is missing fails with status 1. Like OpenVMM, a restore checks its `--virtio-console` against the attachment of the boot diff --git a/nvx-rs/src/host/cli/mod.rs b/nvx-rs/src/host/cli/mod.rs index dd01ce08e..01d9adf3d 100644 --- a/nvx-rs/src/host/cli/mod.rs +++ b/nvx-rs/src/host/cli/mod.rs @@ -16,8 +16,9 @@ //! `nvx.py download` that it replaced; it leaves `nvx-rs` itself alone. //! - `egress compile PATH` compiles an egress policy file of `--network-egress-policy-file` and //! prints its native rules as JSON. -//! - `run` boots or restores a microVM on the terminal, with the options, checks, outputs, and -//! exit statuses of the `nvx.py run` that it replaced. +//! - `run` boots a microVM on the terminal, with the options, checks, outputs, and exit statuses +//! of the `nvx.py run` that it replaced, or restores a named snapshot once, as +//! `snapshot restore` restores one. //! - `sandbox` runs or manages a sandbox, with the options, outputs, exit statuses, outcome //! reports, and state directories of the `nvx.py sandbox` that it replaced. //! - `snapshot capture` captures a blockless VM into an immutable named snapshot, and diff --git a/nvx-rs/src/host/cli/network.rs b/nvx-rs/src/host/cli/network.rs index 08c9f76e9..1525bbd49 100644 --- a/nvx-rs/src/host/cli/network.rs +++ b/nvx-rs/src/host/cli/network.rs @@ -1,7 +1,6 @@ //! The network options that `nvx-rs sandbox` and `nvx-rs run` share with `nvx.py`, ported from -//! the parsers of `scripts/nvx.py`, `_resolve_network_egress_rules`, and -//! `_extend_network_arguments`: their values, the egress rules of a -//! `--network-egress-policy-file`, and the OpenVMM options that they become. +//! the parsers of `scripts/nvx.py` and `_resolve_network_egress_rules`: their values and the +//! egress rules of a `--network-egress-policy-file`. use std::path::Path; @@ -116,42 +115,4 @@ impl NetworkOptions { deny: compiled.deny, }) } - - /// Appends the OpenVMM options of these network options and `rules` to `command`, as - /// `_extend_network_arguments` does, for `run`, except that `--network-profile` goes without - /// `--net` too, as a restore passes it; `sandbox` lowers them through its typed network policy - /// instead. - pub(crate) fn extend_arguments(&self, command: &mut Vec, rules: &NativeRules) { - let mut push = |name: &str, value: &str| { - command.push(name.to_owned()); - command.push(value.to_owned()); - }; - if let Some(net) = &self.net { - push("--net", net); - } - if let Some(profile) = self.network_profile { - push("--network-profile", profile); - } - if let Some(egress) = self.network_egress { - push("--network-egress", egress); - } - if let Some(ingress) = self.network_ingress { - push("--network-ingress", ingress); - } - for rule in &rules.allow { - push("--network-egress-allow", rule); - } - for rule in &rules.deny { - push("--network-egress-deny", rule); - } - if let Some(host_loopback) = self.host_loopback { - push("--host-loopback", host_loopback); - } - if let Some(proxy) = &self.network_proxy { - push("--network-proxy", proxy); - } - for forward in &self.host_loopback_forward { - push("--host-loopback-forward", forward); - } - } } diff --git a/nvx-rs/src/host/cli/run.rs b/nvx-rs/src/host/cli/run.rs index 571ef734b..c426dcc54 100644 --- a/nvx-rs/src/host/cli/run.rs +++ b/nvx-rs/src/host/cli/run.rs @@ -1,40 +1,37 @@ //! `nvx-rs run`: the port of the former `nvx.py run`, from `command_run` and the `run` parser of //! `scripts/nvx.py`, with its options, defaults, checks, outputs, and exit statuses. //! -//! It boots a guest in a microVM, or restores a microVM from a snapshot, on the terminal: it -//! prints OpenVMM's command as `>> COMMAND`, then runs OpenVMM with the standard streams, unless -//! `--dry-run` is given, and exits with OpenVMM's status. The parser reads the options of -//! `nvx.py run` with their names, defaults, types, and choices, and ends a usage error with -//! status 2. +//! It boots a guest in a microVM on the terminal: it prints OpenVMM's command as `>> COMMAND`, +//! then runs OpenVMM with the standard streams, unless `--dry-run` is given, and exits with +//! OpenVMM's status. The parser reads the options of `nvx.py run` with their names, defaults, +//! types, and choices, and ends a usage error with status 2. //! //! A cold boot forwards the terminal through the VM's boot-console listener, `hvc1`, and the //! VM's log to standard error. Its temporary private VM directory is removed after confirmed -//! exit. A restore attaches the boot console as its snapshot had it: it reads the -//! snapshot's manifest, `manifest.bin` in the snapshot's directory, dry run or not, and passes -//! `--virtio-console console` when the manifest's machine contract holds the attachment of the -//! boot console, `console:microvm-virtio0`, with the reconnect policy -//! `require-inherited-attachment`, which a capture with `--virtio-console console` records, and -//! which OpenVMM restores only with that option again. It passes no console option otherwise, which -//! OpenVMM requires of a snapshot without a boot console, and which leaves any other boot console -//! the attachment that its snapshot records. A manifest that it cannot read fails the restore -//! before OpenVMM starts, and before anything is downloaded, since OpenVMM could not restore the -//! snapshot either; the failure follows that of a missing OpenVMM, as the shares' do. +//! exit. +//! +//! With `--restore-snapshot NAME`, it restores the named snapshot of `--store-dir`, which +//! `snapshot capture` captured, once, in place of a cold boot, through the path of +//! `snapshot restore`: it provisions a managed sandbox in `--state-dir`, outside the store, with +//! the image, upper, and workload of the options that `snapshot restore` takes for them, which +//! `sandbox` defines and parses, and starts it, under the snapshot's lock, through the blocking +//! restore driver, which forwards the workload's output. It stops the VM once the workload has +//! exited with status 0, removes the sandbox in any case, and exits with the workload's status, as +//! [`sandbox::restore_once`] does. A raw OpenVMM snapshot, which the option named before, is +//! refused before anything is read or launched. A restore refuses the options of a cold boot that +//! its snapshot or its sandbox fixes, such as `--guest`, `--processors`, the shares, `--cmdline`, +//! and `--dry-run`, and a cold boot the options of a restore. //! //! A cold boot takes `--net` and `--network-profile` together. A restore takes the addressing of -//! its NIC from the snapshot, as OpenVMM does, so it refuses `--net`, and passes -//! `--network-profile` alone, which OpenVMM requires, with the network policy that the snapshot -//! was captured with, to restore a snapshot that has a NIC. With `--restore-network-policy`, which -//! it passes on, the restore installs the policy of its network options instead, ingress, -//! explicit host loopback, and forwards included, which OpenVMM admits for a NIC template, a -//! snapshot whose NIC was captured with the complete deny-all policy, and the guest applies the -//! new policy's DNS server and listener announcements before the host releases its input. +//! its NIC from the snapshot, so it refuses `--net`. The network options of the restore of a +//! snapshot with a NIC, which need `--network-profile`, are the restore's own network policy, with +//! its DNS server, egress, forwards, and published listeners, as `sandbox` lowers them; those of a +//! snapshot without a NIC, which `snapshot capture` captures, are refused. //! //! Its shares differ from those of `sandbox`. `--mount-deny`, `--mount-allow`, and `--mount-write` //! name host paths, which must be absolute with several shares or a shared file. A cold boot //! checks their ownership in the filesystem export plan, exports its numbered children, and adds -//! the plan's `nvx_share=` rows to the kernel command line. A restore keeps its existing lowering: -//! one directory uses `--mount`, several shares or a file use an aggregate, and no new share tokens -//! replace the binds that its saved guest already made. +//! the plan's `nvx_share=` rows to the kernel command line. //! //! After OpenVMM exits with a status other than 0 from a cold boot, on a host whose CPU no built-in //! CPU profile serves, it prints the guidance of [`cpu_profile`] to standard error, which says @@ -56,30 +53,30 @@ use super::cpu_profile::{self, HostCpu}; use super::download::{self, Hold}; use super::network::{NativeRules, NetworkOptions}; use super::sandbox::{ - HYPERVISORS, MAX_MEMORY_MIB, MOUNT_OWNERS, copy_openvmm_outcome, error_with_cause, hypervisor, - path_string, path_text, run_openvmm, sandbox_network, validate_outcome_destination, + self, HYPERVISORS, MAX_MEMORY_MIB, MOUNT_OWNERS, copy_openvmm_outcome, error_with_cause, + hypervisor, path_string, path_text, sandbox_network, validate_outcome_destination, }; use super::session::{AttachmentCancellation, Session}; use super::shell::format_command; +use super::snapshots::{self, Name}; use super::{Exit, Result, check_interrupt, output}; use crate::ErrorCode; use crate::activation::{MachineIdMode, View}; use crate::host::artifacts::{ArtifactDir, Guest, require_artifact}; -use crate::host::config::{Boot, ConsoleAttachment, NetworkConfig, RootConfig, VmConfig}; +use crate::host::config::{ + Boot, ConsoleAttachment, NetworkConfig, NicConfig, RootConfig, VmConfig, +}; use crate::host::filesystem::{ self, ExportPlan, ExportRequest, Mapping, Owner, PathList, ReservedTargets, Target, }; use crate::host::platform::{self, Transport}; use crate::host::runtime::{ArtifactIdentity, LaunchOptions, Runtime, RuntimeIdentity}; +use crate::host::sandbox::SANDBOX_COMMAND_LINE_MAX_SIZE; use crate::host::sandbox::mount::{ - AGGREGATE_MOUNT_TARGET, MountAccess, MountKind, MountOwner, POLICY_KINDS, PolicyKind, - aggregate_child_argument, aggregate_child_name, aggregate_command_line_fragment, - aggregate_share_tokens, export_command_line_fragment, mount_argument, mount_kind, - parse_mount_value, require_mount_owner_supported, + MountAccess, MountKind, MountOwner, POLICY_KINDS, PolicyKind, export_command_line_fragment, + mount_kind, parse_mount_value, require_mount_owner_supported, }; -use crate::host::sandbox::options::validate_run_share_targets; -use crate::host::sandbox::{SANDBOX_COMMAND_LINE_MAX_SIZE, has_kernel_separator}; -use crate::host::template::read_attachments; +use crate::host::sandbox::options::{SandboxOperation, validate_run_share_targets}; use crate::host::vm::{self, VmDir}; /// The usage and help of the parser. @@ -94,7 +91,7 @@ options: --machine {microvm} --memory-mib MEMORY_MIB guest RAM in MiB (default by --guest: alpine 128, - ubuntu 512, azurelinux 512) + ubuntu 512, azurelinux 512; for a restore: 256) --memory-capacity-mib MEMORY_CAPACITY_MIB --processors {1,2,4,8} --cpu-profile ID OpenVMM CPU profile: auto (OpenVMM's default) for the @@ -142,14 +139,26 @@ options: --outcome-report OUTCOME_REPORT write a bounded local JSON outcome report --cmdline CMDLINE - --restore-snapshot RESTORE_SNAPSHOT - --restore-processors {1,2,4,8} - --restore-memory-mib RESTORE_MEMORY_MIB - --restore-ready-path RESTORE_READY_PATH - --restore-network-policy - install the network options' policy in place of the - complete deny-all policy of a snapshot's NIC - (doc/run.md) + --restore-snapshot NAME + restore the named snapshot of --store-dir, which + snapshot capture captured, once, in place of a cold + boot, run the workload of the options below, and exit + with its status; a raw OpenVMM snapshot is refused + --store-dir PATH the named snapshot store + --state-dir STATE_DIR + the restore's sandbox, outside the store, on its + filesystem, which the run removes once it ends + --layer ROLE,PATH,IMAGE_UUID + --layer-format {erofs,gpt} + --scratch SCRATCH + --upper {scratch,ram} + --ram-upper-percent PERCENT + --entrypoint ENTRYPOINT + --arg SANDBOX_ARG + --hostname HOSTNAME + --workload-user UID:GID + --timeout TIMEOUT seconds for each wait for a lock, and for the restore + and its workload (default: 60) --dry-run", }; @@ -161,20 +170,48 @@ const GUESTS: [&str; 3] = [ ]; /// The choices of `--machine`. const MACHINES: [&str; 1] = ["microvm"]; -/// The choices of `--processors` and `--restore-processors`, the processor counts that a microVM -/// takes. +/// The choices of `--processors`, the processor counts that a microVM takes. const PROCESSORS: [&str; 4] = ["1", "2", "4", "8"]; -/// The stable ID of the attachment of a microVM's boot console, `hvc1`, in a snapshot's machine -/// contract. -const BOOT_CONSOLE: &str = "console:microvm-virtio0"; +/// The options of `snapshot restore` that a restore of `run` takes, which `sandbox` defines and +/// parses for both: the sandbox's state directory, the image, its upper, the workload, and the +/// timeout of the restore. +const RESTORE_INPUTS: [&str; 11] = [ + "--state-dir", + "--layer", + "--layer-format", + "--scratch", + "--upper", + "--ram-upper-percent", + "--entrypoint", + "--arg", + "--hostname", + "--workload-user", + "--timeout", +]; -/// The reconnect policy of a boot console that a capture attached to the terminal with -/// `--virtio-console console`, which OpenVMM restores only with `--virtio-console console` again. -const INHERITED_ATTACHMENT: &str = "require-inherited-attachment"; +/// The options of a cold boot that a restore refuses: its snapshot fixes the guest, the processors, +/// the CPU profile, and the NIC's address, and its sandbox takes no shares, kernel command line, +/// outcome report, or dry run. +const COLD_ONLY: [&str; 13] = [ + "--guest", + "--memory-capacity-mib", + "--processors", + "--cpu-profile", + "--mount", + "--mount-deny", + "--mount-allow", + "--mount-write", + "--mount-owner", + "--net", + "--outcome-report", + "--cmdline", + "--dry-run", +]; -/// The options of `nvx-rs run`, which parse with the defaults of the former `nvx.py run`. -pub(super) const OPTIONS: [Spec; 30] = [ +/// The options of `nvx-rs run` but [`RESTORE_INPUTS`], which parse with the defaults of the +/// former `nvx.py run`. +const OPTIONS: [Spec; 27] = [ Spec::one("--guest"), Spec::one("--hypervisor"), Spec::one("--machine"), @@ -200,15 +237,25 @@ pub(super) const OPTIONS: [Spec; 30] = [ Spec::one("--outcome-report"), Spec::one("--cmdline"), Spec::one("--restore-snapshot"), - Spec::one("--restore-processors"), - Spec::one("--restore-memory-mib"), - Spec::one("--restore-ready-path"), - Spec::flag("--restore-network-policy"), + Spec::one("--store-dir"), Spec::flag("--dry-run"), ]; +/// Returns the options of `nvx-rs run`: its own, and [`RESTORE_INPUTS`] as `snapshot restore` +/// takes them, so that a restore of `run` takes none that `snapshot restore` refuses. +pub(super) fn options() -> Vec { + OPTIONS + .into_iter() + .chain( + snapshots::restore_options() + .into_iter() + .filter(|spec| RESTORE_INPUTS.contains(&spec.name)), + ) + .collect() +} + /// The parsed options of `nvx-rs run`: the namespace of the former `nvx.py run`. -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug, Clone, PartialEq)] pub(crate) struct Options { /// `--guest`, Alpine by default. pub(crate) guest: Guest, @@ -236,16 +283,16 @@ pub(crate) struct Options { pub(crate) outcome_report: Option, /// `--cmdline`, empty by default. pub(crate) cmdline: String, - /// `--restore-snapshot`, as given. + /// `--restore-snapshot`, as given: the name of a snapshot of `--store-dir`. pub(crate) restore_snapshot: Option, - /// `--restore-processors`. - pub(crate) restore_processors: Option, - /// `--restore-memory-mib`. - pub(crate) restore_memory_mib: Option, - /// `--restore-ready-path`, as given. - pub(crate) restore_ready_path: Option, - /// `--restore-network-policy`. - pub(crate) restore_network_policy: bool, + /// `--store-dir`, as given. + pub(crate) store_dir: Option, + /// The options of [`RESTORE_INPUTS`], as `sandbox` parses them, for the sandbox that a restore + /// provisions. + pub(crate) restore: sandbox::Options, + /// The names of the options given, in their order, which [`check`] refuses for a cold boot or + /// a restore that does not take them. + pub(crate) given: Vec<&'static str>, /// `--dry-run`. pub(crate) dry_run: bool, } @@ -267,10 +314,12 @@ impl Default for Options { outcome_report: None, cmdline: String::new(), restore_snapshot: None, - restore_processors: None, - restore_memory_mib: None, - restore_ready_path: None, - restore_network_policy: false, + store_dir: None, + restore: sandbox::Options { + operation: SandboxOperation::Provision, + ..sandbox::Options::default() + }, + given: Vec::new(), dry_run: false, } } @@ -289,6 +338,7 @@ fn memory(value: &str) -> std::result::Result { impl Receiver for Options { fn take(&mut self, name: &'static str, value: Option<&str>) -> std::result::Result<(), String> { + self.given.push(name); let text = || value.unwrap_or_default().to_owned(); match name { "--guest" => { @@ -317,11 +367,9 @@ impl Receiver for Options { "--outcome-report" => self.outcome_report = Some(text()), "--cmdline" => self.cmdline = text(), "--restore-snapshot" => self.restore_snapshot = Some(text()), - "--restore-processors" => self.restore_processors = Some(processors(&text())?), - "--restore-memory-mib" => self.restore_memory_mib = Some(memory(&text())?), - "--restore-ready-path" => self.restore_ready_path = Some(text()), - "--restore-network-policy" => self.restore_network_policy = true, + "--store-dir" => self.store_dir = Some(text()), "--dry-run" => self.dry_run = true, + _ if RESTORE_INPUTS.contains(&name) => return self.restore.take(name, value), _ => return self.network.take(name, &text()), } Ok(()) @@ -332,61 +380,37 @@ impl Receiver for Options { /// not recognize. pub(super) fn parse(arguments: &[String]) -> Result<(Options, Vec)> { let mut options = Options::default(); - let unrecognized = Parser::new(&OPTIONS, &[]) + let unrecognized = Parser::new(&self::options(), &[]) .parse(arguments, &mut options) .map_err(|stop| Exit::from_stop(stop, &SYNTAX))?; Ok((options, unrecognized)) } -/// Appends `name` and `value` to `command`. -fn push(command: &mut Vec, name: &str, value: impl Into) { - command.push(name.to_owned()); - command.push(value.into()); -} - -/// Checks `options` in the order of `command_run`, before anything that it looks up, except that -/// a restore refuses `--net` and takes `--network-profile` alone. +/// Checks `options` before anything that it looks up: a cold boot refuses the options that only a +/// restore takes, and a restore those of [`COLD_ONLY`], in the order given. fn check(options: &Options) -> Result { let refused = |message: String| Err(Exit::error(ErrorCode::Unsupported, message)); - let network = &options.network; let restore = options.restore_snapshot.is_some(); - if restore && network.net.is_some() { - return refused( - "--net is not accepted for restore; the snapshot already fixes the network addressing" - .to_owned(), - ); - } - if !restore && network.net.is_some() != network.network_profile.is_some() { - return refused("--net and --network-profile must be specified together".to_owned()); - } - for (given, option) in [ - (options.restore_ready_path.is_some(), "--restore-ready-path"), - (options.restore_processors.is_some(), "--restore-processors"), - (options.restore_memory_mib.is_some(), "--restore-memory-mib"), - (options.restore_network_policy, "--restore-network-policy"), - ] { - if given && !restore { + for option in &options.given { + if restore && COLD_ONLY.contains(option) { + return refused(format!( + "{option} is not accepted for restore; the snapshot and the restore's own \ + sandbox fix it" + )); + } + if !restore && (*option == "--store-dir" || RESTORE_INPUTS.contains(option)) { return refused(format!("{option} requires --restore-snapshot")); } } - if options.memory_capacity_mib.is_some() && restore { - return refused("--memory-capacity-mib is only valid for a fresh boot".to_owned()); - } - if restore && options.guest != Guest::Alpine { - return refused( - "--guest is not accepted for restore; the snapshot already fixes the guest".to_owned(), - ); + let network = &options.network; + if !restore && network.net.is_some() != network.network_profile.is_some() { + return refused("--net and --network-profile must be specified together".to_owned()); } if let Some(capacity) = &options.memory_capacity_mib && *capacity < memory_mib(options) { return refused("--memory-capacity-mib cannot be below --memory-mib".to_owned()); } - if let Some(restored) = &options.restore_processors - && *restored > options.processors - { - return refused("--restore-processors cannot exceed --processors capacity".to_owned()); - } Ok(()) } @@ -397,95 +421,96 @@ fn memory_mib(options: &Options) -> u64 { .unwrap_or_else(|| u64::from(options.guest.default_memory_mib())) } -/// Returns whether the restore of the snapshot in the directory `snapshot` attaches the boot -/// console to the terminal: whether the machine contract of the snapshot's manifest holds the -/// attachment of the boot console with the reconnect policy of one that a capture attached to the -/// terminal. OpenVMM restores such a console only with `--virtio-console console` again, and -/// refuses that option for any other snapshot. -fn restores_terminal_console(snapshot: &str) -> Result { - let attachments = - read_attachments(Path::new(&path_text(snapshot))).map_err(error_with_cause)?; - // OpenVMM restores the first attachment of the boot console. - Ok(attachments - .iter() - .find(|attachment| attachment.stable_id == BOOT_CONSOLE) - .is_some_and(|attachment| attachment.reconnect_policy == INHERITED_ATTACHMENT)) -} - /// Runs the microVM of `options`, as `command_run` did, once the release of a checkout without -/// artifacts is downloaded, as [`download::ensure`] decides, unless the options are refused. +/// artifacts is downloaded, as [`download::ensure`] decides, unless the options are refused, or +/// restores the named snapshot of `--restore-snapshot`, as [`restore`] does. pub(super) fn run(root: Option<&str>, options: Options) -> Result { check(&options)?; + if options.restore_snapshot.is_some() { + return restore(root, options); + } let rules = options.network.egress_rules()?; - // A restore's console depends on its snapshot's manifest, which a restore that cannot read it - // reports after a missing artifact, as the refusals of the shares and the network below are. - let terminal_console = options - .restore_snapshot - .as_deref() - .map(restores_terminal_console); // The shares and the network follow the artifacts on the command line, and a missing artifact // is reported before them, as `command_run` did, but a command that they refuse downloads // nothing, though it reads the artifacts under the lock of downloads all the same. Their // arguments are built again once the artifacts are in place, as a share's kind depends on what // its host path holds, which a download may install. - let refused = if options.restore_snapshot.is_some() { - terminal_console.as_ref().is_some_and(Result::is_err) - || extend_shares_and_network(&mut Vec::new(), &options, &rules).is_err() - } else { - cold_preflight(&options, &rules).is_err() - }; + let refused = cold_preflight(&options, &rules).is_err(); let located = locate(root)?; - // A restore boots no kernel or initramfs: the snapshot holds the guest. - let guest = options.restore_snapshot.is_none().then_some(options.guest); let hold = if refused { download::hold(&located)? } else { - download::ensure(&located, guest, options.hypervisor)? + download::ensure(&located, Some(options.guest), options.hypervisor)? }; let Located { dir, openvmm } = located; require_artifact(&openvmm, "OpenVMM release binary")?; - if options.restore_snapshot.is_none() { - return run_cold(dir, openvmm, &options, &rules, hold); + run_cold(dir, openvmm, &options, &rules, hold) +} + +/// Restores the named snapshot of `--store-dir` once, through the path of `snapshot restore`, as +/// [`sandbox::restore_once`] does, with the source that [`snapshots::restore_source`] checks: in +/// a sandbox in `--state-dir`, which must not exist, since the run removes it once it ends, and +/// must lie outside the store, on the store's filesystem, since the restore links the snapshot's +/// files into it, with the image, upper, and workload of the options, and the run's network policy. +fn restore(root: Option<&str>, options: Options) -> Result { + let refused = |message: String| Exit::error(ErrorCode::Unsupported, message); + let raw = "it names a snapshot of --store-dir, which `snapshot capture` captured, and no \ + longer a raw OpenVMM snapshot"; + let Some(store_dir) = &options.store_dir else { + return Err(refused(format!( + "--restore-snapshot requires --store-dir: {raw}" + ))); + }; + let given = options.restore_snapshot.clone().unwrap_or_default(); + let name = Name::try_from(given.clone()).map_err(|error| { + refused(format!( + "--restore-snapshot {given:?} is not a snapshot name: {raw}; {}", + error.message() + )) + })?; + let mut sandbox = options.restore; + let Some(state_dir) = sandbox.state_dir.as_deref().filter(|dir| !dir.is_empty()) else { + return Err(refused( + "--restore-snapshot requires --state-dir: the directory of the restore's sandbox, \ + outside the store, on its filesystem" + .to_owned(), + )); + }; + // The run removes the directory once it ends, so it takes only one that it creates. + if fs::symlink_metadata(state_dir).is_ok() { + return Err(refused(format!( + "the restore's --state-dir {state_dir} exists: the run creates it, and removes it \ + once it ends" + ))); } - let mut command = vec![ - path_text(&path_string(&openvmm)?), - "--single-process".to_owned(), - ]; - push(&mut command, "--machine", options.machine); - push(&mut command, "--processors", options.processors.to_string()); - push(&mut command, "--hypervisor", hypervisor(options.hypervisor)); - if let Some(snapshot) = &options.restore_snapshot { - push(&mut command, "--restore-snapshot", path_text(snapshot)); - if let Some(processors) = &options.restore_processors { - push(&mut command, "--restore-processors", processors.to_string()); - } - if let Some(memory) = &options.restore_memory_mib { - push(&mut command, "--restore-memory", format!("{memory}M")); - } - if let Some(ready) = &options.restore_ready_path { - push(&mut command, "--restore-ready-path", path_text(ready)); + let (source, nic) = snapshots::restore_source(name, store_dir, &sandbox)?; + sandbox.network = restored_network(options.network, nic)?; + sandbox.hypervisor = options.hypervisor; + sandbox.memory_mib = options.memory_mib.unwrap_or(sandbox.memory_mib); + sandbox::restore_once(root, sandbox, source) +} + +/// Returns the network options of the sandbox that restores a snapshot whose NIC is `nic`, which +/// the snapshot fixes: `network`, the run's own, at the NIC's address, which need +/// `--network-profile`, as the policy of the restore's NIC, or none for a snapshot without a NIC. +pub(super) fn restored_network( + mut network: NetworkOptions, + nic: Option, +) -> Result { + let refused = |message: &str| Err(Exit::error(ErrorCode::Unsupported, message)); + match (nic, network.network_profile) { + (None, _) if network.any() => { + refused("the snapshot has no NIC, so its restore takes no network options") } - if options.restore_network_policy { - command.push("--restore-network-policy".to_owned()); + (Some(_), None) => { + refused("the snapshot has a NIC, so its restore needs --network-profile") } - if terminal_console.transpose()? == Some(true) { - push(&mut command, "--virtio-console", "console"); + (Some(nic), Some(_)) => { + network.net = Some(nic.guest_network.to_string()); + Ok(network) } + (None, _) => Ok(network), } - if let Some(profile) = &options.cpu_profile { - push(&mut command, "--cpu-profile", profile.as_str()); - } - let cmdline = extend_shares_and_network(&mut command, &options, &rules)?; - if !cmdline.is_empty() { - push(&mut command, "--cmdline", cmdline); - } - output::print_stdout(&format!(">> {}", format_command(&command))) - .map_err(Exit::from_os_error)?; - if options.dry_run { - return Ok(()); - } - let status = run_openvmm(&command, &[], hold)?; - Err(Exit::Status(status)) } /// Checks only options, not paths that artifact installation may create or replace. @@ -1381,124 +1406,3 @@ fn run_cold( #[cfg(test)] mod tests; - -/// Appends the shares of `options`, their policy paths and owner, the network options with the -/// egress `rules`, and the outcome report to `command`, as `command_run` did, and returns the -/// kernel command line: the one of `--cmdline`, and on the cold boot of an aggregate, the -/// `nvx_share=` tokens of its children, which with OpenVMM's tokens must fit the 1024-byte budget. -fn extend_shares_and_network( - command: &mut Vec, - options: &Options, - rules: &NativeRules, -) -> Result { - let shares = options - .mounts - .iter() - .map(|value| parse_mount_value(value)) - .collect::>>()?; - // A file is shared as the child of an aggregate, even alone, because the guest mounts the - // device's root, which is a directory. - let kinds: Vec = shares - .iter() - .map(|(_, host_path, _)| mount_kind(Path::new(host_path))) - .collect(); - let aggregate = shares.len() > 1 || kinds.contains(&MountKind::File); - match shares.as_slice() { - [] => {} - [(target, host_path, access)] if !aggregate => { - if has_kernel_separator(target) { - return Err(Exit::error( - ErrorCode::Unsupported, - format!( - "--mount target {target:?} must be one kernel token without whitespace, control characters, or quotes" - ), - )); - } - push( - command, - "--mount", - mount_argument(target, host_path, *access), - ); - } - _ => { - let targets: Vec<&str> = shares - .iter() - .map(|(target, _, _)| target.as_str()) - .collect(); - validate_run_share_targets(&targets)?; - push(command, "--mount-aggregate", AGGREGATE_MOUNT_TARGET); - // A restore names the children as the boot did, so OpenVMM refuses a restore whose - // targets differ from those that the guest bound. - for (index, ((target, host_path, access), kind)) in - shares.iter().zip(&kinds).enumerate() - { - let name = aggregate_child_name(index, target); - let child = aggregate_child_argument(&name, host_path, *access, *kind); - push(command, "--mount-child", child); - } - } - } - if options.mount_owner.is_some() && options.mounts.is_empty() { - return Err(Exit::error( - ErrorCode::Unsupported, - "--mount-owner requires --mount", - )); - } - for kind in POLICY_KINDS { - for (_, given) in options.policy_paths.iter().filter(|(of, _)| *of == kind) { - let path = path_text(given); - // OpenVMM attributes each path of an aggregate to the child whose host directory - // contains it. - if aggregate && !Path::new(&path).is_absolute() { - return Err(Exit::error( - ErrorCode::Unsupported, - format!( - "with several --mount shares, or a shared file, {} {path} must be an \ - absolute host path", - kind.option(), - ), - )); - } - push(command, kind.option(), path); - } - } - if let Some(owner) = options.mount_owner { - require_mount_owner_supported(owner)?; - push(command, "--mount-owner", owner.as_str()); - } - options.network.extend_arguments(command, rules); - if let Some(report) = &options.outcome_report { - push(command, "--microvm-report", path_text(report)); - } - // A restored guest keeps the binds that its boot made, at the targets that its children's - // names pin. - if !aggregate || options.restore_snapshot.is_some() { - return Ok(options.cmdline.clone()); - } - let tokens = aggregate_share_tokens( - &shares - .iter() - .zip(&kinds) - .map(|((target, _, access), kind)| (target.as_str(), *access, *kind)) - .collect::>(), - ); - let cmdline = std::iter::once(options.cmdline.as_str()) - .chain(tokens.iter().map(String::as_str)) - .filter(|token| !token.is_empty()) - .collect::>() - .join(" "); - let fragment = aggregate_command_line_fragment( - &shares - .iter() - .map(|(_, _, access)| *access) - .collect::>(), - ); - if cmdline.len() + fragment.len() + 1 > SANDBOX_COMMAND_LINE_MAX_SIZE { - return Err(Exit::error( - ErrorCode::Unsupported, - "the --mount shares and --cmdline exceed the 1024-byte budget of the kernel command \ - line; attach fewer shares or use shorter targets", - )); - } - Ok(cmdline) -} diff --git a/nvx-rs/src/host/cli/run/tests.rs b/nvx-rs/src/host/cli/run/tests.rs index 59aa215e6..26fb615a5 100644 --- a/nvx-rs/src/host/cli/run/tests.rs +++ b/nvx-rs/src/host/cli/run/tests.rs @@ -1,5 +1,5 @@ -//! Cold-run shares stay kernel tokens through the checked VM configuration; restore keeps its -//! existing policy-path lowering. +//! Cold-run shares stay kernel tokens through the checked VM configuration; a restore takes the +//! restore options of `snapshot restore` and the network of its snapshot's NIC. use proptest::prelude::*; @@ -147,6 +147,88 @@ fn cold_preflight_defers_installable_sources_but_the_launch_checks_them() { } } +/// Returns the options of `run ARGUMENTS`. +fn run_options(arguments: &[&str]) -> Options { + let arguments: Vec = arguments + .iter() + .map(|&argument| argument.to_owned()) + .collect(); + let (options, unrecognized) = parse(&arguments).unwrap(); + assert!(unrecognized.is_empty(), "{unrecognized:?}"); + options +} + +#[test] +fn restore_inputs_parse_as_those_of_snapshot_restore() { + let inputs = [ + "--state-dir", + "state", + "--layer", + "distro,image.img,11111111-1111-1111-1111-111111111111", + "--layer-format", + "gpt", + "--upper", + "ram", + "--ram-upper-percent", + "30", + "--entrypoint", + "/bin/echo", + "--arg", + "restored", + "--hostname", + "restored", + "--workload-user", + "1000:1001", + "--timeout", + "5", + ]; + let restore = run_options(&[&["--restore-snapshot", "base"][..], &inputs].concat()).restore; + let provision: Vec = std::iter::once("provision") + .chain(inputs) + .map(str::to_owned) + .collect(); + let (expected, unrecognized) = sandbox::parse(&provision).unwrap(); + assert!(unrecognized.is_empty()); + assert_eq!(restore, expected); +} + +#[test] +fn a_restore_takes_the_nic_of_its_snapshot_with_the_policy_of_its_options() { + let nic = NicConfig { + guest_network: crate::net::Ipv4Cidr::parse("10.42.0.2/24").unwrap(), + }; + let network = |arguments: &[&str]| run_options(arguments).network; + // A snapshot without a NIC takes no network option, and one with a NIC needs the profile. + for (arguments, nic, refusal) in [ + (&["--network-egress", "deny"][..], None, "no NIC"), + (&["--network-profile", "portable"][..], None, "no NIC"), + (&[][..], Some(nic), "needs --network-profile"), + ( + &["--network-egress", "deny"][..], + Some(nic), + "needs --network-profile", + ), + ] { + let error = restored_network(network(arguments), nic).unwrap_err(); + assert_eq!(error.code(), Some(ErrorCode::Unsupported), "{error:?}"); + assert!(error.message().unwrap().contains(refusal), "{error:?}"); + } + assert_eq!( + restored_network(NetworkOptions::default(), None).unwrap(), + NetworkOptions::default() + ); + // The NIC keeps the address of the snapshot, and its policy is the restore's own. + let options = network(&["--network-profile", "portable", "--network-egress", "deny"]); + let restored = restored_network(options.clone(), Some(nic)).unwrap(); + assert_eq!( + restored, + NetworkOptions { + net: Some("10.42.0.2/24".to_owned()), + ..options + } + ); +} + #[test] fn attachment_keeps_forwarding_until_the_handshake_completes() { let (observed, progress) = mpsc::sync_channel(0); @@ -599,52 +681,6 @@ proptest! { } } - #[test] - fn restore_policy_paths_attach_to_their_shares( - events in prop::collection::vec((any::(), 0_u8..3, any::()), 0..40), - ) { - let directory = tempfile::tempdir().unwrap(); - let roots = [directory.path().join("first"), directory.path().join("second")]; - for root in &roots { - std::fs::create_dir(root).unwrap(); - } - let mut arguments = Vec::new(); - let mut policies: [Vec; 3] = Default::default(); - for (sequence, (second, kind, mount_first)) in events.into_iter().enumerate() { - let root = &roots[usize::from(second)]; - let path = root.join(format!("path-{sequence}")).to_str().unwrap().to_owned(); - let option = ["--mount-deny", "--mount-allow", "--mount-write"][usize::from(kind)]; - let mount = format!("/share-{sequence},{},rw", root.display()); - let policy = [option.to_owned(), path.clone()]; - let share = ["--mount".to_owned(), mount]; - if mount_first { - arguments.extend(share); - arguments.extend(policy); - } else { - arguments.extend(policy); - arguments.extend(share); - } - policies[usize::from(kind)].push(path); - } - // Two fixed shares exercise aggregate attribution even with an empty event sequence. - // Restore avoids the unrelated cold-boot command-line budget. - arguments.extend([ - "--mount".to_owned(), format!("/first,{},rw", roots[0].display()), - "--mount".to_owned(), format!("/second,{},rw", roots[1].display()), - "--restore-snapshot".to_owned(), "snapshot".to_owned(), - ]); - let (options, unrecognized) = parse(&arguments).unwrap(); - prop_assert!(unrecognized.is_empty()); - let rules = options.network.egress_rules().unwrap(); - let mut command = Vec::new(); - extend_shares_and_network(&mut command, &options, &rules).unwrap(); - for (kind, expected) in ["--mount-deny", "--mount-allow", "--mount-write"].into_iter().zip(policies) { - let actual: Vec<_> = command.windows(2) - .filter(|pair| pair[0] == kind).map(|pair| pair[1].clone()).collect(); - prop_assert_eq!(actual, expected); - } - } - #[test] fn aggregate_policy_paths_must_be_absolute(relative in "[a-z]{1,20}") { for option in ["--mount-deny", "--mount-allow", "--mount-write"] { diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs b/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs index eacdf1ce8..4dae1ee6d 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs @@ -51,11 +51,19 @@ use crate::protocol::workload::WorkloadIdentity; use crate::protocol::{self, ExitCategory, HostMap, WorkloadEnvironment}; use crate::{ErrorCode, ErrorKind}; -/// Starts a provisioned sandbox, as `start` does. -pub(crate) fn start(state_path: &str, artifacts: &Artifacts<'_>, timeout: f64) -> Result { +/// Starts a provisioned sandbox, as `start` does. A workload of a restored sandbox that does not +/// exit with status 0 fails the start with an error that names how it ended, or, if +/// `forward_status`, as `run --restore-snapshot` asks, ends it with [`Exit::Status`] of its +/// status, as `sandbox run` ends: 128 plus the signal's number for a workload that a signal ends. +pub(crate) fn start( + state_path: &str, + artifacts: &Artifacts<'_>, + timeout: f64, + forward_status: bool, +) -> Result { let state_dir = prepare_state_directory(state_path, false)?; let mut lock = LifecycleLock::acquire(&state_dir, timeout, false)?; - start_locked(&state_dir, &mut lock, artifacts, timeout) + start_locked(&state_dir, &mut lock, artifacts, timeout, forward_status) } /// Starts the sandbox of a locked state directory, as `_start` does, through the launch journal @@ -65,6 +73,7 @@ fn start_locked( lock: &mut LifecycleLock, artifacts: &Artifacts<'_>, timeout: f64, + forward_status: bool, ) -> Result { let config_path = state_dir.join(CONFIG_NAME); require_config(state_dir)?; @@ -122,6 +131,7 @@ fn start_locked( let restore = source.as_ref().map(|source| Restore { source, identity: launch.settings().workload_identity, + forward_status, }); let config = config.vm_config(&launch)?; let started = launch_locked(lock, &runtime, config, restore, scratch, timeout, vm::spawn); @@ -130,12 +140,16 @@ fn start_locked( } /// What a start restores in place of a cold boot: the named snapshot of a restored sandbox, and -/// the identity of the workload that the restore runs. +/// the identity of the workload that the restore runs, and how a workload that does not exit with +/// status 0 ends it. struct Restore<'a> { /// The snapshot and the workload, as the sandbox's configuration records them. source: &'a Source, /// The workload's UID and GID. identity: (u32, u32), + /// Whether a workload that does not exit with status 0, such as one that a signal ends, ends + /// the start with [`Exit::Status`] of its status, rather than with an error that names it. + forward_status: bool, } /// Launches the VM of the sandbox of a locked state directory through the launch journal of its @@ -374,7 +388,8 @@ fn check_ready(saved: &SavedVm, runtime: &Runtime, timeout: f64) -> Result return Ok(()), + Ok(ExecEvent::Exit { status, .. }) if restore.forward_status => { + return Err(Exit::Status(status)); + } Ok(ExecEvent::Exit { category, status }) => { return Err(Exit::error( ErrorCode::Other, diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs b/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs index a863bb2fa..0536ef688 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs @@ -13,10 +13,13 @@ use crate::host::artifacts::{ ArtifactDir, Guest, INITRAMFS_PROVENANCE_NAME, KERNEL_PROVENANCE_NAME, OPENVMM_PROVENANCE_NAME, }; use crate::host::cli::hex; +use crate::host::cli::run; use crate::host::cli::sandbox::lifecycle::{ProvisionConfig, deprovision}; +use crate::host::cli::sandbox::sandbox_network; use crate::host::cli::sandbox::state::{RecordedDistro, serialize_launch, write_json}; use crate::host::cli::snapshots::{self, Name}; use crate::host::cli::store::StateDir; +use crate::host::config::NetworkConfig; use crate::host::filesystem::{Bind, ExportChild, ExportPlan, Owner}; use crate::host::network::{Access, NetworkPolicy}; use crate::host::paths::resolve; @@ -28,6 +31,7 @@ use crate::host::sandbox::options::sandbox_mounts; use crate::host::sandbox::{SandboxLaunch, SandboxSettings, SandboxUpper}; use crate::host::snapshot::tests::RestoreFixture; use crate::host::snapshot::{DESCRIPTION_FILE, KEY_FILE, OPENVMM_DIR}; +use crate::net::Ipv4Cidr; use crate::protocol::root::RootImage; use crate::protocol::{ APP_ACTIVATE, APP_EXEC, APP_EXIT, APP_HEADER_LEN, APP_MAPS, APP_READY, APP_STDOUT, OUTER_DATA, @@ -106,12 +110,26 @@ impl Fixture { /// Restores the snapshot into the state directory `state`, whose fake VM does as `behavior` /// says, as a start of a sandbox restored from it does. fn start(&self, state: &str, config: VmConfig, behavior: &str, timeout: f64) -> Result { + self.start_with(state, config, behavior, timeout, false) + } + + /// Restores the snapshot as [`start`](Self::start) does, forwarding the status of a workload + /// that does not exit with status 0 if `forward_status`, as `run --restore-snapshot` asks. + fn start_with( + &self, + state: &str, + config: VmConfig, + behavior: &str, + timeout: f64, + forward_status: bool, + ) -> Result { let state_dir = self.state(state); let mut lock = LifecycleLock::acquire(&state_dir, timeout, true).unwrap(); let source = self.source(); let restore = Restore { source: &source, identity: (1000, 1001), + forward_status, }; launch_locked( &mut lock, @@ -330,6 +348,145 @@ fn named_snapshots_restore_each_image_format_and_upper_with_their_own_policy_and } } +/// Returns `run ARGUMENTS`, parsed. +fn run_options(arguments: &[&str]) -> run::Options { + let arguments: Vec = arguments + .iter() + .map(|&argument| argument.to_owned()) + .collect(); + run::parse(&arguments).unwrap().0 +} + +#[test] +fn a_one_off_restore_installs_the_network_options_of_run_as_its_own_policy() { + let fixture = Fixture::new(true); + let options = run_options(&[ + "--restore-snapshot", + "source", + "--network-profile", + "portable", + "--network-egress", + "deny", + "--network-egress-allow", + "10.42.0.1/32:udp:53", + "--host-loopback", + "allow", + "--host-loopback-forward", + "tcp:18080:8080", + ]); + // The run's options at the NIC of the snapshot, as `sandbox` lowers them and a start of the + // sandbox records them. + let nic = fixture + .store + .lock(&name(), Deadline::after(5.0)) + .unwrap() + .open() + .unwrap() + .nic(); + let network = run::restored_network(options.network, nic).unwrap(); + let rules = network.egress_rules().unwrap(); + let network = sandbox_network(&network, &rules).unwrap().unwrap(); + let mut config = fixture.restore.config.clone(); + config.network = Some(NetworkConfig { + guest_network: Some(Ipv4Cidr::parse(network.guest_network()).unwrap()), + policy: network.policy().clone(), + }); + fixture.start("state", config, "ready", 30.0).unwrap(); + // The guest learns the DNS server that the policy lets it reach, and announces its listeners, + // which the policy publishes on host loopback. + let requests = fixture.requests("state"); + assert_eq!(requests[0].0, APP_ACTIVATE); + assert_eq!( + activation(&requests[0].1).0.configuration().network, + Some(Network { + dns: Some("10.42.0.1".parse().unwrap()), + publication: Publication::Published, + proxy: None, + }) + ); + // OpenVMM installs the policy, egress, forwards, and host loopback, in place of the + // snapshot's, at the NIC's address, which the restore does not name. + let [(_, arguments, _)] = &fixture.spawned()[..] else { + panic!("one VM restores"); + }; + for [option, value] in [ + ["--network-egress", "deny"], + ["--network-egress-allow", "10.42.0.1/32:udp:53"], + ["--host-loopback", "allow"], + ["--host-loopback-forward", "tcp:18080:8080"], + ] { + assert!( + arguments + .windows(2) + .any(|pair| pair[0] == option && pair[1] == value), + "{option} {value}: {arguments:?}" + ); + } + assert!( + arguments + .iter() + .any(|argument| argument == "--restore-network-policy") + ); + assert!(!arguments.iter().any(|argument| argument == "--net")); +} + +#[test] +fn a_one_off_restore_refuses_the_network_options_of_a_snapshot_without_a_nic_before_it_provisions() +{ + let fixture = Fixture::new(false); + let store = fixture.root.path().join("store"); + let state = fixture.state("state"); + let options = run_options(&[ + "--restore-snapshot", + "source", + "--store-dir", + store.to_str().unwrap(), + "--state-dir", + state.to_str().unwrap(), + "--layer", + "distro,distro.erofs,11111111-1111-1111-1111-111111111111", + "--upper", + "ram", + "--network-profile", + "portable", + ]); + let error = run::run(None, options).unwrap_err(); + assert_eq!(error.code(), Some(ErrorCode::Unsupported), "{error:?}"); + assert!(error.message().unwrap().contains("no NIC"), "{error:?}"); + assert!(!state.exists()); + assert!(fixture.spawned().is_empty()); +} + +#[test] +fn a_one_off_restore_ends_with_the_status_of_its_workload_once_its_vm_and_launch_have_ended() { + let fixture = Fixture::new(false); + let config = &fixture.restore.config; + // A forwarded status changes nothing for a workload that exits with status 0. + fixture + .start_with("ready", config.clone(), "ready", 30.0, true) + .unwrap(); + // Another status ends the start with that status, in place of the error of `snapshot + // restore`, once the VM and its launch have ended, before the snapshot goes free, and so does + // a signal's, 128 plus its number, as `sandbox run` ends. + for (index, (state, status)) in [("failing", 3), ("signaled", 143)].into_iter().enumerate() { + let error = fixture + .start_with(state, config.clone(), state, 30.0, true) + .unwrap_err(); + assert!( + matches!(error, Exit::Status(found) if found == status), + "{error:?}" + ); + assert!(fixture.exited(index + 1)); + let locked = StateDir::open(&fixture.state(state)) + .unwrap() + .lock(Deadline::after(5.0)) + .unwrap(); + assert!(locked.journal().unwrap().is_none()); + assert!(locked.runtime().unwrap().is_none()); + assert!(fixture.snapshot_free()); + } +} + #[test] fn concurrent_restores_of_one_snapshot_wait_for_its_lock_until_the_first_vm_has_exited() { let fixture = Fixture::new(false); @@ -402,6 +559,7 @@ fn a_restore_whose_spawn_cannot_confirm_that_its_vm_ended_recovers_it_before_its let restore = Restore { source: &source, identity: (1000, 1001), + forward_status: false, }; let error = launch_locked( &mut lock, @@ -593,7 +751,7 @@ fn a_restored_start_refuses_a_share_of_its_snapshot_store() { let artifacts = Artifacts { root: Some(root.to_str().unwrap()), }; - let error = start(state_dir.to_str().unwrap(), &artifacts, 5.0).unwrap_err(); + let error = start(state_dir.to_str().unwrap(), &artifacts, 5.0, false).unwrap_err(); assert_eq!(error.code(), Some(ErrorCode::Unsupported), "{error:?}"); let message = error.message().unwrap_or_default(); assert!( @@ -700,6 +858,55 @@ fn snapshot_restore_records_its_source_and_starts_a_restore_of_it() { assert!(!state.exists()); } +/// `run --restore-snapshot` takes the workloads that `snapshot restore` takes, exec requests that +/// may hold whitespace and quotes, which no boot token can: its one-off provisions the sandbox and +/// starts a restore of the snapshot, which memory that the snapshot refuses ends before it +/// launches or links anything, and then removes the sandbox. +#[test] +fn a_one_off_restore_takes_the_workloads_of_snapshot_restore() { + let fixture = Fixture::new(false); + let root = artifact_root(&fixture.state("checkout")); + let layer = fixture.state("distro.erofs"); + fs::write(&layer, "distro").unwrap(); + let layer = format!( + "distro,{},11111111-1111-1111-1111-111111111111", + layer.display() + ); + let store = fixture.root.path().join("store"); + let state = fixture.state("one-off"); + let options = run_options(&[ + "--restore-snapshot", + "source", + "--store-dir", + store.to_str().unwrap(), + "--state-dir", + state.to_str().unwrap(), + "--layer", + &layer, + "--upper", + "ram", + "--hypervisor", + if cfg!(windows) { "whp" } else { "kvm" }, + "--memory-mib", + "2176", + "--entrypoint", + "/bin/echo", + "--arg", + "a \"restored\" sandbox", + "--timeout", + "10", + ]); + let error = run::run(Some(&root), options).unwrap_err(); + assert_eq!(error.code(), Some(ErrorCode::Unsupported), "{error:?}"); + let message = error.message().unwrap_or_default(); + assert!( + message.contains("snapshot memory must be from boot"), + "{message}" + ); + assert!(!state.exists()); + assert!(fixture.snapshot_free()); +} + /// Reads one record of the control endpoint, or `None` once the host has gone. fn read_record(stream: &mut impl Read) -> Option<(u8, Vec)> { let mut header = [0; OUTER_HEADER_LEN]; @@ -744,18 +951,15 @@ fn serve(stream: &mut (impl Read + Write), capability: &[u8], epoch: u64, recipe vec![(APP_READY, 0, ready.encode())] } APP_MAPS => vec![(APP_READY, 0, Vec::new())], - // The workload writes its output, then exits, with status 3 if it fails. + // The workload writes its output, then exits, with status 3 if it fails, or SIGTERM + // ends it. _ => vec![ (APP_STDOUT, 0, b"restored\n".to_vec()), - ( - APP_EXIT, - if recipe["behavior"] == "failing" { - 3 - } else { - 0 - }, - b"exit".to_vec(), - ), + match recipe["behavior"].as_str() { + Some("failing") => (APP_EXIT, 3, b"exit".to_vec()), + Some("signaled") => (APP_EXIT, 143, b"signal".to_vec()), + _ => (APP_EXIT, 0, b"exit".to_vec()), + }, ], }; for (kind, status, payload) in replies { diff --git a/nvx-rs/src/host/cli/sandbox/mod.rs b/nvx-rs/src/host/cli/sandbox/mod.rs index d7fbc23be..88254b554 100644 --- a/nvx-rs/src/host/cli/sandbox/mod.rs +++ b/nvx-rs/src/host/cli/sandbox/mod.rs @@ -17,7 +17,6 @@ mod lifecycle; mod lock; -mod one_shot; mod options; mod state; @@ -41,7 +40,6 @@ use crate::protocol::root::RootUpper; pub(crate) use self::lifecycle::ProvisionConfig; pub(super) use self::lifecycle::{copy_openvmm_outcome, validate_outcome_destination}; -pub(super) use self::one_shot::run_openvmm; pub(super) use self::options::sandbox_network; pub(super) use self::options::{ HYPERVISORS, MAX_MEMORY_MIB, MOUNT_OWNERS, OPTIONS, hypervisor, parse, path_text, @@ -148,9 +146,48 @@ pub(super) fn restore(root: Option<&str>, options: Options, source: Source) -> R .expect("provision required --state-dir"), &Artifacts { root }, timeout, + false, ) } +/// Restores `source` once, as `run --restore-snapshot` does: provisions the sandbox of `options` +/// and starts it as [`restore`] does, but a workload that does not exit with status 0 ends the +/// start with its status, stops the VM once the workload has exited with status 0, and +/// deprovisions the sandbox once provisioned, whatever happened, so that the command ends as the +/// workload or the restore did. A sandbox that `options` names already is never removed. +/// +/// A command that dies before it deprovisions the sandbox leaves it, which `sandbox stop` and +/// `sandbox deprovision` of its directory end and remove, and so does Ctrl-C while the stop waits +/// for the VM, after which the deprovision refuses a VM that still runs. +pub(super) fn restore_once(root: Option<&str>, options: Options, source: Source) -> Result { + let (state_dir, timeout) = ( + options.state_dir.clone().unwrap_or_default(), + options.timeout, + ); + run_from(root, options, Some(&source))?; + let artifacts = Artifacts { root }; + let restored = lifecycle::start(&state_dir, &artifacts, timeout, true) + .and_then(|()| lifecycle::stop(&state_dir, &artifacts, timeout)); + let removed = lifecycle::deprovision(&state_dir, &artifacts, timeout).map_err(|error| { + match error.message() { + Some(message) => Exit::error( + error.code().unwrap_or(ErrorCode::Io), + format!("cannot remove the restore's sandbox {state_dir}: {message}"), + ), + None => error, + } + }); + match (restored, removed) { + (Err(error), Err(removal)) => { + if let Some(message) = removal.message() { + output::print_stderr(&format!("warning: {message}")); + } + Err(error) + } + (restored, removed) => restored.and(removed), + } +} + /// Runs the operation of `options`, recording `source` with the sandbox that it provisions. fn run_from(root: Option<&str>, options: Options, source: Option<&Source>) -> Result { let operation = options.operation; @@ -308,7 +345,7 @@ fn run_from(root: Option<&str>, options: Options, source: Option<&Source>) -> Re ) } (SandboxOperation::Start, _) => { - lifecycle::start(state_dir("start")?, &artifacts, options.timeout) + lifecycle::start(state_dir("start")?, &artifacts, options.timeout, false) } (SandboxOperation::Exec, _) => { let state_dir = state_dir("exec")?; diff --git a/nvx-rs/src/host/cli/sandbox/one_shot.rs b/nvx-rs/src/host/cli/sandbox/one_shot.rs deleted file mode 100644 index a15218957..000000000 --- a/nvx-rs/src/host/cli/sandbox/one_shot.rs +++ /dev/null @@ -1,161 +0,0 @@ -//! The OpenVMM process of a one-shot run, which runs on the terminal until it exits or the run is -//! interrupted. - -#[cfg(not(windows))] -use std::process::{Child, Command}; -use std::thread; -use std::time::Duration; -#[cfg(not(windows))] -use std::time::Instant; - -use crate::ErrorCode; -use crate::host::cli::download::Hold; -use crate::host::cli::{Exit, Result}; -use crate::host::platform; -use crate::host::supervisor::Inherited; - -/// Runs OpenVMM on the terminal, as `subprocess.run(command)` does, and returns its status. The -/// command keeps `hold` until OpenVMM has started, so that no download replaces the artifacts that -/// it read for OpenVMM before then, and not while OpenVMM runs, however long it runs. -/// -/// OpenVMM inherits the descriptors or handles `inherit`, under their numbers in this process, and -/// no other of this process beyond its standard streams, as with `subprocess.run`, which closes -/// the others on Linux and passes none on Windows: `inherit` names the checked files of the run, -/// and on Windows the claim's handle of its scratch image too, so that OpenVMM's handle list holds -/// copies of this process's standard handles and those handles alone. On Windows, OpenVMM opens -/// the checked files by their final paths, which the checked files, and OpenVMM's copies of them, -/// hold in place. `nvx-rs run`, which holds no checked files, passes none, as `nvx.py run` passed -/// none to `subprocess.run`. -/// -/// On Linux, OpenVMM starts with this process's umask without the owner's bits, where -/// `subprocess.run` passed the umask unchanged, so that its outcome report keeps the owner's -/// permissions that OpenVMM gives it whatever the umask, outside a directory with a default ACL, -/// as [`spawn`](crate::host::spawn::spawn) describes. -/// -/// OpenVMM shares the terminal and its process group, so Ctrl-C reaches it too. As -/// `subprocess.run` does, the CLI then gives it 0.25 s to exit on Unix before it kills it, and on -/// Windows, where Python cannot interrupt its wait, waits for it to exit; either way the command -/// ends as interrupted, also when OpenVMM exited first, since Python's wait raises -/// `KeyboardInterrupt` once Ctrl-C arrives. -pub(in crate::host::cli) fn run_openvmm( - command: &[String], - inherit: &[Inherited<'_>], - hold: Hold, -) -> Result { - let mut openvmm = OneShot::spawn(command, inherit)?; - drop(hold); - loop { - let status = openvmm.poll()?; - if platform::interrupted() { - if status.is_none() { - end_interrupted(&mut openvmm)?; - } - return Err(Exit::Interrupted); - } - if let Some(status) = status { - return Ok(status); - } - thread::sleep(Duration::from_millis(10)); - } -} - -/// The OpenVMM process of a one-shot run. -struct OneShot { - /// The process, which `std::process` started. - #[cfg(not(windows))] - child: Child, - /// The process, which [`platform::spawn_foreground`] started. - #[cfg(windows)] - process: platform::LaunchedProcess, -} - -impl OneShot { - /// Starts OpenVMM with `command`, with the descriptors or handles `inherit`, as - /// [`run_openvmm`] says. - fn spawn(command: &[String], inherit: &[Inherited<'_>]) -> Result { - let failed = |error: std::io::Error| { - // The host's own refusals of a command, such as of a batch file on Windows, carry no - // OS error. - let refused = - error.kind() == std::io::ErrorKind::InvalidInput && error.raw_os_error().is_none(); - let code = if refused { - ErrorCode::Unsupported - } else { - ErrorCode::Io - }; - Exit::error(code, spawn_error(&error, &command[0])) - }; - #[cfg(windows)] - { - let arguments: Vec = command[1..].iter().map(Into::into).collect(); - platform::spawn_foreground(std::path::Path::new(&command[0]), &arguments, inherit) - .map(|process| Self { process }) - .map_err(failed) - } - #[cfg(not(windows))] - { - let mut child = Command::new(&command[0]); - child.args(&command[1..]); - platform::inherit_only(&mut child, inherit).map_err(failed)?; - child.spawn().map(|child| Self { child }).map_err(failed) - } - } - - /// Returns OpenVMM's status if it has exited: `Popen.returncode`, which on Unix is the - /// negated signal of a process that a signal ended, and on Windows the status of - /// `GetExitCodeProcess`, as `std::process::ExitStatus::code` reads it. - fn poll(&mut self) -> Result> { - #[cfg(windows)] - { - let status = self - .process - .wait_timeout(Duration::ZERO) - .map_err(Exit::from_os_error)?; - Ok(status.map(|status| status as i32)) - } - #[cfg(not(windows))] - { - let status = self.child.try_wait().map_err(Exit::from_os_error)?; - Ok(status.map(|status| { - #[cfg(unix)] - { - use std::os::unix::process::ExitStatusExt; - - status - .code() - .unwrap_or_else(|| -status.signal().unwrap_or_default()) - } - #[cfg(not(unix))] - status.code().unwrap_or_default() - })) - } - } -} - -/// Ends the OpenVMM of an interrupted run as `subprocess.run` does: on Unix, it waits 0.25 s for -/// OpenVMM to exit and then kills it, and on Windows, it waits for OpenVMM to exit. -fn end_interrupted(openvmm: &mut OneShot) -> Result { - #[cfg(not(windows))] - let deadline = Instant::now() + Duration::from_millis(250); - while openvmm.poll()?.is_none() { - #[cfg(not(windows))] - if cfg!(unix) && Instant::now() >= deadline { - let _ = openvmm.child.kill(); - let _ = openvmm.child.wait(); - break; - } - thread::sleep(Duration::from_millis(10)); - } - Ok(()) -} - -/// Returns the message of the I/O error with which the host fails to start `program`: the -/// program, quoted, and the error, as [`io_message`](crate::host::cli::io_message) writes them, -/// or the error alone for a refusal of the host's own, such as of a batch file on Windows, which -/// names the program itself. -pub(crate) fn spawn_error(error: &std::io::Error, program: &str) -> String { - match error.raw_os_error() { - Some(_) => crate::host::cli::io_message(error, std::path::Path::new(program)), - None => error.to_string(), - } -} diff --git a/nvx-rs/src/host/cli/snapshots.rs b/nvx-rs/src/host/cli/snapshots.rs index d7abb378b..65d3d785a 100644 --- a/nvx-rs/src/host/cli/snapshots.rs +++ b/nvx-rs/src/host/cli/snapshots.rs @@ -26,6 +26,7 @@ use super::sandbox::{self, Source}; use super::session::{ExecRequest, Session}; use super::store::{LockedState, StateDir}; use super::{Exit, Result, hex, io_error, temporary_path}; +use crate::host::config::NicConfig; use crate::host::sandbox::options::SandboxOperation; use crate::host::snapshot::{OPENVMM_DIR, SNAPSHOT_ARTIFACTS, Snapshot}; use crate::host::vm::VmDir; @@ -180,7 +181,7 @@ pub(super) fn parse(arguments: &[String]) -> Result<(Command, Vec)> { /// Returns the options of `snapshot restore`: those of `sandbox`, but those that only an /// execution or a cold boot takes, and `--store-dir`. -fn restore_options() -> Vec { +pub(super) fn restore_options() -> Vec { sandbox::OPTIONS .into_iter() .filter(|spec| !NOT_RESTORED.contains(&spec.name)) @@ -196,13 +197,8 @@ pub(super) fn run(root: Option<&str>, command: Command) -> Result { } } -/// Restores a new managed sandbox from a named snapshot, as `snapshot restore` does: opens the -/// snapshot, so that a missing one provisions nothing, then provisions the sandbox with its -/// source, whose workload is the sandbox's entrypoint and arguments, and starts it. Before it -/// provisions anything, it refuses a state directory in the store, as each start and deprovision -/// of the sandbox refuses one before it removes anything, and a workload that is not a valid exec -/// request: each restored start sends the workload as one, not as boot tokens, so it may contain -/// whitespace, which `sandbox provision` refuses. +/// Restores a new managed sandbox from a named snapshot, as `snapshot restore` does: provisions +/// the sandbox with the source that [`restore_source`] checks, and starts it. fn restore(root: Option<&str>, options: RestoreOptions) -> Result { let RestoreOptions { name, @@ -210,7 +206,25 @@ fn restore(root: Option<&str>, options: RestoreOptions) -> Result { sandbox: options, } = options; let snapshot = name.expect("the parser requires a snapshot name"); - let store = paths::resolve(Path::new(&store_dir)).map_err(Exit::from_os_error)?; + let (source, _) = restore_source(snapshot, &store_dir, &options)?; + sandbox::restore(root, options, source) +} + +/// Returns the source of a sandbox that restores the snapshot `snapshot` of the store at +/// `store_dir` with `options`, as `snapshot restore` and `run --restore-snapshot` provision one, +/// and the snapshot's NIC. Its workload is the sandbox's entrypoint and arguments. +/// +/// Before the caller provisions anything, it refuses a state directory in the store, as each start +/// and deprovision of the sandbox refuses one before it removes anything, and a workload that is +/// not a valid exec request: each restored start sends the workload as one, not as boot tokens, so +/// it may contain whitespace, which `sandbox provision` refuses. It then opens the snapshot, so +/// that a missing one provisions nothing. +pub(super) fn restore_source( + snapshot: Name, + store_dir: &str, + options: &sandbox::Options, +) -> Result<(Source, Option)> { + let store = paths::resolve(Path::new(store_dir)).map_err(Exit::from_os_error)?; outside_store( &store, Path::new(options.state_dir.as_deref().unwrap_or_default()), @@ -225,15 +239,16 @@ fn restore(root: Option<&str>, options: RestoreOptions) -> Result { environment: None, inherit_default_environment: false, })?; - Store::open(&store)? + let nic = Store::open(&store)? .lock(&snapshot, Deadline::after(options.timeout))? - .open()?; + .open()? + .nic(); let source = Source { snapshot, store, workload, }; - sandbox::restore(root, options, source) + Ok((source, nic)) } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] diff --git a/nvx-rs/src/host/cli/tests.rs b/nvx-rs/src/host/cli/tests.rs index 48a58d3ff..0b0de8791 100644 --- a/nvx-rs/src/host/cli/tests.rs +++ b/nvx-rs/src/host/cli/tests.rs @@ -18,7 +18,7 @@ fn usages_are_one_line_and_helps_list_every_option() { (&download::SYNTAX, &download::OPTIONS[..]), (&egress::SYNTAX, &[][..]), (&egress::COMPILE_SYNTAX, &[][..]), - (&run::SYNTAX, &run::OPTIONS[..]), + (&run::SYNTAX, &run::options()[..]), (&sandbox::tests::SYNTAX, &sandbox::tests::OPTIONS[..]), ] { let usage = syntax.usage(); diff --git a/nvx-rs/src/host/platform/windows/mod.rs b/nvx-rs/src/host/platform/windows/mod.rs index 2003f8d41..a39767529 100644 --- a/nvx-rs/src/host/platform/windows/mod.rs +++ b/nvx-rs/src/host/platform/windows/mod.rs @@ -46,8 +46,6 @@ pub use self::process::process_start_time; pub(crate) use self::process::{ ProcessHandle, boot_id, host_id, monotonic_ns, open_process, own_bytes_read, }; -#[cfg(feature = "cli")] -pub(crate) use self::spawn::spawn_foreground; pub(crate) use self::spawn::{LaunchedProcess, SpawnMode, spawn_with_stdio}; /// Whether this host can run NVX microVMs. diff --git a/nvx-rs/src/host/platform/windows/spawn.rs b/nvx-rs/src/host/platform/windows/spawn.rs index 4ceb78880..310524909 100644 --- a/nvx-rs/src/host/platform/windows/spawn.rs +++ b/nvx-rs/src/host/platform/windows/spawn.rs @@ -9,20 +9,10 @@ use std::path::Path; use std::time::Duration; use std::{mem, ptr}; -#[cfg(feature = "cli")] -use windows_sys::Win32::Foundation::{ - DUPLICATE_SAME_ACCESS, DuplicateHandle, INVALID_HANDLE_VALUE, -}; use windows_sys::Win32::Foundation::{ GetHandleInformation, HANDLE, HANDLE_FLAG_INHERIT, SetHandleInformation, WAIT_OBJECT_0, WAIT_TIMEOUT, }; -#[cfg(feature = "cli")] -use windows_sys::Win32::System::Console::{ - GetStdHandle, STD_ERROR_HANDLE, STD_HANDLE, STD_INPUT_HANDLE, STD_OUTPUT_HANDLE, -}; -#[cfg(feature = "cli")] -use windows_sys::Win32::System::Threading::GetCurrentProcess; use windows_sys::Win32::System::Threading::{ CREATE_BREAKAWAY_FROM_JOB, CREATE_NEW_PROCESS_GROUP, CREATE_NO_WINDOW, CREATE_UNICODE_ENVIRONMENT, CreateProcessW, DeleteProcThreadAttributeList, @@ -278,58 +268,6 @@ pub(crate) enum SpawnMode { }, } -/// Starts OpenVMM in the caller's console, process group, and working directory, as -/// `subprocess.run` starts a child, with copies of the caller's standard handles as its own. -/// -/// Unlike `std::process::Command`, which lets the child inherit every inheritable handle of the -/// caller, it passes the child no other handle of the caller than the handles `inherit`, which -/// keep their values in the child, through the handle list of [`spawn_with_stdio`], with the same -/// window, under the same lock, for processes that the caller spawns concurrently without one. -/// Unlike `std::process::Command` too, which escapes the arguments of a batch file for `cmd.exe`, -/// it fails with [`io::ErrorKind::InvalidInput`] for a batch file, as [`refuse_batch_file`] says. -#[cfg(feature = "cli")] -pub(crate) fn spawn_foreground( - program: &Path, - arguments: &[OsString], - inherit: &[Inherited<'_>], -) -> io::Result { - let stdio = [ - standard_handle(STD_INPUT_HANDLE)?, - standard_handle(STD_OUTPUT_HANDLE)?, - standard_handle(STD_ERROR_HANDLE)?, - ]; - spawn_listed(program, arguments, None, stdio, inherit, None, 0) -} - -/// Returns a copy of the caller's standard handle `which`, or `None` if the caller has none, as -/// `std::process::Command` gives a child the standard handles of its caller. -#[cfg(feature = "cli")] -fn standard_handle(which: STD_HANDLE) -> io::Result> { - // SAFETY: GetStdHandle has no preconditions. - let handle = unsafe { GetStdHandle(which) }; - if handle.is_null() || handle == INVALID_HANDLE_VALUE { - return Ok(None); - } - let mut copy = ptr::null_mut(); - // SAFETY: the source and target are this process, and the output pointer is valid; a - // standard handle that is not open fails the copy. - if unsafe { - DuplicateHandle( - GetCurrentProcess(), - handle, - GetCurrentProcess(), - &mut copy, - 0, - 0, - DUPLICATE_SAME_ACCESS, - ) - } == 0 - { - return Err(io::Error::last_os_error()); - } - Ok(owned(copy)) -} - /// Starts `program` with `arguments` in `working_dir`, or in the caller's working directory, with /// the creation flags `flags`, the handles `stdio` as its standard handles, a missing one null, /// and the caller's handles `inherit`, and no other handle of the caller, in the job object diff --git a/nvx-rs/src/host/platform/windows/tests.rs b/nvx-rs/src/host/platform/windows/tests.rs index ef63f5c01..f9d6c0779 100644 --- a/nvx-rs/src/host/platform/windows/tests.rs +++ b/nvx-rs/src/host/platform/windows/tests.rs @@ -407,38 +407,6 @@ fn children_inherit_the_allowed_handles_while_they_are_created() { assert!(!allowed_inheritable && kept_inheritable); } -#[cfg(feature = "cli")] -#[test] -fn foreground_children_inherit_only_the_allowed_handles_besides_their_standard_ones() { - // A handle that this process leaves inheritable, as its parent may have passed it, and - // that `std::process::Command` would pass on. - let (_other_reader, other) = io::pipe().unwrap(); - make_inheritable(&other); - let (_allowed_reader, allowed) = io::pipe().unwrap(); - let system = std::env::var_os("SystemRoot").unwrap(); - let cmd = Path::new(&system).join("System32").join("cmd.exe"); - // The child shares this process's console and standard handles, so it writes nothing. - let child = spawn_foreground( - &cmd, - &["/c", "ping -n 30 127.0.0.1 >NUL"].map(OsString::from), - &[Inherited::new(&allowed)], - ) - .unwrap(); - let value = |handle: &io::PipeWriter| handle.as_raw_handle() as HANDLE; - let (allowed_held, other_held) = ( - holds(&child.process, value(&allowed), &allowed), - holds(&child.process, value(&other), &other), - ); - let allowed_inheritable = inheritable(&allowed); - assert!(child.terminate(Duration::from_secs(10))); - assert!(allowed_held, "an allowed handle was not inherited"); - assert!(!other_held, "a handle that was not allowed was inherited"); - assert!( - !allowed_inheritable, - "the spawn left the allowed handle inheritable" - ); -} - #[test] fn batch_files_are_refused_before_cmd_could_read_their_arguments() { // Batch files that would leave a marker if they ran, with an argument that cmd.exe would @@ -462,8 +430,6 @@ fn batch_files_are_refused_before_cmd_could_read_their_arguments() { let error = result.err().expect("a batch file started"); assert_eq!(error.kind(), io::ErrorKind::InvalidInput, "{error}"); }; - #[cfg(feature = "cli")] - refused(spawn_foreground(&program, &arguments, &[])); let (stdin, _stdin_writer) = io::pipe().unwrap(); let (_stdout_reader, stdout) = io::pipe().unwrap(); let stderr = stdout.try_clone().unwrap(); diff --git a/nvx-rs/src/host/snapshot/manifest.rs b/nvx-rs/src/host/snapshot/manifest.rs index f3ec3fe5d..20af1c9a0 100644 --- a/nvx-rs/src/host/snapshot/manifest.rs +++ b/nvx-rs/src/host/snapshot/manifest.rs @@ -1,6 +1,4 @@ -//! The manifest of a snapshot: a template's capture, which [`SnapshotManifest::read`] reads, and -//! the host attachments of any microVM snapshot, which `read_attachments` reads for the restores -//! of `nvx-rs run`. +//! The manifest of a snapshot: a template's capture, which [`SnapshotManifest::read`] reads. //! //! OpenVMM writes `manifest.bin` into the directory of a snapshot that it publishes: one message //! of the protobuf wire format, as its `mesh` encodes `openvmm.snapshot.SnapshotManifest`, of at @@ -786,7 +784,7 @@ impl SnapshotManifest { /// A host attachment of a snapshot's machine contract, `openvmm.snapshot.SnapshotAttachment`: an /// endpoint outside the VM, such as the backend of a console, whose identity and reconnect -/// policy snapshot compatibility and `nvx-rs run` read for a restore. +/// policy snapshot compatibility reads for a restore. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub(crate) struct Attachment { /// The stable ID of the device that it attaches, such as `console:microvm-virtio0`. @@ -795,29 +793,6 @@ pub(crate) struct Attachment { pub(crate) reconnect_policy: String, } -/// Reads the host attachments of the snapshot in the directory `snapshot`, in the order of the -/// machine contract of its manifest, [`MANIFEST_NAME`]: those of any microVM snapshot, a -/// template's or not, which `nvx-rs run` restores. -/// -/// It reads the manifest as [`SnapshotManifest::read`] does, and checks its format and the form of -/// its fields as [`SnapshotManifest::parse`] does, but nothing that makes it a template's. A -/// manifest without a machine contract holds no attachments. -/// -/// Fails as [`SnapshotManifest::read`] does, but for a manifest that is not a template's. -#[cfg(feature = "cli")] -pub(crate) fn read_attachments(snapshot: &Path) -> Result> { - let (bytes, description) = read_manifest(snapshot)?; - let raw = decode_described( - &bytes, - &description, - "the snapshot again with the OpenVMM of this build", - )?; - Ok(raw - .contract - .map(|contract| contract.attachments) - .unwrap_or_default()) -} - /// What a template's capture mends of a manifest of a format that this build does not read. const CAPTURE_TEMPLATE: &str = "the template again with the runtime of this build"; diff --git a/nvx-rs/src/host/snapshot/manifest/tests.rs b/nvx-rs/src/host/snapshot/manifest/tests.rs index 517b46cb2..506bff41e 100644 --- a/nvx-rs/src/host/snapshot/manifest/tests.rs +++ b/nvx-rs/src/host/snapshot/manifest/tests.rs @@ -1,5 +1,4 @@ -//! Tests of [`SnapshotManifest`]: what it reads of a template's manifest, and what it refuses, and -//! of `read_attachments`, which reads the host attachments of any snapshot's manifest for the CLI. +//! Tests of [`SnapshotManifest`]: what it reads of a template's manifest, and what it refuses. use std::fs; use std::path::Path; @@ -882,7 +881,6 @@ fn the_manifests_that_openvmm_writes_are_read() { /// Returns what a restore reads of the attachment of the device `stable_id`, which `policy` /// reconnects. -#[cfg(feature = "cli")] fn attachment(stable_id: &str, policy: &str) -> Attachment { Attachment { stable_id: stable_id.to_owned(), @@ -890,7 +888,6 @@ fn attachment(stable_id: &str, policy: &str) -> Attachment { } } -#[cfg(feature = "cli")] #[test] fn the_attachments_that_openvmm_writes_are_read_in_their_order() { // OpenVMM writes the attachment of the NIC, if there is one, and then those of the consoles. @@ -899,115 +896,39 @@ fn the_attachments_that_openvmm_writes_are_read_in_their_order() { attachment("console:microvm-control0", "broker-authenticated-listener"), ]; assert_eq!( - read_attachments(&fixture("platform-kvm")).unwrap(), + SnapshotManifest::read(&fixture("platform-kvm")) + .unwrap() + .attachments, consoles ); let mut networked = vec![attachment("net:microvm0", "recreate-endpoint")]; networked.extend(consoles); assert_eq!( - read_attachments(&fixture("platform-mshv-network")).unwrap(), + SnapshotManifest::read(&fixture("platform-mshv-network")) + .unwrap() + .attachments, networked ); } -/// Returns the manifest of a snapshot that is no template's: a capture at no tier, of a boot whose -/// boot console was on the terminal, as `--virtio-console console` records it. -#[cfg(feature = "cli")] -fn terminal_console_snapshot() -> ManifestSpec { - let mut spec = ManifestSpec::template(); - spec.tier.clear(); - spec.restore_policy.clear(); - spec.consumed_sections = 0; - let console = &mut spec.contract().attachments[0]; - assert_eq!(console.stable_id, "console:microvm-virtio0"); - console.required = true; - console.reconnect_policy = "require-inherited-attachment".to_owned(); - console.identity_kind = "provider".to_owned(); - console.identity = b"console".to_vec(); - spec -} - -#[cfg(feature = "cli")] #[test] -fn the_attachments_of_any_snapshot_are_read() { - let spec = terminal_console_snapshot(); - let directory = snapshot(&spec.encode()); - let (message, _) = read_refusal(directory.path(), ErrorCode::Io); - assert!(message.contains("is not a template's"), "{message}"); - assert_eq!( - read_attachments(directory.path()).unwrap(), - [ - attachment("console:microvm-virtio0", "require-inherited-attachment"), - attachment("console:microvm-control0", "broker-authenticated-listener"), - ] - ); - // A machine without attachments, or without a contract, has none. - let mut bare = spec.clone(); - bare.contract().attachments.clear(); - let directory = snapshot(&bare.encode()); - assert!(read_attachments(directory.path()).unwrap().is_empty()); - bare.contract = None; - let directory = snapshot(&bare.encode()); - assert!(read_attachments(directory.path()).unwrap().is_empty()); +fn malformed_attachments_are_refused() { // The contract holds at most its bound of attachments. - let mut crowded = spec; + let mut crowded = ManifestSpec::template(); let control = crowded.contract().attachments[1].clone(); crowded .contract() .attachments .resize(MAX_ATTACHMENTS + 1, control); - let directory = snapshot(&crowded.encode()); - let path = directory.path().join(MANIFEST_NAME); - let error = read_attachments(directory.path()).unwrap_err(); + let error = SnapshotManifest::parse(&crowded.encode()).unwrap_err(); assert_eq!(error.code(), ErrorCode::Io, "{error}"); assert_eq!( error.message(), format!( - "the snapshot manifest {} is malformed: its attachments hold more than \ - {MAX_ATTACHMENTS} items", - path.display() - ) - ); -} - -#[cfg(feature = "cli")] -#[test] -fn the_attachments_of_unreadable_manifests_are_refused() { - // A missing manifest is not found, as a template's is. - let directory = tempfile::tempdir().unwrap(); - let error = read_attachments(directory.path()).unwrap_err(); - assert_eq!(error.code(), ErrorCode::NotFound, "{error}"); - let path = directory.path().join(MANIFEST_NAME); - assert_eq!( - error.message(), - format!("cannot read the snapshot manifest {}", path.display()) - ); - assert!(std::error::Error::source(&error).is_some()); - // A manifest of another format asks for a capture of the snapshot, not of a template. - let mut spec = terminal_console_snapshot(); - spec.version = 3; - let directory = snapshot(&spec.encode()); - let error = read_attachments(directory.path()).unwrap_err(); - assert_eq!(error.code(), ErrorCode::FormatUnsupported, "{error}"); - assert_eq!( - error.message(), - format!( - "the snapshot manifest {} has format version 3, which this build does not read; \ - capture the snapshot again with the OpenVMM of this build", - directory.path().join(MANIFEST_NAME).display() + "the snapshot manifest is malformed: its attachments hold more than \ + {MAX_ATTACHMENTS} items" ) ); - // A malformed field is refused anywhere in the manifest, as a template's is. - let directory = - snapshot(&terminal_console_snapshot().encode_with(Proto::default().varint(5, 2))); - let error = read_attachments(directory.path()).unwrap_err(); - assert_eq!(error.code(), ErrorCode::Io, "{error}"); - assert!( - error - .message() - .ends_with("is malformed: it holds field 5 more than once"), - "{error}" - ); let attachment = Proto::default().bytes(1, &[0xff]).finish(); assert_eq!( decode_attachment(&attachment, 0), diff --git a/nvx-rs/src/host/template/mod.rs b/nvx-rs/src/host/template/mod.rs index 8c37873a2..4d8d208c5 100644 --- a/nvx-rs/src/host/template/mod.rs +++ b/nvx-rs/src/host/template/mod.rs @@ -86,8 +86,6 @@ pub use self::clone::{ READY_TO_START_TIMEOUT, RESTORE_READY_EVENT, RESTORE_TO_READY_TIMEOUT, ReattachedClone, RestoreDeadlines, Restoring, Stage, Started, Teardown, reattach_clone, status_evidence, }; -#[cfg(feature = "cli")] -pub(crate) use self::manifest::read_attachments; pub use self::plan::{MEMORY_BLOCK_SIZE, SCRATCH_NAME, TemplatePlan, TemplateResources}; pub use self::store::{ Acquired, CAPTURE_RECORD_NAME, CAPTURES_NAME, Capture, Created, GENERATIONS_NAME, Generation, diff --git a/nvx-rs/tests/cli.rs b/nvx-rs/tests/cli.rs index 7d0b95fcb..48a490237 100644 --- a/nvx-rs/tests/cli.rs +++ b/nvx-rs/tests/cli.rs @@ -2617,19 +2617,21 @@ fn run_guides_failed_cold_boots_on_cpus_that_no_built_in_profile_serves() { } // A cold boot that succeeds, a dry run, the restores that fail, the options that nvx-rs // refuses, its usage errors, and the help print none. - let snapshot = root.path.join("snapshot"); - fs::create_dir(&snapshot).unwrap(); - fs::write(snapshot.join("manifest.bin"), &*openvmm::NO_CONSOLE).unwrap(); - let snapshot = snapshot.to_str().unwrap(); + let store = root.path.join("store"); + let store = store.to_str().unwrap(); + let state = root.path.join("state"); + let state = state.to_str().unwrap(); for (options, status) in [ (&[][..], 0), (&["--cmdline", "fake_exit_status=1", "--dry-run"][..], 0), ( &[ "--restore-snapshot", - snapshot, - "--cmdline", - "fake_exit_status=1", + "base", + "--store-dir", + store, + "--state-dir", + state, ][..], 1, ), diff --git a/nvx-rs/tests/cli/download.rs b/nvx-rs/tests/cli/download.rs index eb734cd18..2f6860e00 100644 --- a/nvx-rs/tests/cli/download.rs +++ b/nvx-rs/tests/cli/download.rs @@ -192,21 +192,30 @@ fn commands_download_nothing_for_options_that_they_refuse() { } #[test] -fn restores_download_nothing_for_snapshots_that_they_cannot_read() { - // A restore reads its snapshot's manifest first, and one that it cannot read fails it as - // options that it refuses do: after the missing artifact, without the download that would - // provide it, and with the checkout as it was. - for arguments in [ - vec!["run", "--restore-snapshot", "missing", "--dry-run"], - vec!["run", "--restore-snapshot", "missing"], +fn restores_download_nothing_for_snapshots_that_they_cannot_open() { + // A restore refuses a raw OpenVMM snapshot, and opens its snapshot's store, before it reads + // anything else: one that it cannot open fails it before the download that a provision of + // its sandbox would make, and leaves the checkout as it was. + for (arguments, code) in [ + (vec!["run", "--restore-snapshot", "missing"], "unsupported"), + ( + vec![ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "store", + "--state-dir", + "state", + ], + "not-found", + ), ] { let root = checkout(&INSTALLED); let before = root.snapshot(); let outcome = root.nvx_rs(&arguments); - outcome.assert_error("artifact-missing", &["OpenVMM"]); - let mut after = root.snapshot(); - after.remove("build/.nvx-download.lock"); - assert_eq!(after, before, "{arguments:?}"); + outcome.assert_error(code, &[]); + assert_eq!(root.snapshot(), before, "{arguments:?}"); } } diff --git a/nvx-rs/tests/cli/dry_run.rs b/nvx-rs/tests/cli/dry_run.rs index 2ab0ab87f..4b1bcf90c 100644 --- a/nvx-rs/tests/cli/dry_run.rs +++ b/nvx-rs/tests/cli/dry_run.rs @@ -7,10 +7,7 @@ use proptest::test_runner::{Config, TestRunner}; use crate::harness::{Content, Hosts, LAYER, OPENVMM, Outcome, Root, SCRATCH, in_parallel, native}; use crate::lifecycle::exported; -use crate::openvmm::{ - DISCONNECTED_CONSOLE, NO_CONSOLE, Options, TERMINAL_CONSOLE, child, cold_boot, managed_run, - restore, share_token, snapshot, -}; +use crate::openvmm::{Options, child, cold_boot, managed_run, share_token}; /// An egress policy file with an allowed range, an exclusion, a port, and a denied protocol, which /// compiles into the rules [`COMPILED_ALLOW`] and [`COMPILED_DENY`]. @@ -46,9 +43,7 @@ struct DryRun { refusal: Option<&'static str>, } -/// Returns the dry run `name` of `arguments`, which prints an OpenVMM command line of `options`, -/// in a root that holds a snapshot without a boot console at `{root/snapshot}`, which a restore -/// restores unless the case writes another manifest there. +/// Returns the dry run `name` of `arguments`, which prints an OpenVMM command line of `options`. fn dry_run( name: &'static str, arguments: impl IntoIterator, @@ -57,7 +52,7 @@ fn dry_run( DryRun { name, hosts: Hosts::Every, - files: vec![snapshot(&NO_CONSOLE)], + files: Vec::new(), removed: Vec::new(), arguments: arguments .into_iter() @@ -80,12 +75,6 @@ impl DryRun { self } - /// Returns the case with `manifest` as the manifest of the snapshot `{root/snapshot}`. - fn with_snapshot(self, manifest: &'static [u8]) -> Self { - let (relative, content) = snapshot(manifest); - self.with_file(relative, content) - } - /// Returns the case with the file `relative` removed first. fn without(mut self, relative: &'static str) -> Self { self.removed.push(relative); @@ -366,20 +355,6 @@ fn run_dry_runs() -> Vec { share_token(1, "/other", "ro", false), ] }; - let mut many = vec![ - "--restore-snapshot".to_owned(), - "{root/snapshot}".to_owned(), - ]; - for index in 0..20 { - many.extend([ - "--mount".to_owned(), - format!("/share/{index:02},{{root/share}}"), - ]); - } - let many: Vec<&str> = many.iter().map(String::as_str).collect(); - let many_children: Vec = (0..20) - .map(|index| mount_child(index, &format!("/share/{index:02}"), "{root/share}", "ro")) - .collect(); let fits = "x".repeat(crate::openvmm::two_share_command_line_room()); vec![ dry_run("run: Alpine default", run(&[]), cold_boot()), @@ -528,130 +503,6 @@ fn run_dry_runs() -> Vec { run(&["--cpu-profile", "host"]), cold_boot().with("--cpu-profile", ["host"]), ), - dry_run( - "run: restore", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore(), - ), - // A restore reads its snapshot's manifest, dry or not, and attaches the boot console to - // the terminal again where the snapshot's capture did, as OpenVMM requires, with the - // other options of a restore. - dry_run( - "run: restore of a boot console on the terminal", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore().with("--virtio-console", ["console"]), - ) - .with_snapshot(&TERMINAL_CONSOLE), - dry_run( - "run: restore of a boot console on the terminal with every option", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--restore-processors", - "1", - "--restore-network-policy", - "--network-profile", - "portable", - "--outcome-report", - "{root/reports/restore.json}", - ]), - restore() - .with("--restore-processors", ["1"]) - .flag("--restore-network-policy") - .with("--network-profile", ["portable"]) - .with("--virtio-console", ["console"]) - .with("--microvm-report", ["{root/reports/restore.json}"]), - ) - .with_snapshot(&TERMINAL_CONSOLE), - dry_run( - "run: restore of a disconnected boot console", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore(), - ) - .with_snapshot(&DISCONNECTED_CONSOLE), - // A restore needs no guest kernel or initramfs. - dry_run( - "run: restore without guest artifacts", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore(), - ) - .without("build/vmlinux") - .without("build/initramfs.cpio.gz"), - dry_run( - "run: restore with every option", - run(&[ - "--hypervisor", - "mshv", - "--processors", - "4", - "--restore-snapshot", - "{root/snapshot}", - "--restore-processors", - "2", - "--restore-memory-mib", - "1024", - "--restore-ready-path", - "{root/ready.sock}", - "--cpu-profile", - "host", - "--outcome-report", - "{root/reports/restore.json}", - "--cmdline", - "quiet", - ]), - restore() - .with("--processors", ["4"]) - .with("--hypervisor", ["mshv"]) - .with("--restore-processors", ["2"]) - .with("--restore-memory", ["1024M"]) - .with("--restore-ready-path", ["{root/ready.sock}"]) - .with("--cpu-profile", ["host"]) - .with("--microvm-report", ["{root/reports/restore.json}"]) - .with("--cmdline", ["quiet"]), - ), - dry_run( - "run: restore of the Alpine guest", - run(&["--guest", "alpine", "--restore-snapshot", "{root/snapshot}"]), - restore(), - ), - dry_run( - "run: restore with equal processors", - run(&[ - "--processors", - "2", - "--restore-processors", - "2", - "--restore-snapshot", - "{root/snapshot}", - ]), - restore() - .with("--processors", ["2"]) - .with("--restore-processors", ["2"]), - ), - dry_run( - "run: restore ignores the memory", - run(&[ - "--memory-mib", - "512", - "--restore-snapshot", - "{root/snapshot}", - ]), - restore(), - ), - // Paths print with their components alone, as `Path::components` reads them, relative - // ones included: a leading `.` stays. - dry_run( - "run: restore paths", - run(&[ - "--restore-snapshot", - "./snapshot/", - "--restore-ready-path", - "run//ready.sock", - ]), - restore() - .with("--restore-snapshot", [native("./snapshot")]) - .with("--restore-ready-path", [native("run/ready.sock")]), - ), dry_run( "run: one share", run(&["--mount", "/data,{root/share}"]), @@ -885,53 +736,6 @@ fn run_dry_runs() -> Vec { .with("--mount-owner", ["vmm"]) .kernel(&two_tokens()), ), - // A restore adds no boot tokens: its guest keeps the binds that its boot made. - dry_run( - "run: restore with shares", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--mount", - "/data,{root/share},rw", - "--mount", - "/other,{root/share2}", - "--mount", - "/etc/single.txt,{root/single.txt}", - ]), - restore() - .with("--mount-aggregate", ["/run/nvx/shares"]) - .with( - "--mount-child", - [ - mount_child(0, "/data", "{root/share}", "rw"), - mount_child(1, "/other", "{root/share2}", "ro"), - mount_child(2, "/etc/single.txt", "{root/single.txt}", "ro,file"), - ], - ), - ), - dry_run( - "run: restore with shares and a command line", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--mount", - "/data,{root/share},rw", - "--mount", - "/other,{root/share2}", - "--cmdline", - "quiet", - ]), - restore() - .with("--mount-aggregate", ["/run/nvx/shares"]) - .with( - "--mount-child", - [ - mount_child(0, "/data", "{root/share}", "rw"), - mount_child(1, "/other", "{root/share2}", "ro"), - ], - ) - .with("--cmdline", ["quiet"]), - ), // A proxy is exclusive of direct egress allow rules. dry_run( "run: network", @@ -1035,94 +839,6 @@ fn run_dry_runs() -> Vec { cold_boot().with("--network-egress", ["deny"]), ) .refusing("network policy options require --net"), - // A restore takes the addressing of its NIC from the snapshot, so it passes the profile - // and the policy without `--net`. - dry_run( - "run: restore of a networked snapshot", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--network-profile", - "portable", - ]), - restore().with("--network-profile", ["portable"]), - ), - dry_run( - "run: restore with a network policy", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--network-profile", - "portable", - "--network-egress", - "deny", - "--network-egress-allow", - "10.0.0.0/8:tcp:443", - "--network-egress-deny", - "10.1.0.0/16", - "--host-loopback", - "deny", - "--network-proxy", - "10.0.0.1:3128", - ]), - restore() - .with("--network-profile", ["portable"]) - .with("--network-egress", ["deny"]) - .with("--network-egress-allow", ["10.0.0.0/8:tcp:443"]) - .with("--network-egress-deny", ["10.1.0.0/16"]) - .with("--host-loopback", ["deny"]) - .with("--network-proxy", ["10.0.0.1:3128"]), - ), - dry_run( - "run: restore with an egress policy file", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--network-profile", - "portable", - "--network-egress", - "deny", - "--network-egress-policy-file", - "{root/policy.json}", - ]), - restore() - .with("--network-profile", ["portable"]) - .with("--network-egress", ["deny"]) - .with("--network-egress-allow", COMPILED_ALLOW) - .with("--network-egress-deny", COMPILED_DENY), - ) - .with_file("policy.json", POLICY), - // With `--restore-network-policy`, a restore installs the policy of its network options - // in place of a NIC template's deny-all, ingress, explicit host loopback, and forwards - // included, which OpenVMM admits only with the option. - dry_run( - "run: restore that replaces the network policy", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--restore-network-policy", - "--network-profile", - "portable", - "--network-egress", - "deny", - "--network-egress-allow", - "10.0.0.1/32:udp:53", - "--network-ingress", - "allow", - "--host-loopback", - "allow", - "--host-loopback-forward", - "tcp:8080:80", - ]), - restore() - .flag("--restore-network-policy") - .with("--network-profile", ["portable"]) - .with("--network-egress", ["deny"]) - .with("--network-egress-allow", ["10.0.0.1/32:udp:53"]) - .with("--network-ingress", ["allow"]) - .with("--host-loopback", ["allow"]) - .with("--host-loopback-forward", ["tcp:8080:80"]), - ), dry_run( "run: host loopback forward", run(&["--host-loopback-forward", "tcp:8080:80"]), @@ -1188,16 +904,6 @@ fn run_dry_runs() -> Vec { .refusing( "boot.command_line cannot contain double quotes or a standalone -- with the OpenVMM control console", ), - dry_run( - "run: restore retains quoted command line and delimiter", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--cmdline", - "a=\"b c\" -- d", - ]), - restore().with("--cmdline", ["a=\"b c\" -- d"]), - ), dry_run( "run: command line starting with a dash", run(&["--cmdline=-x"]), @@ -1223,13 +929,6 @@ fn run_dry_runs() -> Vec { run(&["--cmdline", "a", "--cmdline", "b"]), cold_boot().with("--cmdline", ["b"]), ), - dry_run( - "run: restore without the budget", - run(&many), - restore() - .with("--mount-aggregate", ["/run/nvx/shares"]) - .with("--mount-child", many_children), - ), // The command line and the boot tokens fill the 1024 bytes that OpenVMM leaves them. dry_run( "run: budget fits", @@ -1291,11 +990,7 @@ fn dry_runs_print_the_openvmm_command_and_launch_nothing() { let cold = case .arguments .first() - .is_some_and(|command| command == "run") - && !case - .arguments - .iter() - .any(|argument| argument == "--restore-snapshot"); + .is_some_and(|command| command == "run"); let outcome = root.nvx_rs(&arguments); eprintln!("case: {}", case.name); if let Some(message) = case.refusal { diff --git a/nvx-rs/tests/cli/one_shot.rs b/nvx-rs/tests/cli/one_shot.rs index d49dc5356..fca1c2af5 100644 --- a/nvx-rs/tests/cli/one_shot.rs +++ b/nvx-rs/tests/cli/one_shot.rs @@ -5,8 +5,7 @@ use crate::harness::{Content, OPENVMM, Root, in_parallel, native}; use crate::openvmm::{ - DISCONNECTED_CONSOLE, NO_CONSOLE, NO_NETWORK, NetworkPolicy, Options, TERMINAL_CONSOLE, - assert_outcome_report, child, cold_boot, restore, share_token, snapshot, + NO_NETWORK, NetworkPolicy, Options, assert_outcome_report, child, cold_boot, share_token, }; /// The outcome report that OpenVMM writes for a run that asks for one. @@ -130,10 +129,6 @@ fn fails_a_cold_boot(case: &Launch) -> bool { .first() .is_some_and(|command| command == "run") && case.status != 0 - && !case - .arguments - .iter() - .any(|argument| argument == "--restore-snapshot") } /// Returns `nvx-rs run OPTIONS`. @@ -148,7 +143,6 @@ fn run(options: &[&str]) -> Vec { /// Returns the runs of `nvx-rs run`. fn run_launches() -> Vec { let booted = "openvmm: booted the microVM\n"; - let restored = "openvmm: restored the microVM\n"; vec![ launch("run: boot", run(&[]), cold_boot(), 0, booted), launch( @@ -199,65 +193,6 @@ fn run_launches() -> Vec { 0, booted, ), - // A restore records its command line next to its snapshot, since it has no kernel. - launch( - "run: restore", - run(&[ - "--processors", - "4", - "--restore-snapshot", - "{root/snapshot}", - "--restore-processors", - "2", - "--restore-memory-mib", - "1024", - "--restore-ready-path", - "{root/ready.sock}", - "--cpu-profile", - "auto", - ]), - restore() - .with("--processors", ["4"]) - .with("--restore-processors", ["2"]) - .with("--restore-memory", ["1024M"]) - .with("--restore-ready-path", ["{root/ready.sock}"]) - .with("--cpu-profile", ["auto"]), - 0, - restored, - ) - .with_file(snapshot(&NO_CONSOLE)) - .recorded_in(Some("fake-openvmm-.json")), - launch( - "run: restore without guest artifacts", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore(), - 0, - restored, - ) - .with_file(snapshot(&NO_CONSOLE)) - .without("build/vmlinux") - .without("build/initramfs.cpio.gz") - .recorded_in(Some("fake-openvmm-.json")), - // A restore attaches the boot console to the terminal again where its capture did, as - // OpenVMM requires, and leaves any other boot console as its snapshot records it. - launch( - "run: restore of a boot console on the terminal", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore().with("--virtio-console", ["console"]), - 0, - restored, - ) - .with_file(snapshot(&TERMINAL_CONSOLE)) - .recorded_in(Some("fake-openvmm-.json")), - launch( - "run: restore of a disconnected boot console", - run(&["--restore-snapshot", "{root/snapshot}"]), - restore(), - 0, - restored, - ) - .with_file(snapshot(&DISCONNECTED_CONSOLE)) - .recorded_in(Some("fake-openvmm-.json")), launch( "run: outcome report", run(&["--outcome-report", "{root/reports/openvmm-run.json}"]), @@ -272,108 +207,6 @@ fn run_launches() -> Vec { status: 0, network: NO_NETWORK, }), - launch( - "run: restore outcome report", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--outcome-report", - "{root/reports/openvmm-run.json}", - "--cmdline", - "fake_exit_status=4", - ]), - restore() - .with("--microvm-report", ["{root/reports/openvmm-run.json}"]) - .with("--cmdline", ["fake_exit_status=4"]), - 4, - restored, - ) - .with_file(snapshot(&NO_CONSOLE)) - .recorded_in(Some("fake-openvmm-reports.json")) - .reporting(Report { - path: "reports/openvmm-run.json", - category: "guest-exit", - status: 4, - network: NO_NETWORK, - }), - // A restore passes the network profile and policy of a snapshot that has a NIC without - // `--net`, which OpenVMM refuses on a restore. - launch( - "run: restore of a networked snapshot", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--network-profile", - "portable", - "--network-egress", - "deny", - "--network-egress-allow", - "10.0.0.0/8:tcp:443", - "--host-loopback", - "deny", - "--outcome-report", - "{root/reports/openvmm-run.json}", - ]), - restore() - .with("--network-profile", ["portable"]) - .with("--network-egress", ["deny"]) - .with("--network-egress-allow", ["10.0.0.0/8:tcp:443"]) - .with("--host-loopback", ["deny"]) - .with("--microvm-report", ["{root/reports/openvmm-run.json}"]), - 0, - restored, - ) - .with_file(snapshot(&NO_CONSOLE)) - .recorded_in(Some("fake-openvmm-reports.json")) - .reporting(Report { - path: "reports/openvmm-run.json", - category: "success", - status: 0, - network: ("applied", "rules", 1, 0, "deny"), - }), - // A restore that replaces the policy of a NIC template may publish the guest's listeners - // and forward host ports, which OpenVMM refuses at every other restore. - launch( - "run: restore that replaces the network policy", - run(&[ - "--restore-snapshot", - "{root/snapshot}", - "--restore-network-policy", - "--network-profile", - "portable", - "--network-egress", - "deny", - "--network-egress-allow", - "10.0.0.1/32:udp:53", - "--network-ingress", - "allow", - "--host-loopback", - "allow", - "--host-loopback-forward", - "tcp:8080:80", - "--outcome-report", - "{root/reports/openvmm-run.json}", - ]), - restore() - .flag("--restore-network-policy") - .with("--network-profile", ["portable"]) - .with("--network-egress", ["deny"]) - .with("--network-egress-allow", ["10.0.0.1/32:udp:53"]) - .with("--network-ingress", ["allow"]) - .with("--host-loopback", ["allow"]) - .with("--host-loopback-forward", ["tcp:8080:80"]) - .with("--microvm-report", ["{root/reports/openvmm-run.json}"]), - 0, - restored, - ) - .with_file(snapshot(&NO_CONSOLE)) - .recorded_in(Some("fake-openvmm-reports.json")) - .reporting(Report { - path: "reports/openvmm-run.json", - category: "success", - status: 0, - network: ("applied", "rules", 1, 0, "allow"), - }), launch( "run: network outcome report", run(&[ @@ -516,11 +349,7 @@ fn one_shot_runs_exit_with_the_workload_or_openvmm_status() { let cold = case .arguments .first() - .is_some_and(|command| command == "run") - && !case - .arguments - .iter() - .any(|argument| argument == "--restore-snapshot"); + .is_some_and(|command| command == "run"); let before = case.refusal.map(|_| root.snapshot()); let outcome = root.nvx_rs_with(&arguments, &environment); eprintln!("case: {}", case.name); diff --git a/nvx-rs/tests/cli/openvmm.rs b/nvx-rs/tests/cli/openvmm.rs index ffba28faf..a22e37327 100644 --- a/nvx-rs/tests/cli/openvmm.rs +++ b/nvx-rs/tests/cli/openvmm.rs @@ -1,17 +1,15 @@ //! OpenVMM command lines: their options, each with its values, and the options with which the //! launches of `nvx-rs` start OpenVMM, as its documentation describes them, from which the tests //! derive what they expect, and the checks of the images that a launch hands OpenVMM, of the -//! outcome reports that OpenVMM writes, and of the control endpoints of managed starts. It also -//! writes the manifests of the snapshots that `nvx-rs run` restores, as OpenVMM writes them. +//! outcome reports that OpenVMM writes, and of the control endpoints of managed starts. use std::collections::{BTreeMap, BTreeSet}; use std::fmt::Write; use std::path::{Path, PathBuf}; -use std::sync::LazyLock; use serde_json::{Value, json}; -use crate::harness::{Content, HYPERVISOR, OPENVMM, Root, is_hex}; +use crate::harness::{HYPERVISOR, OPENVMM, Root, is_hex}; /// Caller bytes left by the two-directory fixture, its native bootstrap, and a terminating NUL. pub(crate) fn two_share_command_line_room() -> usize { @@ -679,106 +677,6 @@ pub(crate) fn cold_boot() -> Options { .with("--virtio-console", ["console"]) } -/// Returns the options with which `nvx-rs run --restore-snapshot {root/snapshot}` restores the -/// snapshot, without a guest kernel or initramfs. -pub(crate) fn restore() -> Options { - Options::default() - .flag("--single-process") - .with("--machine", ["microvm"]) - .with("--processors", ["1"]) - .with("--hypervisor", [HYPERVISOR]) - .with("--restore-snapshot", ["{root/snapshot}"]) -} - -/// The manifest of a snapshot whose boot console a capture attached to the terminal, with -/// `--virtio-console console`, which OpenVMM restores only with that option again. -pub(crate) static TERMINAL_CONSOLE: LazyLock> = LazyLock::new(|| { - snapshot_manifest(Some(( - true, - "require-inherited-attachment", - "provider", - "console", - ))) -}); - -/// The manifest of a snapshot whose boot console a capture left disconnected, with -/// `--virtio-console none`, as a template's. -pub(crate) static DISCONNECTED_CONSOLE: LazyLock> = LazyLock::new(|| { - snapshot_manifest(Some(( - false, - "discard-while-disconnected", - "disconnected", - "discard", - ))) -}); - -/// The manifest of a snapshot of a boot without a boot console, which OpenVMM restores only -/// without `--virtio-console`. -pub(crate) static NO_CONSOLE: LazyLock> = LazyLock::new(|| snapshot_manifest(None)); - -/// Returns the file that holds `manifest` as the manifest of the snapshot `{root/snapshot}`. -pub(crate) fn snapshot(manifest: &'static [u8]) -> (&'static str, Content) { - ("snapshot/manifest.bin", Content::Bytes(manifest)) -} - -/// Returns the manifest of a snapshot whose machine contract holds the attachment of the boot -/// console that `console` describes, whether a restore requires it, its reconnect policy, the kind -/// of its identity, and its identity, and the attachment of the control console of a managed VM, -/// or the latter alone without `console`. -/// -/// It holds what `nvx-rs run` reads, its format version and magic and its contract, which -/// `openvmm.snapshot.SnapshotManifest`, `SnapshotMachineContract`, and `SnapshotAttachment` of -/// OpenVMM's `openvmm_helpers` define, as `mesh` encodes them: each a protobuf message, which omits -/// a field whose value is zero or empty. -fn snapshot_manifest(console: Option<(bool, &str, &str, &str)>) -> Vec { - fn put_varint(bytes: &mut Vec, mut value: u64) { - while value >= 0x80 { - bytes.push(value as u8 | 0x80); - value >>= 7; - } - bytes.push(value as u8); - } - fn bytes(message: &mut Vec, number: u64, value: &[u8]) { - put_varint(message, number << 3 | 2); - put_varint(message, value.len() as u64); - message.extend_from_slice(value); - } - fn attachment(stable_id: &str, kind: &str, fields: (bool, &str, &str, &str)) -> Vec { - let (required, policy, identity_kind, identity) = fields; - let mut message = Vec::new(); - bytes(&mut message, 1, stable_id.as_bytes()); - bytes(&mut message, 2, kind.as_bytes()); - if required { - message.extend([3 << 3, 1]); - } - bytes(&mut message, 4, policy.as_bytes()); - bytes(&mut message, 5, identity_kind.as_bytes()); - bytes(&mut message, 6, identity.as_bytes()); - message - } - let mut contract = Vec::new(); - bytes(&mut contract, 1, b"microvm"); - if let Some(console) = console { - let console = attachment("console:microvm-virtio0", "virtio-console", console); - bytes(&mut contract, 10, &console); - } - let control = attachment( - "console:microvm-control0", - "virtio-control-console", - ( - false, - "broker-authenticated-listener", - "unix-socket", - "/run/nvx/control.sock", - ), - ); - bytes(&mut contract, 10, &control); - let mut manifest = vec![1 << 3, 2]; - bytes(&mut manifest, 9, &contract); - bytes(&mut manifest, 10, b"OPENVMM_SNAPSHOT_V2\0"); - manifest -} - /// Returns the name of the aggregate child at `index` that the guest binds at `target`: the /// index, a hyphen, and the first 32 hexadecimal digits of the SHA-256 digest of the target, which /// a snapshot pins. diff --git a/nvx-rs/tests/cli/usage.rs b/nvx-rs/tests/cli/usage.rs index 9c790a12d..a1e676945 100644 --- a/nvx-rs/tests/cli/usage.rs +++ b/nvx-rs/tests/cli/usage.rs @@ -424,10 +424,16 @@ const USAGE_ERRORS: &[UsageError] = &[ fragments: &["--memory-capacity-mib", "\"1e3\""], }, UsageError { - name: "run: restore memory not an integer", - arguments: &["run", "--restore-memory-mib", ""], - code: "argument-invalid", - fragments: &["--restore-memory-mib", "\"\""], + name: "run: restore memory option removed", + arguments: &[ + "run", + "--restore-snapshot", + "base", + "--restore-memory-mib", + "512", + ], + code: "argument-unrecognized", + fragments: &["--restore-memory-mib", "512"], }, UsageError { name: "run: memory in other digits", @@ -507,17 +513,62 @@ const USAGE_ERRORS: &[UsageError] = &[ code: "argument-invalid", fragments: &["--processors", "\"-5\""], }, + // A restore takes the processors that its snapshot had online, its own readiness endpoint, + // and always installs the network policy of its options, so these options are gone. + UsageError { + name: "run: restore processors option removed", + arguments: &[ + "run", + "--restore-snapshot", + "base", + "--restore-processors", + "2", + ], + code: "argument-unrecognized", + fragments: &["--restore-processors", "2"], + }, + UsageError { + name: "run: restore ready path option removed", + arguments: &[ + "run", + "--restore-snapshot", + "base", + "--restore-ready-path", + "ready.sock", + ], + code: "argument-unrecognized", + fragments: &["--restore-ready-path", "ready.sock"], + }, UsageError { - name: "run: restore processors not an integer", - arguments: &["run", "--restore-processors", "x"], + name: "run: restore network policy option removed", + arguments: &[ + "run", + "--restore-snapshot", + "base", + "--restore-network-policy", + ], + code: "argument-unrecognized", + fragments: &["--restore-network-policy"], + }, + // A restore takes no option that `snapshot restore` refuses, such as the exec-only `--wait`. + UsageError { + name: "run: restore wait option", + arguments: &["run", "--restore-snapshot", "base", "--wait", "5"], + code: "argument-unrecognized", + fragments: &["--wait", "5"], + }, + // The restore options parse as `snapshot restore` parses them. + UsageError { + name: "run: restore upper not a choice", + arguments: &["run", "--restore-snapshot", "base", "--upper", "disk"], code: "argument-invalid", - fragments: &["--restore-processors", "\"x\""], + fragments: &["--upper", "\"disk\""], }, UsageError { - name: "run: restore processors not a choice", - arguments: &["run", "--restore-processors", "16"], + name: "run: zero restore timeout", + arguments: &["run", "--restore-snapshot", "base", "--timeout", "0"], code: "argument-invalid", - fragments: &["--restore-processors", "\"16\""], + fragments: &["--timeout", "greater than 0"], }, UsageError { name: "run: invalid mount owner", @@ -556,7 +607,7 @@ const USAGE_ERRORS: &[UsageError] = &[ fragments: &["--memory-mib", "one argument"], }, UsageError { - name: "run: missing path", + name: "run: missing snapshot name", arguments: &["run", "--restore-snapshot"], code: "argument-missing", fragments: &["--restore-snapshot", "one argument"], @@ -702,7 +753,10 @@ const HELP: [Help; 6] = [ usage: "usage: nvx-rs run [OPTIONS]", options: &[ "--guest {alpine,ubuntu,azurelinux}", - "--restore-snapshot", + "--restore-snapshot NAME", + "--store-dir PATH", + "--state-dir STATE_DIR", + "--layer ROLE,PATH,IMAGE_UUID", "--dry-run", ], }, diff --git a/nvx-rs/tests/cli/validation.rs b/nvx-rs/tests/cli/validation.rs index d287b7e2a..631160a69 100644 --- a/nvx-rs/tests/cli/validation.rs +++ b/nvx-rs/tests/cli/validation.rs @@ -775,57 +775,51 @@ fn run_refusals() -> Vec { refusal( "run: net of a restore", "unsupported", - run_dry(&[ + [ + "run", "--restore-snapshot", - "{root/snapshot}", + "base", + "--store-dir", + "{root/store}", "--net", "10.0.0.2/24", "--network-profile", "portable", - ]), - &["--net", "restore", "network addressing"], + ], + &["--net is not accepted for restore"], ), refusal( "run: net without profile of a restore", "unsupported", - run_dry(&[ + [ + "run", "--restore-snapshot", - "{root/snapshot}", + "base", + "--store-dir", + "{root/store}", "--net", "10.0.0.2/24", - ]), - &["--net", "restore", "network addressing"], + ], + &["--net is not accepted for restore"], ), + // A cold boot takes none of the options of a restore. refusal( - "run: ready path without snapshot", + "run: store without snapshot", "unsupported", - run_dry(&["--restore-ready-path", "{root/ready.sock}"]), - &["--restore-ready-path requires --restore-snapshot"], + run_dry(&["--store-dir", "{root/store}"]), + &["--store-dir requires --restore-snapshot"], ), - // A cold boot installs the policy of its options anyway. refusal( - "run: network policy replacement without snapshot", + "run: restore inputs without snapshot", "unsupported", - run_dry(&[ - "--restore-network-policy", - "--net", - "10.0.0.2/24", - "--network-profile", - "portable", - ]), - &["--restore-network-policy requires --restore-snapshot"], + run_dry(&["--layer", LAYER, "--upper", "ram"]), + &["--layer requires --restore-snapshot"], ), refusal( - "run: restore processors without snapshot", + "run: state directory without snapshot", "unsupported", - run_dry(&["--restore-processors", "1"]), - &["--restore-processors requires --restore-snapshot"], - ), - refusal( - "run: restore memory without snapshot", - "unsupported", - run_dry(&["--restore-memory-mib", "1024"]), - &["--restore-memory-mib requires --restore-snapshot"], + run_dry(&["--state-dir", "{root/state}"]), + &["--state-dir requires --restore-snapshot"], ), refusal( "run: memory capacity of a restore", @@ -836,7 +830,7 @@ fn run_refusals() -> Vec { "--memory-capacity-mib", "1024", ]), - &["--memory-capacity-mib", "fresh boot"], + &["--memory-capacity-mib is not accepted for restore"], ), refusal( "run: Ubuntu guest of a restore", @@ -869,58 +863,176 @@ fn run_refusals() -> Vec { &["--memory-capacity-mib", "--memory-mib"], ) .with_file("build/initramfs-ubuntu.cpio.gz", INITRAMFS), + // A restore names a snapshot of a store, and no longer a raw OpenVMM snapshot, which it + // refuses before it reads anything. refusal( - "run: restore processors above processors", + "run: raw snapshot without a store", "unsupported", - run_dry(&[ - "--processors", - "2", + ["run", "--restore-snapshot", "{root/snapshot}"], + &[ + "--restore-snapshot requires --store-dir", + "no longer a raw OpenVMM snapshot", + ], + ) + .with_file("snapshot/manifest.bin", Content::Bytes(&[1 << 3])), + refusal( + "run: raw snapshot in place of a name", + "unsupported", + [ + "run", "--restore-snapshot", "{root/snapshot}", - "--restore-processors", - "4", - ]), - &["--restore-processors", "--processors"], - ), - // A restore reads its snapshot's manifest, dry or not, to attach the boot console that - // the snapshot had, and refuses one that it cannot read, which OpenVMM could not restore. + "--store-dir", + "{root}", + ], + &["is not a snapshot name", "no longer a raw OpenVMM snapshot"], + ) + .with_file("snapshot/manifest.bin", Content::Bytes(&[1 << 3])), refusal( - "run: restore of a missing snapshot", + "run: restore from a missing store", "not-found", - run_dry(&["--restore-snapshot", "{root/missing}"]), - &["cannot read the snapshot manifest {root/missing/manifest.bin}: "], + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/missing}", + "--state-dir", + "{root/state}", + ], + &["{root/missing}"], ), + // The restore's sandbox needs a directory that the run creates, outside the store, and + // removes once it ends. refusal( - "run: restore of a missing snapshot without a dry run", - "not-found", - ["run", "--restore-snapshot", "{root/missing}"], - &["cannot read the snapshot manifest {root/missing/manifest.bin}: "], + "run: restore without a state directory", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + ], + &["--restore-snapshot requires --state-dir"], ), + // An empty path names the working directory, which the run would provision and remove. refusal( - "run: restore of a snapshot without a manifest", - "not-found", - run_dry(&["--restore-snapshot", "{root/snapshot}"]), - &["cannot read the snapshot manifest {root/snapshot/manifest.bin}: "], - ) - .with_file("snapshot/state.bin", Content::Text("state\n")), + "run: restore into an empty state directory path", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + "--state-dir=", + ], + &["--restore-snapshot requires --state-dir"], + ), refusal( - "run: restore of a malformed manifest", - "io", - run_dry(&["--restore-snapshot", "{root/snapshot}"]), - &["the snapshot manifest {root/snapshot/manifest.bin} is malformed: it is truncated"], - ) - .with_file("snapshot/manifest.bin", Content::Bytes(&[1 << 3])), + "run: restore into an existing state directory", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + "--state-dir", + "{root}", + ], + &["--state-dir {root} exists"], + ), refusal( - "run: restore of a manifest of another format", - "format-unsupported", - run_dry(&["--restore-snapshot", "{root/snapshot}"]), - &[ - "the snapshot manifest {root/snapshot/manifest.bin} has format version 3, which \ - this build does not read; capture the snapshot again with the OpenVMM of this \ - build", + "run: restore into the store", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + "--state-dir", + "{root/store/run}", ], - ) - .with_file("snapshot/manifest.bin", Content::Bytes(&[1 << 3, 3])), + &["state directory must be outside the snapshot store"], + ), + // The restore's workload is an exec request, which the run checks as `snapshot restore` + // does, before it opens the store or provisions anything. + refusal( + "run: restore of a relative entrypoint", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/missing}", + "--state-dir", + "{root/state}", + "--entrypoint", + "echo", + ], + &["managed exec entrypoint must be absolute"], + ), + // The snapshot and the restore's own sandbox fix what a cold boot sets, and a restore + // provisions a sandbox, which no dry run shows. + refusal( + "run: dry run of a restore", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + "--dry-run", + ], + &["--dry-run is not accepted for restore"], + ), + refusal( + "run: share of a restore", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + "--mount", + "/data,{root/share}", + ], + &["--mount is not accepted for restore"], + ), + refusal( + "run: command line of a restore", + "unsupported", + [ + "run", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + "--cmdline", + "quiet", + ], + &["--cmdline is not accepted for restore"], + ), + refusal( + "run: processors of a restore", + "unsupported", + [ + "run", + "--processors", + "2", + "--restore-snapshot", + "base", + "--store-dir", + "{root/store}", + ], + &["--processors is not accepted for restore"], + ), refusal( "run: policy file with allow rules", "unsupported", diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index dd15bf727..d5002a80b 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -56,7 +56,6 @@ "restore-downtime": 69.2, "restore-memory": 4.1, "restore-processors": 21.4, - "run-console-restore": 1.3, "sandbox-blocks": 0.7, "scratch-snapshot": 3.4, "smp": 21.2, @@ -74,7 +73,7 @@ }, "shards": [ { - "seconds": 269.1, + "seconds": 267.8, "scenarios": [ "command-line-limit", "console-snapshot", @@ -89,7 +88,6 @@ "restore-downtime", "restore-memory", "restore-processors", - "run-console-restore", "smp-snapshot", "snapshot-core", "snapshot-tiers", @@ -166,7 +164,6 @@ "restore-downtime": 71.0, "restore-memory": 4.0, "restore-processors": 21.3, - "run-console-restore": 1.1, "sandbox-blocks": 0.6, "scratch-snapshot": 3.2, "smp": 22.6, @@ -209,7 +206,7 @@ ] }, { - "seconds": 249.8, + "seconds": 248.7, "scenarios": [ "console-exit", "cpu-profile-fallback", @@ -231,7 +228,6 @@ "platform-network-policy", "private-network-ingress", "restore-memory", - "run-console-restore", "sandbox-blocks", "structured-outcome", "time-abi-conformance", @@ -276,7 +272,6 @@ "restore-downtime": 84.7, "restore-memory": 12.5, "restore-processors": 28.8, - "run-console-restore": 5.8, "sandbox-blocks": 1.2, "scratch-snapshot": 8.8, "smp": 30.9, @@ -294,22 +289,21 @@ }, "shards": [ { - "seconds": 356.1, + "seconds": 354.1, "scenarios": [ "denied-filesystem-paths", "directional-network-policy", "endpoint-policy-snapshot", - "filesystem-comma-paths", "filesystem-policy", "host-loopback-policy", "interrupt-map", "network-snapshot", + "platform-aggregate-export", "platform-filesystem", "platform-restore-processors", "restore-downtime", "restore-memory", "restore-processors", - "run-console-restore", "scratch-snapshot", "smp", "smp-snapshot", @@ -318,12 +312,13 @@ ] }, { - "seconds": 357.5, + "seconds": 353.7, "scenarios": [ "command-line-limit", "console-exit", "console-snapshot", "cpu-profile-fallback", + "filesystem-comma-paths", "filesystem-owner", "filesystem-shares", "filesystem-snapshot", @@ -335,7 +330,6 @@ "l3-l4-egress-protocols", "lifecycle", "managed-lifecycle", - "platform-aggregate-export", "platform-network-policy", "platform-zero-fill", "private-network-ingress", diff --git a/scripts/nvx_tools/microvm_test_scripts/run-console-restore.sh b/scripts/nvx_tools/microvm_test_scripts/run-console-restore.sh deleted file mode 100644 index 50e9cabe0..000000000 --- a/scripts/nvx_tools/microvm_test_scripts/run-console-restore.sh +++ /dev/null @@ -1,5 +0,0 @@ -set -eu -# The restored shell read this script from the terminal, through the boot -# console that the restore of `nvx-rs run` attached to it again. -echo NVX-RUN-CONSOLE-RESTORE-OK -nvx-exit 0 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index e4dc5e1e8..be89fbd7c 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -130,7 +130,6 @@ "restore-downtime", "restore-memory", "restore-processors", - "run-console-restore", "sandbox-blocks", "sandbox-lifecycle", "sandbox-proxy-only", @@ -394,7 +393,6 @@ class MicrovmVirtioDevice(NamedTuple): BOOT_MARKER = b"NVX-GUEST-BOOT-OK:" GUEST_BOOT_COMPLETION_MARKER = b"NVX-GUEST-BOOT-CHECK-OK" GUEST_IDENTITY_COMPLETION_MARKER = b"NVX-GUEST-IDENTITY-OK" -RUN_CONSOLE_RESTORE_MARKER = b"NVX-RUN-CONSOLE-RESTORE-OK" RESTORE_PROCESSORS_FAILURE_MARKER = b"NVX-RESTORE-PROCESSORS-FAIL" # A platform snapshot opts into restore-time vCPU activation as an untiered one # does, by capturing its capacity booted with maxcpus=. A capacity of 4 leaves @@ -8692,73 +8690,6 @@ def run_console_snapshot( ) -def run_run_console_restore( - executable: Path, - kernel: Path, - initrd: Path, - backend: str, - *, - memory_mib: int, - timeout: float, - output_dir: Path, -) -> None: - """Restore through `nvx-rs run` a snapshot whose boot console was on the - terminal, and type into the restored guest's shell. - - A direct cold boot attaches the boot console to OpenVMM's standard - streams with `--virtio-console console`, and the guest captures itself - there. OpenVMM restores such a snapshot only with `--virtio-console - console` again, which `nvx-rs run` must pass, as its snapshot's manifest - asks, so that the restored shell takes the terminal's input: the scenario - types a script into it, which shuts the guest down.""" - with tempfile.TemporaryDirectory(prefix="nvx-run-console-restore-") as temporary: - snapshot_path = Path(temporary) / "snapshot" - boot_command = workload_boot_command( - executable, - backend, - kernel, - initrd, - memory_mib, - "quiet loglevel=0", - ) - # The restored guest keeps its shell after the restore marker, for the - # script that the restore types. - capture_snapshot( - [*boot_command, "--snapshot-destination", str(snapshot_path)], - snapshot_path, - timeout=timeout, - teardown_mode="host-terminate", - log_path=output_dir / "run-console-restore-capture.log", - ) - fingerprint = _snapshot_fingerprint(snapshot_path) - command = [ - *nvx_rs.command_line(("run",)), - "--hypervisor", - backend, - "--restore-snapshot", - str(snapshot_path), - ] - result = run_guest_script( - command, - _read_script("run-console-restore.sh"), - RUN_CONSOLE_RESTORE_MARKER, - timeout=timeout, - log_path=output_dir / "run-console-restore.log", - boot_marker=RESTORE_MARKER, - contain_process_tree=True, - ) - printed = result["text"].split("\n", 1)[0].removesuffix("\r") - if not printed.startswith(">> ") or not printed.endswith( - " --virtio-console console" - ): - raise RuntimeError( - "nvx-rs run did not attach the boot console of the snapshot to " - f"the terminal: {printed}" - ) - if _snapshot_fingerprint(snapshot_path) != fingerprint: - raise RuntimeError("the restore of nvx-rs run modified the snapshot") - - def run_endpoint_policy_snapshot( executable: Path, kernel: Path, @@ -10968,21 +10899,6 @@ def _run_scenarios( timeout=args.timeout, output_dir=output_dir, ) - if "run-console-restore" in scenarios: - report.start( - "run-console-restore", - "Running nvx-rs run restore of a terminal boot console on " - f"OpenVMM/{args.backend}", - ) - run_run_console_restore( - executable, - kernel, - initrd, - args.backend, - memory_mib=args.memory_mib, - timeout=args.timeout, - output_dir=output_dir, - ) if "cpu-profile-fallback" in scenarios: report.start( "cpu-profile-fallback", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 34640cd12..dd9befaf8 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -4036,129 +4036,6 @@ def require(path: Path, _description: str) -> Path: self.assertIn("nvx-exit 0", script) -class RunConsoleRestoreScenarioTests(unittest.TestCase): - """The run-console-restore scenario, with its capture, `nvx-rs run`, and - the guest simulated.""" - - def _run( - self, - *, - printed: str = ">> openvmm --restore-snapshot snapshot --virtio-console console", - fingerprints: tuple[str, str] = ("snapshot", "snapshot"), - ) -> tuple[MagicMock, MagicMock, MagicMock]: - result = {"text": f"{printed}\r\nNVX-RUN-CONSOLE-RESTORE-OK\r\n"} - with ( - tempfile.TemporaryDirectory() as temporary, - patch.object( - microvm_tests, - "workload_boot_command", - return_value=["openvmm", "--virtio-console", "console"], - ) as boot_command, - patch.object(microvm_tests, "capture_snapshot") as capture_snapshot, - patch.object( - microvm_tests, "_snapshot_fingerprint", side_effect=fingerprints - ), - patch.object( - microvm_tests, "run_guest_script", return_value=result - ) as run_guest_script, - patch.object(nvx_rs, "require_binary", return_value=_FAKE_NVX_RS), - ): - microvm_tests.run_run_console_restore( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "kvm", - memory_mib=128, - timeout=1.0, - output_dir=Path(temporary), - ) - return boot_command, capture_snapshot, run_guest_script - - def test_scenario_restores_a_terminal_console_through_nvx_rs_run(self): - boot_command, capture_snapshot, run_guest_script = self._run() - # The capture's boot console is on OpenVMM's standard streams, and its - # restored guest keeps its shell for the script. - boot_command.assert_called_once_with( - Path("openvmm"), - "kvm", - Path("vmlinux"), - Path("initrd"), - 128, - "quiet loglevel=0", - ) - capture, snapshot = capture_snapshot.call_args.args - self.assertEqual( - capture, - [ - "openvmm", - "--virtio-console", - "console", - "--snapshot-destination", - str(snapshot), - ], - ) - self.assertEqual( - capture_snapshot.call_args.kwargs["teardown_mode"], "host-terminate" - ) - # nvx-rs run restores it without a console option of its own, and the - # console types the script once the restored guest prints its marker. - command, script, marker = run_guest_script.call_args.args - self.assertEqual( - command, - [ - *_FAKE_PUBLIC_RUN, - "--hypervisor", - "kvm", - "--restore-snapshot", - str(snapshot), - ], - ) - self.assertEqual(marker, microvm_tests.RUN_CONSOLE_RESTORE_MARKER) - self.assertEqual(script, microvm_tests._read_script("run-console-restore.sh")) - self.assertIn("echo NVX-RUN-CONSOLE-RESTORE-OK\nnvx-exit 0\n", script) - kwargs = run_guest_script.call_args.kwargs - self.assertEqual(kwargs["boot_marker"], microvm_tests.RESTORE_MARKER) - self.assertIs(kwargs["contain_process_tree"], True) - self.assertIn("run-console-restore", microvm_tests.MICROVM_TEST_SCENARIOS) - - def test_scenario_requires_the_boot_console_of_the_restore(self): - with self.assertRaisesRegex(RuntimeError, "did not attach the boot console"): - self._run(printed=">> openvmm --restore-snapshot snapshot") - - def test_scenario_requires_the_snapshot_unchanged(self): - with self.assertRaisesRegex(RuntimeError, "modified the snapshot"): - self._run(fingerprints=("snapshot", "changed")) - - def test_runner_dispatches_the_scenario(self): - def require(path: Path, _description: str) -> Path: - return path - - with ( - tempfile.TemporaryDirectory() as temporary, - patch.object(microvm_tests, "validate_openvmm_test_backend"), - patch.object(microvm_tests, "require_file", side_effect=require), - patch.object(microvm_tests, "run_run_console_restore") as scenario, - ): - args = nvx.parse_args( - [ - "test-microvm", - "--backend", - "mshv", - "--scenario", - "run-console-restore", - "--output-dir", - temporary, - ] - ) - self.assertEqual(microvm_tests.run(args), 0) - scenario.assert_called_once() - self.assertEqual(scenario.call_args.args[3], "mshv") - self.assertEqual( - scenario.call_args.kwargs, - {"memory_mib": 128, "timeout": 60.0, "output_dir": Path(temporary)}, - ) - - class _PortExclusionSockets: """Create sockets that refuse ports excluded per protocol, as Windows does.""" diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 1348a34dd..306ad8926 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -15,7 +15,6 @@ import os import queue import re -import select import shlex import shutil import signal @@ -2544,76 +2543,6 @@ def test_the_real_nvx_rs_prints_what_nvx_py_reads(self): dashed, nvx_rs.EgressRules(allow=("192.0.2.0/24",), deny=()) ) - @unittest.skipUnless( - os.environ.get("NVX_RS"), "set NVX_RS to an nvx-rs executable to run" - ) - def test_run_leaves_openvmm_the_terminal_to_restore(self): - """A guest-triggered shutdown can end OpenVMM while its console holds - the terminal in raw mode, and OpenVMM restores the terminal settings - only when its standard error is a terminal, so nvx-rs run must leave - it one.""" - if sys.platform != "linux": - self.skipTest("needs a Linux pseudo-terminal") - import pty - import termios - - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary).resolve() - # OpenVMM saves the settings through standard error, if it is a - # terminal, and its console makes standard input raw until the - # guest shuts down. A restore needs no kernel or initramfs. - openvmm = root / "openvmm" - openvmm.write_text( - f"#!{sys.executable}\n" - "import os, termios, tty\n" - "saved = termios.tcgetattr(2) if os.isatty(2) else None\n" - "tty.setraw(0)\n" - "if saved is not None:\n" - " termios.tcsetattr(2, termios.TCSAFLUSH, saved)\n", - encoding="utf-8", - ) - openvmm.chmod(0o755) - # nvx-rs run reads the snapshot's manifest before it starts - # OpenVMM, to attach the boot console that the snapshot recorded. - # This one, its format version and magic alone, records none. - snapshot = root / "snapshot" - snapshot.mkdir() - (snapshot / "manifest.bin").write_bytes( - b"\x08\x02\x52\x14OPENVMM_SNAPSHOT_V2\x00" - ) - primary, secondary = pty.openpty() - try: - before = termios.tcgetattr(secondary) - completed = subprocess.run( - [ - os.environ["NVX_RS"], - "--root", - str(root), - "run", - "--restore-snapshot", - str(snapshot), - ], - stdin=secondary, - stdout=secondary, - stderr=secondary, - cwd=root, - env={**os.environ, "NVX_OPENVMM": str(openvmm)}, - timeout=60, - check=False, - ) - after = termios.tcgetattr(secondary) - output = b"" - while select.select([primary], [], [], 0)[0]: - chunk = os.read(primary, 65536) - if not chunk: - break - output += chunk - finally: - os.close(secondary) - os.close(primary) - self.assertEqual(completed.returncode, 0, output.decode(errors="replace")) - self.assertEqual(after, before, "the terminal was left in raw mode") - class CiTests(unittest.TestCase): def test_openvmm_unit_tests_run_nextest_and_doctests(self): From ce25a7fa114a8516823dba6c6efb05d1ca5996b2 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 14:55:29 -0800 Subject: [PATCH 2/2] doc: retire run-console-restore in the case matrix (SL3.8b) SL3.8b removes `nvx-rs run`'s restore of a raw OpenVMM snapshot and the boot console that it reattached on the terminal, which the `run-console-restore` microVM scenario checked, so the case matrix gives that scenario the fate retired, and its summary of the microVM scenarios counts 48 kept and 1 retired. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/acceptance-case-matrix.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index 512b4fd0c..d88fec333 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -28,7 +28,7 @@ its own in turn. | --- | ---: | ---: | ---: | ---: | | [The adapter's scenarios](#the-adapters-scenarios) | 157 | 0 | 114 | 43 | | [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 | -| [MicroVM scenarios](#microvm-scenarios) | 52 | 49 | 3 | 0 | +| [MicroVM scenarios](#microvm-scenarios) | 52 | 48 | 3 | 1 | | [CI](#ci) | 69 | 43 | 3 | 23 | | [Docs](#docs) | 36 | 35 | 1 | 0 | @@ -330,7 +330,7 @@ suite in its shards, and the public sandbox scenarios in its guest and sandbox j | `restore-downtime` | kept | | | `restore-memory` | kept | | | `restore-processors` | kept | | -| `run-console-restore` | kept | | +| `run-console-restore` | retired | SL3.8b removes `nvx-rs run`'s restore of a raw OpenVMM snapshot and the boot console that it reattached on the terminal: `run --restore-snapshot` restores a named snapshot through `snapshot restore`'s path | | `sandbox-blocks` | kept | | | `sandbox-lifecycle` | kept | SL5.1b keeps it running, or gives it its fate | | `sandbox-proxy-only` | kept | SL5.1b keeps it running, or gives it its fate |