From 32ef4e09a926ead8b8d087751ce8df1545784e41 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 14:12:21 -0800 Subject: [PATCH 1/2] cli: let a named snapshot carry a NIC (SL3.7c) `snapshot capture` gave its VM no NIC, and a snapshot restores only with the NIC presence that it captured, so no named snapshot could restore with a network: the restore's own DNS server, egress, forwards, and published listeners, which `snapshot restore` (SL3.8a) and `run --restore-snapshot` (SL3.8b) install, reached no real guest. `snapshot capture --net IPV4/PREFIX` now gives the capture VM a NIC at that static address under the complete deny-all policy, with egress, ingress, and host-loopback access denied and no rules, forwards, proxy, or DNS server, the one policy whose NIC a restore may give a policy of its own. The library already attaches a capture's NIC under that policy and records the NIC in the capture context, the key, and `snapshot.json`, so `Snapshot::nic` returns it. Without `--net`, the capture is unchanged. `snapshot restore` now takes its network at the snapshot's NIC, as `run --restore-snapshot` does: both take the NIC that `snapshots::restore_source` returns, and the network options of the restore through `run::restored_network`. The network options of a snapshot with a NIC, which need `--network-profile`, become the restore's own policy at the NIC's address, through the same lowering, and those of a snapshot without a NIC are refused, before anything is provisioned, where the start's `Snapshot::accepts` refused them after the provision. `--net`, which only a cold boot takes, joins the options that `snapshot restore` leaves unrecognized: a restore keeps the address that its snapshot captured. The help of `snapshot restore`, `snapshot capture` and `run`, and the docs, say that a restore's NIC comes from its snapshot, and that `snapshot capture --net` gives a snapshot its NIC. Not here: re-addressing a restored VM's NIC, and snapshots of the running workloads of blockless VMs. Tests: the capture parser takes for `--net` only an address that OpenVMM accepts; a capture with `--net` attaches its VM's NIC under the complete deny-all policy alone, and its snapshot keeps the NIC and its address, in its key and description, while one without has no NIC; and `snapshot restore` refuses the network options of a snapshot without a NIC and a snapshot with a NIC without `--network-profile` before it provisions anything, and gives the options of one with a NIC the NIC's address. The fake VMM of the capture tests writes a manifest with a NIC for a VM that has one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/run.md | 6 +- doc/usage.md | 69 +++++--- nvx-rs/README.md | 48 +++--- nvx-rs/src/host/cli/run.rs | 17 +- nvx-rs/src/host/cli/snapshots.rs | 28 +++- nvx-rs/src/host/cli/snapshots/capture.rs | 24 ++- .../src/host/cli/snapshots/capture/tests.rs | 154 +++++++++++++++++- nvx-rs/src/host/cli/snapshots/tests.rs | 52 +++++- 8 files changed, 331 insertions(+), 67 deletions(-) diff --git a/doc/run.md b/doc/run.md index b448ee753..9edab4384 100644 --- a/doc/run.md +++ b/doc/run.md @@ -598,9 +598,11 @@ grants it connectivity that it never had. OpenVMM rejects the option for a cold boot, for a snapshot without a NIC or whose NIC was captured with any other policy, and for a `workload-start` or `instance-checkpoint` snapshot. A [platform NIC template](#platform-nic-templates) takes the option too, and so -does each restore of a named snapshot with a NIC through +does each restore of a named snapshot whose NIC `nvx-rs snapshot capture --net` +captured under the complete deny-all policy, through `nvx-rs run --restore-snapshot` or `nvx-rs snapshot restore`, which installs the -policy of its network options; see [`run`](usage.md#run). +policy of its network options; see +[`snapshot capture`](usage.md#snapshot-capture) and [`run`](usage.md#run). ### Platform NIC templates diff --git a/doc/usage.md b/doc/usage.md index 38fe88dcc..89fb66367 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -176,7 +176,7 @@ nvx-rs/target/release/nvx-rs run | `--mount-allow HOST_PATH` | none | Expose one existing file or directory inside a `--mount-deny` path again; the hidden directories on the way list only the entries that lead to it. Repeat to allow multiple paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. See [Access policy](run.md#access-policy). | | `--mount-write HOST_PATH` | none | Make one existing file or directory one of the only writable parts of an `rw` mapping; the guest's writes anywhere else in it fail with `EROFS`. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. | | `--mount-owner {vmm,caller}` | `vmm` | Select the host identity of the guest's operations on every `--mount` directory or file. `caller` performs each one as the guest caller's UID and GID and squashes guest root to the owner of the shared directory or file; it requires a Linux host. See [Run](run.md#file-ownership). | -| `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. A restore takes the addresses from the snapshot and rejects this option. | +| `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. A restore takes its NIC, with these addresses, from the snapshot, which [`snapshot capture --net`](#snapshot-capture) gives one, and rejects this option. | | `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net` on a fresh boot, and without it to restore a snapshot with a NIC. | | `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | | `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture rejects `allow`. | @@ -242,11 +242,13 @@ exists or lies in the store, a workload that `snapshot restore` refuses, and the options that the snapshot or its sandbox fixes: `--guest`, `--memory-capacity-mib`, `--processors`, `--cpu-profile`, `--net`, the share options, `--outcome-report`, `--cmdline`, and `--dry-run`. A cold boot refuses -`--store-dir`, `--state-dir`, and the restore options. The snapshot fixes the -NIC: the restore of a snapshot without one, as `snapshot capture` captures -them, takes no network options, and that of a snapshot with a NIC installs the -policy of its network options, which need `--network-profile`, as the policy -of the NIC, with its DNS server, egress, forwards, and published listeners. +`--store-dir`, `--state-dir`, and the restore options. A restore's NIC comes +from the snapshot: `--net` belongs to the cold boot and to +[`snapshot capture`](#snapshot-capture), which gives a snapshot its NIC. The +restore of a snapshot without a NIC takes no network options, and that of a +snapshot with a NIC installs the policy of its network options, which need +`--network-profile`, as the policy of the NIC, with its DNS server, egress, +forwards, and published listeners. A `run` that dies before it removes its sandbox, as `kill -9` ends it, leaves `--state-dir` with the sandbox's files and its view of the snapshot, and its @@ -408,6 +410,16 @@ dormant scratch is `--scratch-mib` (64 by default). `--processors` (1) and `--memory-capacity-mib` may reserve larger capacities. RAM capacity must be a multiple of 128 MiB. `--cpu-profile` selects the OpenVMM profile. +The VM has no NIC unless `--net IPV4/PREFIX` gives it one, at that static +address, as for [`run`](#run): the NIC has the complete deny-all policy, without +egress, ingress, host-loopback access, forwards, a proxy, or a DNS server, so +the guest learns no DNS server and announces no listeners, and nothing that the +snapshot holds depends on the policy. The snapshot keeps the NIC and its +address. `--net` is the only way to give a snapshot's restores a NIC: each +restore takes its NIC from the snapshot and refuses `--net`, so it cannot give +the NIC another address, and installs a network policy of its own on it, as +[`snapshot restore`](#snapshot-restore) and [`run`](#run) describe. + Only a successful capture whose own VMM exits within `--timeout` (60 whole seconds by default) can publish. The immutable named directory contains exactly `key.bin`, `snapshot.json` and `openvmm/`; an existing name is never replaced. @@ -435,32 +447,37 @@ nvx-rs/target/release/nvx-rs snapshot restore baseline \ options of [`sandbox provision`](#sandbox), but for those that only an execution or a cold boot takes, `--cwd`, `--environment`, `--environment-file`, `--inherit-default-environment`, `--exec-timeout-ms`, `--wait`, -`--outcome-report`, `--cmdline`, and `--dry-run`, then starts it from the named -snapshot of `--store-dir`. The sandbox's configuration records the snapshot, -its store, and the sandbox's workload, `--entrypoint` with its `--arg` values, -and each later `sandbox start` of the sandbox restores the snapshot again. Each -restore runs the workload over the control protocol, as `run` and `exec` do, -not from the kernel command line, so its entrypoint and arguments may contain -whitespace and double quotes, which `sandbox provision` refuses; the command -checks them as `exec` checks its workload before it provisions anything. The -state directory must be outside the store. The guest of a restore accepts only -a fresh scratch image, whose upper no earlier root wrote, such as a new empty -ext4 image, so a later start of a restored sandbox with a scratch image refuses -once an earlier start has created the sandbox's view of the snapshot, as each -start does before its guest can use the image: deprovision the sandbox and -restore the snapshot again with a fresh one. A missing snapshot provisions -nothing; a sandbox whose start fails stays provisioned, as after -`sandbox provision`. +`--outcome-report`, `--cmdline`, `--dry-run`, and `--net`, then starts it from +the named snapshot of `--store-dir`. The sandbox's configuration records the +snapshot, its store, and the sandbox's workload, `--entrypoint` with its `--arg` +values, and each later `sandbox start` of the sandbox restores the snapshot +again. Each restore runs the workload over the control protocol, as `run` and +`exec` do, not from the kernel command line, so its entrypoint and arguments +may contain whitespace and double quotes, which `sandbox provision` refuses; +the command checks them as `exec` checks its workload before it provisions +anything. The state directory must be outside the store. The guest of a +restore accepts only a fresh scratch image, whose upper no earlier root wrote, +such as a new empty ext4 image, so a later start of a restored sandbox with a +scratch image refuses once an earlier start has created the sandbox's view of +the snapshot, as each start does before its guest can use the image: +deprovision the sandbox and restore the snapshot again with a fresh one. A +missing snapshot provisions nothing; a sandbox whose start fails stays +provisioned, as after `sandbox provision`. A restore attaches the sandbox's image and upper in place of the snapshot's placeholder image and dormant scratch: the image must fit the placeholder, with its block sizes, and a scratch upper must be as long as the dormant scratch. The VM keeps the processors that the snapshot had online, and `--memory-mib` must be a multiple of 128 between the snapshot's boot memory and -its capacity. The snapshot fixes the NIC: one that `snapshot capture` captured -has none, so its restores take no `--net`. A restore of a snapshot with a NIC -installs the restore's own network policy, whose DNS server and listener -announcements the guest applies as it activates. +its capacity. The sandbox's NIC comes from the snapshot, with its presence and +its address: `--net` belongs to [`snapshot capture`](#snapshot-capture), which +gives a snapshot its NIC, and a restore takes no `--net`. A restore of a +snapshot without a NIC refuses the network options before it provisions +anything. That of a snapshot with a NIC needs `--network-profile`, and installs +the policy of its network options, at the NIC's address, as the restore's own +network policy: OpenVMM enforces its egress, ingress, host-loopback access, and +forwards, and the guest applies its DNS server and announces its listeners, +when it publishes them, as it activates. Each start holds the snapshot's exclusive lock, which it waits for up to `--timeout`, from before it links the snapshot's files into the sandbox's view diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 9a4e6edbf..513d02f27 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -2195,10 +2195,12 @@ names the parser after the usage, as in `nvx-rs run: error[CODE]: MESSAGE`. layered root or workload exists. It journals the launch before spawning, requests capture through authenticated NVXC, and requires that VM's timely exit before finishing and publishing exactly the strict snapshot's three -entries. The next capture using that store ends an interrupted attempt through -ordinary launch recovery, never resuming it. See -[`snapshot capture`](../doc/usage.md#snapshot-capture) for its resource options -and recovery behavior. +entries. With `--net IPV4/PREFIX`, the VM has a NIC at that address under the +complete deny-all policy, which the snapshot keeps, without the policy, so that +each restore installs a policy of its own on it. The next capture using that +store ends an interrupted attempt through ordinary launch recovery, never +resuming it. See [`snapshot capture`](../doc/usage.md#snapshot-capture) for its +resource options and recovery behavior. `snapshot restore NAME --store-dir PATH --state-dir PATH` provisions a managed sandbox, with the options of `sandbox provision` that a restore takes, whose @@ -2207,13 +2209,18 @@ then starts it: that start, and each later one, creates the VM with `host::vm::create` from the snapshot, through the store's launch journal, and drives its restore with the blocking restore driver, which activates the guest with the start's own network policy, maps the sandbox's shares, and runs the -workload, whose output the start forwards until it exits. A restore's guest -accepts only a fresh upper, so a later start refuses a scratch image once an -earlier start has created its view, as each start does before its guest can use -the image. Each start holds the snapshot's exclusive lock from before it links -the snapshot's files into the sandbox's `openvmm/` view until the start ends or -the VM has exited, and `deprovision` holds it while it removes the view; a start -whose command dies leaves a journal whose recovery ends the VM. No index of a +workload, whose output the start forwards until it exits. The restore's NIC +comes from the snapshot, which `snapshot capture --net` gives one, so the +restore takes no `--net`: the network options of a snapshot with a NIC, which +need `--network-profile`, are the restore's own policy at the NIC's address, as +for `run --restore-snapshot`, and a snapshot without a NIC refuses them before +anything is provisioned. A restore's guest accepts only a fresh upper, so a +later start refuses a scratch image once an earlier start has created its view, +as each start does before its guest can use the image. Each start holds the +snapshot's exclusive lock from before it links the snapshot's files into the +sandbox's `openvmm/` view until the start ends or the VM has exited, and +`deprovision` holds it while it removes the view; a start whose command dies +leaves a journal whose recovery ends the VM. No index of a snapshot's sandboxes or record of a restore in progress exists. See [`snapshot restore`](../doc/usage.md#snapshot-restore). @@ -2395,11 +2402,12 @@ status; a restore does none of these, as the second item says. reaches the guest through it, not through the byte channel, `hvc0`, whose output, such as the kernel's early messages, goes to standard error. - A cold boot takes `--net` and `--network-profile` together. A restore takes - the addressing of its NIC from the snapshot, so it refuses `--net`. Its other - network options are the restore's own policy, at the NIC's address, which - `sandbox` lowers, with its DNS server, egress, forwards, and published - listeners; a snapshot with a NIC needs `--network-profile` with them, and one - without, as `snapshot capture` captures, refuses them. + its NIC, with its addressing, from the snapshot, which + `snapshot capture --net` gives one, so it refuses `--net`. Its other network + options are the restore's own policy, at the NIC's address, which `sandbox` + lowers, with its DNS server, egress, forwards, and published listeners; a + snapshot with a NIC needs `--network-profile` with them, and one without + refuses them. - One directory `--mount` is OpenVMM's `--mount`. Several, or a shared file, are the children of an aggregate at `/run/nvx/shares`, whose guest targets must be absolute, must not overlap, and must survive the kernel command line, @@ -2692,10 +2700,10 @@ guest agents of NVX advertise bits 0, 4, and 6 in sandbox mode. `snapshot capture` published, and restores it once through the path of `snapshot restore`, where `nvx.py run` handed OpenVMM the directory of a raw snapshot, which `nvx-rs run` refuses before it reads anything. It refuses - `--net`, whose addressing a restored NIC takes from the snapshot, and takes - none of the options of a raw restore, `--restore-processors`, - `--restore-memory-mib`, and `--restore-ready-path`, whose values the snapshot - and the restore fix. + `--net`: a restored NIC, with its addressing, comes from the snapshot, which + `snapshot capture --net` gives one. It takes none of the options of a raw + restore, `--restore-processors`, `--restore-memory-mib`, and + `--restore-ready-path`, whose values the snapshot and the restore fix. ### Test hooks diff --git a/nvx-rs/src/host/cli/run.rs b/nvx-rs/src/host/cli/run.rs index 98015c134..ab9f18c93 100644 --- a/nvx-rs/src/host/cli/run.rs +++ b/nvx-rs/src/host/cli/run.rs @@ -22,11 +22,11 @@ //! its snapshot or its sandbox fixes, such as `--guest`, `--processors`, the shares, `--cmdline`, //! and `--dry-run`, and a cold boot the options of a restore. //! -//! A cold boot takes `--net` and `--network-profile` together. A restore takes the addressing of -//! its NIC from the snapshot, so it refuses `--net`. The network options of the restore of a -//! snapshot with a NIC, which need `--network-profile`, are the restore's own network policy, with -//! its DNS server, egress, forwards, and published listeners, as `sandbox` lowers them; those of a -//! snapshot without a NIC, which `snapshot capture` captures, are refused. +//! A cold boot takes `--net` and `--network-profile` together. A restore takes its NIC, with its +//! addressing, from the snapshot, which `snapshot capture --net` gives one, so it refuses `--net`. +//! The network options of the restore of a snapshot with a NIC, which need `--network-profile`, +//! are the restore's own network policy, with its DNS server, egress, forwards, and published +//! listeners, as `sandbox` lowers them; those of a snapshot without a NIC are refused. //! //! Its shares differ from those of `sandbox`. `--mount-deny`, `--mount-allow`, and `--mount-write` //! name host paths, which must be absolute with several shares or a shared file. A cold boot @@ -114,7 +114,9 @@ options: host identity of the share's file operations: vmm (default) or caller, with guest root squashed to the owner of the shared directory or file (Linux only) - --net IPV4/PREFIX + --net IPV4/PREFIX a cold boot's NIC at this address; a restore takes its + NIC from the snapshot, which snapshot capture --net + gives one --network-profile {portable} --network-egress {allow,deny} --network-ingress {allow,deny} @@ -489,8 +491,9 @@ fn restore(root: Option<&str>, options: Options) -> Result { } /// Returns the network options of the sandbox that restores a snapshot whose NIC is `nic`, which -/// the snapshot fixes: `network`, the run's own, at the NIC's address, which need +/// the snapshot fixes: `network`, the restore's own, at the NIC's address, which need /// `--network-profile`, as the policy of the restore's NIC, or none for a snapshot without a NIC. +/// A restore of `run` and one of `snapshot restore` both take their network from it. pub(super) fn restored_network( mut network: NetworkOptions, nic: Option, diff --git a/nvx-rs/src/host/cli/snapshots.rs b/nvx-rs/src/host/cli/snapshots.rs index 8cb46544d..061c92022 100644 --- a/nvx-rs/src/host/cli/snapshots.rs +++ b/nvx-rs/src/host/cli/snapshots.rs @@ -11,6 +11,10 @@ //! `snapshot capture` publishes a named snapshot, as [`capture`] describes. `snapshot restore` //! provisions a managed sandbox whose configuration records the snapshot, its store and a //! workload, then starts it, and each later start of the sandbox restores the snapshot again. +//! The sandbox's NIC comes from the snapshot, which `snapshot capture --net` gives one, so a +//! restore takes no `--net`: the network options of a restore of a snapshot with a NIC are the +//! restore's own policy, at the NIC's address, as for `run --restore-snapshot`, and a snapshot +//! without a NIC takes none. pub(super) mod capture; @@ -55,7 +59,11 @@ const RESTORE_SYNTAX: Syntax = Syntax { Provision a managed sandbox with the options of sandbox provision, then start it from a named snapshot, under the snapshot's lock, and run its workload, the entrypoint and its arguments, which must exit with status 0 within the -timeout. Each later start of the sandbox restores the snapshot again. +timeout. Each later start of the sandbox restores the snapshot again. The +sandbox's NIC comes from the snapshot, which snapshot capture --net gives one, +so a restore takes no --net: a snapshot with a NIC needs --network-profile, +and its network options are the restore's own policy, at the NIC's address; +a snapshot without a NIC takes no network options. positional arguments: NAME the snapshot's name @@ -81,7 +89,6 @@ options: --mount-allow HOST_PATH --mount-write HOST_PATH --mount-owner {vmm,caller} - --net IPV4/PREFIX --network-profile {portable} --network-egress {allow,deny} --network-ingress {allow,deny} @@ -93,8 +100,10 @@ options: --host-loopback-forward HOST_LOOPBACK_FORWARD", }; -/// The options of `sandbox` that only an execution or a cold boot takes, which a restore lacks. -const NOT_RESTORED: [&str; 9] = [ +/// The options of `sandbox` that only an execution or a cold boot takes, which a restore lacks: +/// a restore's NIC comes from its snapshot, which `snapshot capture --net` gives one, so it takes +/// no `--net`. +const NOT_RESTORED: [&str; 10] = [ "--cwd", "--environment", "--environment-file", @@ -104,6 +113,7 @@ const NOT_RESTORED: [&str; 9] = [ "--outcome-report", "--cmdline", "--dry-run", + "--net", ]; /// A parsed `snapshot` command. @@ -198,15 +208,19 @@ pub(super) fn run(root: Option<&str>, command: Command) -> Result { } /// Restores a new managed sandbox from a named snapshot, as `snapshot restore` does: provisions -/// the sandbox with the source that [`restore_source`] checks, and starts it. +/// the sandbox with the source that [`restore_source`] checks, and its network at the snapshot's +/// NIC, as [`run::restored_network`](super::run::restored_network) gives it, and starts it. It +/// refuses the network options of a snapshot without a NIC, and those of a snapshot with one +/// without `--network-profile`, before it provisions anything. fn restore(root: Option<&str>, options: RestoreOptions) -> Result { let RestoreOptions { name, store_dir, - sandbox: options, + sandbox: mut options, } = options; let snapshot = name.expect("the parser requires a snapshot name"); - let (source, _) = restore_source(snapshot, &store_dir, &options)?; + let (source, nic) = restore_source(snapshot, &store_dir, &options)?; + options.network = super::run::restored_network(options.network, nic)?; sandbox::restore(root, options, source) } diff --git a/nvx-rs/src/host/cli/snapshots/capture.rs b/nvx-rs/src/host/cli/snapshots/capture.rs index 9a8376b55..6cefcb62b 100644 --- a/nvx-rs/src/host/cli/snapshots/capture.rs +++ b/nvx-rs/src/host/cli/snapshots/capture.rs @@ -1,4 +1,8 @@ //! A blockless capture, kept in the ordinary launch journal until it ends. +//! +//! With `--net`, the capture VM has a NIC under the complete deny-all policy, and the snapshot +//! keeps the NIC and its address, without a policy: each restore installs a policy of its own on +//! it. use std::fs; use std::path::Path; @@ -16,14 +20,16 @@ use crate::host::cli::store::{LockedState, StateDir, StateFile}; use crate::host::cli::{Exit, Result, check_interrupt, io_error, output}; use crate::host::config::{ BlockGeometry, Boot, Capacity, CaptureConfig, ConsoleAttachment, DiskCapture, Invariants, - RootConfig, VmConfig, + NetworkConfig, RootConfig, VmConfig, }; +use crate::host::network::{Access, NetworkPolicy}; use crate::host::process::{Exited, KILL_TIMEOUT, Process}; use crate::host::runtime::{LaunchOptions, Runtime}; use crate::host::session::{ControlSession, SessionError}; use crate::host::snapshot::{DESCRIPTION_FILE, KEY_FILE, OPENVMM_DIR, finish_capture}; use crate::host::transport; use crate::host::vm::{self, Prepared, VmDir, VmOutputs}; +use crate::net::Ipv4Cidr; /// The capture command parser. const CAPTURE_SYNTAX: Syntax = Syntax { @@ -50,12 +56,15 @@ options: --image-capacity-mib MIB longest restore image (default: 1024) --scratch-mib MIB dormant scratch length (default: 64) + --net IPV4/PREFIX a NIC at this address, which the snapshot keeps for its + restores, under a deny-all policy that each restore + replaces with its own (default: none) --timeout SECONDS positive whole seconds for launch, capture and exit, and each lock wait (default: 60)", }; /// Capture's small, fixed option set. -const OPTIONS: [Spec; 10] = [ +const OPTIONS: [Spec; 11] = [ Spec::one("--store-dir"), Spec::one("--hypervisor"), Spec::one("--cpu-profile"), @@ -65,6 +74,7 @@ const OPTIONS: [Spec; 10] = [ Spec::one("--memory-capacity-mib"), Spec::one("--image-capacity-mib"), Spec::one("--scratch-mib"), + Spec::one("--net"), Spec::one("--timeout"), ]; @@ -90,6 +100,8 @@ pub(in crate::host::cli) struct Options { image_mib: u64, /// Dormant scratch size. scratch_mib: u64, + /// The address of the VM's NIC, if it has one. + net: Option, /// Bounded operation time. timeout: u32, } @@ -107,6 +119,7 @@ impl Default for Options { memory_capacity_mib: None, image_mib: 1024, scratch_mib: 64, + net: None, timeout: 60, } } @@ -139,6 +152,7 @@ impl Receiver for Options { } "--image-capacity-mib" => self.image_mib = integer(value, 1..=MAX_MEMORY_MIB)?, "--scratch-mib" => self.scratch_mib = integer(value, 1..=MAX_MEMORY_MIB)?, + "--net" => self.net = Some(Ipv4Cidr::parse(value)?), "--timeout" => self.timeout = integer(value, 1..=u32::MAX)?, _ => unreachable!("the capture parser has a fixed option set"), } @@ -228,7 +242,11 @@ fn capture( hostname: b"nvx-capture".to_vec(), machine_id: MachineIdMode::Image, }, - network: None, + // The complete deny-all policy is the one that a restore may replace with its own. + network: options.net.map(|address| NetworkConfig { + guest_network: Some(address), + policy: NetworkPolicy::new(Access::Deny), + }), filesystem: None, console: ConsoleAttachment::Disconnected, boot: Some(Boot { diff --git a/nvx-rs/src/host/cli/snapshots/capture/tests.rs b/nvx-rs/src/host/cli/snapshots/capture/tests.rs index f2ba1838d..09f8ac9c2 100644 --- a/nvx-rs/src/host/cli/snapshots/capture/tests.rs +++ b/nvx-rs/src/host/cli/snapshots/capture/tests.rs @@ -1,5 +1,6 @@ #![cfg(any(target_os = "linux", windows))] +use std::ffi::OsString; use std::io::{Read, Write}; use std::path::PathBuf; use std::process::{Child, Command, Stdio}; @@ -9,6 +10,7 @@ use std::time::Duration; use serde_json::{Value, json}; use super::*; +use crate::host::config::NicConfig; use crate::host::platform; use crate::host::process::{BootId, Opened, ProcessRef}; use crate::host::snapshot::tests::{ManifestSpec, SnapshotFixture, class}; @@ -238,7 +240,11 @@ fn write_recipe(root: &Path, prepared: &Prepared, behavior: &str) { ); let context = prepared.capture_context().unwrap(); let DiskCapture::Fresh { image, scratch } = context.capture.disks; - let mut manifest = ManifestSpec::template(); + // The fake's manifest has a NIC, at 10.0.0.2/24, only for a VM that has one. + let mut manifest = match context.nic { + Some(_) => ManifestSpec::with_network(), + None => ManifestSpec::template(), + }; let contract = manifest.contract(); contract.source_hypervisor = runtime(root).hypervisor.to_string(); contract.blocks[0].physical_block_size = image.physical_block_size; @@ -318,6 +324,152 @@ fn capture_parser_refuses_bad_names_and_invalid_options() { CAPTURE_SYNTAX.assert_lists(&OPTIONS); } +#[test] +fn the_capture_parser_takes_for_net_only_an_address_that_openvmm_accepts() { + let parsed = |extra: &[&str]| { + let arguments: Vec = ["name", "--store-dir", "store"] + .iter() + .chain(extra) + .map(|value| (*value).to_owned()) + .collect(); + parse(&arguments) + }; + assert_eq!(parsed(&[]).unwrap().0.net, None); + assert_eq!( + parsed(&["--net", "192.0.2.2/24"]).unwrap().0.net, + Some(Ipv4Cidr::parse("192.0.2.2/24").unwrap()) + ); + // The network's own, broadcast and gateway addresses, an address without a prefix, a prefix + // longer than /30, and IPv6. + for refused in [ + "192.0.2.0/24", + "192.0.2.255/24", + "192.0.2.1/24", + "192.0.2.2", + "192.0.2.2/31", + "fd00::2/120", + "", + ] { + assert!( + matches!( + parsed(&["--net", refused]), + Err(Exit::Usage { + code: ErrorCode::ArgumentInvalid, + .. + }) + ), + "{refused:?}" + ); + } +} + +#[test] +fn a_capture_with_net_gives_its_vm_a_deny_all_nic_which_its_snapshot_keeps() { + let fixture = Fixture::new(); + let store = fixture.root.path().join("store"); + let arguments = [ + "networked", + "--store-dir", + store.to_str().unwrap(), + "--processors", + "2", + "--memory-mib", + "512", + "--memory-capacity-mib", + "2048", + "--net", + "10.0.0.2/24", + "--timeout", + "20", + ] + .map(str::to_owned); + let (networked, extras) = parse(&arguments).unwrap(); + assert!(extras.is_empty()); + let address = Ipv4Cidr::parse("10.0.0.2/24").unwrap(); + let mut launched = Vec::new(); + capture(&networked, &runtime(fixture.root.path()), |prepared| { + assert_eq!( + prepared.capture_context().unwrap().nic, + Some(NicConfig { + guest_network: address + }) + ); + launched = prepared.arguments().to_vec(); + Ok(fake_spawn(fixture.root.path(), prepared, "publish")) + }) + .unwrap(); + // OpenVMM attaches the NIC under the complete deny-all policy alone: no egress, ingress, + // host loopback, forward, proxy or DNS server. + let deny_all = [ + "--net", + "10.0.0.2/24", + "--network-profile", + "portable", + "--network-egress", + "deny", + "--network-ingress", + "deny", + "--host-loopback", + "deny", + ] + .map(OsString::from); + assert_eq!( + launched + .windows(deny_all.len()) + .filter(|window| *window == deny_all) + .count(), + 1, + "{launched:?}" + ); + for option in [ + "--network-egress-allow", + "--network-egress-deny", + "--host-loopback-forward", + "--network-proxy", + ] { + assert!( + !launched.iter().any(|argument| argument == option), + "{option}" + ); + } + let store = fixture.store(); + let held = store + .lock(&name("networked"), Deadline::after(10.0)) + .unwrap(); + assert_eq!( + held.open().unwrap().nic(), + Some(NicConfig { + guest_network: address + }) + ); + let description: Value = + serde_json::from_slice(&fs::read(held.dir.join(DESCRIPTION_FILE)).unwrap()).unwrap(); + assert_eq!( + description["nic"], + json!({ "guest_network": "10.0.0.2/24" }) + ); + drop(held); + assert!(!fixture.vm_dir().exists()); + // Without --net, a capture's VM has no NIC, and neither has its snapshot. + capture( + &fixture.options("plain"), + &runtime(fixture.root.path()), + |prepared| { + assert_eq!(prepared.capture_context().unwrap().nic, None); + assert!( + !prepared + .arguments() + .iter() + .any(|argument| argument.to_str().unwrap().starts_with("--net")) + ); + Ok(fake_spawn(fixture.root.path(), prepared, "publish")) + }, + ) + .unwrap(); + let held = store.lock(&name("plain"), Deadline::after(10.0)).unwrap(); + assert_eq!(held.open().unwrap().nic(), None); +} + #[test] fn capture_publishes_only_three_strict_entries_after_its_own_exit_and_removes_its_vm() { let fixture = Fixture::new(); diff --git a/nvx-rs/src/host/cli/snapshots/tests.rs b/nvx-rs/src/host/cli/snapshots/tests.rs index 44651379a..906bff988 100644 --- a/nvx-rs/src/host/cli/snapshots/tests.rs +++ b/nvx-rs/src/host/cli/snapshots/tests.rs @@ -2,7 +2,7 @@ use std::fs; use std::path::Path; use super::*; -use crate::host::snapshot::tests::SnapshotFixture; +use crate::host::snapshot::tests::{RestoreFixture, SnapshotFixture}; use crate::host::snapshot::{DESCRIPTION_FILE, KEY_FILE, MANIFEST_NAME, OPENVMM_DIR}; use crate::host::tests::released; @@ -408,6 +408,56 @@ fn a_restore_refuses_a_workload_that_is_not_a_valid_exec_request_before_it_provi assert!(!state.exists()); } +#[test] +fn a_restore_takes_its_network_at_the_nic_of_its_snapshot_before_it_provisions() { + let root = tempfile::tempdir().unwrap(); + let store_dir = root.path().join("store"); + let store = Store::create(&store_dir).unwrap(); + for (snapshot, nic) in [("plain", false), ("networked", true)] { + let fixture = RestoreFixture::new(nic); + let opened = fixture.opened(); + let held = store.lock(&name(snapshot), Deadline::after(0.0)).unwrap(); + let staged = held.stage().unwrap(); + for entry in [KEY_FILE, DESCRIPTION_FILE, OPENVMM_DIR] { + fs::rename(opened.dir.join(entry), staged.dir().join(entry)).unwrap(); + } + held.publish(staged).unwrap(); + } + let state = root.path().join("state"); + let refusal = |snapshot: &str, network: &[&str]| { + let mut arguments = vec![ + "restore", + snapshot, + "--store-dir", + store_dir.to_str().unwrap(), + "--state-dir", + state.to_str().unwrap(), + "--upper", + "ram", + ]; + arguments.extend(network); + let Ok((Command::Restore(options), extras)) = command(&arguments) else { + panic!("{arguments:?} parses"); + }; + assert!(extras.is_empty()); + let error = restore(None, *options).unwrap_err(); + assert_eq!(error.code(), Some(ErrorCode::Unsupported), "{error:?}"); + assert!(!state.exists(), "{arguments:?}"); + error.message().unwrap().to_owned() + }; + // A snapshot without a NIC takes no network option, and one with a NIC needs the profile. + assert!(refusal("plain", &["--network-profile", "portable"]).contains("no NIC")); + assert!(refusal("networked", &[]).contains("needs --network-profile")); + // With the profile, the options take the address of the snapshot's NIC, which the restore does + // not name: the provision gets past its check of a network without one, and refuses the + // missing image. + let message = refusal( + "networked", + &["--network-profile", "portable", "--network-egress", "deny"], + ); + assert!(message.contains("requires --layer"), "{message}"); +} + /// Returns a store with the published snapshot `first`, and a VM directory with a view of it. fn restored_view(root: &Path) -> (Store, VmDir) { fs::create_dir_all(root).unwrap(); From 9a91a4fe077f0a026ad9567034bfd55a64c89de8 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 14:12:43 -0800 Subject: [PATCH 2/2] tests: run untiered-network-policy on named snapshots (SL3.7c) The untiered-network-policy scenario drove OpenVMM directly: it booted a VM with a NIC under the complete deny-all policy, captured a raw snapshot of it with a post-restore script, and restored that snapshot with `--restore-network-policy` and four network policies. The CLI no longer restores raw snapshots, and `snapshot capture --net` now captures a named snapshot with such a NIC, so the scenario now runs on named snapshots, through the CLI's restore paths. It captures the template with `nvx-rs snapshot capture --net` and restores it four times, each into a sandbox of the Ubuntu layer over a RAM upper, with the network options of a policy at the NIC's captured address, and the scenario's checks as the restored workload: `snapshot restore`, whose sandboxes the scenario then stops and deprovisions, restores the template's own deny-all and private-network ingress with allowed egress, which show the restore's DNS server and announcer (SL3.8a's check); and `run --restore-snapshot` restores egress rules that admit DNS over IPv4 and one host service, and explicit host loopback with a forward under egress rules that admit DNS over IPv6 alone, which show its DNS server, egress, forwards, and published listeners (SL3.8b's checks). Each restored workload reports its resolver, whether the announcer of its listeners runs, and which host services it reaches, then serves the host through the forward and the published port, as many clients as the scenario names, or without one, holds its port for six seconds, while the host checks that nothing publishes it. The host cannot end the hold, as a restored workload takes no input and deny-all has no way in, so it must check within two seconds of seeing the server listen, and see the hold end at least two seconds after its check: a check that a late marker could have moved past the hold fails instead of passing. The workload's script is its `sh -c` argument, whitespace and all, as a restore sends its workload as an exec request. The scenario keeps its checks of those reports, of the requests that the host services receive, of where each policy publishes the guest's port, and that no restore changes the template, and adds that no restore leaves its sandbox or an OpenVMM process behind. Only the raw parts retire: the raw capture and restores, the guest's repair marker, the console release, and OpenVMM's refusals of a raw restore that keeps the template's policy but passes another or passes private-network ingress without `--restore-network-policy`, which the CLI cannot issue. In their place, the CLI must refuse, before it provisions anything, a restore of the template without `--network-profile`, and one that names the NIC's address with `--net`. A failed run stops the sandboxes that it left, ends the OpenVMM processes that name its fixture, keeps the VM log of each sandbox, and then deprovisions the sandboxes, as the sandbox lifecycle acceptance cleans up, and a failed restore's error names its cause. The scenario now reads the Ubuntu layer, so it becomes a public sandbox scenario: CI runs it in each backend's guest and sandbox job, beside the other public scenarios, and no longer in the shards of the default suite, which plan-microvm-shards re-planned. The case matrix now gives the scenario's rewrite, and that of the tests of the adapter's clones that it covers, to SL3.7c. Tests: the unit tests of the scenario run it against a simulated nvx-rs, whose restores' workloads request the real host services as their policies admit, and keep checking the reports, the host services, the publications, and the template, and now the commands of the capture, the restores, their stops and deprovisions, and the refusals, a script that fits one argument of an exec request, the timing of the host's check of an unpublished port against the guest's hold, also with markers that reach the host late, the cleanup after a failure, and the leftovers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/run-nvx-microvm-tests.yml | 2 + doc/acceptance-case-matrix.md | 27 +- doc/ci.md | 57 +- doc/usage.md | 8 +- scripts/nvx_tools/microvm_shards.json | 34 +- .../untiered-network-policy.sh.in | 85 +- scripts/nvx_tools/microvm_tests.py | 573 +++++++++---- scripts/test_microvm_tests.py | 795 +++++++++++++----- scripts/test_nvx_tools.py | 1 + 9 files changed, 1105 insertions(+), 477 deletions(-) diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index 0e83643c2..454f3a481 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -222,6 +222,7 @@ jobs: --scenario vm-lifecycle --scenario snapshot-core --scenario snapshot-tiers + --scenario untiered-network-policy --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run public managed sandbox acceptance on Windows @@ -238,6 +239,7 @@ jobs: --scenario vm-lifecycle --scenario snapshot-core --scenario snapshot-tiers + --scenario untiered-network-policy --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run Ubuntu guest smoke tests on Linux diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index 9ac21e0fe..d04744f7d 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -16,7 +16,9 @@ vm-lifecycle` creates, spawns, reattaches to and stops a VM through the public A as [Cold boots](#cold-boots) lists. SL5.1c (#1118) added those of snapshots, which capture a snapshot, open it and restore it with nvx-init as the guest, run concurrent restores of one snapshot under the CLI's lock, and check that a failed restore reports its attribution, as -[Snapshots](#snapshots) lists. +[Snapshots](#snapshots) lists. SL3.7c (#1114) let a named snapshot keep a NIC, and runs the +network policies of its restores on real VMs with +`test-microvm --scenario untiered-network-policy`, as [Snapshots](#snapshots) lists too. The inventory is fixed: it lists the tests as they are where this matrix was added, from `tests/runtime-adapter/scenarios.txt` and the adapter's sources, `MICROVM_TEST_SCENARIOS` and @@ -30,7 +32,7 @@ its own in turn. | --- | ---: | ---: | ---: | ---: | | [The adapter's scenarios](#the-adapters-scenarios) | 157 | 0 | 114 | 43 | | [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 | -| [MicroVM scenarios](#microvm-scenarios) | 54 | 51 | 2 | 1 | +| [MicroVM scenarios](#microvm-scenarios) | 54 | 50 | 3 | 1 | | [CI](#ci) | 69 | 43 | 3 | 23 | | [Docs](#docs) | 36 | 28 | 2 | 6 | @@ -212,13 +214,13 @@ The adapter's README said what each proved. | `a_removal_reports_busy_while_leases_keep_the_generation` | retired | Removals, leases, tombstones and generations leave with the old clone path (#1121) | | `a_template_whose_processors_are_all_online_at_boot_limits_none` | rewritten | `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_treats_zero_boot_processors_as_all_capacity_online`; `nvx-rs/src/host/config/tests.rs::fresh_captures_pass_with_supported_processors_and_aligned_memory`; `test-microvm --scenario smp-snapshot` | | `a_template_without_a_network_has_no_nic` | rewritten | `nvx-rs/src/host/vm/tests.rs::capture_creation_records_sparse_disks_devices_and_a_complete_serde_context`; `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_refuses_both_nic_presence_mismatches_and_each_named_address_mismatch`. A store that serves it again retires (#1121) | -| `the_networks_of_every_policy_at_one_guest_s_address_share_one_template` | rewritten | `nvx-rs/src/host/network/tests.rs::templates_attach_their_nic_under_the_complete_deny_all_policy`; `test-microvm --scenario untiered-network-policy`, which SL3.7c (#1114) rewrites on a strict snapshot with a NIC. A cached template retires (#1121) | +| `the_networks_of_every_policy_at_one_guest_s_address_share_one_template` | rewritten | `nvx-rs/src/host/network/tests.rs::templates_attach_their_nic_under_the_complete_deny_all_policy`; `test-microvm --scenario untiered-network-policy`, whose four restores of one named snapshot with a NIC each install a network policy of their own. A cached template retires (#1121) | ### `vm_clone` (36) | Scenario | Fate | Coverage, or why it retires | | --- | --- | --- | -| `a_clone_binds_the_resolver_of_its_policy_s_dns_server_read_only` | rewritten | `test-microvm --scenario untiered-network-policy`, which SL3.7c (#1114) rewrites on a strict snapshot with a NIC; `test-microvm --scenario network-snapshot`; `nvx-rs/src/guest/root/tests/view.rs::actual_resolver_failures_keep_the_network_class_and_root_owned_bind` | +| `a_clone_binds_the_resolver_of_its_policy_s_dns_server_read_only` | rewritten | `test-microvm --scenario untiered-network-policy`, whose restored guests' resolvers name exactly the DNS server of their policies; `test-microvm --scenario network-snapshot`; `nvx-rs/src/guest/root/tests/view.rs::actual_resolver_failures_keep_the_network_class_and_root_owned_bind` | | `a_clone_maps_an_aggregate_of_directories_and_files_beneath_its_root_before_its_sandbox_starts` | rewritten | `test-microvm --scenario platform-aggregate-export`; `nvx-rs/src/guest/mappings/table/tests.rs::mappings_keep_the_paths_and_the_access_of_their_entries` | | `a_clone_of_a_template_with_a_nic_maps_its_export_beside_its_resolver` | rewritten | `nvx-rs/src/guest/activation/state/apply/tests.rs::the_record_s_view_is_applied_once_with_its_own_dns_and_returned_machine_id`; `test-microvm --scenario filesystem-snapshot` | | `a_clone_reads_no_more_of_its_disks_than_its_root_needs` | rewritten | A restore reads no content of its disks or snapshot: `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_never_reopens_snapshot_paths_or_reads_runtime_artifacts`; `nvx-rs/src/host/sandbox/distro/tests.rs::the_staged_check_reads_no_content`; `nvx-rs/src/host/sandbox/scratch/tests.rs::checks_claim_the_image_without_reading_it`. The adapter's in-guest read statistics retire | @@ -234,7 +236,7 @@ The adapter's README said what each proved. | `a_generation_is_eligible_by_the_clock_of_its_capture` | rewritten | `nvx-rs/src/host/snapshot/eligibility/tests.rs::within_one_boot_the_downtime_is_the_host_monotonic_clocks`; `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_reports_early_and_expired_time_as_unavailable` | | `a_generation_refuses_clones_of_fewer_processors_than_its_template_booted_with` | rewritten | `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_refuses_a_blockless_root_and_each_processor_mismatch`; `test-microvm --scenario restore-processors` | | `a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity` | rewritten | `nvx-rs/tests/security_checklist.rs::a_saved_vm_runs_until_its_process_exits_and_another_runtime_is_refused_without_a_signal`; `nvx-rs/src/host/vm/tests.rs::stop_forces_only_its_verified_process_after_grace`. Leases retire (#1121) | -| `a_run_in_a_template_of_a_network_finds_its_policy_s_proxy` | rewritten | `test-microvm --scenario platform-network-policy`; `test-microvm --scenario snapshot-core`, whose runs restore a snapshot without a NIC until SL3.7c (#1114). The template's `run` retires (#1121) | +| `a_run_in_a_template_of_a_network_finds_its_policy_s_proxy` | rewritten | `test-microvm --scenario platform-network-policy`; `test-microvm --scenario untiered-network-policy`, whose runs restore a named snapshot with a NIC under a network policy of their own, though with no proxy. The template's `run` retires (#1121) | | `a_run_runs_one_execution_in_its_clone_then_tears_the_clone_down` | rewritten | `test-microvm --scenario snapshot-core`, whose `run --restore-snapshot` runs one workload in its restored VM and removes its sandbox. The template's `run` and leases retire (#1121) | | `a_run_s_execution_finds_its_shares_beneath_the_root` | rewritten | `test-microvm --scenario platform-aggregate-export` | | `a_run_whose_clone_does_not_start_runs_nothing` | retired | Template selections and the template's `run` leave with the old clone path (#1121) | @@ -250,7 +252,7 @@ The adapter's README said what each proved. | `one_generation_serves_clones_of_each_processor_count_from_its_boot_online_count_to_its_capacity` | rewritten | `test-microvm --scenario restore-processors`; `test-microvm --scenario platform-restore-processors` | | `one_generation_serves_clones_of_each_upper_and_memory_target` | rewritten | `test-microvm --scenario scratch-snapshot`; `test-microvm --scenario restore-memory`. Leases retire (#1121) | | `one_generation_serves_clones_of_either_format_over_either_upper` | rewritten | `test-microvm --scenario snapshot-tiers`; `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_matching_real_disk_geometry_and_every_resource_boundary` | -| `one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address` | rewritten | `test-microvm --scenario untiered-network-policy`, which SL3.7c (#1114) rewrites on a strict snapshot with a NIC; `test-microvm --scenario platform-network-policy` | +| `one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address` | rewritten | `test-microvm --scenario untiered-network-policy`, whose four restores of one named snapshot with a NIC each install a network policy of their own at the NIC's address; `test-microvm --scenario platform-network-policy` | | `openvmm_reads_none_of_the_host_s_input_until_the_repair_acknowledges_the_restore` | retired | It watches OpenVMM's own gating of the host's input through the repair delay of the adapter's templates, which leaves with them. NVX's side stays: `nvx-rs/src/guest/activation/state/tests.rs::the_deadline_starts_at_the_acknowledgement_and_retries_never_extend_it`; `nvx-rs/src/guest/init/boot/tests.rs::restored_sessions_keep_the_deadline_of_the_actual_repair_acknowledgement` | | `restore_exclusion::hard_linked_restore_views_obey_the_callers_exclusion` (linux) | rewritten | `test-microvm --scenario snapshot-core`, whose concurrent restores of one snapshot take turns at the CLI's lock and leave no link to its files. The view that a restore hard-links, which `vm::create` documents: `nvx-rs/src/host/snapshot/tests.rs::a_restore_view_hard_links_only_the_fixed_openvmm_inventory`; `nvx-rs/src/host/vm/tests.rs::snapshot_creation_owns_the_fixed_view_and_uses_captured_nic_with_requested_policy`; `nvx-rs/src/host/vm/tests.rs::snapshot_creation_refuses_a_true_cross_filesystem_view_without_copying`; `nvx-rs/src/host/vm/tests.rs::snapshot_creation_never_replaces_existing_outputs_views_or_listeners`; `nvx-rs/src/host/vm/tests.rs::the_same_prebound_listener_keeps_its_queue_through_both_launch_paths_without_inheritance`. OpenVMM's own checks before its restore-ready connection, which fail a restore whose view changed or whose caller died, and which `vm::create` documents too, leave NVX's tests | | `the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store` | retired | Leases and clone records leave with the old clone path (#1121) | @@ -298,7 +300,8 @@ checked the adapter's own code, and CI ran in its configurations of the adapter. suite in its shards, and the public sandbox scenarios in its guest and sandbox jobs. The public `sandbox-gpt-layer` and `vm-lifecycle` scenarios came with SL5.1b, after this matrix; [Cold boots](#cold-boots) says what they check. SL5.1c rewrote `snapshot-core` and -`snapshot-tiers` as public scenarios; [Snapshots](#snapshots) says what they check. +`snapshot-tiers` as public scenarios, and SL3.7c `untiered-network-policy`; +[Snapshots](#snapshots) says what they check. | Scenario | Fate | Note | | --- | --- | --- | @@ -351,7 +354,7 @@ suite in its shards, and the public sandbox scenarios in its guest and sandbox j | `structured-outcome` | kept | | | `time-abi-conformance` | kept | | | `time-host-correction` | kept | | -| `untiered-network-policy` | kept | On a raw snapshot, until a strict snapshot can carry a NIC: SL3.7c (#1114) rewrites it on one. It keeps the coverage of a restore's DNS, egress, forwards and listener publication until then | +| `untiered-network-policy` | rewritten | On named snapshots with a NIC, which `snapshot capture --net` captures, as [Snapshots](#snapshots) says, keeping its checks of a restore's DNS, egress, forwards and listener publication. Its raw-snapshot parts retire: the raw capture and restores, the guest's repair marker, the console release, and OpenVMM's refusals of raw restores that the CLI cannot issue, in whose place the CLI must refuse a restore without `--network-profile`, and one with `--net` | | `virtio-features` | kept | | | `virtio-net` | kept | | | `vm-lifecycle` | kept | A [cold boot](#cold-boots) through the public API: `nvx-rs/tests/vm_lifecycle.rs::a_cold_vm_is_reattached_from_saved_values_and_stopped` | @@ -396,13 +399,13 @@ capture`, whose VM nvx-init runs, and restore it with `nvx-rs snapshot restore` | A real VM is captured, opened as a strict snapshot and restored with nvx-init | `snapshot-core`: two one-off restores and three managed ones, whose guests keep the snapshot's uptime, read the host's clock, and take fresh random values and machine IDs. `snapshot-tiers`: a managed restore of each image format, EROFS and GPT, over each upper, scratch and RAM | | Concurrent restores of one snapshot are serialized by the CLI's lock | `snapshot-core`: three restores at once, whose workloads end only when the scenario releases them. While one holds the lock, no other links its view of the snapshot, and a restore that cannot wait for the lock fails with `sandbox-busy` before its workload runs. The three then run at once, each with its own share, upper and machine ID | | The selected failed restore reports its attribution | `snapshot-core`: a scratch image that the snapshot cannot serve is refused before any VM starts, with an error that names the lengths of the requested scratch and of the snapshot's, which attributes the failure to the request; the managed sandbox of the refused restore holds its configuration and its lock alone, with no trace of a launch. A restore that fails in its guest ends its VM, frees the lock, and reports `guest-exit status 125` | +| A restore of a snapshot with a NIC installs a network policy of its own | `untiered-network-policy`: a snapshot that `snapshot capture --net` captures under the complete deny-all policy, restored four times with `snapshot restore` and `run --restore-snapshot`. Each guest names exactly its policy's DNS server and runs the announcer of its listeners only when its policy publishes them, reaches exactly the host services that its egress admits, and serves the host through a forward and through the port that its policy publishes, on host loopback alone or to the host's private networks. The CLI refuses a restore without `--network-profile`, and one with `--net`, before it provisions anything | The CLI prints no attribution of a restore that fails in its guest beyond the guest's exit status: nvx-init's status for each class of failure, and the CLI's report of the restore's -attribution, are a follow-up on #1118. A snapshot that `snapshot capture` captures has no NIC, so -its restores take no network option: `untiered-network-policy` keeps the coverage of a restore's -DNS, egress, forwards and listener publication on raw snapshots until SL3.7c (#1114) lets a -capture keep a NIC. +attribution, are a follow-up on #1118. A snapshot that `snapshot capture` captures without +`--net` has no NIC, so its restores take no network option; one captured with `--net` keeps its +NIC, and each of its restores installs the network policy of its own options on it. ## CI diff --git a/doc/ci.md b/doc/ci.md index ebe75e301..7d803e9b6 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -275,28 +275,36 @@ forward whose host port another socket holds, the option on a platform snapshot captured with plain deny-all egress, which leaves host loopback allowed, and the option on a cold boot, and no restore may change the template. -The `untiered-network-policy` scenario covers a NIC template without a tier, a -snapshot without sandbox blocks, as the non-sandbox mode captures one, whose -NIC was captured with the complete deny-all policy. It restores the template -with `--restore-network-policy` four times: under the template's own deny-all, -under egress rules that admit DNS over IPv4 and one host service, with -explicit host loopback and a forward under egress rules that admit DNS over -IPv6 alone, and with private-network ingress and allowed egress. The -template's guest learned no DNS server and runs no announcer of its listeners, -so each restore's repair applies those of the restore's policy, which the -restore packet's network settings carry, before it acknowledges the restore: -the guest's resolver must name exactly the policy's DNS server, and no gateway -without one, and the announcer must run exactly when the policy publishes -the guest's listeners. Each restore must reach through its gateway exactly the -host-loopback services that its policy admits, which must receive no other -request, and the host must reach the guest's server through the forward and -through the port that the policy publishes: on host loopback alone, which no -other host address reaches, or to the host's private networks, through a host -address other than loopback. It must find the server on no host loopback port -without a publication. Before the guest runs, OpenVMM must refuse a restore -that keeps the template's policy but passes another, and one that passes -private-network ingress without the option, and no restore may change the -template. +The `untiered-network-policy` scenario, one of the public sandbox scenarios +that the guest and sandbox jobs run (see below), covers a NIC template without +a tier: a named snapshot that `nvx-rs snapshot capture --net` captures, of a +blockless VM whose NIC has the complete deny-all policy. It restores the +template four times, with the network options of a sandbox of the Ubuntu +layer over a RAM upper, at the NIC's captured address, as the restore's own +policy, and runs the scenario's checks as the restored workload: through +`nvx-rs snapshot restore`, under the template's own deny-all and with +private-network ingress and allowed egress, and through +`nvx-rs run --restore-snapshot`, under egress rules that admit DNS over IPv4 +and one host service, and with explicit host loopback and a forward under +egress rules that admit DNS over IPv6 alone. The template's guest learned no +DNS server and runs no announcer of its listeners, so each restore's +activation applies those of the restore's policy: the guest's resolver must +name exactly the policy's DNS server, and no gateway without one, and the +announcer must run exactly when the policy publishes the guest's listeners. +Each restore must reach through its gateway exactly the host-loopback services +that its policy admits, which must receive no other request, and the host must +reach the guest's server through the forward and through the port that the +policy publishes: on host loopback alone, which no other host address reaches, +or to the host's private networks, through a host address other than loopback. +It must find the server on no host loopback port without a publication, while +the guest holds the server's port, which the host cannot release: it must +check within two seconds of seeing the server listen, and see the six-second +hold end at least two seconds after its check, so that a check that a late +marker could have moved past the hold fails instead of passing. Before it +provisions anything, the CLI must refuse a restore of the template without +`--network-profile`, and one that names the NIC's address with `--net`. No +restore may change the template, or leave its sandbox or an OpenVMM process +behind; a failed run keeps the VM log of each sandbox that it leaves. The `platform-filesystem` scenario covers a share that a platform restore attaches to the filesystem slot that the capture left dormant, as a clone's export does. It captures a platform snapshot without `--mount` and restores it @@ -565,7 +573,10 @@ a VM through the public API of `nvx`. The public `snapshot-core` and [Run](run.md#named-snapshots)): restores of each image format over each upper, repeated restores with fresh entropy and machine IDs, clocks that follow the host's across the downtime, concurrent restores that take turns at the -snapshot's lock, failed restores, and snapshots that no restore changes. +snapshot's lock, failed restores, and snapshots that no restore changes. The +public `untiered-network-policy` scenario restores a named snapshot with a NIC, +which `snapshot capture --net` captures, under four network policies, as +described above. Their bounded evidence is part of the backend's failure logs. The same entrypoint then verifies a live virtio-fs share inside the container: a read-write `/workspace` share with a denied diff --git a/doc/usage.md b/doc/usage.md index 89fb66367..3aba1ee65 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1270,8 +1270,9 @@ which runs only when named: it repeats `smp` and also asserts that every CPU uses the one-shot counting LAPIC. `managed-exec-config`, `sandbox-lifecycle`, `sandbox-proxy-only`, `sandbox-ram-upper`, and `sandbox-gpt-layer`, the public sandbox scenarios, drive the public `nvx-rs sandbox` lifecycle commands (see -[Run](run.md#managed-lifecycle)), `snapshot-core` and `snapshot-tiers` capture -and restore named snapshots with `nvx-rs snapshot` and `nvx-rs run` (see +[Run](run.md#managed-lifecycle)), `snapshot-core`, `snapshot-tiers`, and +`untiered-network-policy` capture and restore named snapshots, the last with a +NIC, with `nvx-rs snapshot` and `nvx-rs run` (see [Run](run.md#named-snapshots)), and `vm-lifecycle` drives a VM through the public API of `nvx`; they also read the Ubuntu distro layer, its manifest, and the sandbox scratch template, which @@ -1285,7 +1286,8 @@ the sandbox-control-dependent `snapshot-core`, `snapshot-tiers`, `platform-restore-processors`, `platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`, `platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, `sandbox-proxy-only`, `sandbox-ram-upper`, -`sandbox-gpt-layer`, and `vm-lifecycle` scenarios. Ubuntu also rejects +`sandbox-gpt-layer`, `vm-lifecycle`, and `untiered-network-policy` scenarios. +Ubuntu also rejects the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without `--scenario`, it runs diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index e5f4e4bc5..1fabd9971 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -64,20 +64,20 @@ "structured-outcome": 0.6, "time-abi-conformance": 0.7, "time-host-correction": 14.0, - "untiered-network-policy": 14.0, "virtio-features": 0.7, "virtio-net": 3.8, "workload-identity": 0.7 }, "shards": [ { - "seconds": 258.5, + "seconds": 251.7, "scenarios": [ "command-line-limit", "console-snapshot", "directional-network-policy", "filesystem-comma-paths", "filesystem-snapshot", + "l3-l4-egress-policy", "lifecycle", "managed-lifecycle", "network-snapshot", @@ -86,14 +86,13 @@ "restore-memory", "restore-processors", "smp", - "smp-snapshot", "time-abi-conformance", "time-host-correction", "virtio-features" ] }, { - "seconds": 258.4, + "seconds": 251.2, "scenarios": [ "console-exit", "cpu-profile-fallback", @@ -107,7 +106,6 @@ "host-loopback-policy", "interrupt-map", "ipv6-egress-policy", - "l3-l4-egress-policy", "l3-l4-egress-port-ranges", "l3-l4-egress-protocols", "platform-aggregate-export", @@ -117,9 +115,9 @@ "private-network-ingress", "sandbox-blocks", "scratch-snapshot", + "smp-snapshot", "snapshot-rollback", "structured-outcome", - "untiered-network-policy", "virtio-net", "workload-identity" ] @@ -168,14 +166,13 @@ "structured-outcome": 0.5, "time-abi-conformance": 0.8, "time-host-correction": 14.0, - "untiered-network-policy": 14.1, "virtio-features": 0.6, "virtio-net": 3.7, "workload-identity": 0.6 }, "shards": [ { - "seconds": 239.8, + "seconds": 233.0, "scenarios": [ "command-line-limit", "console-snapshot", @@ -189,16 +186,15 @@ "platform-restore-processors", "platform-zero-fill", "restore-downtime", + "restore-processors", "scratch-snapshot", "smp", "smp-snapshot", - "snapshot-rollback", - "time-host-correction", - "untiered-network-policy" + "snapshot-rollback" ] }, { - "seconds": 240.0, + "seconds": 232.7, "scenarios": [ "console-exit", "cpu-profile-fallback", @@ -219,10 +215,10 @@ "platform-network-policy", "private-network-ingress", "restore-memory", - "restore-processors", "sandbox-blocks", "structured-outcome", "time-abi-conformance", + "time-host-correction", "virtio-features", "virtio-net", "workload-identity" @@ -272,21 +268,20 @@ "structured-outcome": 0.8, "time-abi-conformance": 1.6, "time-host-correction": 14.0, - "untiered-network-policy": 18.8, "virtio-features": 1.5, "virtio-net": 4.5, "workload-identity": 1.1 }, "shards": [ { - "seconds": 339.3, + "seconds": 329.8, "scenarios": [ "denied-filesystem-paths", "directional-network-policy", "endpoint-policy-snapshot", "filesystem-policy", "filesystem-shares", - "host-loopback-policy", + "l3-l4-egress-protocols", "network-snapshot", "platform-aggregate-export", "platform-filesystem", @@ -296,12 +291,11 @@ "restore-processors", "scratch-snapshot", "smp", - "smp-snapshot", - "untiered-network-policy" + "smp-snapshot" ] }, { - "seconds": 339.3, + "seconds": 330.0, "scenarios": [ "command-line-limit", "console-exit", @@ -312,11 +306,11 @@ "filesystem-snapshot", "guest-boot", "guest-identity", + "host-loopback-policy", "interrupt-map", "ipv6-egress-policy", "l3-l4-egress-policy", "l3-l4-egress-port-ranges", - "l3-l4-egress-protocols", "lifecycle", "managed-lifecycle", "platform-network-policy", diff --git a/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in b/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in index e1a72ce5b..4670bcd3c 100644 --- a/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in +++ b/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in @@ -1,23 +1,22 @@ set -eu -on_exit() { - status=$? - if [ "$status" -ne 0 ]; then - nvx-exit "$status" - fi -} -trap on_exit EXIT fail() { echo "NVX-UNTIERED-NETWORK-FAIL $1" exit "$2" } -# The nameservers of the guest's resolver, which the restore's repair names -# when the restore's policy has a DNS server. +# The number of the host's exchanges with the server below, through each way +# in that the restore's policy gives the host. +exchanges=$1 +case "$exchanges" in +'' | *[!0-9]*) fail "exchanges $exchanges" 84 ;; +esac +# The nameservers of the guest's resolver, which the restore's activation +# names when the restore's policy has a DNS server. resolver=$(awk '$1 == "nameserver" { list = list sep $2; sep = "," } END { print (list == "" ? "none" : list) }' /etc/resolv.conf 2>/dev/null) || resolver=none -# The repair starts the announcer of the guest's listeners when the restore's -# policy publishes them. -if pidof nvx-ingress >/dev/null; then +# The activation starts the announcer of the guest's listeners when the +# restore's policy publishes them. +if busybox pidof nvx-ingress >/dev/null; then announcer=running else announcer=none @@ -25,11 +24,8 @@ fi # Sets result to whether the guest reached the host service that answers $1 # at port $2 of the gateway, which the endpoint maps to host loopback. probe() { - rm -f /tmp/nvx-untiered-network-probe - if wget -T @PROBE_TIMEOUT@ -qO /tmp/nvx-untiered-network-probe \ - "http://@GATEWAY_IPV4@:$2/$1" 2>/dev/null; then - [ "$(cat /tmp/nvx-untiered-network-probe)" = "NVX-PLATFORM-NETWORK-$1" ] || - fail "unexpected $1 answer" 81 + if answer=$(wget -T @PROBE_TIMEOUT@ -qO- "http://@GATEWAY_IPV4@:$2/$1" 2>/dev/null); then + [ "$answer" = "NVX-PLATFORM-NETWORK-$1" ] || fail "unexpected $1 answer" 81 result=reached else result=denied @@ -40,18 +36,24 @@ allowed=$result probe other @OTHER_PORT@ other=$result echo "NVX-UNTIERED-NETWORK-REPORT resolver=$resolver announcer=$announcer allowed=$allowed other=$other" -# The guest's TCP server answers the host, one client at a time, through the -# ports that the restore's policy publishes or forwards. -rm -f /tmp/nvx-untiered-network-inbound-* -( - client=0 - while [ "$client" -lt @MAX_CLIENTS@ ]; do - printf 'NVX-HOST-LOOPBACK-TCP-ACK\n' | - nc -l -p @GUEST_PORT@ -w 20 \ - >"/tmp/nvx-untiered-network-inbound-$client" 2>/dev/null || true - client=$((client + 1)) - done -) & +# The guest's TCP server answers the host, one client at a time, each of which +# it waits for up to @CLIENT_TIMEOUT@ seconds, and keeps the line that each +# sends. Without a client, it listens on while the host checks that nothing +# publishes its port. +inbound=$(mktemp -d) || fail "no directory for the inbound lines" 87 +if [ "$exchanges" -eq 0 ]; then + busybox nc -l -p @GUEST_PORT@ /dev/null 2>&1 & +else + ( + client=0 + while [ "$client" -lt "$exchanges" ]; do + printf 'NVX-HOST-LOOPBACK-TCP-ACK\n' | + busybox nc -l -p @GUEST_PORT@ -w @CLIENT_TIMEOUT@ \ + >"$inbound/$client" 2>/dev/null || true + client=$((client + 1)) + done + ) /dev/null 2>&1 & +fi server=$! port_hex=$(printf '%04X' @GUEST_PORT@) tries=0 @@ -62,27 +64,16 @@ while ! awk -v pattern=":$port_hex\$" \ tries=$((tries + 1)) sleep 0.05 done -# The terminal would echo the host's release into the lines after it. -stty -echo || fail "the terminal kept echoing" 85 echo NVX-UNTIERED-NETWORK-LISTENING -# The host's release says how many lines it exchanged with the server. The -# last one may still be on the way when the release arrives, so the guest -# waits for each before it stops the server. -read -r release exchanges || fail "the host did not release the guest" 83 -[ "$release" = NVX-UNTIERED-NETWORK-CHECKED ] || fail "release $release" 84 -case "$exchanges" in -'' | *[!0-9]*) fail "release $release $exchanges" 84 ;; -esac +if [ "$exchanges" -eq 0 ]; then + sleep @HOLD_SECONDS@ + kill "$server" 2>/dev/null || true +fi +wait "$server" || true client=0 while [ "$client" -lt "$exchanges" ]; do - tries=0 - while ! grep -qx NVX-HOST-LOOPBACK-INBOUND \ - "/tmp/nvx-untiered-network-inbound-$client" 2>/dev/null; do - [ "$tries" -lt 100 ] || fail "client $client sent no line" 86 - tries=$((tries + 1)) - sleep 0.05 - done + grep -qx NVX-HOST-LOOPBACK-INBOUND "$inbound/$client" || + fail "client $client sent no line" 86 client=$((client + 1)) done -kill "$server" 2>/dev/null || true echo "NVX-UNTIERED-NETWORK-INBOUND $exchanges" diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index e49da312e..115586abc 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -72,6 +72,7 @@ from .managed_exec_tests import ( WorkloadArtifacts, bounded_text, + format_errors, load_workload_artifacts, run_managed_exec_configuration, workload_artifact_paths, @@ -205,6 +206,7 @@ "scratch-snapshot", "snapshot-core", "snapshot-tiers", + "untiered-network-policy", "virtio-features", "vm-lifecycle", ) @@ -223,6 +225,7 @@ "sandbox-ram-upper", "snapshot-core", "snapshot-tiers", + "untiered-network-policy", "vm-lifecycle", ) ) @@ -519,14 +522,14 @@ class MicrovmVirtioDevice(NamedTuple): PLATFORM_NETWORK_LISTENING_MARKER, RESTORE_MARKER, ) -# A NIC template without a tier is a snapshot without sandbox blocks whose NIC -# was captured with the complete deny-all policy. Each restore installs a -# policy of its own, and its guest's repair applies the DNS server and the -# listener announcement of that policy, which the restore packet carries. -UNTIERED_POST_RESTORE_MARKER = b"NVX-POST-RESTORE-OK: tier=legacy" +# A NIC template without a tier is a named snapshot of a blockless VM, which +# `nvx-rs snapshot capture --net` captures with its NIC under the complete +# deny-all policy. Each restore installs a policy of its own, and the restored +# guest's activation applies the DNS server and the listener announcement of +# that policy. +UNTIERED_NETWORK_SNAPSHOT = "untiered-network" UNTIERED_NETWORK_FAILURE_MARKER = b"NVX-UNTIERED-NETWORK-FAIL" UNTIERED_NETWORK_LISTENING_MARKER = b"NVX-UNTIERED-NETWORK-LISTENING" -UNTIERED_NETWORK_CHECKED = b"NVX-UNTIERED-NETWORK-CHECKED" UNTIERED_NETWORK_REPORT_PREFIX = b"NVX-UNTIERED-NETWORK-REPORT " UNTIERED_NETWORK_INBOUND_PREFIX = b"NVX-UNTIERED-NETWORK-INBOUND " # The template's NIC is on the directional network, whose documentation range @@ -538,21 +541,20 @@ class MicrovmVirtioDevice(NamedTuple): # The gateway's IPv6 address, which a policy that admits DNS over IPv6 alone # names as the guest's DNS server. UNTIERED_NETWORK_GATEWAY_IPV6 = DIRECTIONAL_NETWORK_GATEWAY_IPV6 -# The most clients that the guest's server answers, as many as a restore's -# host reaches it through: its forward and its published port. -UNTIERED_NETWORK_MAX_CLIENTS = 2 -# The restores of the NIC template that OpenVMM refuses before the guest runs, -# and why. +# How long the restored workload's server listens when no way in reaches it, +# while the host checks that nothing publishes its port, which +# _check_untiered_network_hold bounds. +UNTIERED_NETWORK_HOLD_SECONDS = 6 +# The restores of the NIC template that the CLI refuses before it provisions +# anything: its options, and what its error says. UNTIERED_NETWORK_REJECTED = ( - ("kept-policy", b"restore-time egress policy does not match the snapshot"), - ("ingress", b"microVM snapshots do not support --network-ingress allow"), -) -# Lines that only a restored guest prints, so that a refused restore prints -# none of them. -UNTIERED_NETWORK_GUEST_MARKERS = ( - UNTIERED_POST_RESTORE_MARKER, - UNTIERED_NETWORK_LISTENING_MARKER, - RESTORE_MARKER, + ("without-profile", ("snapshot", "restore"), (), "needs --network-profile"), + ( + "readdressed", + ("run", "--restore-snapshot"), + ("--net", UNTIERED_NETWORK_CIDR, "--network-profile", "portable"), + "--net is not accepted for restore", + ), ) # A platform restore attaches a host directory to the filesystem slot that the # capture left dormant. The clone mounts it after the restore's input gate @@ -9034,33 +9036,39 @@ def check_template(context: str) -> None: def untiered_network_policy_script( - *, allowed_port: int, other_port: int, guest_port: int + *, allowed_port: int, other_port: int, guest_port: int, timeout: float ) -> str: - """Return the checks that each restore of the NIC template without a tier - runs: it reports the nameservers of its resolver, whether the announcer of - its listeners runs, and whether it reaches the host services at - ``allowed_port`` and ``other_port`` of the gateway, then serves the host at - ``guest_port`` until the host releases it.""" + """Return the checks that the workload of each restore of the NIC template + without a tier runs, as `sh -c SCRIPT NAME EXCHANGES`: they report the + nameservers of the guest's resolver, whether the announcer of its + listeners runs, and whether it reaches the host services at + ``allowed_port`` and ``other_port`` of the gateway, then serve the host at + ``guest_port`` until they have exchanged a line with EXCHANGES clients, + each of which they wait for up to ``timeout`` seconds, or without one, for + UNTIERED_NETWORK_HOLD_SECONDS.""" return _render_script( "untiered-network-policy.sh.in", GATEWAY_IPV4=UNTIERED_NETWORK_GATEWAY_IPV4, ALLOWED_PORT=str(allowed_port), OTHER_PORT=str(other_port), GUEST_PORT=str(guest_port), - MAX_CLIENTS=str(UNTIERED_NETWORK_MAX_CLIENTS), PROBE_TIMEOUT=str(PLATFORM_NETWORK_PROBE_TIMEOUT_SECONDS), + CLIENT_TIMEOUT=str(math.ceil(timeout)), + HOLD_SECONDS=str(UNTIERED_NETWORK_HOLD_SECONDS), ) @dataclass(frozen=True) class UntieredNetworkRestore: """A restore of the NIC template without a tier: the policy that it - installs; the DNS server that its guest learns from the restore packet; - whether its guest reaches the allowed and the other host service; and how - the host reaches the guest's server: through a forward, and through the - port that the policy publishes on host loopback, ``host-loopback``, or to - the host's private networks, ``private-networks``, for which the guest - announces its listeners.""" + installs; the DNS server that its guest learns from its activation; + whether its guest reaches the allowed and the other host service; how the + host reaches the guest's server: through a forward, and through the port + that the policy publishes on host loopback, ``host-loopback``, or to the + host's private networks, ``private-networks``, for which the guest + announces its listeners; and whether `nvx-rs run --restore-snapshot` + restores it once, ``one_off``, or `nvx-rs snapshot restore` restores a + sandbox that runs on, which the scenario stops and deprovisions.""" name: str policy: tuple[str, ...] @@ -9069,6 +9077,7 @@ class UntieredNetworkRestore: other: bool forward: bool publication: str | None + one_off: bool @property def announces(self) -> bool: @@ -9076,12 +9085,6 @@ def announces(self) -> bool: policy publishes them.""" return self.publication is not None - @property - def network_settings(self) -> bool: - """Return whether the guest's repair applies network settings, after - which it reports the restore's completion.""" - return self.dns_server is not None or self.announces - @property def exchanges(self) -> int: """Return how many lines the host exchanges with the guest's server: @@ -9097,7 +9100,10 @@ def untiered_network_restores( egress rules that admit DNS over IPv4 and the allowed service; explicit host loopback with a forward from ``host_port`` to ``guest_port``, under egress rules that admit DNS over IPv6 alone; and private-network ingress - with allowed egress.""" + with allowed egress. `snapshot restore` restores the first and the last, + whose guests learn a DNS server and an announcer of their listeners or + neither, and `run --restore-snapshot` the others, whose guests learn their + DNS servers, egress, forwards and published listeners.""" gateway = UNTIERED_NETWORK_GATEWAY_IPV4 return ( UntieredNetworkRestore( @@ -9108,6 +9114,7 @@ def untiered_network_restores( other=False, forward=False, publication=None, + one_off=False, ), UntieredNetworkRestore( "egress-dns", @@ -9124,6 +9131,7 @@ def untiered_network_restores( other=False, forward=False, publication=None, + one_off=True, ), UntieredNetworkRestore( "host-loopback", @@ -9142,6 +9150,7 @@ def untiered_network_restores( other=False, forward=True, publication="host-loopback", + one_off=True, ), UntieredNetworkRestore( "private-ingress", @@ -9158,6 +9167,7 @@ def untiered_network_restores( other=True, forward=False, publication="private-networks", + one_off=False, ), ) @@ -9235,13 +9245,20 @@ def _untiered_network_exchanges( """Exchange a line with the guest's server through each way that ``restore``'s policy gives the host, and return how many it exchanged. - The forward binds ``host_port`` on host loopback before the guest runs. A - published port is ``guest_port``, which the gateway publishes once the - guest's announcer has announced the server: on host loopback alone, which - no other host address reaches, or on every host address, through one of - which, other than loopback, the host reaches it as a peer on its private - networks would. Without an announcer, nothing publishes the guest's port. + Without an announcer, nothing publishes the guest's port, which the host + checks first, while the server waits for its clients, or without one, + holds the port, as _check_untiered_network_hold then checks. The forward + binds ``host_port`` on host loopback before the guest runs. A published + port is ``guest_port``, which the gateway publishes once the guest's + announcer has announced the server: on host loopback alone, which no other + host address reaches, or on every host address, through one of which, + other than loopback, the host reaches it as a peer on its private networks + would. """ + if restore.publication is None and _holds_loopback( + socket.AF_INET, socket.SOCK_STREAM, guest_port + ): + raise RuntimeError(f"guest port {guest_port} was published without ingress") exchanges = 0 if restore.forward: _host_loopback_tcp_exchange("127.0.0.1", host_port, timeout) @@ -9277,11 +9294,41 @@ def _untiered_network_exchanges( ) _host_loopback_tcp_exchange(addresses[0], guest_port, timeout) exchanges += 1 - elif _holds_loopback(socket.AF_INET, socket.SOCK_STREAM, guest_port): - raise RuntimeError(f"guest port {guest_port} was published without ingress") return exchanges +def _check_untiered_network_hold( + guest_port: int, *, listening: float, checked: float, held: float +) -> None: + """Check that the host checked that nothing publishes ``guest_port`` + while the guest held the port, as far as the host can tell from the + monotonic times at which it saw the guest's markers. + + The host cannot tell the guest that it has checked, as a restored + workload takes no input and the template's deny-all policy has no way + in, and a marker that reaches the host late can let the guest's hold end + before the check, which would then miss a port that the hold published. + So the host must have checked by ``checked``, within a third of + UNTIERED_NETWORK_HOLD_SECONDS after it saw the guest listen, at + ``listening``, and must have seen the hold end, at ``held``, at least a + third of it later: the check then misses the hold only if both markers + reached the host seconds late. + """ + margin = UNTIERED_NETWORK_HOLD_SECONDS / 3 + if checked - listening > margin: + raise RuntimeError( + f"the host checked guest port {guest_port} {checked - listening:.1f}s " + "after it saw the guest listen, too late to fall within the guest's " + f"{UNTIERED_NETWORK_HOLD_SECONDS}s hold" + ) + if held - checked < margin: + raise RuntimeError( + f"the host saw the guest's hold of port {guest_port} end " + f"{held - checked:.1f}s after its check, too soon to show that the " + "check fell within the hold" + ) + + def _run_untiered_network_restore( command: list[str], restore: UntieredNetworkRestore, @@ -9293,52 +9340,59 @@ def _run_untiered_network_restore( log_path: Path, ) -> None: """Restore the NIC template with ``command``, which installs - ``restore``'s policy, exchange lines with its guest's server while the - guest serves the host, and check what the guest reports, whether its - repair applied network settings, and the requests that the allowed and the + ``restore``'s policy and runs the scenario's checks as its workload, + exchange lines with the workload's server while it serves the host, and + check what the workload reports and the requests that the allowed and the other host service in ``services`` received.""" context = f"untiered network policy restore {restore.name}" + + def failure() -> str: + lines = [ + line.decode("utf-8", "replace") + for line in _output_lines( + log_path.read_bytes() if log_path.is_file() else b"" + ) + if line.startswith(UNTIERED_NETWORK_FAILURE_MARKER) + ] + return f": {lines[0]}" if lines else "" + try: with OpenvmmProcess(command, log_path) as process: process.wait_for_line(UNTIERED_NETWORK_LISTENING_MARKER, timeout) - exchanges = _untiered_network_exchanges( + listening = time.monotonic() + _untiered_network_exchanges( restore, host_port=host_port, guest_port=guest_port, timeout=timeout ) - process.send_line(f"{UNTIERED_NETWORK_CHECKED.decode()} {exchanges}") + if not restore.exchanges: + # A server without a client holds its port until the guest + # reports that it received no line. + checked = time.monotonic() + process.wait_for_line(UNTIERED_NETWORK_INBOUND_PREFIX + b"0", timeout) + _check_untiered_network_hold( + guest_port, + listening=listening, + checked=checked, + held=time.monotonic(), + ) result = process.wait(timeout) except Exception as error: - failures = [ - line.decode("utf-8", "replace") - for line in _output_lines( - log_path.read_bytes() if log_path.is_file() else b"" - ) - if line.startswith(UNTIERED_NETWORK_FAILURE_MARKER) - ] + cause = next( + (line for line in str(error).splitlines() if line.strip()), + type(error).__name__, + ) raise RuntimeError( - f"{context} failed{f': {failures[0]}' if failures else ''}; see {log_path}" + f"{context} failed{failure()} ({cause}); see {log_path}" ) from error if result.returncode != 0: - raise RuntimeError(f"{context} exited with {result.returncode}") + raise RuntimeError( + f"{context} exited with {result.returncode}{failure()}; see {log_path}" + ) _check_untiered_network_report( restore, _untiered_network_report(result.output, context), context ) - for marker in ( - UNTIERED_NETWORK_INBOUND_PREFIX + str(restore.exchanges).encode(), - RESTORE_MARKER, - ): - if not contains_output_line(result.output, marker): - raise RuntimeError(f"{context} did not report {marker.decode()!r}") - # A restore without network settings to apply finishes quietly, as a plain - # restore without a tier does. - if ( - contains_output_line(result.output, UNTIERED_POST_RESTORE_MARKER) - != restore.network_settings - ): - raise RuntimeError( - f"{context}: the guest's repair " - f"{'did not report' if restore.network_settings else 'reported'} " - f"{UNTIERED_POST_RESTORE_MARKER.decode()!r}" - ) + inbound = UNTIERED_NETWORK_INBOUND_PREFIX + str(restore.exchanges).encode() + if not contains_output_line(result.output, inbound): + raise RuntimeError(f"{context} did not report {inbound.decode()!r}") requests = tuple(service.take_requests() for service in services) expected = (int(restore.allowed), int(restore.other)) if requests != expected: @@ -9348,26 +9402,169 @@ def _run_untiered_network_restore( ) +def _untiered_network_cli( + arguments: Sequence[str], *, timeout: float +) -> subprocess.CompletedProcess[bytes]: + """Run `nvx-rs ARGUMENTS` on the artifacts of this checkout, within twice + ``timeout`` and half a minute, and return its status and output.""" + return subprocess.run( + nvx_rs.command_line(arguments), + stdin=subprocess.DEVNULL, + capture_output=True, + timeout=2 * timeout + 30, + check=False, + ) + + +def _check_untiered_network_cli( + arguments: Sequence[str], *, timeout: float, log_path: Path, context: str +) -> None: + """Run `nvx-rs ARGUMENTS` as _untiered_network_cli does, keep its output + at ``log_path``, and fail unless it succeeds.""" + result = _untiered_network_cli(arguments, timeout=timeout) + log_path.write_bytes(result.stdout + result.stderr) + if result.returncode != 0: + reported = nvx_rs.reported_error(result.stderr) + error = ( + "" if reported is None else f": error[{reported.code}]: {reported.message}" + ) + raise RuntimeError( + f"{context} exited with {result.returncode}{error}; see {log_path}" + ) + + +def _untiered_network_sandbox_options( + *, backend: str, store: Path, state: Path, layer: str, timeout: float +) -> tuple[str, ...]: + """Return the options with which a restore of the NIC template of + ``store`` provisions its sandbox at ``state``: the distro layer ``layer`` + over a RAM upper, with ``timeout`` seconds for each wait.""" + return ( + "--store-dir", + os.fspath(store), + "--state-dir", + os.fspath(state), + "--hypervisor", + backend, + "--layer", + layer, + "--upper", + "ram", + "--timeout", + str(timeout), + ) + + +def _untiered_network_restore_command( + restore: UntieredNetworkRestore, *, script: str, sandbox: Sequence[str] +) -> list[str]: + """Return the command that restores the NIC template with ``restore``'s + policy into a sandbox of the options ``sandbox``, and runs ``script`` with + the number of the host's exchanges as its workload: `nvx-rs run + --restore-snapshot` for a one-off, and `nvx-rs snapshot restore` + otherwise. A restore sends its workload as an exec request, so the script + is one argument, whitespace and all.""" + command = ( + ("run", "--restore-snapshot") if restore.one_off else ("snapshot", "restore") + ) + return nvx_rs.command_line( + ( + *command, + UNTIERED_NETWORK_SNAPSHOT, + *sandbox, + "--network-profile", + "portable", + *restore.policy, + "--entrypoint", + "/bin/sh", + "--arg=-c", + f"--arg={script}", + "--arg=untiered-network-policy", + f"--arg={restore.exchanges}", + ) + ) + + +def _clean_up_untiered_network_policy( + root: Path, timeout: float, output_dir: Path +) -> list[Exception]: + """Stop each sandbox that a failed run left in ``root``, end each OpenVMM + process that names ``root``, keep each sandbox's VM log in + ``output_dir``, and then deprovision the sandboxes, as the sandbox + lifecycle acceptance cleans up; return the failures.""" + errors: list[Exception] = [] + states = sorted( + path for path in root.iterdir() if path.is_dir() and path.name != "store" + ) + + def sandbox(operation: str, state: Path) -> subprocess.CompletedProcess[bytes]: + return _untiered_network_cli( + ( + "sandbox", + operation, + "--state-dir", + os.fspath(state), + "--timeout", + str(timeout), + ), + timeout=timeout, + ) + + for state in states: + try: + sandbox("stop", state) + except Exception as error: + errors.append(error) + # Every OpenVMM process whose arguments name the fixture is the scenario's, + # even one that a stop could not end. + try: + for pid in openvmm_processes(root.name): + with suppress(OSError): + terminate_process(pid) + deadline = time.monotonic() + timeout + while strays := openvmm_processes(root.name): + if time.monotonic() >= deadline: + raise RuntimeError(f"OpenVMM processes {list(strays)} survived") + time.sleep(0.05) + except Exception as error: + errors.append(error) + for state in states: + try: + log = state / LOG + if log.is_file(): + shutil.copyfile( + log, + output_dir / f"untiered-network-policy-{state.name}-openvmm.log", + ) + # A deprovision ends an unfinished start, clears the runtime record + # of an OpenVMM that exited, and removes the sandbox. + result = sandbox("deprovision", state) + if result.returncode != 0: + raise RuntimeError( + f"cannot deprovision {state}: {bounded_text(result.stderr).strip()}" + ) + except Exception as error: + errors.append(error) + return errors + + def run_untiered_network_policy( - executable: Path, - kernel: Path, - initrd: Path, - backend: str, - *, - memory_mib: int, - timeout: float, - output_dir: Path, + backend: str, *, timeout: float, output_dir: Path ) -> None: """Restore one NIC template without a tier with four network policies. - The template is a snapshot without sandbox blocks, as the non-sandbox - mode captures one, whose NIC was captured with a complete deny-all - policy, so its guest learned no DNS server and announces no listeners. - Each restore installs another policy with ``--restore-network-policy``: - the template's deny-all; egress rules that admit DNS over IPv4 and one - host service; explicit host loopback with a forward, under egress rules - that admit DNS over IPv6 alone; and private-network ingress with allowed - egress. Each restore's guest must name the DNS server of its policy in its + The template is a named snapshot that `nvx-rs snapshot capture --net` + captures, of a blockless VM whose NIC has the complete deny-all policy, + so its guest learned no DNS server and announces no listeners. Each + restore installs another policy, the network options of a sandbox of the + Ubuntu layer over a RAM upper, at the NIC's captured address, and runs the + scenario's checks as its workload: the template's deny-all and + private-network ingress with allowed egress through `nvx-rs snapshot + restore`, whose sandboxes the scenario then stops and deprovisions, and + egress rules that admit DNS over IPv4 and one host service, and explicit + host loopback with a forward, under egress rules that admit DNS over IPv6 + alone, through `nvx-rs run --restore-snapshot`, which removes its + sandbox. Each restored guest must name the DNS server of its policy in its resolver, and no gateway without one, run the announcer of its listeners exactly when its policy publishes them, and reach exactly the host services that its policy admits, which must receive no other request. The @@ -9375,11 +9572,52 @@ def run_untiered_network_policy( port that the policy publishes: on host loopback alone, which no other host address reaches, or to the host's private networks, through a host address other than loopback. It must find the server on no host loopback - port without a publication. OpenVMM must refuse, before the guest runs, a - restore that keeps the template's policy but passes another, and one that - passes private-network ingress without the option. No restore may change - the template. + port without a publication, while the guest holds the server's port, as + far as the host can tell. Before it provisions anything, the CLI must + refuse a restore of the template without `--network-profile`, and one + that names the NIC's address with `--net`. No restore may change the + template, or leave its sandbox or an OpenVMM process behind; a failed run + keeps the VM log of each sandbox that it leaves in ``output_dir``. """ + artifacts = load_workload_artifacts() + output_dir.mkdir(parents=True, exist_ok=True) + root = Path(tempfile.mkdtemp(prefix="nvx-untiered-network-")) + try: + _run_untiered_network_policy( + backend, + layer=f"distro,{artifacts.distro},{artifacts.layer_uuid}", + root=root, + timeout=timeout, + output_dir=output_dir, + ) + except BaseException as error: + errors = _clean_up_untiered_network_policy(root, timeout, output_dir) + if any(path.name != "store" for path in root.iterdir()): + errors.append(RuntimeError(f"fixture preserved for recovery: {root}")) + else: + shutil.rmtree(root, ignore_errors=True) + if errors and isinstance(error, Exception): + raise RuntimeError(f"{error}; cleanup: {format_errors(errors)}") from error + raise + shutil.rmtree(root) + + +def _run_untiered_network_policy( + backend: str, *, layer: str, root: Path, timeout: float, output_dir: Path +) -> None: + """Run the untiered-network-policy scenario in ``root``, whose sandboxes + run the distro layer ``layer``, as run_untiered_network_policy says.""" + store = root / "store" + + def sandbox(state: str) -> tuple[str, ...]: + return _untiered_network_sandbox_options( + backend=backend, + store=store, + state=root / state, + layer=layer, + timeout=timeout, + ) + with ExitStack() as stack: services = ( _HostLoopbackService("allowed", timeout), @@ -9395,61 +9633,57 @@ def run_untiered_network_policy( restores = untiered_network_restores( allowed_port=allowed.port, host_port=host_port, guest_port=guest_port ) - root = Path( - stack.enter_context( - tempfile.TemporaryDirectory(prefix="nvx-untiered-network-policy-") - ) - ) - template = root / "template" - capture_snapshot( - [ - *workload_boot_command( - executable, - backend, - kernel, - initrd, - memory_mib, - "", - network=UNTIERED_NETWORK_CIDR, - ), - *PLATFORM_NETWORK_TEMPLATE_POLICY, - "--snapshot-destination", - str(template), - ], - template, + script = untiered_network_policy_script( + allowed_port=allowed.port, + other_port=other.port, + guest_port=guest_port, timeout=timeout, - processors=1, - post_restore_script=untiered_network_policy_script( - allowed_port=allowed.port, - other_port=other.port, - guest_port=guest_port, + ) + _check_untiered_network_cli( + ( + "snapshot", + "capture", + UNTIERED_NETWORK_SNAPSHOT, + "--store-dir", + os.fspath(store), + "--hypervisor", + backend, + "--net", + UNTIERED_NETWORK_CIDR, + "--timeout", + str(math.ceil(timeout)), ), + timeout=timeout, log_path=output_dir / "untiered-network-policy-capture.log", + context="the capture of the NIC template", + ) + template = ( + store / "snapshots" / UNTIERED_NETWORK_SNAPSHOT.encode().hex() / "openvmm" ) fingerprint = _snapshot_fingerprint(template) - def restore_command(*arguments: str) -> list[str]: - return [ - *snapshot_restore_command( - executable, backend, template, network_profile="portable" - ), - *arguments, - ] - def check_template(context: str) -> None: if _snapshot_fingerprint(template) != fingerprint: raise RuntimeError(f"{context} modified the NIC template") errors = [error for service in services for error in service.errors] if errors: raise RuntimeError(f"a host service failed by {context}") from errors[0] + processes = openvmm_processes(root.name) + if processes: + raise RuntimeError( + f"OpenVMM processes {list(processes)} outlived {context}" + ) check_template("the capture") # The template's guest probed nothing before its capture. if any(service.take_requests() for service in services): raise RuntimeError("the NIC template's capture reached a host service") for restore in restores: + state = root / restore.name _run_untiered_network_restore( - restore_command("--restore-network-policy", *restore.policy), + _untiered_network_restore_command( + restore, script=script, sandbox=sandbox(restore.name) + ), restore, services=services, host_port=host_port, @@ -9457,22 +9691,55 @@ def check_template(context: str) -> None: timeout=timeout, log_path=output_dir / f"untiered-network-policy-{restore.name}.log", ) + # The sandbox of `snapshot restore` runs on once its workload has + # exited, until the scenario stops it. + if not restore.one_off: + for operation in ("stop", "deprovision"): + _check_untiered_network_cli( + ( + "sandbox", + operation, + "--state-dir", + os.fspath(state), + "--timeout", + str(timeout), + ), + timeout=timeout, + log_path=output_dir + / f"untiered-network-policy-{restore.name}-{operation}.log", + context=f"sandbox {operation} after restore {restore.name}", + ) + if state.exists(): + raise RuntimeError(f"restore {restore.name} left its sandbox {state}") check_template(f"restore {restore.name}") - rejected_commands = { - "kept-policy": restore_command("--network-egress", "allow"), - "ingress": restore_command( - *PLATFORM_NETWORK_TEMPLATE_POLICY, "--network-ingress", "allow" - ), - } - for name, reason in UNTIERED_NETWORK_REJECTED: - _expect_process_failure( - rejected_commands[name], - output_dir / f"untiered-network-policy-rejected-{name}.log", - timeout, - reason, - forbidden_markers=UNTIERED_NETWORK_GUEST_MARKERS, + for name, command, options, message in UNTIERED_NETWORK_REJECTED: + log_path = output_dir / f"untiered-network-policy-rejected-{name}.log" + result = _untiered_network_cli( + ( + *command, + UNTIERED_NETWORK_SNAPSHOT, + *sandbox(name), + *options, + "--entrypoint", + "/bin/true", + ), + timeout=timeout, ) + log_path.write_bytes(result.stdout + result.stderr) + reported = nvx_rs.reported_error(result.stderr) + if ( + result.returncode != 1 + or reported is None + or reported.code != "unsupported" + or message not in reported.message + ): + raise RuntimeError( + f"the CLI did not refuse restore {name} with {message!r}: " + f"status {result.returncode}; see {log_path}" + ) + if (root / name).exists(): + raise RuntimeError(f"the refused restore {name} provisioned a sandbox") check_template("a refused restore") if any(service.take_requests() for service in services): raise RuntimeError("a refused restore reached a host service") @@ -12532,17 +12799,11 @@ def _run_scenarios( if "untiered-network-policy" in scenarios: report.start( "untiered-network-policy", - "Running microVM untiered-snapshot network policy replacement on " + "Running public nvx-rs network policies of named snapshot restores on " f"OpenVMM/{args.backend}", ) run_untiered_network_policy( - executable, - kernel, - initrd, - args.backend, - memory_mib=args.memory_mib, - timeout=args.timeout, - output_dir=output_dir, + args.backend, timeout=args.timeout, output_dir=output_dir ) if "platform-filesystem" in scenarios: report.start( diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 48a8ce11d..5af17c567 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2660,11 +2660,6 @@ def test_scripts_awaiting_a_release_turn_the_echo_off_first(self): f"echo {microvm_tests.PLATFORM_NETWORK_LISTENING_MARKER.decode()}", "read -r release exchange", ), - ( - "untiered-network-policy.sh.in", - f"echo {microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER.decode()}", - "read -r release exchanges", - ), ): with self.subTest(name=name): script = microvm_tests._read_script(name) @@ -14238,19 +14233,20 @@ def test_platform_network_policy_keeps_the_template_intact(self): Path(temporary), fingerprints=fingerprints ) - def test_untiered_network_policy_script_reports_the_repair_and_serves_the_host( + def test_untiered_network_policy_script_reports_the_policy_and_serves_the_host( self, ): script = microvm_tests.untiered_network_policy_script( - allowed_port=40001, other_port=40002, guest_port=50001 + allowed_port=40001, other_port=40002, guest_port=50001, timeout=59.5 ) self.assertNotRegex(script, r"@[A-Z0-9_]+@") - # The guest reports the resolver and the announcer that its repair set - # up, and probes both host services. + # The workload takes the number of the host's exchanges. + self.assertIn("\nexchanges=$1\n", script) + # The workload reports the resolver and the announcer that its + # activation set up, and probes both host services. self.assertIn("' /etc/resolv.conf 2>/dev/null) ||\n resolver=none", script) - self.assertIn("if pidof nvx-ingress >/dev/null; then", script) - self.assertIn('"http://192.0.2.1:$2/$1"', script) - self.assertIn("wget -T 2 -qO /tmp/nvx-untiered-network-probe", script) + self.assertIn("if busybox pidof nvx-ingress >/dev/null; then", script) + self.assertIn('wget -T 2 -qO- "http://192.0.2.1:$2/$1"', script) self.assertIn("\nprobe allowed 40001\n", script) self.assertIn("\nprobe other 40002\n", script) self.assertIn( @@ -14259,28 +14255,31 @@ def test_untiered_network_policy_script_reports_the_repair_and_serves_the_host( + 'resolver=$resolver announcer=$announcer allowed=$allowed other=$other"', script, ) - # The guest's server answers the host, one client at a time, as many - # as a restore's host reaches it through. + # The workload's server answers each client that the host exchanges a + # line with, which it waits for up to the scenario's timeout, or + # without one, holds its port while the host checks that nothing + # publishes it. ack = microvm_tests.HOST_LOOPBACK_TCP_ACK.decode().removesuffix("\n") - self.assertIn('while [ "$client" -lt 2 ]; do', script) - self.assertIn(f"printf '{ack}\\n' |\n nc -l -p 50001 -w 20", script) - self.assertIn("printf '%04X' 50001", script) - # Once released, the guest waits for each line that the host sent, - # which the release may overtake, before it stops its server. - request = microvm_tests.HOST_LOOPBACK_TCP_REQUEST.decode().strip() - self.assertIn("read -r release exchanges ||", script) + self.assertIn('while [ "$client" -lt "$exchanges" ]; do', script) self.assertIn( - f'grep -qx {request} \\\n "/tmp/nvx-untiered-network-inbound-$client"', + f"printf '{ack}\\n' |\n busybox nc -l -p 50001 -w 60 ", script, ) + self.assertIn("busybox nc -l -p 50001 /dev/null 2>&1 &", script) + self.assertIn("printf '%04X' 50001", script) + self.assertIn( + f"\n sleep {microvm_tests.UNTIERED_NETWORK_HOLD_SECONDS}\n", script + ) + # It checks each client's line once its server has served them all. + request = microvm_tests.HOST_LOOPBACK_TCP_REQUEST.decode().strip() + listening = microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER.decode() self.assertLess( - script.index("read -r release exchanges"), script.index('kill "$server"') + script.index(f"echo {listening}"), script.index('wait "$server"') + ) + self.assertLess( + script.index('wait "$server"'), + script.index(f'grep -qx {request} "$inbound/$client"'), ) - for marker in ( - microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER, - microvm_tests.UNTIERED_NETWORK_CHECKED, - ): - self.assertIn(marker.decode(), script) self.assertTrue( script.endswith( 'echo "' @@ -14293,6 +14292,17 @@ def test_untiered_network_policy_script_reports_the_repair_and_serves_the_host( script, ) + def test_untiered_network_policy_script_fits_one_argument_of_an_exec_request( + self, + ): + # A restore sends its workload as an exec request, whose arguments + # hold at most 4096 bytes each, so the longest script that the + # scenario renders must fit one. + script = microvm_tests.untiered_network_policy_script( + allowed_port=65535, other_port=65535, guest_port=65535, timeout=3600 + ) + self.assertLessEqual(len(script.encode()), 4096) + def test_untiered_network_restores_name_what_their_policies_give_the_guest(self): restores = microvm_tests.untiered_network_restores( allowed_port=40001, host_port=40003, guest_port=50001 @@ -14308,7 +14318,7 @@ def test_untiered_network_restores_name_what_their_policies_give_the_guest(self) restore.forward, restore.publication, restore.exchanges, - restore.network_settings, + restore.one_off, ) for restore in restores ], @@ -14335,7 +14345,7 @@ def test_untiered_network_restores_name_what_their_policies_give_the_guest(self) False, "private-networks", 1, - True, + False, ), ], ) @@ -14544,10 +14554,13 @@ def values(option: str) -> list[str]: def _simulate_untiered_network_guest( self, command: list[str], ports: dict[str, int] ) -> list[bytes]: - """Return the lines of the guest that ``command`` restores before its - release, after it requests each host service in ``ports`` that its - policy admits.""" - policy = command[command.index("--restore-network-policy") + 1 :] + """Return the lines that the workload of the restore that ``command`` + runs prints before it listens, after it requests each host service in + ``ports`` that its policy, the network options between + `--network-profile portable` and its `--entrypoint`, admits.""" + policy = command[ + command.index("--network-profile") + 2 : command.index("--entrypoint") + ] dns_server, announces, allowed, other = self._untiered_network_settings( policy, ports["allowed"] ) @@ -14566,10 +14579,7 @@ def _simulate_untiered_network_guest( self.assertTrue( answer.endswith(f"\r\n\r\nNVX-PLATFORM-NETWORK-{name}".encode()) ) - lines: list[bytes] = [] - if dns_server is not None or announces: - lines.append(microvm_tests.UNTIERED_POST_RESTORE_MARKER) - lines.append( + return [ microvm_tests.UNTIERED_NETWORK_REPORT_PREFIX + ( f"resolver={dns_server or 'none'} " @@ -14577,50 +14587,81 @@ def _simulate_untiered_network_guest( f"allowed={'reached' if allowed else 'denied'} " f"other={'reached' if other else 'denied'}" ).encode() - ) - return lines + ] def _run_untiered_network_policy( self, - output_dir: Path, + root: Path, *, guest: Callable[[list[str], dict[str, int]], list[bytes]] | None = None, + cli: Callable[[tuple[str, ...]], subprocess.CompletedProcess[bytes] | None] + | None = None, fingerprints: Sequence[tuple[str, str, str]] | None = None, + processes: Sequence[tuple[int, ...]] | None = None, host_addresses: Sequence[str] = ("192.0.2.10",), accepts_tcp: bool = False, exchange: Callable[[str, int, float], None] | None = None, + hold: Callable[..., None] | None = None, + record: dict[str, Any] | None = None, ) -> dict[str, Any]: - """Run the scenario with a simulated OpenVMM, whose restores request - the scenario's real host services as their policies admit, on a host - whose addresses other than loopback are ``host_addresses``, through - which a TCP connection completes if ``accepts_tcp``, and with - ``exchange`` standing for each exchange with the guest's server, and - return its calls.""" - captures: list[tuple[list[str], dict[str, Any]]] = [] + """Run the scenario in ``root``, with a simulated nvx-rs, whose + restores' workloads request the scenario's real host services as their + policies admit, and which ``cli`` may answer first, on a host whose + OpenVMM processes are ``processes``, whose addresses other than + loopback are ``host_addresses``, through which a TCP connection + completes if ``accepts_tcp``, with ``exchange`` standing for each + exchange with the workload's server, and ``hold`` for the check that + the host checked a port within the guest's hold of it, which passes + unless given; return its calls, which ``record`` also receives, even + when the scenario fails. + + The scenario's sandboxes and logs go to ``root``'s `fixture` and + `logs`.""" + calls: list[tuple[str, ...]] = [] restores: list[list[str]] = [] - releases: list[str] = [] + awaited: list[tuple[str, bytes]] = [] ports: dict[str, int] = {} - - def capture(command: list[str], *_args: object, **kwargs: Any) -> None: - captures.append((command, kwargs)) - script = kwargs["post_restore_script"] - for name in ("allowed", "other"): - match = re.search(rf"\nprobe {name} (\d+)\n", script) - assert match is not None - ports[name] = int(match[1]) - match = re.search(r"nc -l -p (\d+) ", script) - assert match is not None - ports["guest"] = int(match[1]) - + fixture = root / "fixture" + output_dir = root / "logs" simulated_guest = guest or self._simulate_untiered_network_guest + def run_cli( + arguments: Sequence[str], *, timeout: float + ) -> subprocess.CompletedProcess[bytes]: + del timeout + call = tuple(arguments) + calls.append(call) + if cli is not None and (result := cli(call)) is not None: + return result + if call[:2] == ("snapshot", "capture"): + return subprocess.CompletedProcess(call, 0, b"Snapshot captured\n", b"") + if call[:2] == ("sandbox", "deprovision"): + shutil.rmtree(call[call.index("--state-dir") + 1]) + if call[0] == "sandbox": + return subprocess.CompletedProcess(call, 0, b"", b"") + refusal = next( + message + for name, _, _, message in microvm_tests.UNTIERED_NETWORK_REJECTED + if os.fspath(fixture / name) in call + ) + return subprocess.CompletedProcess( + call, 1, b"", f"error[unsupported]: {refusal}\n".encode() + ) + class FakeProcess: def __init__(self, command: list[str], log_path: Path) -> None: restores.append(command) + script = command[command.index("--arg=-c") + 1].removeprefix("--arg=") + for name in ("allowed", "other"): + match = re.search(rf"\nprobe {name} (\d+)\n", script) + assert match is not None + ports[name] = int(match[1]) + match = re.search(r"nc -l -p (\d+) ", script) + assert match is not None + ports["guest"] = int(match[1]) self.command = command self.log_path = log_path self.lines: list[bytes] = [] - self.exchanges: bytes | None = None def __enter__(self) -> "FakeProcess": return self @@ -14629,33 +14670,58 @@ def __exit__(self, *_exception: object) -> None: return None def wait_for_line(self, marker: bytes, _timeout: float) -> None: - assert marker == microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER - self.lines = simulated_guest(self.command, ports) - - def send_line(self, line: str) -> None: - releases.append(line) - self.exchanges = line.split()[1].encode() + state = Path(self.command[self.command.index("--state-dir") + 1]) + awaited.append((state.name, marker)) + if marker != microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER: + # Without a client, the guest holds its port until it + # reports that it received no line. + assert self.lines, marker + assert ( + marker == microvm_tests.UNTIERED_NETWORK_INBOUND_PREFIX + b"0" + ) + return + # The restore provisions its sandbox in its state directory. + state.mkdir() + self.lines = [*simulated_guest(self.command, ports), marker] def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: - assert self.exchanges is not None + exchanges = self.command[-1].removeprefix("--arg=").encode() lines = [ *self.lines, - microvm_tests.UNTIERED_NETWORK_INBOUND_PREFIX + self.exchanges, - benchmark.RESTORE_MARKER, + microvm_tests.UNTIERED_NETWORK_INBOUND_PREFIX + exchanges, ] output = b"".join(line + b"\r\n" for line in lines) self.log_path.write_bytes(output) + # `run --restore-snapshot` removes its sandbox once its workload + # has exited. + if "--restore-snapshot" in self.command: + shutil.rmtree(self.command[self.command.index("--state-dir") + 1]) return openvmm_process.OpenvmmProcessResult(0, output) fingerprint = ("manifest", "state", "memory") - output_dir.mkdir(parents=True, exist_ok=True) + root.mkdir(parents=True, exist_ok=True) + observed: dict[str, Any] = {} if record is None else record + observed.update( + calls=calls, + restores=restores, + awaited=awaited, + ports=ports, + fixture=fixture, + output_dir=output_dir, + ) with ( patch.object( microvm_tests, - "workload_boot_command", - return_value=["openvmm", "boot"], - ) as workload_boot_command, - patch.object(microvm_tests, "capture_snapshot", side_effect=capture), + "load_workload_artifacts", + return_value=managed_exec_tests.WorkloadArtifacts( + Path("distro.erofs"), + "11111111-1111-1111-1111-111111111111", + Path("scratch.ext4"), + ), + ), + patch.object(microvm_tests.tempfile, "mkdtemp", return_value=str(fixture)), + patch.object(nvx_rs, "require_binary", return_value=_FAKE_NVX_RS), + patch.object(microvm_tests, "_untiered_network_cli", side_effect=run_cli), patch.object(microvm_tests, "OpenvmmProcess", FakeProcess), patch.object( microvm_tests, "_host_loopback_tcp_exchange", side_effect=exchange @@ -14670,106 +14736,206 @@ def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: patch.object( microvm_tests, "_accepts_tcp", return_value=accepts_tcp ) as accepts_tcp_mock, + patch.object( + microvm_tests, "_check_untiered_network_hold", side_effect=hold + ) as hold_check, patch.object( microvm_tests, "_snapshot_fingerprint", side_effect=fingerprints or itertools.repeat(fingerprint), + ) as snapshot_fingerprint, + patch.object( + microvm_tests, + "openvmm_processes", + side_effect=itertools.chain(processes or (), itertools.repeat(())), ), - patch.object(microvm_tests, "_expect_process_failure") as expect_failure, + patch.object(microvm_tests, "terminate_process") as terminate, ): + observed.update( + exchange=exchange_mock, + wait_until=wait_until, + holds_loopback=holds_loopback, + accepts_tcp=accepts_tcp_mock, + hold_check=hold_check, + snapshot_fingerprint=snapshot_fingerprint, + terminate=terminate, + ) + fixture.mkdir() microvm_tests.run_untiered_network_policy( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "kvm", - memory_mib=128, - timeout=60, - output_dir=output_dir, + "kvm", timeout=60, output_dir=output_dir ) - return { - "workload_boot_command": workload_boot_command, - "captures": captures, - "restores": restores, - "releases": releases, - "exchange": exchange_mock, - "wait_until": wait_until, - "holds_loopback": holds_loopback, - "accepts_tcp": accepts_tcp_mock, - "expect_failure": expect_failure, - "ports": ports, - } + return observed - def test_untiered_network_policy_restores_one_template_with_four_policies(self): + def test_untiered_network_policy_restores_one_named_template_with_four_policies( + self, + ): with tempfile.TemporaryDirectory() as temporary: - output_dir = Path(temporary) / "logs" - calls = self._run_untiered_network_policy(output_dir) - logs = sorted(path.name for path in output_dir.iterdir()) + calls = self._run_untiered_network_policy(Path(temporary)) + logs = sorted(path.name for path in calls["output_dir"].iterdir()) + fixture_removed = not calls["fixture"].exists() + # A public scenario, which reads the Ubuntu layer, and which CI runs in + # its guest and sandbox jobs. self.assertIn("untiered-network-policy", microvm_tests.MICROVM_TEST_SCENARIOS) - # Every guest runs it: the non-sandbox mode captures the template. - self.assertNotIn( + self.assertIn("untiered-network-policy", microvm_tests.PUBLIC_SANDBOX_SCENARIOS) + self.assertIn( "untiered-network-policy", microvm_tests.SANDBOX_CONTROL_SCENARIOS ) - self.assertNotIn( - "untiered-network-policy", microvm_tests.UBUNTU_UNSUPPORTED_SCENARIOS - ) - boot_call = calls["workload_boot_command"].call_args - self.assertEqual(boot_call.args[5], "") - self.assertEqual( - boot_call.kwargs["network"], microvm_tests.UNTIERED_NETWORK_CIDR - ) + self.assertNotIn("untiered-network-policy", microvm_tests.default_suite()) + fixture: Path = calls["fixture"] + store = os.fspath(fixture / "store") - # The template is a snapshot without a tier or sandbox blocks, whose - # NIC has the complete deny-all policy. - ((template, template_kwargs),) = calls["captures"] + # The template is a named snapshot of a VM whose NIC the capture gives + # the complete deny-all policy. + capture, *rest = calls["calls"] self.assertEqual( - template[:-2], - [ - "openvmm", - "boot", - *microvm_tests.PLATFORM_NETWORK_TEMPLATE_POLICY, - ], + capture, + ( + "snapshot", + "capture", + microvm_tests.UNTIERED_NETWORK_SNAPSHOT, + "--store-dir", + store, + "--hypervisor", + "kvm", + "--net", + microvm_tests.UNTIERED_NETWORK_CIDR, + "--timeout", + "60", + ), + ) + template = ( + fixture + / "store" + / "snapshots" + / microvm_tests.UNTIERED_NETWORK_SNAPSHOT.encode().hex() + / "openvmm" ) - self.assertEqual(template[-2], "--snapshot-destination") - self.assertNotIn("--snapshot-tier", template) - self.assertNotIn("--microvm-sandbox-block", template) - ports = calls["ports"] self.assertEqual( - template_kwargs["post_restore_script"], - microvm_tests.untiered_network_policy_script( - allowed_port=ports["allowed"], - other_port=ports["other"], - guest_port=ports["guest"], - ), + {entry.args for entry in calls["snapshot_fingerprint"].call_args_list}, + {(template,)}, ) - self.assertEqual(template_kwargs["processors"], 1) - # Each restore installs its own policy on the template. - restores = calls["restores"] - names = ("deny-all", "egress-dns", "host-loopback", "private-ingress") + def sandbox(state: str) -> list[str]: + return [ + "--store-dir", + store, + "--state-dir", + os.fspath(fixture / state), + "--hypervisor", + "kvm", + "--layer", + "distro,distro.erofs,11111111-1111-1111-1111-111111111111", + "--upper", + "ram", + "--timeout", + "60", + ] + + # Each restore installs its own policy on the template's NIC, without + # naming its address, through `snapshot restore` or a one-off `run`, + # and runs the scenario's checks as its workload. + ports = calls["ports"] + script = microvm_tests.untiered_network_policy_script( + allowed_port=ports["allowed"], + other_port=ports["other"], + guest_port=ports["guest"], + timeout=60, + ) + restores = microvm_tests.untiered_network_restores( + allowed_port=ports["allowed"], + host_port=0, + guest_port=ports["guest"], + ) forward_port = None - for command, name in zip(restores, names, strict=True): - with self.subTest(restore=name): - self.assertEqual( - command[command.index("--restore-snapshot") + 1], template[-1] + for command, restore in zip(calls["restores"], restores, strict=True): + with self.subTest(restore=restore.name): + head = ( + ["run", "--restore-snapshot"] + if restore.one_off + else ["snapshot", "restore"] ) self.assertEqual( - command[command.index("--network-profile") + 1], "portable" + command[:3], + [os.fspath(_FAKE_NVX_RS), "--root", os.fspath(nvx_rs.ROOT)], ) - self.assertNotIn("--microvm-sandbox-block", command) - self.assertNotIn("--restore-absent-scratch", command) - self.assertIn("--restore-network-policy", command) - if "--host-loopback-forward" in command: - forward = command[command.index("--host-loopback-forward") + 1] + policy = command[ + command.index("--network-profile") + 2 : command.index( + "--entrypoint" + ) + ] + if "--host-loopback-forward" in policy: + forward = policy[policy.index("--host-loopback-forward") + 1] forward_port = int(forward.split(":")[1]) self.assertEqual(forward, f"tcp:{forward_port}:{ports['guest']}") + else: + self.assertEqual(tuple(policy), restore.policy) + self.assertEqual( + command[3:], + [ + *head, + microvm_tests.UNTIERED_NETWORK_SNAPSHOT, + *sandbox(restore.name), + "--network-profile", + "portable", + *policy, + "--entrypoint", + "/bin/sh", + "--arg=-c", + f"--arg={script}", + "--arg=untiered-network-policy", + f"--arg={restore.exchanges}", + ], + ) + self.assertNotIn("--net", command) self.assertIsNotNone(forward_port) self.assertNotEqual(forward_port, ports["guest"]) - # The host reaches the guest's server through the forward and the port - # that a policy publishes, on host loopback alone, which no other host - # address reaches, or through another host address for private-network - # ingress, and finds it on no host loopback port when the policy - # publishes nothing. + # The scenario stops and deprovisions the sandboxes of `snapshot + # restore`, which run on once their workloads have exited, and then + # the CLI refuses a restore without the profile of a policy, and one + # that names an address for the NIC, before either provisions anything. + self.assertEqual( + rest, + [ + *( + ( + "sandbox", + operation, + "--state-dir", + os.fspath(fixture / name), + "--timeout", + "60", + ) + for name in ("deny-all", "private-ingress") + for operation in ("stop", "deprovision") + ), + ( + "snapshot", + "restore", + microvm_tests.UNTIERED_NETWORK_SNAPSHOT, + *sandbox("without-profile"), + "--entrypoint", + "/bin/true", + ), + ( + "run", + "--restore-snapshot", + microvm_tests.UNTIERED_NETWORK_SNAPSHOT, + *sandbox("readdressed"), + "--net", + microvm_tests.UNTIERED_NETWORK_CIDR, + "--network-profile", + "portable", + "--entrypoint", + "/bin/true", + ), + ], + ) + # The host reaches the workload's server through the forward and the + # port that a policy publishes, on host loopback alone, which no other + # host address reaches, or through another host address for + # private-network ingress, and finds it on no host loopback port when + # the policy publishes nothing. self.assertEqual( [entry.args for entry in calls["exchange"].call_args_list], [ @@ -14787,42 +14953,53 @@ def test_untiered_network_policy_restores_one_template_with_four_policies(self): [entry.args for entry in calls["holds_loopback"].call_args_list], [(socket.AF_INET, socket.SOCK_STREAM, ports["guest"])] * 2, ) + # Without a client, the host waits for the end of the guest's hold of + # its port, and checks that its check of the port fell within it. + listening = microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER + held = microvm_tests.UNTIERED_NETWORK_INBOUND_PREFIX + b"0" self.assertEqual( - calls["releases"], + calls["awaited"], [ - "NVX-UNTIERED-NETWORK-CHECKED 0", - "NVX-UNTIERED-NETWORK-CHECKED 0", - "NVX-UNTIERED-NETWORK-CHECKED 2", - "NVX-UNTIERED-NETWORK-CHECKED 1", + ("deny-all", listening), + ("deny-all", held), + ("egress-dns", listening), + ("egress-dns", held), + ("host-loopback", listening), + ("private-ingress", listening), ], ) self.assertEqual( - logs, - sorted(f"untiered-network-policy-{name}.log" for name in names), - ) - - # OpenVMM refuses the other restores before the guest runs. - failures = calls["expect_failure"].call_args_list - self.assertEqual( - [entry.args[3] for entry in failures], - [reason for _, reason in microvm_tests.UNTIERED_NETWORK_REJECTED], - ) - for entry in failures: - self.assertEqual( - entry.kwargs["forbidden_markers"], - microvm_tests.UNTIERED_NETWORK_GUEST_MARKERS, - ) - self.assertNotIn("--restore-network-policy", entry.args[0]) - kept_policy, ingress = (entry.args[0] for entry in failures) - self.assertEqual(kept_policy[-2:], ["--network-egress", "allow"]) - self.assertEqual( - ingress[-6:], [ - *microvm_tests.PLATFORM_NETWORK_TEMPLATE_POLICY, - "--network-ingress", - "allow", + (entry.args, sorted(entry.kwargs)) + for entry in calls["hold_check"].call_args_list ], + [((ports["guest"],), ["checked", "held", "listening"])] * 2, + ) + for entry in calls["hold_check"].call_args_list: + times = entry.kwargs + self.assertLessEqual(times["listening"], times["checked"]) + self.assertLessEqual(times["checked"], times["held"]) + names = ("deny-all", "egress-dns", "host-loopback", "private-ingress") + self.assertEqual( + logs, + sorted( + [ + "untiered-network-policy-capture.log", + *(f"untiered-network-policy-{name}.log" for name in names), + *( + f"untiered-network-policy-{name}-{operation}.log" + for name in ("deny-all", "private-ingress") + for operation in ("stop", "deprovision") + ), + *( + f"untiered-network-policy-rejected-{name}.log" + for name, *_ in microvm_tests.UNTIERED_NETWORK_REJECTED + ), + ] + ), ) + calls["terminate"].assert_not_called() + self.assertTrue(fixture_removed) def test_untiered_network_policy_requires_what_each_policy_gives_the_guest(self): def guest_reporting( @@ -14836,51 +15013,19 @@ def guest(command: list[str], ports: dict[str, int]) -> list[bytes]: return guest - def guest_dropping( - dropped: bytes, - ) -> Callable[[list[str], dict[str, int]], list[bytes]]: - def guest(command: list[str], ports: dict[str, int]) -> list[bytes]: - return [ - line - for line in self._simulate_untiered_network_guest(command, ports) - if line != dropped - ] - - return guest - - def guest_reporting_completion( - command: list[str], ports: dict[str, int] - ) -> list[bytes]: - lines = self._simulate_untiered_network_guest(command, ports) - if microvm_tests.UNTIERED_POST_RESTORE_MARKER in lines: - return lines - return [microvm_tests.UNTIERED_POST_RESTORE_MARKER, *lines] - for guest, message in ( - # A guest whose repair did not name the policy's DNS server. + # A guest whose activation did not name the policy's DNS server. ( guest_reporting(b"resolver=192.0.2.1", b"resolver=none"), r"^untiered network policy restore egress-dns: the guest's " r"resolver names none, not 192\.0\.2\.1$", ), - # A guest whose repair started no announcer. + # A guest whose activation started no announcer. ( guest_reporting(b"announcer=running", b"announcer=none"), "^untiered network policy restore host-loopback: the guest " "reported announcer=none$", ), - # A guest whose repair applied settings without reporting it. - ( - guest_dropping(microvm_tests.UNTIERED_POST_RESTORE_MARKER), - "^untiered network policy restore egress-dns: the guest's repair " - "did not report 'NVX-POST-RESTORE-OK: tier=legacy'$", - ), - # A guest without settings whose repair did not finish quietly. - ( - guest_reporting_completion, - "^untiered network policy restore deny-all: the guest's repair " - "reported 'NVX-POST-RESTORE-OK: tier=legacy'$", - ), ): with ( self.subTest(message=message), @@ -14915,24 +15060,100 @@ def guest_reaching_more( def test_untiered_network_policy_reports_guest_failures(self): def failing_guest(_command: list[str], _ports: dict[str, int]) -> list[bytes]: - log_path = output_dir / "untiered-network-policy-deny-all.log" + log_path = root / "logs" / "untiered-network-policy-deny-all.log" log_path.write_bytes( b"NVX-UNTIERED-NETWORK-FAIL the TCP server did not listen\r\n" ) raise TimeoutError("line marker was not observed") with tempfile.TemporaryDirectory() as temporary: - output_dir = Path(temporary) + root = Path(temporary) with self.assertRaises(RuntimeError) as raised: - self._run_untiered_network_policy(output_dir, guest=failing_guest) + self._run_untiered_network_policy(root, guest=failing_guest) + log_path = root / "logs" / "untiered-network-policy-deny-all.log" self.assertEqual( str(raised.exception), "untiered network policy restore deny-all failed: " - "NVX-UNTIERED-NETWORK-FAIL the TCP server did not listen; see " - f"{output_dir / 'untiered-network-policy-deny-all.log'}", + "NVX-UNTIERED-NETWORK-FAIL the TCP server did not listen (line marker " + f"was not observed); see {log_path}", ) self.assertIsInstance(raised.exception.__cause__, TimeoutError) + def test_untiered_network_policy_stops_and_removes_what_a_failure_leaves(self): + def failing_guest(command: list[str], _ports: dict[str, int]) -> list[bytes]: + state = Path(command[command.index("--state-dir") + 1]) + (state / "openvmm.log").write_bytes(b"the VM's log\n") + raise TimeoutError("line marker was not observed") + + # The sandbox that the failed restore provisioned is stopped, the + # OpenVMM process that names the fixture ends, then the sandbox's VM + # log is kept and the sandbox is deprovisioned, and the fixture goes. + record: dict[str, Any] = {} + killed_before_deprovision: list[bool] = [] + + def deprovision( + arguments: tuple[str, ...], + ) -> subprocess.CompletedProcess[bytes] | None: + if arguments[:2] == ("sandbox", "deprovision"): + killed_before_deprovision.append(record["terminate"].called) + return None + + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + with self.assertRaisesRegex( + RuntimeError, + r"^untiered network policy restore deny-all failed \(line marker " + r"was not observed\); see ", + ) as raised: + self._run_untiered_network_policy( + root, + guest=failing_guest, + cli=deprovision, + processes=[(), (4242,)], + record=record, + ) + fixture_exists = (root / "fixture").exists() + vm_log = ( + root / "logs" / "untiered-network-policy-deny-all-openvmm.log" + ).read_bytes() + self.assertNotIn("cleanup", str(raised.exception)) + state = os.fspath(root / "fixture" / "deny-all") + self.assertEqual( + record["calls"][1:], + [ + ("sandbox", operation, "--state-dir", state, "--timeout", "60") + for operation in ("stop", "deprovision") + ], + ) + record["terminate"].assert_called_once_with(4242) + self.assertEqual(killed_before_deprovision, [True]) + self.assertEqual(vm_log, b"the VM's log\n") + self.assertFalse(fixture_exists) + + # A sandbox that its deprovision cannot remove keeps the fixture, + # which the error names. + def refusing_deprovision( + arguments: tuple[str, ...], + ) -> subprocess.CompletedProcess[bytes] | None: + if arguments[:2] != ("sandbox", "deprovision"): + return None + return subprocess.CompletedProcess( + arguments, 1, b"", b"error[sandbox-running]: still running\n" + ) + + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + with self.assertRaisesRegex( + RuntimeError, + r"^untiered network policy restore deny-all failed \(.*\); see .*; " + "cleanup: .*cannot deprovision .*still running.*fixture preserved " + "for recovery", + ): + self._run_untiered_network_policy( + root, guest=failing_guest, cli=refusing_deprovision + ) + self.assertTrue((root / "fixture" / "deny-all").is_dir()) + def test_untiered_network_policy_checks_where_ingress_publishes_the_guest_port( self, ): @@ -14972,11 +15193,81 @@ def loopback_only(address: str, port: int, _timeout: float) -> None: ): with self.assertRaisesRegex( RuntimeError, - f"^untiered network policy restore {restore} failed; see ", + rf"^untiered network policy restore {restore} failed \(.+\); see ", ) as raised: self._run_untiered_network_policy(Path(temporary), **options) self.assertRegex(str(raised.exception.__cause__), cause) + def test_untiered_network_exchanges_check_first_that_nothing_publishes_the_port( + self, + ): + deny_all, _, host_loopback, _ = microvm_tests.untiered_network_restores( + allowed_port=40001, host_port=40003, guest_port=50001 + ) + + def exchanges(restore: microvm_tests.UntieredNetworkRestore) -> int: + return microvm_tests._untiered_network_exchanges( + restore, host_port=40003, guest_port=50001, timeout=60 + ) + + with patch.object(microvm_tests, "_holds_loopback", return_value=False): + self.assertEqual(exchanges(deny_all), 0) + with patch.object(microvm_tests, "_holds_loopback", return_value=True): + with self.assertRaisesRegex( + RuntimeError, "^guest port 50001 was published without ingress$" + ): + exchanges(deny_all) + # A guest that the host reaches serves until the host's exchanges. + with ( + patch.object(microvm_tests, "_host_loopback_tcp_exchange") as exchange, + patch.object(microvm_tests, "_wait_until"), + patch.object(microvm_tests, "_host_ipv4_addresses", return_value=[]), + ): + self.assertEqual(exchanges(host_loopback), 2) + self.assertEqual(exchange.call_count, 2) + + def test_untiered_network_hold_must_cover_the_hosts_check(self): + hold = microvm_tests.UNTIERED_NETWORK_HOLD_SECONDS + third = hold / 3 + # The host checked at once, and saw the hold end most of it later. + microvm_tests._check_untiered_network_hold( + 50001, listening=10.0, checked=10.1, held=10.0 + hold + ) + for times, message in ( + # The host checked a third of the hold after it saw the guest + # listen, which it may have seen late too. + ( + {"listening": 10.0, "checked": 10.1 + third, "held": 10.0 + 2 * hold}, + f"the host checked guest port 50001 {third + 0.1:.1f}s after it " + "saw the guest listen, too late", + ), + # Both markers reached the host at once, as a late listening + # marker does, so the hold may have ended before the check. + ( + {"listening": 10.0, "checked": 10.1, "held": 10.2}, + "the host saw the guest's hold of port 50001 end 0.1s after its " + "check, too soon", + ), + ): + with ( + self.subTest(message=message), + self.assertRaisesRegex(RuntimeError, f"^{re.escape(message)}"), + ): + microvm_tests._check_untiered_network_hold(50001, **times) + + def test_untiered_network_policy_fails_on_markers_that_reach_the_host_late(self): + # The simulated guest's markers reach the host at once, so the host + # cannot tell that its check fell within the guest's hold. + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex( + RuntimeError, + r"^untiered network policy restore deny-all failed \(the host saw " + r"the guest's hold of port \d+ end \d+\.\ds after its check, too soon ", + ): + self._run_untiered_network_policy( + Path(temporary), hold=microvm_tests._check_untiered_network_hold + ) + def test_untiered_network_policy_keeps_the_template_intact(self): original = ("manifest", "state", "memory") changed = ("manifest", "state", "changed") @@ -14998,6 +15289,78 @@ def test_untiered_network_policy_keeps_the_template_intact(self): Path(temporary), fingerprints=fingerprints ) + def test_untiered_network_policy_leaves_no_openvmm_process_or_sandbox(self): + # An OpenVMM process that outlives a restore. + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex( + RuntimeError, r"^OpenVMM processes \[4242\] outlived restore egress-dns" + ): + self._run_untiered_network_policy( + Path(temporary), processes=[(), (), (4242,), ()] + ) + + # A sandbox that a deprovision reports removed, and keeps. + def keeping_deprovision( + arguments: tuple[str, ...], + ) -> subprocess.CompletedProcess[bytes] | None: + if arguments[:2] != ("sandbox", "deprovision"): + return None + return subprocess.CompletedProcess(arguments, 0, b"", b"") + + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex( + RuntimeError, "^restore deny-all left its sandbox .*deny-all" + ): + self._run_untiered_network_policy( + Path(temporary), cli=keeping_deprovision + ) + + def test_untiered_network_policy_requires_the_clis_refusals(self): + # A restore without the profile of a policy that the CLI provisions. + def provisioning( + arguments: tuple[str, ...], + ) -> subprocess.CompletedProcess[bytes] | None: + if arguments[:2] != ("snapshot", "restore"): + return None + return subprocess.CompletedProcess(arguments, 0, b"", b"") + + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex( + RuntimeError, + "^the CLI did not refuse restore without-profile with " + "'needs --network-profile': status 0; see ", + ): + self._run_untiered_network_policy(Path(temporary), cli=provisioning) + + def test_runner_dispatches_public_untiered_network_policy(self): + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "mshv", + "--scenario", + "untiered-network-policy", + "--output-dir", + str(output_dir), + ] + ) + + def require(path: Path, _description: str) -> Path: + return path + + with ( + patch.object(microvm_tests, "validate_openvmm_test_backend"), + patch.object(microvm_tests, "require_file", side_effect=require), + patch.object(microvm_tests, "run_untiered_network_policy") as run, + redirect_stdout(io.StringIO()), + ): + self.assertEqual(microvm_tests.run(args), 0) + run.assert_called_once_with( + "mshv", timeout=args.timeout, output_dir=output_dir + ) + def test_platform_filesystem_script_uses_the_share_that_the_restore_attaches( self, ): diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index d19ad4d7e..5cb67b1ac 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -12498,6 +12498,7 @@ def test_ci_runs_the_public_sandbox_acceptance_on_every_backend(self): self.assertIn("--scenario vm-lifecycle", acceptance) self.assertIn("--scenario snapshot-core", acceptance) self.assertIn("--scenario snapshot-tiers", acceptance) + self.assertIn("--scenario untiered-network-policy", acceptance) self.assertIn( '--output-dir "build/test-results/public-exec-${{ inputs.backend }}"', acceptance,