From a4c576b1e3efdadf49cc78b42c689f1aec573121 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 11:58:23 -0800 Subject: [PATCH 1/3] tests: run the CLI's cold cells of a GPT disk The CLI's twelve cold cells are an EROFS layer and a GPT disk, each over a scratch upper and a RAM upper, on KVM, MSHV and WHP (#624). The public sandbox scenarios boot the EROFS layer over both uppers on each backend, but no test booted a GPT disk through the CLI: only the runtime adapter's VM tests assembled GPT roots, and NVX deletes the adapter once the tests that replace it pass. The public sandbox-gpt-layer scenario of test-microvm now runs the GPT cells of its backend. It writes a GPT disk as UEFI lays one out, with a first partition of a few bytes and an empty layer partition, which a guest of the control initramfs formats as ext4 and fills with a copy of the tree of the Ubuntu EROFS layer. Then, over a scratch upper and over a RAM upper, a sandbox run of the smoke probe, and a managed sandbox's provision, start, exec, stop and deprovision, must boot the disk over the upper alone in a VM that nvx-init manages, with the disk's GUID and the upper's token alone on the kernel command line, pass the probe's checks, show the guest the disk's layer partition as the lower directory of its root, and stop with a successful outcome and a complete teardown, while the disk keeps its digest. CI's guest and sandbox job of each backend runs it beside the other public sandbox scenarios. Evidence: the scenario passes on Linux/KVM (nvx-kvm-test), Linux/MSHV (nvx-mshv-test) and Windows/WHP (local), with OpenVMM and the guest artifacts of CI run 43075697333; the layer's tree takes about 100 MB of the partition's 256 MiB. Tests: a fake of the CLI drives the scenario through both uppers and the regressions that it must catch, and the disk's tables are checked as UEFI specifies them. Part of #1118 (SL5.1b). Co-authored-by: Enrique Saurez Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/run-nvx-microvm-tests.yml | 2 + doc/ci.md | 10 +- doc/contribute.md | 4 +- doc/run.md | 26 ++ doc/usage.md | 7 +- scripts/nvx_tools/microvm_tests.py | 438 +++++++++++++++++++- scripts/test_microvm_tests.py | 407 ++++++++++++++++++ scripts/test_nvx_tools.py | 1 + 8 files changed, 883 insertions(+), 12 deletions(-) diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index 016699138..7882d45c8 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -237,6 +237,7 @@ jobs: --scenario sandbox-lifecycle --scenario sandbox-proxy-only --scenario sandbox-ram-upper + --scenario sandbox-gpt-layer --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run public managed sandbox acceptance on Windows @@ -249,6 +250,7 @@ jobs: --scenario sandbox-lifecycle --scenario sandbox-proxy-only --scenario sandbox-ram-upper + --scenario sandbox-gpt-layer --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run Ubuntu guest smoke tests on Linux diff --git a/doc/ci.md b/doc/ci.md index 2a90a9df3..27be252cb 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -545,8 +545,8 @@ such an execution: the fixed identity, empty capability sets and overlay root over read-only image layers with its upper directory on scratch. The live-share runs below repeat the profile checks. Each backend also runs the public `managed-exec-config`, `sandbox-lifecycle`, -`sandbox-proxy-only`, and `sandbox-ram-upper` scenarios of `test-microvm`, -which drive `nvx-rs sandbox run`, `provision`, `start`, +`sandbox-proxy-only`, `sandbox-ram-upper`, and `sandbox-gpt-layer` scenarios +of `test-microvm`, which drive `nvx-rs sandbox run`, `provision`, `start`, `exec`, `stop`, and `deprovision` as subprocesses (see [Run](run.md#managed-lifecycle)): request-scoped working directories and environments, timeouts, nonzero exits, stdout and stderr, bounded output and @@ -555,8 +555,10 @@ refusal, fail-closed lifecycle transitions, a crashed OpenVMM's stale runtime state, teardown that leaves no OpenVMM process, control endpoint, capability, or runtime record while the layer and scratch artifacts survive, a proxy-only network, whose guest reaches a test proxy on host loopback at -its gateway and no other destination, and sandboxes whose overlay upper is a -tmpfs in guest memory, with no scratch image. +its gateway and no other destination, sandboxes whose overlay upper is a +tmpfs in guest memory, with no scratch image, and sandboxes whose root is a +GPT disk, over either upper, so that the three backends run the CLI's twelve +cold cells (see the [case matrix](acceptance-case-matrix.md#cold-boots)). Their bounded evidence is part of the backend's failure logs. The same entrypoint then verifies a live virtio-fs share inside the container: a read-write `/workspace` share with a denied diff --git a/doc/contribute.md b/doc/contribute.md index e9271599c..6985c3581 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -130,8 +130,8 @@ devices. Add those scenarios to The scenarios that drive the public `nvx-rs sandbox` commands live in `scripts/nvx_tools/managed_exec_tests.py`, `scripts/nvx_tools/sandbox_lifecycle_tests.py`, -`scripts/nvx_tools/sandbox_proxy_tests.py`, and -`scripts/nvx_tools/sandbox_ram_upper_tests.py`, and `microvm_tests.py` +`scripts/nvx_tools/sandbox_proxy_tests.py`, +`scripts/nvx_tools/sandbox_ram_upper_tests.py`, and `microvm_tests.py`, which registers them. `sandbox` and the control protocol belong to `nvx-rs` ([microsoft/nvx#359](https://github.com/microsoft/nvx/issues/359)), so the scenarios drive them as commands, compile egress policies with diff --git a/doc/run.md b/doc/run.md index fa002887e..921e359f5 100644 --- a/doc/run.md +++ b/doc/run.md @@ -1087,6 +1087,32 @@ bounded command observations, the runtime record, the output of the run, of the probe, and of the guest's view, and the outcome reports of the run and of both stops. +The `sandbox-gpt-layer` scenario of `test-microvm --backend BACKEND` is the +acceptance for a GPT disk, over either upper. It writes a GPT disk in its +temporary directory, whose first partition holds a few bytes and whose second, +its one layer, a guest of the Alpine control initramfs formats as ext4 and +fills with a copy of the tree of the Ubuntu EROFS layer. Then, over a scratch +upper and then a RAM upper, a run of `/sbin/nvx-sandbox-smoke` with +`--layer-format gpt` must print an OpenVMM command with the disk as its +read-only distro block device, and a scratch block device for a scratch upper +alone, whose kernel command line names the disk's GUID, +`nvx_layer=distro,0xd0003000,gpt:GUID`, the upper's token alone, +`nvx_sandbox=1`, and `nvx_mode=managed`. The probe must pass its checks of an +execution of nvx-init, and the run's `--outcome-report` must report the +success of the VM's stop and a complete teardown. A managed sandbox must record +the disk's format and GUID and the upper, pass the probe's managed checks, and +show its guest the same tokens, one virtio block device more for a scratch +upper, and the disk's layer partition as the lower directory of its root, +`/run/nvx/layers/p2`, before it stops with a successful outcome report and a +complete teardown and is deprovisioned. The disk keeps its digest throughout. +The scenario's evidence, with a `sandbox-gpt-layer-` prefix, holds the console +of the guest that wrote the disk, the bounded command observations, the runtime +record, the output of each run, probe, and view of the guest, and the outcome +reports of each run and stop. With `sandbox-lifecycle` and +`managed-exec-config`, which boot the EROFS layer over a scratch upper, and +`sandbox-ram-upper`, CI runs it on KVM, MSHV, and WHP: the CLI's twelve cold +cells, which the [case matrix](acceptance-case-matrix.md#cold-boots) lists. + ### Live host shares `sandbox run` and `sandbox provision` accept repeatable diff --git a/doc/usage.md b/doc/usage.md index 7c6bc704b..0a4f26b60 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1251,8 +1251,8 @@ against the public OpenVMM CLI. Repeat `--scenario` to select a subset; without it, every scenario supported by the selected guest runs, except `smp-lapic`, which runs only when named: it repeats `smp` and also asserts that every CPU uses the one-shot counting LAPIC. `managed-exec-config`, `sandbox-lifecycle`, -`sandbox-proxy-only`, and `sandbox-ram-upper`, the public sandbox scenarios, -drive the public `nvx-rs sandbox` lifecycle commands (see +`sandbox-proxy-only`, `sandbox-ram-upper`, and `sandbox-gpt-layer`, the public +sandbox scenarios, drive the public `nvx-rs sandbox` lifecycle commands (see [Run](run.md#managed-lifecycle)) and also read the Ubuntu distro layer, its manifest, and the sandbox scratch template, which [`download`](#download) installs and the guest-artifact build makes; a run @@ -1264,7 +1264,8 @@ and `virtio-features` scenarios and the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`, `platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`, `platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, -`sandbox-proxy-only`, and `sandbox-ram-upper` scenarios. Ubuntu also rejects +`sandbox-proxy-only`, `sandbox-ram-upper`, and `sandbox-gpt-layer` +scenarios. Ubuntu also rejects the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without `--scenario`, it runs diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index be89fbd7c..05bdc2f31 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -21,6 +21,7 @@ import threading import time import uuid +import zlib from collections.abc import Callable, Collection, Sequence from contextlib import ExitStack, suppress from dataclasses import dataclass @@ -65,15 +66,38 @@ sha256_file, ) from .guests import GUEST_NAMES, GuestDescriptor, guest_descriptor -from .managed_exec_tests import run_managed_exec_configuration, workload_artifact_paths +from .managed_exec_tests import ( + WorkloadArtifacts, + load_workload_artifacts, + run_managed_exec_configuration, + workload_artifact_paths, +) from .microvm_durations import REPORT_NAME, DurationReport from .microvm_shards import configure_parser as configure_shards_parser from .microvm_shards import load_partition, shard_scenarios from .nvx_rs import EgressRules, compile_egress_policy from .openvmm_process import OpenvmmProcess, TcpConsole -from .sandbox_lifecycle_tests import run_sandbox_lifecycle +from .sandbox_lifecycle_tests import ( + CONFIG, + OUTCOME, + PROBE, + PROVISIONED_ENTRIES, + STOPPED_ENTRIES, + LifecycleAcceptance, + check_managed_report, + managed_report, + run_acceptance, + run_sandbox_lifecycle, +) from .sandbox_proxy_tests import run_sandbox_proxy_only -from .sandbox_ram_upper_tests import run_sandbox_ram_upper +from .sandbox_ram_upper_tests import ( + MANAGED_TOKENS, + RAM_UPPER_TOKEN, + SCRATCH_TOKEN, + check_run_report, + probe_markers, + run_sandbox_ram_upper, +) from .shard_plans import parse_shard from .time_abi import ( ABI_VERSION, @@ -131,6 +155,7 @@ "restore-memory", "restore-processors", "sandbox-blocks", + "sandbox-gpt-layer", "sandbox-lifecycle", "sandbox-proxy-only", "sandbox-ram-upper", @@ -159,6 +184,7 @@ "platform-restore-processors", "platform-zero-fill", "sandbox-blocks", + "sandbox-gpt-layer", "sandbox-lifecycle", "sandbox-proxy-only", "sandbox-ram-upper", @@ -174,6 +200,7 @@ PUBLIC_SANDBOX_SCENARIOS = frozenset( ( "managed-exec-config", + "sandbox-gpt-layer", "sandbox-lifecycle", "sandbox-proxy-only", "sandbox-ram-upper", @@ -1501,6 +1528,398 @@ def run_managed_lifecycle( ) +# The sandbox-gpt-layer scenario's GPT disk, which it lays out as UEFI does and +# as nvx-init reads a layered root's disk: its logical block size, the type of +# its partitions, Linux's filesystem data, and the bytes of its first +# partition, which configure an adapter and which nvx-init reads without +# interpreting them. Its second partition is the root's one layer: an ext4 +# copy of the tree of the Ubuntu EROFS layer, which a guest writes. +GPT_SECTOR = 512 +GPT_LINUX_DATA = uuid.UUID("0fc63daf-8483-4772-8e79-3d69d8477de4") +GPT_FIRST_PARTITION = b"nvx sandbox-gpt-layer\n" +# Partitions start on 1 MiB boundaries, as partitioning tools align them. +GPT_ALIGNMENT = 2048 +# The partition entry array: its number of entries and their size, in bytes. +GPT_ENTRIES = 128 +GPT_ENTRY_SIZE = 128 +# The layer partition leaves the tree room: at least GPT_LAYER_MIN_BYTES, and +# GPT_LAYER_EXPANSION times the compressed EROFS layer, whose tree takes about +# twice the layer's size in ext4. +GPT_LAYER_MIN_BYTES = 256 << 20 +GPT_LAYER_EXPANSION = 4 +GPT_LAYER_MEMORY_MIB = 512 +GPT_LAYER_READY_MARKER = b"NVX-GPT-LAYER-READY" +# The guest that writes the layer has the EROFS layer as its distro block +# device, vda, and the GPT disk as its scratch, vdb, whose layer partition +# Linux reads from the disk's table as vdb2. +GPT_LAYER_SCRIPT = """set -eu +fail() { + echo "NVX-GPT-LAYER-FAIL code=$1" + nvx-exit "$1" + exit "$1" +} +for name in vda vdb2; do + tries=0 + while [ ! -b "/dev/$name" ] && [ "$tries" -lt 200 ]; do + sleep 0.05 + tries=$((tries + 1)) + done + [ -b "/dev/$name" ] || fail 20 +done +mkdir -p /mnt/nvx-image /mnt/nvx-layer +mount -t erofs -o ro /dev/vda /mnt/nvx-image || fail 21 +mkfs.ext4 -q -F -O ^has_journal /dev/vdb2 || fail 22 +mount -t ext4 /dev/vdb2 /mnt/nvx-layer || fail 23 +cp -a /mnt/nvx-image/. /mnt/nvx-layer/ || fail 24 +df -k /mnt/nvx-layer +umount /mnt/nvx-layer || fail 25 +umount /mnt/nvx-image || fail 26 +echo NVX-GPT-LAYER-READY +nvx-exit 0 +""" +# The kernel command-line token of a scratch upper, on the scratch block +# device's whole ext4 filesystem. +SCRATCH_UPPER_TOKEN = "nvx_scratch=0xd0006000,ext4" +# The guest's block devices, one per line, the lower directory of its root's +# overlay, and its kernel command line. +GPT_VIEW_SCRIPT = ( + "ls /sys/block && grep ' / / ' /proc/self/mountinfo | " + "grep -o 'lowerdir=[^,]*' && cat /proc/cmdline" +) +# The overlay's lower directory of the disk's layer, its second partition. +GPT_LOWER_DIRECTORY = "lowerdir=/run/nvx/layers/p2" + + +def write_gpt_disk(path: Path, layer_bytes: int) -> str: + """Write a new GPT disk at ``path`` and return its GUID. + + The disk has a protective MBR, the primary header and its array of + GPT_ENTRIES entries, the first partition, 4 KiB that hold + GPT_FIRST_PARTITION, an empty layer partition of ``layer_bytes``, whole + sectors, and, after a gap of GPT_ALIGNMENT sectors, the backup array and + header that end it. Each partition takes a GUID of its own, and starts on + a GPT_ALIGNMENT boundary. + """ + if layer_bytes <= 0 or layer_bytes % GPT_SECTOR: + raise ScriptError("a GPT disk's layer partition takes whole sectors") + first = (GPT_ALIGNMENT, GPT_ALIGNMENT + 4096 // GPT_SECTOR - 1) + layer = (2 * GPT_ALIGNMENT, 2 * GPT_ALIGNMENT + layer_bytes // GPT_SECTOR - 1) + sectors = (layer[1] // GPT_ALIGNMENT + 2) * GPT_ALIGNMENT + array_sectors = GPT_ENTRIES * GPT_ENTRY_SIZE // GPT_SECTOR + guid = uuid.uuid4() + entries = bytearray(GPT_ENTRIES * GPT_ENTRY_SIZE) + for index, (start, end) in enumerate((first, layer)): + struct.pack_into( + "<16s16sQQQ", + entries, + index * GPT_ENTRY_SIZE, + GPT_LINUX_DATA.bytes_le, + uuid.uuid4().bytes_le, + start, + end, + 0, + ) + + def header(own: int, alternate: int, array: int) -> bytes: + block = bytearray(GPT_SECTOR) + struct.pack_into( + "<8sIIIIQQQQ16sQIII", + block, + 0, + b"EFI PART", + 0x0001_0000, + 92, + 0, + 0, + own, + alternate, + 2 + array_sectors, + sectors - 2 - array_sectors, + guid.bytes_le, + array, + GPT_ENTRIES, + GPT_ENTRY_SIZE, + zlib.crc32(entries), + ) + struct.pack_into(" None: + super().__init__( + backend, + timeout=timeout, + output_dir=output_dir, + artifacts=artifacts, + root=root, + ) + self.executable = executable + self.kernel = kernel + self.initrd = initrd + self.disk = root / "distro.gpt" + self.guid = "" + + def run(self) -> None: + self.write_disk() + digest = sha256_file(self.disk) + for upper in ("scratch", "ram"): + self.check_run(upper) + self.check_managed(upper) + if sha256_file(self.disk) != digest: + raise RuntimeError("the GPT sandboxes modified their disk") + self.require_processes((), "after the GPT sandboxes") + + def write_disk(self) -> None: + """Writes the GPT disk, then has a guest of the control initramfs + format its layer partition as ext4 and copy the EROFS layer's tree + into it.""" + layer_bytes = max( + GPT_LAYER_MIN_BYTES, + GPT_LAYER_EXPANSION * self.artifacts.distro.stat().st_size, + ) + layer_bytes += -layer_bytes % (1 << 20) + self.guid = write_gpt_disk(self.disk, layer_bytes) + command = workload_boot_command( + self.executable, + self.backend, + self.kernel, + self.initrd, + GPT_LAYER_MEMORY_MIB, + "quiet loglevel=0", + ) + command.extend( + ( + "--microvm-sandbox-block", + _block_arg("distro", self.artifacts.distro, read_only=True), + "--microvm-sandbox-block", + _block_arg("scratch", self.disk, read_only=False), + ) + ) + run_guest_script( + command, + GPT_LAYER_SCRIPT, + GPT_LAYER_READY_MARKER, + timeout=self.timeout, + log_path=self.output_dir / f"{self.evidence_prefix}disk.log", + ) + + def options(self, upper: str, scratch_name: str) -> list[str]: + """Returns the options of `run` and `provision` of the disk over + `upper`, with a new copy of the scratch template at `scratch_name` in + the fixture for a scratch upper.""" + options = [ + "--hypervisor", + self.backend, + "--layer", + f"distro,{self.disk},{self.guid}", + "--layer-format", + "gpt", + "--memory-mib", + "256", + ] + if upper == "ram": + return [*options, "--upper", "ram"] + scratch = self.root / scratch_name + shutil.copyfile(self.artifacts.scratch_template, scratch) + return [*options, "--scratch", str(scratch)] + + def check_boot(self, command_line: str, upper: str, context: str) -> None: + """Checks that `command_line`, or a quoted command that holds it, + boots the disk's layer over `upper` alone, in a VM that nvx-init + manages.""" + tokens = [token.strip("'\"") for token in command_line.split()] + layers = [token for token in tokens if token.startswith("nvx_layer=")] + selected, other = ( + (SCRATCH_UPPER_TOKEN, RAM_UPPER_TOKEN) + if upper == "scratch" + else (RAM_UPPER_TOKEN, SCRATCH_TOKEN) + ) + if ( + layers != [f"nvx_layer=distro,0xd0003000,gpt:{self.guid}"] + or tokens.count(selected) != 1 + or any(token.startswith(other) for token in tokens) + or not all(token in tokens for token in MANAGED_TOKENS) + ): + raise RuntimeError( + f"{context} does not boot the GPT disk over the {upper} upper " + f"in a VM that nvx-init manages: {command_line!r}" + ) + + def check_probe( + self, result: subprocess.CompletedProcess[bytes], what: str + ) -> None: + """Checks that `result`, of the smoke probe of a managed execution, + reports that its checks passed.""" + for marker in probe_markers(entry=True): + if marker not in result.stdout: + raise RuntimeError(f"{what} did not report {marker.decode()}") + + def check_run(self, upper: str) -> None: + """A run, which keeps no state, boots the disk over `upper`, passes + the probe's checks of an execution of nvx-init once, and forwards the + report of its VM's graceful stop.""" + report = self.root / f"run-{upper}-outcome.json" + result = self.invoke( + "run", + *self.options(upper, "run-scratch.ext4"), + "--outcome-report", + str(report), + "--entrypoint", + PROBE, + "--arg=init", + "--arg=entry", + state=None, + ) + self.save_evidence(f"run-{upper}.log", result.stdout + result.stderr) + lines = result.stdout.splitlines() + command = lines[0].decode(errors="replace") if lines else "" + # The disk's path names the fixture, so it names the run's OpenVMM to + # the process checks and to cleanup. The run may name the disk as it + # resolved its path. + attached = { + f"distro:file:{disk},ro" for disk in (self.disk, self.disk.resolve()) + } + if ( + not command.startswith(">> ") + or not any(disk in command for disk in attached) + or command.count("--microvm-sandbox-block") + != (2 if upper == "scratch" else 1) + or ("scratch:file:" in command) != (upper == "scratch") + ): + raise RuntimeError( + f"the GPT/{upper} run did not attach its disk and upper alone: " + f"{command!r}" + ) + self.check_boot(command, upper, f"the GPT/{upper} run's OpenVMM command") + self.check_probe(result, f"the GPT/{upper} run's probe") + self.copy_evidence(report, f"run-{upper}-outcome.json") + check_run_report(report, self.backend) + if self.state.exists(): + raise RuntimeError("the run created sandbox state") + self.require_processes((), f"after the GPT/{upper} run") + + def check_managed(self, upper: str) -> None: + """A managed sandbox records the disk and `upper`, boots them, passes + the probe's managed checks with the disk's layer as its root's lower + directory, and stops and is deprovisioned as any other does.""" + self.invoke( + "provision", *self.options(upper, "managed-scratch.ext4"), state=self.state + ) + self.require_entries( + self.state, PROVISIONED_ENTRIES, f"after the GPT/{upper} provision" + ) + config = json.loads((self.state / CONFIG).read_text(encoding="utf-8")) + image, recorded = config.get("image"), config.get("upper") + if ( + not isinstance(image, dict) + or cast(dict[str, object], image).get("format") != "gpt" + or cast(dict[str, object], image).get("identity") != self.guid + or not isinstance(recorded, dict) + or cast(dict[str, object], recorded).get("kind") != upper + ): + raise RuntimeError( + f"provision recorded the GPT/{upper} root as {image!r} over {recorded!r}" + ) + self.start(description=f"GPT/{upper} start") + result = self.exec(PROBE, "--arg=init", "--arg=entry") + self.save_evidence(f"{upper}-probe.log", result.stdout + result.stderr) + self.check_probe(result, f"the managed GPT/{upper} probe") + result = self.exec( + "/bin/sh", "--arg=-c", f"--arg={GPT_VIEW_SCRIPT}", expected=None + ) + self.save_evidence(f"{upper}-view.log", result.stdout + result.stderr) + lines = result.stdout.decode(errors="replace").splitlines() + if result.returncode != 0 or len(lines) < 2: + raise RuntimeError( + f"the guest view of GPT/{upper} returned {result.returncode}" + ) + devices = [name for name in lines[:-2] if name.startswith("vd")] + expected = ["vda", "vdb"] if upper == "scratch" else ["vda"] + if devices != expected or lines[-2] != GPT_LOWER_DIRECTORY: + raise RuntimeError( + f"the guest of GPT/{upper} has the virtio block devices {devices} " + f"and the root's {lines[-2]!r}" + ) + self.check_boot( + lines[-1], upper, f"the GPT/{upper} guest's kernel command line" + ) + self.invoke("stop", state=self.state) + self.require_entries(self.state, STOPPED_ENTRIES, f"after the GPT/{upper} stop") + self.copy_evidence(self.state / OUTCOME, f"{upper}-outcome.json") + check_managed_report(managed_report(self.state), self.backend) + self.require_processes((), f"after the GPT/{upper} stop") + self.invoke("deprovision", state=self.state) + if self.state.exists(): + raise RuntimeError(f"deprovision kept the GPT/{upper} sandbox's state") + + +def run_sandbox_gpt_layer( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + timeout: float, + output_dir: Path, +) -> None: + """Run the CLI's cold cells of a GPT disk, over a scratch upper and a RAM + upper, whose disk a guest of `executable`, `kernel`, and `initrd` writes + from the Ubuntu EROFS layer.""" + artifacts = load_workload_artifacts() + output_dir.mkdir(parents=True, exist_ok=True) + root = Path(tempfile.mkdtemp(prefix="nvx-public-gpt-layer-")) + run_acceptance( + GptLayerAcceptance( + backend, + timeout=timeout, + output_dir=output_dir, + artifacts=artifacts, + root=root, + executable=executable, + kernel=kernel, + initrd=initrd, + ) + ) + + def run_structured_outcome( executable: Path, kernel: Path, @@ -11042,6 +11461,19 @@ def _run_scenarios( f"Running public RAM-upper sandboxes on OpenVMM/{args.backend}", ) run_sandbox_ram_upper(args.backend, timeout=args.timeout, output_dir=output_dir) + if "sandbox-gpt-layer" in scenarios: + report.start( + "sandbox-gpt-layer", + f"Running public GPT-layer sandboxes on OpenVMM/{args.backend}", + ) + run_sandbox_gpt_layer( + executable, + kernel, + initrd, + args.backend, + timeout=args.timeout, + output_dir=output_dir, + ) if "managed-lifecycle" in scenarios: report.start( "managed-lifecycle", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index dd9befaf8..96e2785e7 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -17,12 +17,15 @@ import shlex import shutil import socket +import struct import subprocess import sys import tempfile import threading import time import unittest +import uuid +import zlib from collections.abc import Callable, Collection, Mapping, Sequence from contextlib import ExitStack, nullcontext, redirect_stderr, redirect_stdout from datetime import datetime @@ -2965,6 +2968,374 @@ def test_acceptance_detects_ram_upper_regressions(self): ) +class _FakeGptLayerSandboxCli(_FakeSandboxCli): + """Emulates `nvx-rs sandbox --layer-format gpt` and its guest, over a + scratch upper and a RAM upper: the disk's layer token and the upper's on + the kernel command line, the disk's layer partition as the lower + directory of the root's overlay, and a scratch block device for a scratch + upper alone. + + Each fault names a regression of the planner, the guest, or the record + that the acceptance must detect. + """ + + @staticmethod + def _upper(values: dict[str, str]) -> str: + return "ram" if values.get("--upper") == "ram" else "scratch" + + def _command_line(self, guid: str, upper: str) -> str: + image = "erofs" if "boots-erofs" in self.faults else "gpt" + tokens = [f"nvx_layer=distro,0xd0003000,{image}:{guid}"] + if "declares-both-uppers" in self.faults: + tokens += [microvm_tests.SCRATCH_UPPER_TOKEN, microvm_tests.RAM_UPPER_TOKEN] + elif upper == "ram": + tokens += [microvm_tests.RAM_UPPER_TOKEN, "nvx_ram_upper_percent=50"] + else: + tokens += [microvm_tests.SCRATCH_UPPER_TOKEN, "nvx_scratch_journal=allowed"] + return " ".join( + (*tokens, "nvx_sandbox=1", "nvx_mode=managed", "nvx_identity=65534:65534") + ) + + def _run( + self, state: Path | None, values: dict[str, str], arguments: list[str] + ) -> _CliResult: + assert state is None + assert values["--layer-format"] == "gpt" + assert values["--entrypoint"] == sandbox_lifecycle_tests.PROBE + assert arguments == ["init", "entry"] + _role, disk, guid = values["--layer"].split(",") + upper = self._upper(values) + blocks = f"--microvm-sandbox-block distro:file:{disk}" + if "attaches-disk-writable" not in self.faults: + blocks += ",ro" + if upper == "scratch" or "attaches-scratch" in self.faults: + scratch = values.get("--scratch", "scratch.ext4") + blocks += f" --microvm-sandbox-block scratch:file:{scratch}" + if "writes-disk" in self.faults: + with Path(disk).open("r+b") as image: + image.seek(microvm_tests.GPT_ALIGNMENT * microvm_tests.GPT_SECTOR) + image.write(b"written") + outcome = ( + {"operation": "run", "category": "guest-exit", "status_code": 1} + if "run-reports-failure" in self.faults + else sandbox_ram_upper_tests.RUN_SUCCESS + ) + Path(values["--outcome-report"]).write_text( + json.dumps( + { + "backend": values["--hypervisor"], + "outcome": outcome, + "teardown": dict.fromkeys( + microvm_tests.OUTCOME_TEARDOWN_FIELDS, True + ), + } + ), + encoding="utf-8", + ) + command = ( + f">> openvmm {blocks} --cmdline '{self._command_line(guid, upper)}' " + "--microvm-report /tmp/nvx-sandbox-run-0/outcome.json\n" + ) + output = ( + b"" if "probe-fails" in self.faults else _FakeRamUpperSandboxCli.SMOKE_RUN + ) + return 0, command.encode() + output, b"" + + def _provision( + self, state: Path | None, values: dict[str, str], _: list[str] + ) -> _CliResult: + result = super()._provision(state, values, _) + if result[0] == 0: + assert state is not None + path = state / "config.json" + config = json.loads(path.read_text(encoding="utf-8")) + _role, disk, guid = values["--layer"].split(",") + upper = self._upper(values) + if "records-other-upper" in self.faults: + upper = "ram" if upper == "scratch" else "scratch" + image = "erofs" if "records-erofs" in self.faults else "gpt" + config.update( + format=9, + image={"format": image, "path": disk, "identity": guid}, + upper=( + {"kind": "ram", "max_memory_percent": 50} + if upper == "ram" + else {"kind": "scratch", "path": values.get("--scratch", "")} + ), + ) + path.write_text(json.dumps(config), encoding="utf-8") + return result + + def _exec( + self, state: Path | None, values: dict[str, str], arguments: list[str] + ) -> _CliResult: + assert state is not None + if values["--entrypoint"] != "/bin/sh": + return super()._exec(state, values, arguments) + _, refusal = self._running(state) + if refusal is not None: + return refusal + assert arguments == ["-c", microvm_tests.GPT_VIEW_SCRIPT] + config = json.loads((state / "config.json").read_text(encoding="utf-8")) + upper = config["upper"]["kind"] + devices = ( + "vda\nvdb\n" + if upper == "scratch" or "has-scratch-device" in self.faults + else "vda\n" + ) + layer = "distro" if "lower-is-distro" in self.faults else "p2" + view = ( + f"loop0\n{devices}lowerdir=/run/nvx/layers/{layer}\n" + f"{self._command_line(config['image']['identity'], upper)}\n" + ) + return 0, view.encode(), b"" + + +class PublicSandboxGptLayerTests(unittest.TestCase): + # A run of each upper, then a managed sandbox: the probe and the guest's + # view. + EXPECTED_OPERATIONS = [ + "run", + "provision", + "start", + "exec", + "exec", + "stop", + "deprovision", + ] * 2 + + def _run(self, root: Path, fake: _FakeGptLayerSandboxCli) -> tuple[Path, MagicMock]: + distro = root / "ubuntu-distro.erofs" + distro.write_bytes(b"distro") + template = root / "ubuntu-smoke-scratch.ext4" + template.write_bytes(bytes(4096)) + artifacts = managed_exec_tests.WorkloadArtifacts( + distro, "12345678-1234-1234-1234-123456789abc", template + ) + output_dir = root / "results" + mkdtemp = tempfile.mkdtemp + + def fixture(prefix: str) -> str: + return mkdtemp(prefix=prefix, dir=root) + + with ( + patch.object( + microvm_tests, "load_workload_artifacts", return_value=artifacts + ), + patch.object(microvm_tests, "GPT_LAYER_MIN_BYTES", 1 << 20), + patch.object(microvm_tests, "run_guest_script") as guest, + patch.object(microvm_tests.tempfile, "mkdtemp", side_effect=fixture), + patch.object(sandbox_lifecycle_tests.subprocess, "run", side_effect=fake), + patch.object( + sandbox_lifecycle_tests.nvx_rs, + "require_binary", + return_value=Path("nvx-rs"), + ), + patch.object( + sandbox_lifecycle_tests, + "openvmm_processes", + side_effect=fake.processes, + ), + patch.object( + sandbox_lifecycle_tests, + "control_endpoint_present", + side_effect=fake.endpoint_present, + ), + patch.object( + sandbox_lifecycle_tests, + "terminate_process", + side_effect=fake.terminate, + ), + patch.object( + sandbox_lifecycle_tests, "openvmm_threads_remain", return_value=False + ), + ): + microvm_tests.run_sandbox_gpt_layer( + Path("openvmm"), + Path("vmlinux"), + Path("initramfs.cpio.gz"), + "kvm", + timeout=2, + output_dir=output_dir, + ) + return output_dir, guest + + def test_acceptance_runs_the_cold_cells_of_a_gpt_disk(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fake = _FakeGptLayerSandboxCli() + output_dir, guest = self._run(root, fake) + checks = json.loads( + (output_dir / "sandbox-gpt-layer-checks.json").read_text( + encoding="utf-8" + ) + ) + evidence = {path.name for path in output_dir.iterdir()} + fixtures = list(root.glob("nvx-public-gpt-layer-*")) + + self.assertEqual(fake.operations, self.EXPECTED_OPERATIONS) + self.assertEqual(fixtures, []) + self.assertFalse(any(fake.running.values())) + self.assertEqual( + evidence, + { + f"sandbox-gpt-layer-{name}" + for upper in ("scratch", "ram") + for name in ( + f"run-{upper}.log", + f"run-{upper}-outcome.json", + f"{upper}-probe.log", + f"{upper}-view.log", + f"{upper}-outcome.json", + ) + } + | {"sandbox-gpt-layer-checks.json", "sandbox-gpt-layer-runtime.json"}, + ) + # A guest of the control initramfs writes the layer, with the EROFS + # layer as its distro block device and the disk as its scratch. + guest.assert_called_once() + command = guest.call_args.args[0] + disk = next( + argument.removeprefix("scratch:file:") + for argument in command + if argument.startswith("scratch:file:") + ) + self.assertEqual(Path(disk).name, "distro.gpt") + self.assertIn(f"distro:file:{root / 'ubuntu-distro.erofs'},ro", command) + self.assertEqual( + guest.call_args.args[1:], + ( + microvm_tests.GPT_LAYER_SCRIPT, + microvm_tests.GPT_LAYER_READY_MARKER, + ), + ) + # Each upper's run and provision name the disk's GUID in its format, + # and a scratch image for a scratch upper alone. + launches = [ + record["argv"] + for record in checks + if record["operation"] in {"run", "provision"} + ] + guids: set[str] = set() + for argv, upper in zip( + launches, ("scratch", "scratch", "ram", "ram"), strict=True + ): + role, path, guid = argv[argv.index("--layer") + 1].split(",") + guids.add(guid) + self.assertEqual((role, path), ("distro", disk)) + self.assertEqual(argv[argv.index("--layer-format") + 1], "gpt") + self.assertEqual("--scratch" in argv, upper == "scratch") + self.assertEqual("--upper" in argv, upper == "ram") + self.assertEqual(len(guids), 1) + + def test_acceptance_detects_gpt_layer_regressions(self): + regressions = { + "boots-erofs": "the GPT/scratch run's OpenVMM command does not boot", + "attaches-disk-writable": "the GPT/scratch run did not attach its disk", + "declares-both-uppers": "does not boot the GPT disk over the scratch upper", + "probe-fails": "the GPT/scratch run's probe did not report", + "run-reports-failure": "the run reported", + "records-erofs": "provision recorded the GPT/scratch root as", + "records-other-upper": "provision recorded the GPT/scratch root as", + "lower-is-distro": "and the root's 'lowerdir=/run/nvx/layers/distro'", + "stop-reports-incomplete-teardown": "the managed report has incomplete", + "attaches-scratch": "the GPT/ram run did not attach its disk and upper", + "has-scratch-device": "of GPT/ram has the virtio block devices ['vda', 'vdb']", + "writes-disk": "the GPT sandboxes modified their disk", + } + for fault, message in regressions.items(): + with ( + self.subTest(fault=fault), + tempfile.TemporaryDirectory() as temporary, + ): + root = Path(temporary) + fake = _FakeGptLayerSandboxCli(fault) + with self.assertRaisesRegex(RuntimeError, re.escape(message)): + self._run(root, fake) + # Cleanup ends every OpenVMM process and removes the fixture. + self.assertFalse(any(fake.running.values())) + self.assertEqual(list(root.glob("nvx-public-gpt-layer-*")), []) + self.assertTrue( + (root / "results" / "sandbox-gpt-layer-checks.json").is_file() + ) + + def test_the_disk_is_laid_out_as_uefi_lays_out_a_gpt_disk(self): + sector = microvm_tests.GPT_SECTOR + layer_bytes = 3 << 20 + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "distro.gpt" + guid = uuid.UUID(microvm_tests.write_gpt_disk(path, layer_bytes)) + disk = path.read_bytes() + sectors = len(disk) // sector + self.assertEqual(len(disk) % sector, 0) + + def block(lba: int, count: int = 1) -> bytes: + return disk[lba * sector : (lba + count) * sector] + + # The protective MBR: one record of type 0xEE from LBA 1. + mbr = block(0) + self.assertEqual(mbr[510:], b"\x55\xaa") + self.assertEqual(mbr[446 + 4], 0xEE) + self.assertEqual(struct.unpack_from(" tuple[Any, ...]: + data = block(lba) + fields = struct.unpack_from("<8sIIIIQQQQ16sQIII", data) + checked = bytearray(data[:92]) + checked[16:20] = bytes(4) + self.assertEqual(zlib.crc32(checked), fields[3]) + self.assertEqual(data[92:], bytes(sector - 92)) + return fields + + primary, backup = header(1), header(sectors - 1) + self.assertEqual(primary[:3], (b"EFI PART", 0x0001_0000, 92)) + self.assertEqual(primary[4:9], (0, 1, sectors - 1, 34, sectors - 34)) + self.assertEqual( + (uuid.UUID(bytes_le=primary[9]), primary[10:13]), (guid, (2, 128, 128)) + ) + self.assertEqual(backup[4:7], (0, sectors - 1, 1)) + self.assertEqual((backup[7:10], backup[10]), (primary[7:10], sectors - 33)) + entries = block(2, 32) + self.assertEqual(block(sectors - 33, 32), entries) + self.assertEqual(zlib.crc32(entries), primary[13]) + self.assertEqual(primary[13], backup[13]) + + # Two partitions of Linux's filesystem data, each of its own GUID, on + # 1 MiB boundaries in the usable blocks, and nothing after them. + partitions = [ + struct.unpack_from("<16s16sQQQ", entries, index * 128) for index in range(2) + ] + self.assertEqual(entries[256:], bytes(126 * 128)) + linux_data = uuid.UUID("0fc63daf-8483-4772-8e79-3d69d8477de4") + self.assertEqual( + [uuid.UUID(bytes_le=kind) for kind, *_ in partitions], [linux_data] * 2 + ) + unique = {uuid.UUID(bytes_le=partition[1]) for partition in partitions} + self.assertEqual(len(unique | {guid}), 3) + self.assertEqual( + [partition[2:] for partition in partitions], + [(2048, 2055, 0), (4096, 4096 + layer_bytes // sector - 1, 0)], + ) + self.assertLessEqual(partitions[1][3], sectors - 34) + # The first partition holds its bytes, and the layer nothing yet. + self.assertEqual( + block(2048, 8).rstrip(b"\0"), microvm_tests.GPT_FIRST_PARTITION + ) + self.assertEqual(block(4096, layer_bytes // sector), bytes(layer_bytes)) + + def test_a_layer_of_part_of_a_sector_is_refused_before_the_disk_is_written(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "distro.gpt" + for layer_bytes in (0, 1000): + with ( + self.subTest(layer_bytes=layer_bytes), + self.assertRaisesRegex(common.ScriptError, "whole sectors"), + ): + microvm_tests.write_gpt_disk(path, layer_bytes) + self.assertFalse(path.exists()) + + class GuestIdentityScriptTests(unittest.TestCase): def test_guest_identity_checks_fail_before_success_markers(self): descriptor = microvm_tests.guest_descriptor("ubuntu") @@ -15244,6 +15615,42 @@ def require(path: Path, _description: str) -> Path: ) lifecycle.assert_not_called() + def test_runner_dispatches_public_sandbox_gpt_layer(self): + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "kvm", + "--scenario", + "sandbox-gpt-layer", + "--output-dir", + str(output_dir), + ] + ) + + def require(path: Path, _description: str) -> Path: + return path + + with ( + patch.object(microvm_tests, "validate_openvmm_test_backend"), + patch.object(microvm_tests, "require_file", side_effect=require), + patch.object(microvm_tests, "run_sandbox_gpt_layer") as run, + patch.object(microvm_tests, "run_sandbox_ram_upper") as ram_upper, + ): + self.assertEqual(microvm_tests.run(args), 0) + # The guest that writes the GPT disk boots the run's artifacts. + run.assert_called_once_with( + microvm_tests.openvmm_binary_path(), + microvm_tests.artifact_path("vmlinux"), + microvm_tests.artifact_path("initramfs.cpio.gz"), + "kvm", + timeout=args.timeout, + output_dir=output_dir, + ) + ram_upper.assert_not_called() + def test_a_default_run_runs_the_public_sandbox_scenarios(self): def require(path: Path, _description: str) -> Path: return path diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 306ad8926..e292af29d 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -12812,6 +12812,7 @@ def test_ci_runs_the_public_sandbox_acceptance_on_every_backend(self): self.assertIn("--scenario sandbox-lifecycle", acceptance) self.assertIn("--scenario sandbox-proxy-only", acceptance) self.assertIn("--scenario sandbox-ram-upper", acceptance) + self.assertIn("--scenario sandbox-gpt-layer", acceptance) self.assertIn( '--output-dir "build/test-results/public-exec-${{ inputs.backend }}"', acceptance, From b54950dd1e2e8d9742ca59527ab577403b3e5066 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 11:58:49 -0800 Subject: [PATCH 2/3] tests: reattach to and stop a VM through the public API nvx's public API creates and spawns a cold VM, and reattaches to it and stops it from the values that its caller saved, with no store of its own. Its unit tests check each step with stand-ins for OpenVMM, and the CLI's sandbox scenarios drive the steps through nvx-rs, but only the runtime adapter ran them through the public API alone on a real VM, and NVX deletes the adapter once the tests that replace it pass. nvx-rs/tests/vm_lifecycle.rs now does, in an ignored test that the new public vm-lifecycle scenario of test-microvm runs with Cargo. The test creates a cold launch of nvx-init over the Ubuntu EROFS layer and a scratch upper, as nvx-rs sandbox start configures one, and checks that the creation started nothing; spawns it; saves the values of SavedVm and the VM's outputs, and lets the launch go; reattaches from those values alone, twice, each time with a new authenticated connection that the guest answers; and stops the VM through its guest, which must exit without a forced termination and report its success and a complete teardown. A failure ends the VM from the saved values alone. The capability stays in the test's memory. The scenario gives the test the runtime's files and their digests, then checks the outcome report that it leaves, and that no OpenVMM of its VM remains. Its Cargo command takes the form of managed-lifecycle's, which both now share. CI's guest and sandbox job of each backend runs it. Evidence: the scenario passes on Linux/KVM (nvx-kvm-test), Linux/MSHV (nvx-mshv-test) and Windows/WHP (local), with OpenVMM and the guest artifacts of CI run 43075697333. Tests: a stand-in for Cargo's run of the test checks the inputs that the scenario gives it, its evidence, and the failures that the scenario reports: a failed test, a failed stop, a VM that outlives the test, and a missing Cargo. Part of #1118 (SL5.1b). Co-authored-by: Enrique Saurez Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/run-nvx-microvm-tests.yml | 2 + doc/ci.md | 10 +- doc/contribute.md | 4 +- doc/project-structure.md | 2 +- doc/usage.md | 15 +- nvx-rs/README.md | 11 + nvx-rs/tests/vm_lifecycle.rs | 255 ++++++++++++++++++++ scripts/nvx_tools/microvm_tests.py | 155 +++++++++++- scripts/test_microvm_tests.py | 221 +++++++++++++++++ scripts/test_nvx_tools.py | 1 + 10 files changed, 659 insertions(+), 17 deletions(-) create mode 100644 nvx-rs/tests/vm_lifecycle.rs diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index 7882d45c8..082567b1d 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -238,6 +238,7 @@ jobs: --scenario sandbox-proxy-only --scenario sandbox-ram-upper --scenario sandbox-gpt-layer + --scenario vm-lifecycle --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run public managed sandbox acceptance on Windows @@ -251,6 +252,7 @@ jobs: --scenario sandbox-proxy-only --scenario sandbox-ram-upper --scenario sandbox-gpt-layer + --scenario vm-lifecycle --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run Ubuntu guest smoke tests on Linux diff --git a/doc/ci.md b/doc/ci.md index 27be252cb..3ea335e33 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -558,7 +558,9 @@ a proxy-only network, whose guest reaches a test proxy on host loopback at its gateway and no other destination, sandboxes whose overlay upper is a tmpfs in guest memory, with no scratch image, and sandboxes whose root is a GPT disk, over either upper, so that the three backends run the CLI's twelve -cold cells (see the [case matrix](acceptance-case-matrix.md#cold-boots)). +cold cells (see the [case matrix](acceptance-case-matrix.md#cold-boots)). The +public `vm-lifecycle` scenario then creates, spawns, reattaches to, and stops +a VM through the public API of `nvx`. Their bounded evidence is part of the backend's failure logs. The same entrypoint then verifies a live virtio-fs share inside the container: a read-write `/workspace` share with a denied @@ -689,9 +691,9 @@ platform jobs don't take it, but the package job of each platform downloads both through the [`package-release`](../.github/actions/package-release/action.yml) action, the provenance into `build/`, so that each package ships the `nvx-rs` of its -operating system. The `managed-lifecycle` scenario also builds -and runs the ignored `nvx-rs` test `tests/managed_lifecycle.rs` with Cargo on -the runner. +operating system. The `managed-lifecycle` and `vm-lifecycle` scenarios also +build and run the ignored `nvx-rs` tests `tests/managed_lifecycle.rs` and +`tests/vm_lifecycle.rs` with Cargo on the runner. Once the required artifacts are ready, benchmarks run in parallel with the NVX test layer, each on its backend's diff --git a/doc/contribute.md b/doc/contribute.md index 6985c3581..a7436a07c 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -137,7 +137,9 @@ registers them. `sandbox` and the control protocol belong to `nvx-rs` scenarios drive them as commands, compile egress policies with `nvx-rs egress compile`, and leave the control session to `nvx-rs`: the `managed-lifecycle` scenario builds OpenVMM's command line and runs the ignored -`nvx-rs` test `tests/managed_lifecycle.rs`, which drives the guest. Add no +`nvx-rs` test `tests/managed_lifecycle.rs`, which drives the guest, and the +`vm-lifecycle` scenario runs the ignored test `tests/vm_lifecycle.rs`, which +drives a VM through the public API of `nvx`. Add no Python client of the control protocol, lifecycle, or egress logic. Unit tests for the host tooling live in `scripts/test_*.py`. The `nvx` crate diff --git a/doc/project-structure.md b/doc/project-structure.md index 591368cd7..ca4ee977f 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -108,7 +108,7 @@ nvx/ | | |-- guest/ Programs that run in Linux guests, the guest's NVXS connection, the libraries through which any guest init or agent opens the terminal on which it serves its session, starts a program, collects its children's statuses, settles its executions' cgroups, drives each execution, mounts host mappings beneath the descriptors of the roots that it names, assembles and tears down a layered root, and reads a GPT disk's partition table through its block device, and the typed calls through which it runs the time ABI's steps and the reseed (feature `guest`) | | `-- bin/ nvx-rs CLI, nvx-fake-openvmm test double, and the guest binaries | |-- examples/ The external-launch example, and the documented host and guest workflows, which only compile -| |-- tests/ Fake-OpenVMM instance lifecycle, sandbox store, CLI scenario, NVXS session, guest-binary, and hypervisor managed-lifecycle tests, the security checklist, the store's record fixtures and sandbox configurations of every format, the cases of the export and root boot tokens, and the conformance vectors of the wire formats +| |-- tests/ Fake-OpenVMM instance lifecycle, sandbox store, CLI scenario, NVXS session, guest-binary, and hypervisor managed-lifecycle and public-API VM lifecycle tests, the security checklist, the store's record fixtures and sandbox configurations of every format, the cases of the export and root boot tokens, and the conformance vectors of the wire formats | |-- public-api/ The crate's public API on Linux and on Windows, which CI checks | |-- build.rs Stages the bundled artifacts (feature `bundled`) | `-- artifacts.json Release package pinned for the bundled artifacts diff --git a/doc/usage.md b/doc/usage.md index 0a4f26b60..00c3265a8 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -55,7 +55,8 @@ this order: Without `NVX_RS`, a built executable, or Cargo, they fail and name these choices. They rebuild nothing once the executable exists. The -`managed-lifecycle` scenario also needs Cargo, which runs its `nvx-rs` test. +`managed-lifecycle` and `vm-lifecycle` scenarios also need Cargo, which runs +their `nvx-rs` tests. The provenance of a checkout's `nvx-rs`, `build/nvx-rs.provenance.json`, binds the SHA-256 of its executable to a digest of the `nvx-rs` sources that built @@ -1253,7 +1254,8 @@ which runs only when named: it repeats `smp` and also asserts that every CPU uses the one-shot counting LAPIC. `managed-exec-config`, `sandbox-lifecycle`, `sandbox-proxy-only`, `sandbox-ram-upper`, and `sandbox-gpt-layer`, the public sandbox scenarios, drive the public `nvx-rs sandbox` lifecycle commands (see -[Run](run.md#managed-lifecycle)) and also read the Ubuntu distro layer, its +[Run](run.md#managed-lifecycle)), and `vm-lifecycle` drives a VM through the +public API of `nvx`; they also read the Ubuntu distro layer, its manifest, and the sandbox scratch template, which [`download`](#download) installs and the guest-artifact build makes; a run that selects one of them fails before its first scenario when one of those @@ -1264,8 +1266,8 @@ and `virtio-features` scenarios and the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`, `platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`, `platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, -`sandbox-proxy-only`, `sandbox-ram-upper`, and `sandbox-gpt-layer` -scenarios. Ubuntu also rejects +`sandbox-proxy-only`, `sandbox-ram-upper`, `sandbox-gpt-layer`, and +`vm-lifecycle` scenarios. Ubuntu also rejects the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without `--scenario`, it runs @@ -1304,6 +1306,11 @@ the boot to the managed agent, and runs the ignored `nvx-rs` test `tests/managed_lifecycle.rs` with Cargo, which starts OpenVMM with the scenario's command line and drives the guest's control session; the scenario then checks OpenVMM's outcome report and the guest console. The +`vm-lifecycle` scenario runs the ignored `nvx-rs` test +`tests/vm_lifecycle.rs` with Cargo, which creates and spawns a VM of nvx-init +over the Ubuntu layer and a scratch copy through the public API of `nvx`, +reattaches to it from the values that it saved, and stops it through its +guest; the scenario then checks OpenVMM's outcome report. The `virtio-features` scenario boots a guest with every fixed virtio device: the NIC, the dormant host filesystem slot, both consoles, and both sandbox blocks. The guest reports, for each device, the base of its MMIO window, its virtio diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 56dfc8795..693f7613e 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -3621,6 +3621,17 @@ that it leaves: python3 ../scripts/nvx.py test-microvm --backend kvm --scenario managed-lifecycle ``` +`tests/vm_lifecycle.rs` is an ignored test of a cold VM's lifecycle through +the public API alone, on a real hypervisor: `host::vm::create`, `spawn`, +`reattach` from the values that the caller saved, once the launch is gone, +and `stop`, through a request to the guest, nvx-init, over the Ubuntu layer +and a scratch upper. `nvx.py test-microvm --scenario vm-lifecycle` gives it +its inputs, runs it with Cargo, and checks the outcome report that it leaves: + +```bash +python3 ../scripts/nvx.py test-microvm --backend kvm --scenario vm-lifecycle +``` + ### Specification tests The tests take their expectations from what this README and the documents diff --git a/nvx-rs/tests/vm_lifecycle.rs b/nvx-rs/tests/vm_lifecycle.rs new file mode 100644 index 000000000..109c7c54a --- /dev/null +++ b/nvx-rs/tests/vm_lifecycle.rs @@ -0,0 +1,255 @@ +//! A cold VM's lifecycle through the public API alone, on a real hypervisor. +//! +//! [`vm::create`] checks the launch and prepares it without starting it, and [`vm::spawn`] starts +//! it. The caller then saves the values of [`SavedVm`] and the VM's [`VmOutputs`], and lets the +//! launch go. [`vm::reattach`] finds the VM again from those saved values alone, as a caller does +//! after it restarts, once for each of two connections, as each command of a CLI connects anew, +//! and [`vm::stop`] stops the VM through a request to its guest: the VM must exit on its own, +//! before any forced termination, and report its success and a complete teardown. The guest is +//! nvx-init, NVX's reference init, over the layered root that `nvx-rs sandbox start` boots: the +//! Ubuntu EROFS layer over an ext4 scratch upper. +//! +//! `scripts/nvx.py test-microvm --scenario vm-lifecycle` runs it on the selected backend, with +//! `NVX_VM_LIFECYCLE` set to the JSON object of its [`Inputs`], and then checks the outcome report +//! that OpenVMM wrote. The test keeps the VM's control capability in memory alone. +#![cfg(all(feature = "host", any(target_os = "linux", windows)))] + +use std::env; +use std::path::PathBuf; +use std::time::{Duration, Instant}; + +use nvx::Hypervisor; +use nvx::activation::{Journal, MachineIdMode, OverlayMode, View}; +use nvx::host::config::{Boot, ConsoleAttachment, LayeredRoot, RootConfig, UpperConfig, VmConfig}; +use nvx::host::process::{Opened, Process, ProcessRef}; +use nvx::host::report::Report; +use nvx::host::runtime::{ + ArtifactIdentity, LaunchOptions, Runtime, RuntimeIdentity, probe_hypervisor, +}; +use nvx::host::transport::HostConnection; +use nvx::host::vm::{self, SavedVm, VmDir, VmOutputs}; +use nvx::protocol::root::RootImage; +use nvx::protocol::{self, APP_PING, APP_READY, APP_STOP, APP_STOPPED}; +use nvx::uuid::Uuid; + +/// The caller's kernel command-line tokens, which `nvx-rs sandbox start` gives nvx-init too: it +/// manages the VM, and runs workloads as the fixed identity `65534:65534`. +const COMMAND_LINE: &str = "nvx_sandbox=1 nvx_mode=managed nvx_identity=65534:65534"; + +/// The inputs of the test, which the scenario gives it as a JSON object in `NVX_VM_LIFECYCLE`. +#[derive(serde::Deserialize)] +#[serde(deny_unknown_fields)] +struct Inputs { + /// The selected backend: `kvm`, `mshv`, or `whp`. + backend: String, + /// The OpenVMM executable. + openvmm: PathBuf, + /// The SHA-256 digest of the OpenVMM executable, its identity in the runtime's. + openvmm_sha256: String, + /// The guest kernel. + kernel: PathBuf, + /// The SHA-256 digest of the guest kernel. + kernel_sha256: String, + /// The control initramfs, which holds nvx-init. + initramfs: PathBuf, + /// The SHA-256 digest of the control initramfs. + initramfs_sha256: String, + /// The Ubuntu EROFS layer. + image: PathBuf, + /// The UUID of the layer's filesystem. + image_uuid: String, + /// A new copy of the sandbox scratch template, the upper's image. + scratch: PathBuf, + /// The VM's directory, which the test creates. + vm_dir: PathBuf, + /// The longest wait for each step, in seconds. + timeout_seconds: f64, +} + +/// The VM that the test started, which it ends at once if the test fails before the VM exits. +/// +/// It holds only what the caller saved, and finds the VM through it, as a caller that restarted +/// does. +struct Started { + /// The VM's process. + process: ProcessRef, + /// The VM's output files and endpoints. + outputs: VmOutputs, + /// The longest wait for the VM's exit. + timeout: Duration, +} + +impl Drop for Started { + fn drop(&mut self) { + if let Ok(Opened::Running(process)) = Process::open(&self.process) { + let now = Instant::now(); + if let Err(error) = + vm::stop(&process, &self.outputs, |_| false, now, now + self.timeout) + { + eprintln!("the VM did not stop: {error}"); + } + } + } +} + +/// Sends the request `kind` of the caller's own protocol with its guest, which for nvx-init is +/// NVXC, and checks that the guest acknowledges it with `reply` by `deadline`. +fn acknowledged( + connection: &mut HostConnection, + kind: u8, + request: u64, + reply: u8, + deadline: Instant, +) { + connection + .send(&protocol::encode_app(kind, request, 0, &[]), Some(deadline)) + .expect("the guest takes the request"); + let response = connection + .receive(Some(deadline)) + .expect("the guest answers the request"); + let frame = protocol::decode_app(&response).expect("the answer is an NVXC frame"); + assert_eq!( + (frame.kind, frame.request_id, frame.status, frame.payload), + (reply, request, 0, &[][..]), + "the guest acknowledges request {kind}" + ); +} + +#[test] +#[ignore = "boots a VM; run scripts/nvx.py test-microvm --scenario vm-lifecycle"] +fn a_cold_vm_is_reattached_from_saved_values_and_stopped() { + let inputs = env::var("NVX_VM_LIFECYCLE") + .expect("NVX_VM_LIFECYCLE is set: run scripts/nvx.py test-microvm --scenario vm-lifecycle"); + let inputs: Inputs = serde_json::from_str(&inputs).expect("the inputs are a JSON object"); + let hypervisor = match inputs.backend.as_str() { + "kvm" => Hypervisor::Kvm, + "mshv" => Hypervisor::Mshv, + "whp" => Hypervisor::Whp, + backend => panic!("no hypervisor is named {backend}"), + }; + probe_hypervisor(hypervisor).expect("the backend's hypervisor is usable"); + let timeout = Duration::from_secs_f64(inputs.timeout_seconds); + let identity = |digest: &str| ArtifactIdentity::sha256(digest).expect("a SHA-256 digest"); + let runtime = Runtime { + openvmm: inputs.openvmm, + kernel: inputs.kernel, + initramfs: inputs.initramfs, + identity: RuntimeIdentity::new( + identity(&inputs.openvmm_sha256), + identity(&inputs.kernel_sha256), + identity(&inputs.initramfs_sha256), + ), + hypervisor, + cpu_profile: None, + windows_breakaway: false, + options: LaunchOptions::default(), + }; + let image_uuid = Uuid::parse(&inputs.image_uuid).expect("the layer's UUID"); + let config = VmConfig { + processors: 1, + memory_mib: 256, + root: RootConfig::Layered(LayeredRoot { + image: inputs.image, + format: RootImage::erofs(Some(image_uuid)), + gpt_first_partition_bound: None, + upper: UpperConfig::Scratch { + path: inputs.scratch, + journal: Journal::Allowed, + }, + overlay: OverlayMode::TrustedLowers, + }), + view: View { + hostname: b"nvx-vm-lifecycle".to_vec(), + machine_id: MachineIdMode::Generated, + }, + network: None, + filesystem: None, + console: ConsoleAttachment::Disconnected, + boot: Some(Boot { + command_line: COMMAND_LINE.to_owned(), + memory_capacity_mib: None, + capture: None, + }), + }; + + vm::create_private_dir(&inputs.vm_dir).expect("the VM directory is created private"); + let dir = VmDir::open(&inputs.vm_dir).expect("the VM directory opens"); + let prepared = + vm::create(&runtime, &config, None, &dir).expect("the public API creates the cold launch"); + // Creation makes the log, and starts nothing that writes the outcome report. + assert!(prepared.outputs().log.is_file(), "creation makes the log"); + assert!( + !prepared.outputs().outcome.exists(), + "no VM runs before the spawn" + ); + let launched = vm::spawn(prepared).expect("the public API spawns the cold VM"); + let saved = SavedVm { + vm_dir: dir, + endpoint: launched.outputs.endpoint.clone(), + capability: *launched.capability.as_bytes(), + process: launched.process.reference().clone(), + runtime: runtime.identity.clone(), + }; + let started = Started { + process: saved.process.clone(), + outputs: launched.outputs.clone(), + timeout, + }; + // The caller keeps only what it saved. + drop(launched); + + let mut reattached = vm::reattach(&saved, &runtime, Instant::now() + timeout) + .expect("the saved values reattach the VM"); + assert_eq!(reattached.process.reference(), &saved.process); + acknowledged( + &mut reattached.connection, + APP_PING, + 1, + APP_READY, + Instant::now() + timeout, + ); + drop(reattached); + let mut reattached = vm::reattach(&saved, &runtime, Instant::now() + timeout) + .expect("the saved values reattach the VM again, with a new connection"); + acknowledged( + &mut reattached.connection, + APP_PING, + 2, + APP_READY, + Instant::now() + timeout, + ); + let grace = Instant::now() + timeout; + let stopped = vm::stop( + &reattached.process, + &started.outputs, + |deadline| { + acknowledged( + &mut reattached.connection, + APP_STOP, + 3, + APP_STOPPED, + deadline, + ); + true + }, + grace, + grace + timeout, + ) + .expect("the VM's exit is confirmed"); + assert!( + stopped.self_exited && !stopped.forced, + "the guest stops the VM without a forced termination: {stopped:?}" + ); + assert_eq!( + stopped.report.expect("the outcome report is readable"), + Some(Report { + category: Some("success".to_owned()), + status_code: Some(0), + teardown_complete: Some(true), + }), + ); + assert!(matches!( + Process::open(&saved.process), + Ok(Opened::Exited(_)) + )); +} diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 05bdc2f31..6b6ae5585 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -79,15 +79,20 @@ from .openvmm_process import OpenvmmProcess, TcpConsole from .sandbox_lifecycle_tests import ( CONFIG, + EVIDENCE_LIMIT, + LOG, OUTCOME, PROBE, PROVISIONED_ENTRIES, STOPPED_ENTRIES, LifecycleAcceptance, + bounded_evidence, check_managed_report, managed_report, + openvmm_processes, run_acceptance, run_sandbox_lifecycle, + terminate_process, ) from .sandbox_proxy_tests import run_sandbox_proxy_only from .sandbox_ram_upper_tests import ( @@ -171,6 +176,7 @@ "untiered-network-policy", "virtio-features", "virtio-net", + "vm-lifecycle", "workload-identity", ) UBUNTU_UNSUPPORTED_SCENARIOS = frozenset(("console-snapshot",)) @@ -191,12 +197,14 @@ "scratch-snapshot", "snapshot-tiers", "virtio-features", + "vm-lifecycle", ) ) -# The public `nvx-rs sandbox` scenarios, which also read the Ubuntu distro -# layer and the sandbox scratch template that `nvx-rs download` installs. A -# default run runs them, but CI runs them in its guest and sandbox jobs, which -# take those Ubuntu artifacts, rather than in the shards of the default suite. +# The public scenarios, of `nvx-rs sandbox` and of the public API's VM +# lifecycle, which also read the Ubuntu distro layer and the sandbox scratch +# template that `nvx-rs download` installs. A default run runs them, but CI +# runs them in its guest and sandbox jobs, which take those Ubuntu artifacts, +# rather than in the shards of the default suite. PUBLIC_SANDBOX_SCENARIOS = frozenset( ( "managed-exec-config", @@ -204,6 +212,7 @@ "sandbox-lifecycle", "sandbox-proxy-only", "sandbox-ram-upper", + "vm-lifecycle", ) ) # The same-host restore cases also run on the CI debug kernel (doc/ci.md), @@ -1350,8 +1359,9 @@ def run_workload_identity( MANAGED_LIFECYCLE_TEST = "managed_lifecycle" -def managed_lifecycle_test_command(cargo: str) -> list[str]: - """Return the Cargo command that runs MANAGED_LIFECYCLE_TEST.""" +def ignored_nvx_rs_test_command(cargo: str, test: str) -> list[str]: + """Return the Cargo command that runs the ignored tests of the nvx-rs + test target ``test``.""" return [ cargo, "test", @@ -1359,13 +1369,18 @@ def managed_lifecycle_test_command(cargo: str) -> list[str]: os.fspath(nvx_rs.DIRECTORY / "Cargo.toml"), "--locked", "--test", - MANAGED_LIFECYCLE_TEST, + test, "--", "--ignored", "--nocapture", ] +def managed_lifecycle_test_command(cargo: str) -> list[str]: + """Return the Cargo command that runs MANAGED_LIFECYCLE_TEST.""" + return ignored_nvx_rs_test_command(cargo, MANAGED_LIFECYCLE_TEST) + + def run_managed_lifecycle( executable: Path, kernel: Path, @@ -1528,6 +1543,119 @@ def run_managed_lifecycle( ) +# The ignored nvx-rs test of a cold VM's lifecycle through the public API of +# `nvx`: nvx-rs/tests/vm_lifecycle.rs. +VM_LIFECYCLE_TEST = "vm_lifecycle" +VM_LIFECYCLE_EVIDENCE_PREFIX = "vm-lifecycle-" + + +def end_openvmm_processes(marker: str, timeout: float) -> tuple[int, ...]: + """End every OpenVMM process whose arguments contain ``marker``, wait + ``timeout`` seconds at most until they are gone, and return their IDs.""" + processes = openvmm_processes(marker) + for pid in processes: + with suppress(OSError): + terminate_process(pid) + deadline = time.monotonic() + timeout + while survivors := openvmm_processes(marker): + if time.monotonic() >= deadline: + raise RuntimeError(f"OpenVMM processes {list(survivors)} survived") + time.sleep(0.05) + return processes + + +def run_vm_lifecycle( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + timeout: float, + output_dir: Path, +) -> None: + """Run a cold VM's lifecycle through the public API of `nvx`. + + The nvx-rs test VM_LIFECYCLE_TEST creates and spawns a VM of nvx-init over + the Ubuntu EROFS layer and a new copy of the sandbox scratch template, + lets the launch go, reattaches to the VM from the values that it saved, + and stops it through its guest. This gives the test its inputs, then + checks what it leaves: OpenVMM's outcome report, and no OpenVMM process. + """ + cargo = shutil.which("cargo") + if cargo is None: + raise ScriptError( + "the vm-lifecycle scenario runs the nvx-rs test " + f"{VM_LIFECYCLE_TEST} with Cargo, which is not on PATH; " + "install Rust (https://rustup.rs)" + ) + artifacts = load_workload_artifacts() + # Cargo runs the test in the nvx-rs directory, so every path is absolute. + output_dir = output_dir.resolve() + output_dir.mkdir(parents=True, exist_ok=True) + root = Path(tempfile.mkdtemp(prefix="nvx-vm-lifecycle-")).resolve() + vm_dir = root / "vm" + try: + scratch = root / "scratch.ext4" + shutil.copyfile(artifacts.scratch_template, scratch) + inputs = { + "backend": backend, + "openvmm": os.fspath(executable.resolve()), + "openvmm_sha256": sha256_file(executable), + "kernel": os.fspath(kernel.resolve()), + "kernel_sha256": sha256_file(kernel), + "initramfs": os.fspath(initrd.resolve()), + "initramfs_sha256": sha256_file(initrd), + "image": os.fspath(artifacts.distro.resolve()), + "image_uuid": artifacts.layer_uuid, + "scratch": os.fspath(scratch), + "vm_dir": os.fspath(vm_dir), + "timeout_seconds": timeout, + } + test_command = ignored_nvx_rs_test_command(cargo, VM_LIFECYCLE_TEST) + print(f">> {shlex.join(test_command)}", flush=True) + process = subprocess.Popen( + test_command, + cwd=BuildConstants.REPO_ROOT, + env=dict(os.environ, NVX_VM_LIFECYCLE=json.dumps(inputs)), + ) + try: + returncode = process.wait() + except BaseException: + # As in the managed-lifecycle scenario, a signal that reached + # nvx.py alone leaves Cargo's descendants running. + terminate_process_tree(process, process_group=False) + raise + for name in (LOG, OUTCOME): + source = vm_dir / name + if source.is_file(): + # The end of each file, where a failure shows. + with source.open("rb") as stream: + size = stream.seek(0, os.SEEK_END) + stream.seek(max(0, size - EVIDENCE_LIMIT)) + data = stream.read(EVIDENCE_LIMIT) + (output_dir / f"{VM_LIFECYCLE_EVIDENCE_PREFIX}{name}").write_bytes( + bounded_evidence(data, size - len(data)) + ) + if returncode != 0: + raise RuntimeError( + f"the VM lifecycle test {VM_LIFECYCLE_TEST} failed with exit " + f"status {returncode}; see its output above and " + f"{output_dir / (VM_LIFECYCLE_EVIDENCE_PREFIX + LOG)}" + ) + check_managed_report(managed_report(vm_dir), backend) + finally: + # The test stops its VM even when it fails, but not once it is ended, + # and the VM's arguments name the fixture. + survivors = end_openvmm_processes(root.name, timeout) + shutil.rmtree(root, ignore_errors=True) + if survivors: + raise RuntimeError( + f"OpenVMM processes {list(survivors)} outlived the VM lifecycle test" + ) + if root.exists(): + raise RuntimeError(f"the VM lifecycle's fixture {root} could not be removed") + + # The sandbox-gpt-layer scenario's GPT disk, which it lays out as UEFI does and # as nvx-init reads a layered root's disk: its logical block size, the type of # its partitions, Linux's filesystem data, and the bytes of its first @@ -11474,6 +11602,19 @@ def _run_scenarios( timeout=args.timeout, output_dir=output_dir, ) + if "vm-lifecycle" in scenarios: + report.start( + "vm-lifecycle", + f"Running the public API's cold VM lifecycle on OpenVMM/{args.backend}", + ) + run_vm_lifecycle( + executable, + kernel, + initrd, + args.backend, + timeout=args.timeout, + output_dir=output_dir, + ) if "managed-lifecycle" in scenarios: report.start( "managed-lifecycle", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 96e2785e7..b6f5fcb2d 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3336,6 +3336,191 @@ def test_a_layer_of_part_of_a_sector_is_refused_before_the_disk_is_written(self) self.assertFalse(path.exists()) +class _FakeVmLifecycleTest: + """Emulates Cargo's run of the nvx-rs test vm_lifecycle: it takes its + inputs, and leaves the log and the outcome report of its VM, which + reports `outcome`, and exits with `returncode`.""" + + def __init__( + self, + returncode: int = 0, + outcome: Mapping[str, object] = sandbox_lifecycle_tests.MANAGED_SUCCESS, + log: bytes = b"OpenVMM started\n", + ) -> None: + self.returncode = returncode + self.outcome = outcome + self.log = log + self.calls: list[tuple[list[str], dict[str, str]]] = [] + self.scratch = b"" + + def __call__(self, command: list[str], *, cwd: Path, env: dict[str, str]) -> Any: + self.calls.append((command, env)) + inputs = json.loads(env["NVX_VM_LIFECYCLE"]) + self.scratch = Path(inputs["scratch"]).read_bytes() + vm_dir = Path(inputs["vm_dir"]) + vm_dir.mkdir() + (vm_dir / "openvmm.log").write_bytes(self.log) + (vm_dir / "outcome.json").write_text( + json.dumps( + { + "backend": inputs["backend"], + "outcome": self.outcome, + "network_policy": {"status": "not-requested"}, + "teardown": dict.fromkeys( + microvm_tests.OUTCOME_TEARDOWN_FIELDS, True + ), + } + ), + encoding="utf-8", + ) + process = MagicMock() + process.wait.return_value = self.returncode + return process + + +class PublicVmLifecycleTests(unittest.TestCase): + def _run( + self, + root: Path, + test: _FakeVmLifecycleTest, + processes: Sequence[tuple[int, ...]] = ((),), + cargo: str | None = "cargo", + ) -> tuple[Path, MagicMock]: + distro = root / "ubuntu-distro.erofs" + distro.write_bytes(b"distro") + template = root / "ubuntu-smoke-scratch.ext4" + template.write_bytes(b"scratch template") + for name in ("openvmm", "vmlinux", "initramfs.cpio.gz"): + (root / name).write_bytes(name.encode()) + artifacts = managed_exec_tests.WorkloadArtifacts( + distro, "12345678-1234-1234-1234-123456789abc", template + ) + output_dir = root / "results" + mkdtemp = tempfile.mkdtemp + + def fixture(prefix: str) -> str: + return mkdtemp(prefix=prefix, dir=root) + + listed = iter(processes) + + def listing(_marker: str) -> tuple[int, ...]: + return next(listed, ()) + + with ( + patch.object(microvm_tests.shutil, "which", return_value=cargo), + patch.object( + microvm_tests, "load_workload_artifacts", return_value=artifacts + ), + patch.object(microvm_tests.tempfile, "mkdtemp", side_effect=fixture), + patch.object(microvm_tests.subprocess, "Popen", side_effect=test), + patch.object(microvm_tests, "openvmm_processes", side_effect=listing), + patch.object(microvm_tests, "terminate_process") as terminate, + redirect_stdout(io.StringIO()), + ): + microvm_tests.run_vm_lifecycle( + root / "openvmm", + root / "vmlinux", + root / "initramfs.cpio.gz", + "mshv", + timeout=2, + output_dir=output_dir, + ) + return output_dir, terminate + + def test_the_test_takes_its_inputs_and_leaves_a_successful_stop(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary).resolve() + test = _FakeVmLifecycleTest() + output_dir, terminate = self._run(root, test) + evidence = {path.name for path in output_dir.iterdir()} + fixtures = list(root.glob("nvx-vm-lifecycle-*")) + + [(command, environment)] = test.calls + self.assertEqual( + command, microvm_tests.ignored_nvx_rs_test_command("cargo", "vm_lifecycle") + ) + inputs = json.loads(environment["NVX_VM_LIFECYCLE"]) + self.assertEqual(test.scratch, b"scratch template") + self.assertEqual(inputs.pop("backend"), "mshv") + self.assertEqual(inputs.pop("timeout_seconds"), 2) + self.assertEqual(inputs.pop("image"), str(root / "ubuntu-distro.erofs")) + self.assertEqual( + inputs.pop("image_uuid"), "12345678-1234-1234-1234-123456789abc" + ) + for name, path in ( + ("openvmm", "openvmm"), + ("kernel", "vmlinux"), + ("initramfs", "initramfs.cpio.gz"), + ): + self.assertEqual(inputs.pop(name), str(root / path)) + self.assertEqual( + inputs.pop(f"{name}_sha256"), + hashlib.sha256(path.encode()).hexdigest(), + ) + scratch, vm_dir = Path(inputs.pop("scratch")), Path(inputs.pop("vm_dir")) + self.assertEqual(inputs, {}) + self.assertEqual((scratch.name, vm_dir.name), ("scratch.ext4", "vm")) + self.assertEqual(scratch.parent, vm_dir.parent) + self.assertTrue(scratch.parent.name.startswith("nvx-vm-lifecycle-")) + # The test keeps the capability, which nothing gives it or records. + self.assertNotIn("capability", environment["NVX_VM_LIFECYCLE"]) + self.assertEqual( + evidence, {"vm-lifecycle-openvmm.log", "vm-lifecycle-outcome.json"} + ) + self.assertEqual(fixtures, []) + terminate.assert_not_called() + + def test_a_failed_test_fails_the_scenario_and_keeps_the_evidence_of_its_vm(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + with self.assertRaisesRegex(RuntimeError, "failed with exit status 101"): + self._run(root, _FakeVmLifecycleTest(returncode=101)) + self.assertTrue((root / "results" / "vm-lifecycle-openvmm.log").is_file()) + self.assertEqual(list(root.glob("nvx-vm-lifecycle-*")), []) + + def test_the_evidence_keeps_the_end_of_a_long_log(self): + limit = sandbox_lifecycle_tests.EVIDENCE_LIMIT + log = b"early\n" + bytes(range(256)) * (limit // 256) + b"the failure\n" + with tempfile.TemporaryDirectory() as temporary: + output_dir, _ = self._run(Path(temporary), _FakeVmLifecycleTest(log=log)) + evidence = (output_dir / "vm-lifecycle-openvmm.log").read_bytes() + omitted = len(log) - limit + self.assertEqual( + evidence, + f"... ({omitted} earlier bytes omitted)\n".encode() + log[omitted:], + ) + + def test_a_fixture_that_remains_fails_the_scenario(self): + with ( + tempfile.TemporaryDirectory() as temporary, + patch.object(microvm_tests.shutil, "rmtree"), + self.assertRaisesRegex(RuntimeError, "could not be removed"), + ): + self._run(Path(temporary), _FakeVmLifecycleTest()) + + def test_a_failed_stop_fails_the_scenario(self): + outcome = {"operation": "run", "category": "killed", "status_code": 137} + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex(RuntimeError, "the managed outcome is"): + self._run(Path(temporary), _FakeVmLifecycleTest(outcome=outcome)) + + def test_a_vm_that_outlives_the_test_is_ended_and_fails_the_scenario(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + with self.assertRaisesRegex( + RuntimeError, r"OpenVMM processes \[77\] outlived" + ): + self._run(root, _FakeVmLifecycleTest(), processes=((77,), ())) + self.assertEqual(list(root.glob("nvx-vm-lifecycle-*")), []) + + def test_the_scenario_needs_cargo_before_it_starts_anything(self): + with tempfile.TemporaryDirectory() as temporary: + test = _FakeVmLifecycleTest() + with self.assertRaisesRegex(common.ScriptError, "with Cargo"): + self._run(Path(temporary), test, cargo=None) + self.assertEqual(test.calls, []) + + class GuestIdentityScriptTests(unittest.TestCase): def test_guest_identity_checks_fail_before_success_markers(self): descriptor = microvm_tests.guest_descriptor("ubuntu") @@ -15651,6 +15836,42 @@ def require(path: Path, _description: str) -> Path: ) ram_upper.assert_not_called() + def test_runner_dispatches_public_vm_lifecycle(self): + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "mshv", + "--scenario", + "vm-lifecycle", + "--output-dir", + str(output_dir), + ] + ) + + def require(path: Path, _description: str) -> Path: + return path + + with ( + patch.object(microvm_tests, "validate_openvmm_test_backend"), + patch.object(microvm_tests, "require_file", side_effect=require), + patch.object(microvm_tests, "run_vm_lifecycle") as run, + patch.object(microvm_tests, "run_sandbox_gpt_layer") as gpt_layer, + ): + self.assertEqual(microvm_tests.run(args), 0) + # The VM of the nvx-rs test boots the run's artifacts. + run.assert_called_once_with( + microvm_tests.openvmm_binary_path(), + microvm_tests.artifact_path("vmlinux"), + microvm_tests.artifact_path("initramfs.cpio.gz"), + "mshv", + timeout=args.timeout, + output_dir=output_dir, + ) + gpt_layer.assert_not_called() + def test_a_default_run_runs_the_public_sandbox_scenarios(self): def require(path: Path, _description: str) -> Path: return path diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index e292af29d..6bef5161f 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -12813,6 +12813,7 @@ def test_ci_runs_the_public_sandbox_acceptance_on_every_backend(self): self.assertIn("--scenario sandbox-proxy-only", acceptance) self.assertIn("--scenario sandbox-ram-upper", acceptance) self.assertIn("--scenario sandbox-gpt-layer", acceptance) + self.assertIn("--scenario vm-lifecycle", acceptance) self.assertIn( '--output-dir "build/test-results/public-exec-${{ inputs.backend }}"', acceptance, From 72c6a96a41f4f2a92cfc2835e7c7456cc461beca Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 11:59:13 -0800 Subject: [PATCH 3/3] doc: map the cold cells and the VM lifecycle in the case matrix The case matrix left it to SL5.1b to keep the four public sandbox scenarios of a cold boot running or to give them their fates, and pointed the adapter's cold-boot and reattachment scenarios at SL5.1b's tests, which didn't exist yet. The matrix now lists the CLI's twelve cold cells in a Cold boots section: for each image and upper, the scenario or CI step that boots it through sandbox run, and through a managed sandbox's provision and start, on each backend, and what vm-lifecycle checks through the public API. managed-exec-config, sandbox-lifecycle, sandbox-proxy-only and sandbox-ram-upper are kept, each with the cell that it boots or, for sandbox-proxy-only, the network that it checks. The new sandbox-gpt-layer and vm-lifecycle scenarios are kept too, in rows of their own, which the summary counts. The adapter's erofs_over_* and gpt_over_* scenarios, and its restart and reattachment scenarios, name the tests that cover them now, and the CI rows of the public acceptance and the layer smoke test say what they run. Part of #1118 (SL5.1b). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/acceptance-case-matrix.md | 65 ++++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 17 deletions(-) diff --git a/doc/acceptance-case-matrix.md b/doc/acceptance-case-matrix.md index d88fec333..620e518d7 100644 --- a/doc/acceptance-case-matrix.md +++ b/doc/acceptance-case-matrix.md @@ -10,9 +10,10 @@ docs, and each microVM scenario, one fate, by name: of #1076. Where only a part of what it checks leaves NVX, the row says which. - **retired**: what it checks leaves NVX, as the row says, so nothing replaces it. -The later slices are SL5.1b (#1118), whose smoke tests on real VMs create, spawn, reattach and -stop a VM through the public API, and run the CLI's 12 cold cells, EROFS and GPT images, each over -a scratch or a RAM upper, on KVM, MSHV and WHP; and SL5.1c (#1118), whose smoke tests capture a +SL5.1b (#1118) added the smoke tests on real VMs of a cold boot: `test-microvm --scenario +vm-lifecycle` creates, spawns, reattaches to and stops a VM through the public API, and the CLI's +12 cold cells, EROFS and GPT images, each over a scratch or a RAM upper, run on KVM, MSHV and WHP, +as [Cold boots](#cold-boots) lists. The later slice is SL5.1c (#1118), whose smoke tests capture a snapshot, open it and restore it with nvx-init as the guest, run concurrent restores of one snapshot under the CLI's lock, and check that a failed restore reports its attribution. @@ -28,7 +29,7 @@ its own in turn. | --- | ---: | ---: | ---: | ---: | | [The adapter's scenarios](#the-adapters-scenarios) | 157 | 0 | 114 | 43 | | [The adapter's other tests](#the-adapters-other-tests) | 169 | 0 | 9 | 160 | -| [MicroVM scenarios](#microvm-scenarios) | 52 | 48 | 3 | 1 | +| [MicroVM scenarios](#microvm-scenarios) | 54 | 50 | 3 | 1 | | [CI](#ci) | 69 | 43 | 3 | 23 | | [Docs](#docs) | 36 | 35 | 1 | 0 | @@ -141,7 +142,7 @@ The adapter's README says what each proves. | --- | --- | --- | | `a_host_program_of_another_runtime_finds_the_vm_mismatched_and_signals_nothing` | rewritten | `nvx-rs/tests/security_checklist.rs::a_saved_vm_runs_until_its_process_exits_and_another_runtime_is_refused_without_a_signal`; `nvx-rs/src/host/vm/tests/reattach.rs::verifying_saved_values_checks_runtime_and_process_without_opening_an_endpoint` | | `a_launch_tears_down_a_vm_whose_guest_agent_is_of_another_build` | retired | The adapter's guest agent and the build that its Ready reports | -| `a_restarted_host_program_reattaches_to_the_vm_and_stops_it` | rewritten | SL5.1b's lifecycle smoke test; `nvx-rs/src/host/vm/tests/reattach.rs::reattaching_returns_the_held_owned_child_and_authenticated_connection`; `nvx-rs/src/host/vm/tests.rs::stop_forces_only_its_verified_process_after_grace` | +| `a_restarted_host_program_reattaches_to_the_vm_and_stops_it` | rewritten | `nvx-rs/tests/vm_lifecycle.rs::a_cold_vm_is_reattached_from_saved_values_and_stopped`, which `test-microvm --scenario vm-lifecycle` runs; `nvx-rs/src/host/vm/tests/reattach.rs::reattaching_returns_the_held_owned_child_and_authenticated_connection`; `nvx-rs/src/host/vm/tests.rs::stop_forces_only_its_verified_process_after_grace` | ### `vm_steps` (3) @@ -171,10 +172,10 @@ The adapter's README says what each proves. | `busy_root_fails_the_teardown` | rewritten | `nvx-rs/src/guest/mappings/mapper/tests/privileged.rs::a_teardown_unmounts_each_mapping_and_reports_a_busy_one`; `nvx-rs/src/guest/root/tests.rs::a_teardown_unmounts_the_layers_top_first_and_reports_every_failure` | | `corrupt_layer_fails_the_assembly` | rewritten | `nvx-rs/src/guest/root/tests.rs::a_layer_that_is_not_ext4_or_whose_journal_needs_a_recovery_is_refused_before_any_mount` | | `cyclic_journal_is_not_loaded` | rewritten | `nvx-rs/src/guest/root/block/tests.rs::the_superblock_tells_whether_an_ext4_journal_exists_and_needs_a_recovery`; `nvx-rs/src/guest/root/tests.rs::a_gpt_disk_s_layers_are_mounted_under_the_overlay_top_first` | -| `erofs_over_ram` | rewritten | SL5.1b's cold cells; `nvx-rs/src/guest/root/tests.rs::a_tmpfs_upper_needs_no_scratch_device` | +| `erofs_over_ram` | rewritten | `test-microvm --scenario sandbox-ram-upper`, among the [cold cells](#cold-boots); `nvx-rs/src/guest/root/tests.rs::a_tmpfs_upper_needs_no_scratch_device` | | `erofs_over_ram_maps_the_table` | rewritten | `nvx-rs/src/guest/mappings/table/tests.rs::mappings_keep_the_paths_and_the_access_of_their_entries`; `nvx-rs/src/guest/init/boot/tests.rs::layered_cold_boot_maps_through_handles_opened_before_its_only_switch` | | `erofs_over_ram_with_untrusted_lowers` | rewritten | `nvx-rs/src/guest/root/tests.rs::both_overlay_modes_assemble_on_both_uppers_with_their_own_option_sets` | -| `erofs_over_scratch` | rewritten | SL5.1b's cold cells; `nvx-rs/src/guest/root/tests.rs::assembly_mounts_the_image_and_the_scratch_under_the_overlay`. The adapter's cgroups retire with the exec engine (#1087) | +| `erofs_over_scratch` | rewritten | `test-microvm --scenario sandbox-lifecycle`, among the [cold cells](#cold-boots); `nvx-rs/src/guest/root/tests.rs::assembly_mounts_the_image_and_the_scratch_under_the_overlay`. The adapter's cgroups retire with the exec engine (#1087) | | `erofs_over_scratch_maps_the_table` | rewritten | `nvx-rs/src/guest/mappings/mapper/tests.rs::mappings_are_restricted_copies_attached_in_order`; `nvx-rs/src/guest/init/boot/tests.rs::layered_cold_boot_maps_through_handles_opened_before_its_only_switch` | | `erofs_over_scratch_with_untrusted_lowers` | rewritten | `nvx-rs/src/guest/root/tests.rs::both_overlay_modes_assemble_on_both_uppers_with_their_own_option_sets` | | `executions_run_as_the_workload_identity` | rewritten | `test-microvm --scenario workload-identity`; `test-microvm --scenario managed-exec-config`. An execution's own identity, which the adapter refused, retires with the exec engine (#1087) | @@ -184,10 +185,10 @@ The adapter's README says what each proves. | `forbidden_journal_fails_the_assembly` | rewritten | `nvx-rs/src/guest/root/tests.rs::the_journal_policy_is_checked_in_the_scratch_s_superblock_before_any_mount` | | `full_ram_upper_refuses_writes` | rewritten | The tmpfs takes its cap as its size, which the kernel enforces: `nvx-rs/src/guest/root/tests.rs::both_overlay_modes_assemble_on_both_uppers_with_their_own_option_sets`; `nvx-rs/src/host/config/tests.rs::a_ram_upper_holds_1_to_90_percent_of_the_memory_and_half_by_default` | | `full_scratch_refuses_writes` | retired | ext4 refuses writes to a full scratch; NVX sets nothing of it | -| `gpt_over_ram` | rewritten | SL5.1b's cold cells; `nvx-rs/src/guest/root/tests.rs::a_gpt_disk_s_layers_are_mounted_under_the_overlay_top_first` | +| `gpt_over_ram` | rewritten | `test-microvm --scenario sandbox-gpt-layer`, among the [cold cells](#cold-boots); `nvx-rs/src/guest/root/tests.rs::a_gpt_disk_s_layers_are_mounted_under_the_overlay_top_first` | | `gpt_over_ram_maps_the_table` | rewritten | `nvx-rs/src/guest/mappings/table/tests.rs::mappings_keep_the_paths_and_the_access_of_their_entries`; `nvx-rs/src/guest/init/boot/tests.rs::layered_cold_boot_maps_through_handles_opened_before_its_only_switch` | | `gpt_over_ram_with_untrusted_lowers` | rewritten | `nvx-rs/src/guest/root/tests.rs::both_overlay_modes_assemble_on_both_uppers_with_their_own_option_sets` | -| `gpt_over_scratch` | rewritten | SL5.1b's cold cells; `nvx-rs/src/guest/root/tests.rs::the_first_partition_is_returned_as_opaque_bytes_within_its_bound`; `nvx-rs/src/guest/root/tests.rs::a_gpt_disk_s_layers_are_mounted_under_the_overlay_top_first` | +| `gpt_over_scratch` | rewritten | `test-microvm --scenario sandbox-gpt-layer`, among the [cold cells](#cold-boots); `nvx-rs/src/guest/root/tests.rs::the_first_partition_is_returned_as_opaque_bytes_within_its_bound`; `nvx-rs/src/guest/root/tests.rs::a_gpt_disk_s_layers_are_mounted_under_the_overlay_top_first` | | `gpt_over_scratch_maps_the_table` | rewritten | `nvx-rs/src/guest/mappings/mapper/tests.rs::targets_are_created_only_on_the_root_s_own_mounts`; `nvx-rs/src/guest/init/boot/tests.rs::layered_cold_boot_maps_through_handles_opened_before_its_only_switch` | | `gpt_over_scratch_with_untrusted_lowers` | rewritten | `nvx-rs/src/guest/root/tests.rs::both_overlay_modes_assemble_on_both_uppers_with_their_own_option_sets` | | `invalid_configuration_fails_its_stage` | retired | The adapter's configuration stage and its status | @@ -244,7 +245,7 @@ The adapter's README says what each proves. | `a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease` | retired | Tombstones, generations and leases leave with the old clone path (#1121) | | `clones_at_once_keep_their_own_uppers_and_machine_ids` | rewritten | SL5.1c's concurrent restores of one snapshot; `test-microvm --scenario scratch-snapshot` | | `clones_at_once_see_only_their_own_exports` | rewritten | SL5.1c's concurrent restores of one snapshot; `test-microvm --scenario platform-aggregate-export` | -| `each_command_reattaches_to_the_clone_from_a_new_process` | rewritten | `nvx-rs/src/host/vm/tests/reattach.rs::reattaching_returns_the_held_owned_child_and_authenticated_connection`; SL5.1b's lifecycle smoke test. The adapter's probes retire | +| `each_command_reattaches_to_the_clone_from_a_new_process` | rewritten | `nvx-rs/src/host/vm/tests/reattach.rs::reattaching_returns_the_held_owned_child_and_authenticated_connection`; `nvx-rs/tests/vm_lifecycle.rs::a_cold_vm_is_reattached_from_saved_values_and_stopped`. The adapter's probes retire | | `one_generation_serves_clones_of_each_processor_count_from_its_boot_online_count_to_its_capacity` | rewritten | `test-microvm --scenario restore-processors`; `test-microvm --scenario platform-restore-processors` | | `one_generation_serves_clones_of_each_upper_and_memory_target` | rewritten | `test-microvm --scenario scratch-snapshot`; `test-microvm --scenario restore-memory`. Leases retire (#1121) | | `one_generation_serves_clones_of_either_format_over_either_upper` | rewritten | `test-microvm --scenario snapshot-tiers`, which SL5.1c rewrites; `nvx-rs/src/host/snapshot/tests.rs::snapshot_accepts_matching_real_disk_geometry_and_every_resource_boundary` | @@ -293,7 +294,9 @@ check the adapter's own code, and CI runs in its configurations of the adapter. ## MicroVM scenarios `nvx.py test-microvm` runs these scenarios, which use no adapter: CI runs those of the default -suite in its shards, and the public sandbox scenarios in its guest and sandbox jobs. +suite in its shards, and the public sandbox scenarios in its guest and sandbox jobs. The public +`sandbox-gpt-layer` and `vm-lifecycle` scenarios came with SL5.1b, after this matrix; +[Cold boots](#cold-boots) says what they check. | Scenario | Fate | Note | | --- | --- | --- | @@ -318,7 +321,7 @@ suite in its shards, and the public sandbox scenarios in its guest and sandbox j | `l3-l4-egress-port-ranges` | kept | | | `l3-l4-egress-protocols` | kept | | | `lifecycle` | kept | | -| `managed-exec-config` | kept | SL5.1b keeps it running, or gives it its fate | +| `managed-exec-config` | kept | A [cold cell](#cold-boots): the EROFS layer over a scratch upper, through `provision` and `start` | | `managed-lifecycle` | kept | | | `network-snapshot` | kept | | | `platform-aggregate-export` | kept | | @@ -332,9 +335,10 @@ suite in its shards, and the public sandbox scenarios in its guest and sandbox j | `restore-processors` | kept | | | `run-console-restore` | retired | SL3.8b removes `nvx-rs run`'s restore of a raw OpenVMM snapshot and the boot console that it reattached on the terminal: `run --restore-snapshot` restores a named snapshot through `snapshot restore`'s path | | `sandbox-blocks` | kept | | -| `sandbox-lifecycle` | kept | SL5.1b keeps it running, or gives it its fate | -| `sandbox-proxy-only` | kept | SL5.1b keeps it running, or gives it its fate | -| `sandbox-ram-upper` | kept | SL5.1b keeps it running, or gives it its fate | +| `sandbox-gpt-layer` | kept | The [cold cells](#cold-boots) of a GPT disk, over a scratch upper and over a RAM upper, through `run`, and `provision` and `start` | +| `sandbox-lifecycle` | kept | A [cold cell](#cold-boots): the EROFS layer over a scratch upper, through `provision` and `start`, which also reattach to the VM from a new process for each command | +| `sandbox-proxy-only` | kept | Its proxy-only network, over the EROFS layer and a scratch upper; it is no [cold cell](#cold-boots) of its own | +| `sandbox-ram-upper` | kept | A [cold cell](#cold-boots): the EROFS layer over a RAM upper, through `run`, and `provision` and `start` | | `scratch-snapshot` | kept | | | `smp` | kept | | | `smp-lapic` | kept | Runs only when a run names it | @@ -348,8 +352,35 @@ suite in its shards, and the public sandbox scenarios in its guest and sandbox j | `untiered-network-policy` | rewritten | SL5.1c rewrites it on strict snapshots, keeping its DNS, egress, forwards and publication; only its raw-snapshot parts retire | | `virtio-features` | kept | | | `virtio-net` | kept | | +| `vm-lifecycle` | kept | A [cold boot](#cold-boots) through the public API: `nvx-rs/tests/vm_lifecycle.rs::a_cold_vm_is_reattached_from_saved_values_and_stopped` | | `workload-identity` | kept | | +### Cold boots + +The CLI's twelve cold cells are the two image formats, an EROFS layer and a GPT disk, each over +each upper, a scratch image and RAM, on each of KVM, MSHV and WHP: CI's guest and sandbox job of +each backend runs the public scenarios and the steps that the table names. A cell passes once +`sandbox run`, and a managed sandbox's `provision` and `start`, each boot its image over its upper +in a VM that nvx-init manages, and `/sbin/nvx-sandbox-smoke` passes its checks there. The +scenarios also check that each of their VMs stops with a successful outcome report and a complete +teardown; CI's layer smoke test checks the status of the probe that its run runs. + +| Image | Upper | `run` | `provision` and `start` | +| --- | --- | --- | --- | +| EROFS | scratch | CI's "Run Ubuntu sandbox layer smoke test" step, on Linux and on Windows | `test-microvm --scenario sandbox-lifecycle`; `test-microvm --scenario managed-exec-config` | +| EROFS | RAM | `test-microvm --scenario sandbox-ram-upper` | `test-microvm --scenario sandbox-ram-upper` | +| GPT | scratch | `test-microvm --scenario sandbox-gpt-layer` | `test-microvm --scenario sandbox-gpt-layer` | +| GPT | RAM | `test-microvm --scenario sandbox-gpt-layer` | `test-microvm --scenario sandbox-gpt-layer` | + +`sandbox-gpt-layer` writes its own GPT disk, whose layer partition a guest fills with an ext4 copy +of the tree of the EROFS layer, and checks that the guest's root has that partition as its lower +directory. `test-microvm --scenario vm-lifecycle` cold-boots a VM through the public API instead: +`nvx-rs/tests/vm_lifecycle.rs::a_cold_vm_is_reattached_from_saved_values_and_stopped` creates and +spawns it, with nvx-init over the EROFS layer and a scratch upper, lets the launch go, reattaches +to it twice from the values that it saved, and stops it through its guest, which must exit without +a forced termination and report a complete teardown. No cell or smoke test captures or restores a +snapshot: SL5.1c adds those. + ## CI These are the steps of CI's workflows and actions that check something, and the Python tests of @@ -390,10 +421,10 @@ their own. The tests that a step runs have rows of their own where they concern | `.github/workflows/ci.yml` | crate-release: Check the nvx package | kept | | | `.github/workflows/ci.yml` | nvx-microvm-adapter-whp: the runtime adapter's job on WHP | retired | The adapter's scenarios, whose fates are above | | `.github/workflows/run-nvx-microvm-tests.yml` | Run NVX microVM tests on Linux and on Windows | kept | The default suite's scenarios, by shard and on the debug kernel | -| `.github/workflows/run-nvx-microvm-tests.yml` | Run public managed sandbox acceptance on Linux and on Windows | kept | | +| `.github/workflows/run-nvx-microvm-tests.yml` | Run public managed sandbox acceptance on Linux and on Windows | kept | Runs `sandbox-gpt-layer` and `vm-lifecycle` too, since SL5.1b: see [Cold boots](#cold-boots) | | `.github/workflows/run-nvx-microvm-tests.yml` | Run Ubuntu and Azure Linux guest smoke tests on Linux and on Windows | kept | | | `.github/workflows/run-nvx-microvm-tests.yml` | Run extended Ubuntu guest tests on Linux/KVM | kept | | -| `.github/workflows/run-nvx-microvm-tests.yml` | Run the Ubuntu sandbox layer, live-share, managed live-share, caller-owned live-share, concurrent shares and share access policy tests on Linux and on Windows | kept | | +| `.github/workflows/run-nvx-microvm-tests.yml` | Run the Ubuntu sandbox layer, live-share, managed live-share, caller-owned live-share, concurrent shares and share access policy tests on Linux and on Windows | kept | The layer smoke test is the `run` of a [cold cell](#cold-boots) | | `.github/workflows/run-nvx-microvm-tests.yml` | Build the runtime adapter's guest agent on Linux; Download the runtime adapter's guest agent on Windows | retired | The adapter's guest agent | | `.github/workflows/run-nvx-microvm-tests.yml` | Run runtime adapter scenarios on Linux and on Windows | retired | The adapter's scenarios, whose fates are above | | `.github/workflows/run-platform.yml` | Run benchmark: the acceptance test, the performance suite, the multi-vCPU shell restore and the device operation rates | kept | |