Skip to content

cli: restore named snapshots once with run (SL3.8b) - #1489

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
feature/nvx-refactorfrom
agents/slim-sl3-8b-run-restore
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
feature/nvx-refactorfrom
agents/slim-sl3-8b-run-restore

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1114 (SL3.8b: run --restore-snapshot restores a one-off VM from a strict snapshot).

It builds on SL3.8a1 (#1486) and SL3.8a2 (#1487), which add the restored start and the snapshot restore command whose restore path and options this one-off reuses; both have merged.

What and why

nvx-rs run --restore-snapshot PATH handed a raw OpenVMM snapshot directory to OpenVMM with the options of a cold boot, outside the store, the locks, and the restore driver of the named snapshots that snapshot capture (SL3.7b) publishes and snapshot restore (SL3.8a) restores. This PR does SL3.8b's Scope, as Pedro Henrique Penna (@ppenna) decided it (option A: the layered restore of SL3.8a, which #1114's Notes call for, since a restored VM gets its image, upper and workload at activation):

  • One restore path. run --restore-snapshot NAME --store-dir STORE --state-dir DIR restores the named snapshot once through snapshot restore's own path. sandbox::restore_once provisions a managed sandbox in DIR and starts it from the snapshot, under the snapshot's lock, through the launch journal and the blocking restore driver, which forwards the workload's output. It stops the VM once the workload has exited with status 0, deprovisions the sandbox (removing DIR) once it is provisioned, whatever happened, and ends with the workload's status.

  • --state-dir, and why. The restore hard-links the snapshot's files into the sandbox's directory, so that directory must be on the store's filesystem. SL3.8a2 refuses a restored sandbox's directory inside the store, before snapshot restore provisions it, at each start and at deprovision (outside_store, through LockedSnapshot::remove_view), so the one-off can no longer keep its sandbox below the store, as an earlier draft of this PR did in STORE/runs/<hex>. Only the caller can name a directory outside the store on its filesystem, so run takes snapshot restore's own --state-dir, with its definition and parsing. Its rules: it must lie outside the store (checked with SL3.8a2's outside_store before anything is read or provisioned) and on the store's filesystem; it must not exist, and an empty path, which names the working directory, is refused, since the run creates it and removes it once it ends, so a sandbox or directory that it names already is never removed. The existence check guards against reusing an existing state directory. Preparing the state directory is shared with every sandbox lifecycle command, and outside this PR's scope. On Linux, the VM's sockets in it must fit a Unix socket path (below 108 bytes), as for snapshot restore: a longer one fails at the VM's creation, and the run removes the directory. The 66-byte limit on the store's path, which the earlier draft's runs/ imposed, no longer applies.

  • Exit status. A restored workload that doesn't exit with status 0 still fails the start, which ends the VM and its launch. But the start that run asks for (forward_status, carried in the start's Restore) ends with the workload's status, which run exits with, whatever ended the workload. That's what sandbox run and sandbox exec do, which exit with the guest's status whatever its category: 128 plus the signal's number for a workload that a signal ends. snapshot restore, and each later sandbox start of a restored sandbox, keep SL3.8a's error, such as error[other]: the restored workload ended with Exit status N, and status 1.

  • Restore inputs reuse snapshot restore's options. --state-dir, --layer, --layer-format, --scratch, --upper, --ram-upper-percent, --entrypoint, --arg, --hostname, --workload-user and --timeout are sandbox's option definitions, names and parsing (sandbox::OPTIONS and sandbox::Options::take), taken from snapshot restore's own option list (snapshots::restore_options), so a restore of run takes none that snapshot restore refuses, such as SL3.9's exec-only --wait. run keeps its own --hypervisor and --memory-mib, which defaults to the sandbox's 256 MiB for a restore. It adds no option that neither snapshot restore nor today's run has: --store-dir and --state-dir are snapshot restore's.

  • One preparation, so both restore commands accept and refuse the same workloads. snapshot restore's own preparation moves into snapshots::restore_source, which both commands call before they provision anything. It resolves the store, refuses a state directory in the store, checks the workload as the exec request that each restored start sends, and opens the snapshot, returning the restore's Source and the snapshot's NIC. Since cli: restore managed sandboxes from named snapshots (SL3.8a2) #1487's last fix, a restored sandbox's launch takes the default workload, and its own workload travels over NVXC as an exec request. So run --restore-snapshot, like snapshot restore, takes an entrypoint and arguments with whitespace and quotes, which sandbox provision refuses. And it refuses a workload that isn't a valid exec request, such as a relative entrypoint or 65 arguments, before it opens the store or provisions anything; before this, run would have provisioned and launched the VM first. outside_store keeps its original visibility, since only restore_source calls it now.

  • Refusals. A cold boot refuses --store-dir, --state-dir and the restore inputs. A restore refuses the options of a cold boot that the snapshot or its sandbox fixes: --guest, --memory-capacity-mib, --processors and --cpu-profile (the snapshot's), --net (the snapshot's NIC), the share options (run's policy paths mean other paths than sandbox's, and shares are not in the Scope), --outcome-report, --cmdline and --dry-run.

  • Raw snapshots are refused before launch. --restore-snapshot without --store-dir, or with a value that is no snapshot name, such as a directory, fails before anything is read or launched: --restore-snapshot requires --store-dir: it names a snapshot of --store-dir, which `snapshot capture` captured, and no longer a raw OpenVMM snapshot.

  • Raw-only options go. --restore-network-policy (as the Scope says), and --restore-ready-path, --restore-processors and --restore-memory-mib, which only the raw path used: the named restore owns its restore-ready endpoint, takes its processors from the snapshot's key (the online count that vm::create restores), and its memory target from --memory-mib, a multiple of 128 between the snapshot's boot memory and its capacity.

  • Network. The snapshot fixes the NIC. For a snapshot with a NIC, run's DNS, egress, forwards and published listeners, which need --network-profile, take the NIC's address and become the restore's own network policy through the normal lowering (sandbox_network). snapshot capture gives its VM no NIC (capture.rs, network: None), and Snapshot requires the restore's NIC presence to match, so network options for a snapshot without a NIC are refused before anything is provisioned: the snapshot has no NIC, so its restore takes no network options. Limitation: only fake-backed tests cover the policy that reaches the guest of a networked restore; a NIC for snapshot capture is out of scope (follow-up on nvx-rs slimming 3.2.2: the CLI captures and restores named snapshots #1114).

  • What a run leaves when it cannot clean up (measured on KVM, MSHV and WHP). A run that kill -9 (or Stop-Process -Force on Windows) ends while its workload runs leaves --state-dir (config.json, the launch journal launch.json, control.capability, the control endpoint, openvmm.log, sandbox.lock, the openvmm/ view, whose hard links keep the snapshot's files at 2 links, and on Linux restore-ready.sock) and its VM: OpenVMM was still running 120 s later, 100 s after its workload ended, and nothing ends it but a command of the sandbox. The snapshot's lock goes with the run, so another one-off of the same snapshot restored meanwhile, and a later run refuses the existing directory. The start was unfinished, and

    nvx-rs sandbox deprovision --state-dir DIR

    alone ended that VM, through its lock's recovery of the start's launch journal, and removed DIR, the snapshot's links back to 1. (nvx-rs sandbox stop --state-dir DIR also ends it, through the same recovery, and then reports sandbox-not-running.) Once the workload has exited with status 0, the start has committed and the VM runs as a started sandbox does, which deprovision refuses (sandbox must be stopped before deprovision): a run killed then, or one that Ctrl-C interrupts while it stops the VM, which also leaves the sandbox and its running VM and prints a warning that names the directory, needs

    nvx-rs sandbox stop --state-dir DIR
    nvx-rs sandbox deprovision --state-dir DIR

    as shown with a restored sandbox of snapshot restore on KVM and MSHV; the Ctrl-C case is from the code, since that window is too short to time on a VM. Ctrl-C while the workload runs ends the VM and removes the directory (exit 130). doc/usage.md says all this. A sweep of leftover directories would be the restore reconciliation that SL3.8b's "Not in this PR" excludes.

  • Code that goes with the raw path: run's restore body and console reattachment, NetworkOptions::extend_arguments, and snapshot::manifest::read_attachments with its re-export, which only the raw path used. Since SL3.4b (sandbox: run a workload once through the managed VM API (SL3.4b) #1490) runs sandbox run's workload through the managed VM API, the raw restore was also the last caller of host::cli::sandbox::one_shot, which ran OpenVMM on the terminal. The module goes (run_openvmm, OneShot, end_interrupted and spawn_error, with its mod line and the re-export of run_openvmm in sandbox/mod.rs). So do the Windows platform::spawn_foreground and its helper standard_handle, which only one_shot used, with their cli-gated imports and the re-export in platform/windows/mod.rs. Without them, Clippy with warnings denied fails on this stack: the import of run_openvmm is unused, and run_openvmm, OneShot, end_interrupted, spawn_error, spawn_foreground and standard_handle are never used.

  • Docs: doc/usage.md (run's synopsis, options, restore and leftover paragraphs; snapshot restore's pointer to it), doc/run.md (raw restores go through OpenVMM directly; the networked-snapshot and readiness examples use ./bin/openvmm), nvx-rs/README.md (### run, the differences from nvx.py, the fake's restores).

Not in this PR (per #1114): blockless restores; a NIC in snapshot capture; automatic removal of a killed one-off's sandbox; shares for a restore.

Takeover. Enrique Saurez (@esaurez)'s partial implementation on esaurez/private-sl38b-0325-b2 (ef52ba6b5a, checkpoint "WIP - Enrique Saurez (@esaurez) (partial implementation)" on #1114, untouched since 2026-11-09) restored blockless snapshots through unmerged prerequisite branches (esaurez/private-blockless-driver-0325, esaurez/private-blockless-snapshot-0325). This PR keeps its refusal of raw snapshots and of the options that a restore fixes, and restores through SL3.8a's layered path instead, so it needs nothing unmerged beyond SL3.8a.

Size. 210 core lines (estimate 150, stop 300), measured against feature/nvx-refactor (64ad98afacc): host/cli/run.rs 152, host/cli/sandbox/mod.rs 30 (restore_once), host/cli/snapshots.rs 15 (restore_source, which snapshot restore's preparation moves into, and restore_options made visible to run), and host/cli/sandbox/lifecycle/start.rs 13 (forward_status). Counted files: 4 (the estimate was 3). host/cli/network.rs, host/snapshot/manifest.rs and host/template/mod.rs only lose code that leaves with the raw restore, and don't count, as for SL3.4b. Nor do host/cli/sandbox/one_shot.rs, which this PR deletes, and host/platform/windows/spawn.rs and host/platform/windows/mod.rs, which only lose spawn_foreground and standard_handle. The deletions add no core lines. host/cli/mod.rs changes rustdoc only. BOOT_CONTRACT_VERSION, CONTROL_CONTRACT_REVISION and the public API are unchanged (python3 scripts/public_api.py passes).

Validation

At this head (ce25a7fa114, on feature/nvx-refactor 64ad98afacc, where SL3.8a1 and SL3.8a2 have merged), on Windows and on nvx-kvm-test (KVM, Azure Linux 3, kernel 6.6), these passed:

  • cargo fmt --check and the five Clippy variants below with -D warnings.
  • The unit tests of host::cli::run, host::cli::sandbox (with the restore tests), host::cli::snapshots, host::snapshot::manifest and host::spawn, plus host::platform::windows on Windows: 122 passed on Windows and 92 on Linux.
  • The cli integration tests: 110 passed on Windows and 119 on Linux.
  • NvxRsTests with NVX_RS set (a debug build on Windows, the release build on Linux), public_api.py (on KVM), check_test_layout.py and check_markdown_links.py.
  • A smoke test on a real VM on nvx-kvm-test, with this head's release nvx-rs and Alpine initramfs, all after snapshot capture base:
    • run --restore-snapshot base of the Ubuntu EROFS layer over a RAM upper ran /bin/echo 'a "restored" sandbox' (status 0), and, as daemon, a /bin/sh -c script with spaces that exited with 7, which run forwarded.
    • snapshot restore ran the same workload.
    • Both commands refused a relative entrypoint before provisioning anything, and run refused the raw snapshot.
    • No state directory, VM or extra link to the snapshot's files remained.

At 824fa351753, on SL3.8a2 (12f0f38578f, on feature/nvx-refactor d4edeeb9ab9, with SL3.4b), on Windows and on nvx-kvm-test (KVM, Azure Linux 3), these passed:

  • cargo fmt --check and the five Clippy variants below with -D warnings.
  • The unit tests of host::cli::run, host::cli::sandbox, host::cli::snapshots, host::snapshot::manifest and host::spawn, plus host::platform::windows on Windows: 117 passed on Windows and 88 on Linux.
  • The cli integration tests: 110 passed on Windows and 119 on Linux. One Windows run, at the previous push of this rebase, failed stop_forces_the_verified_vm_after_its_grace_period_even_after_an_acknowledgement (cannot confirm exit of VM process, a sandbox stop --timeout 1 that this PR doesn't change); it passed alone three times and in every full run since.
  • NvxRsTests of scripts/test_nvx_tools.py with NVX_RS set, as CI's CLI jobs run them: a debug build on Windows, the release build on Linux. They include test_usage_documents_every_option_of_nvx_rs, test_run_help_states_each_guest_memory_default and test_the_real_nvx_rs_prints_what_nvx_py_reads, which check run's options, help and docs and run only with NVX_RS set.
  • public_api.py (on KVM), check_test_layout.py, Ruff on test_nvx_tools.py, and the scenario-wiring tests: test_microvm_tests (on Windows and Linux) and test_ci_runs_each_default_microvm_suite_in_its_shards.

The table's rows ran at d2a2ef3266d, on SL3.8a2 bf63b6aa. 824fa351753 carried that change, apart from the test imports that SL3.4b changed and the restore tests that SL3.8a2 moved to start/tests/restore.rs. It added the deletions above and two changes from Copilot's review: the forwarded status of a workload that a signal ends, and the retired Python terminal test. This head adds the shared restore preparation (snapshots::restore_source) and its tests.

Check Result
cargo fmt --check; Clippy -D warnings: --all-targets --all-features, --lib, --lib --no-default-features, --lib --bins --features cli, --lib --bins --no-default-features --features testing Passed on Windows and on Linux (nvx-kvm-test)
cargo test --all-features --no-fail-fast Linux, nvx-mshv-test (kernel 6.18): every target passed (lib 2316, cli 102). Linux, nvx-kvm-test (kernel 6.6): the lib's 28 guest:: tests fail as they do at feature/nvx-refactor on that host (kill ProcessGroup: Invalid argument), and nothing else does. Windows: every test passed in one of two full runs; the others failed only under the full suite's load and pass alone (host::cli::store::tests::recovery::process::a_busy_endpoint_cannot_extend_recovery_or_signal_its_live_process, 312 ms against 250 ms; exec_lock::exec_processes_contend_but_other_sandboxes_and_stop_proceed, a stop with --timeout 0.1), or are the Windows timer bug that 3340bbd7ee4 fixes on the feature branch after SL3.8a2's base (host::restore_driver::tests::ready_has_sixty_seconds_and_start_thirty_from_ready_for_each_pending_accept_and_read, Expired at 59.9998 s)
cargo test (default features), cargo test --lib --no-default-features, cargo doc with -D warnings, MSRV 1.89, macOS check, artifact bundling, python3 scripts/public_api.py, runtime adapter protocol-only and host-only configurations Passed on Linux; tests, doc and MSRV also on Windows
check_test_layout.py, check_markdown_links.py, Ruff check and format, Pyright (Linux and Windows), compileall Passed
python -m unittest test_microvm_tests, test_nvx_tools, test_test_changes, test_test_layout, test_markdown_links (1094 tests), test_adversarial.py, nvx.py ... --help Passed on Windows; microvm_shards.json re-planned with nvx.py plan-microvm-shards
Real VMs: nvx-kvm-test (KVM) and nvx-mshv-test (MSHV), Azure Linux 3 virtual machines, and WHP on a Windows 11 developer machine; OpenVMM 14dc4116, the kernel of release v0.1.0-dev.53ad09678bc1, this tree's Alpine initramfs; this head's tree snapshot capture base, then run --restore-snapshot base --state-dir ... restored the release's Ubuntu EROFS layer and a GPT image of its tree, over a RAM upper and over a fresh scratch (GPT over a scratch on KVM and MSHV), as nobody and as daemon (--workload-user 1:1; also 65534:65534, and snapshot restore as daemon, on KVM and MSHV), forwarded the workload's output and its status (0, 7, and 1 for /usr/bin/false), and left no directory, VM, or link to the snapshot's files behind; snapshot restore of /usr/bin/false still failed with error[other]: the restored workload ended with Exit status 1. Refused before launch: the store's raw OpenVMM snapshot directory with and without --store-dir, network options for the NIC-less snapshot, a missing, existing or in-store --state-dir, and --wait; on KVM and MSHV also --processors, --restore-network-policy, a missing snapshot, and --state-dir without a restore. On KVM and MSHV also: --memory-mib 384 of a snapshot of 512 MiB capacity (MemTotal 372564 kB against 241492 kB), three concurrent one-offs of one snapshot, SIGINT (exit 130, directory removed), and an 89-byte --state-dir, whose socket fails the VM's creation and which the run removes. The kill -9 results above are from KVM, MSHV and WHP at this tree.

Found on the way, not caused here: --workload-user 1000:1000 fails a restore at activation (NVX-MANAGED-ERROR: stage=data status=71, guest exit 125), for run --restore-snapshot as for snapshot restore: the NVX Ubuntu image has only root, daemon and nobody, and a layered root needs an existing account (protocol::features::WORKLOAD_ACCOUNT), so a cold sandbox start fails the same way at the base. It is a known pre-existing limitation, which Pedro Henrique Penna (@ppenna) is recording as a follow-up; the checks above use 1:1 (daemon) and 65534:65534 (nobody).

Left to CI: the microVM scenario matrix (test-microvm on kvm, mshv and whp), the guest builds and their determinism check, and the benchmarks. The openvmm gitlink does not move.

Changes to existing tests (tests-modified)

check_test_changes.py: 50 changes to existing tests and 11 added, none relocated, against feature/nvx-refactor (64ad98afacc). Every other test of the area is kept.

  • host::cli::sandbox::lifecycle::start::tests::restore, where SL3.8a keeps its restore tests and this PR adds its four: the fixture's start delegates to a new start_with, which also takes forward_status (start passes false, SL3.8a's behavior); a_restore_whose_spawn_cannot_confirm_that_its_vm_ended_recovers_it_before_its_snapshot_goes_free names forward_status: false in the Restore that it builds, and a_restored_start_refuses_a_share_of_its_snapshot_store passes it to start, as sandbox start does. No assertion changes. The fake guest's serve gains a signaled behavior, whose workload SIGTERM ends (status 143, category signal), for this PR's test of the statuses that a one-off forwards.
  • host::platform::windows::tests: foreground_children_inherit_only_the_allowed_handles_besides_their_standard_ones, which covered only spawn_foreground, retires with it, and batch_files_are_refused_before_cmd_could_read_their_arguments drops its spawn_foreground case and keeps its spawn_with_stdio one.
  • scripts/test_nvx_tools.py: NvxRsTests.test_run_leaves_openvmm_the_terminal_to_restore retires with the raw restore and run_openvmm, along with the select import that only it used. It ran a raw restore on a pseudo-terminal to show that nvx-rs run left OpenVMM the terminal on its standard error. No nvx-rs run gives OpenVMM the terminal anymore: a cold boot gives it a pipe for its standard input and the run's log for its standard output and error (host::spawn), and a named restore starts it detached through the restore driver. The test ran only with NVX_RS set, as in CI's CLI jobs.
  • host::cli::run::tests::restore_policy_paths_attach_to_their_shares (proptest): retired with the raw restore's share and policy-path arguments; a restore refuses shares.
  • host::cli::tests::usages_are_one_line_and_helps_list_every_option: reads run's options from run::options(), which adds the restore inputs that sandbox defines.
  • host::snapshot::manifest::tests: the_attachments_that_openvmm_writes_are_read_in_their_order and its helper attachment read the attachments through SnapshotManifest::read, the remaining reader, so they no longer need the cli feature. the_attachments_of_any_snapshot_are_read is rewritten as malformed_attachments_are_refused, which keeps its attachment bound and non-UTF-8 checks through SnapshotManifest::parse; its acceptance of non-template manifests, the helper terminal_console_snapshot, and the_attachments_of_unreadable_manifests_are_refused retire with read_attachments, whose missing, foreign-format and malformed manifests SnapshotManifest::read's own tests cover for templates.
  • cli/dry_run: dry_runs_print_the_openvmm_command_and_launch_nothing, run_dry_runs and dry_run retire the raw restores' dry runs, which a restore now refuses: "run: restore", "... of a boot console on the terminal", "... of a boot console on the terminal with every option", "... of a disconnected boot console", "... without guest artifacts", "... with every option", "... of the Alpine guest", "... with equal processors", "... ignores the memory", "... paths", "... with shares", "... with shares and a command line", "... of a networked snapshot", "... with a network policy", "... with an egress policy file", "... that replaces the network policy", "... retains quoted command line and delimiter", "... without the budget"; the helper with_snapshot and the raw snapshot fixture go.
  • cli/one_shot: one_shot_runs_exit_with_the_workload_or_openvmm_status, run_launches and fails_a_cold_boot retire the raw restores' launches ("run: restore", "... without guest artifacts", "... of a boot console on the terminal", "... of a disconnected boot console", "... outcome report", "... of a networked snapshot", "... that replaces the network policy").
  • cli/openvmm: the raw restore's expectations and snapshot builders go (restore, TERMINAL_CONSOLE, DISCONNECTED_CONSOLE, NO_CONSOLE, snapshot, snapshot_manifest).
  • cli/usage: "run: restore memory not an integer" and "run: restore processors not an integer" become "... option removed" (unrecognized arguments); "run: restore processors not a choice" retires; "run: missing path" becomes "run: missing snapshot name"; new: the removed --restore-ready-path and --restore-network-policy, --wait, which snapshot restore refuses too, a restore upper that is not a choice, a zero restore timeout. HELP lists --restore-snapshot NAME, --store-dir PATH, --state-dir STATE_DIR, and the restore inputs.
  • cli/validation (run_refusals): "run: net of a restore" and "run: net without profile of a restore" restore a named snapshot of a store and expect --net is not accepted for restore; "run: memory capacity of a restore" expects --memory-capacity-mib is not accepted for restore; retired with their options or the manifest reading: "run: ready path without snapshot", "run: network policy replacement without snapshot", "run: restore processors without snapshot", "run: restore memory without snapshot", "run: restore processors above processors", "run: restore of a missing snapshot", "... without a dry run", "run: restore of a snapshot without a manifest", "run: restore of a malformed manifest", "run: restore of a manifest of another format"; new: a store, a state directory or restore inputs without a snapshot, a raw snapshot without a store and in place of a name, a missing store, a restore without a state directory, with an empty one, into an existing one and into the store, a workload that isn't a valid exec request (a relative entrypoint, with a missing store, which shows that the check comes first), and a dry run, a share, a command line and processors of a restore.
  • cli::run_guides_failed_cold_boots_on_cpus_that_no_built_in_profile_serves: its restore case, after which no guidance prints, restores a named snapshot of a missing store instead of a raw snapshot.
  • cli/download: restores_download_nothing_for_snapshots_that_they_cannot_read is rewritten as restores_download_nothing_for_snapshots_that_they_cannot_open: a restore without --store-dir and one of a missing store download nothing, in place of an unreadable raw manifest.
  • microVM scenario run-console-restore, which restored a raw snapshot with its boot console on the terminal through nvx-rs run, retires: its script, runner, marker and dispatch in scripts/nvx_tools/microvm_tests.py, its tests (RunConsoleRestoreScenarioTests) in scripts/test_microvm_tests.py, and its entries on kvm, mshv and whp in microvm_shards.json, which nvx.py plan-microvm-shards re-planned. The real-VM checks above cover the named restore. In the case matrix (doc/acceptance-case-matrix.md, nvx-rs: build every feature set, compile the documented examples, and give every old test a fate (SL5.1a) #1488), its row becomes retired, and the summary of the microVM scenarios counts 48 kept and 1 retired. No other row names a test that this PR changes or deletes.

@ppenna Pedro Henrique Penna (ppenna) added tests-modified Changes existing tests, test data, or test wiring feat-nvx-refactor labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 50 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified helper fn start in host::cli::sandbox::lifecycle::start::tests::restore, impl Fixture`` nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:108 nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:112
modified helper fn serve in host::cli::sandbox::lifecycle::start::tests::restore nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:715 nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:922
modified helper fn attachment in host::snapshot::manifest::tests (moved, and it now compiles where feature = "host" and test hold, not where feature = "cli" and feature = "host" and test did) nvx-rs/src/host/snapshot/manifest/tests.rs:886 nvx-rs/src/host/snapshot/manifest/tests.rs:884
modified helper fn dry_run in cli/dry_run (tests) nvx-rs/tests/cli/dry_run.rs:52 nvx-rs/tests/cli/dry_run.rs:47
modified helper fn run_dry_runs in cli/dry_run (tests) nvx-rs/tests/cli/dry_run.rs:346 nvx-rs/tests/cli/dry_run.rs:335
modified helper fn fails_a_cold_boot in cli/one_shot (tests) nvx-rs/tests/cli/one_shot.rs:128 nvx-rs/tests/cli/one_shot.rs:127
modified helper fn run_launches in cli/one_shot (tests) nvx-rs/tests/cli/one_shot.rs:149 nvx-rs/tests/cli/one_shot.rs:144
modified helper const USAGE_ERRORS in cli/usage (tests) nvx-rs/tests/cli/usage.rs:20 nvx-rs/tests/cli/usage.rs:20
modified helper const HELP in cli/usage (tests) nvx-rs/tests/cli/usage.rs:675 nvx-rs/tests/cli/usage.rs:726
modified helper fn run_refusals in cli/validation (tests) nvx-rs/tests/cli/validation.rs:744 nvx-rs/tests/cli/validation.rs:744
modified imports the imports of cli/dry_run (tests) nvx-rs/tests/cli/dry_run.rs:5 nvx-rs/tests/cli/dry_run.rs:5
modified imports the imports of cli/one_shot (tests) nvx-rs/tests/cli/one_shot.rs:6 nvx-rs/tests/cli/one_shot.rs:6
modified imports the imports of cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:7 nvx-rs/tests/cli/openvmm.rs:6
modified imports the imports of scripts/test_nvx_tools.py scripts/test_nvx_tools.py:1 scripts/test_nvx_tools.py:1
modified test fn a_restore_whose_spawn_cannot_confirm_that_its_vm_ended_recovers_it_before_its_snapshot_goes_free in host::cli::sandbox::lifecycle::start::tests::restore nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:396 nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:553
modified test fn a_restored_start_refuses_a_share_of_its_snapshot_store in host::cli::sandbox::lifecycle::start::tests::restore nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:563 nvx-rs/src/host/cli/sandbox/lifecycle/start/tests/restore.rs:721
modified test fn usages_are_one_line_and_helps_list_every_option in host::cli::tests nvx-rs/src/host/cli/tests.rs:14 nvx-rs/src/host/cli/tests.rs:14
modified test fn batch_files_are_refused_before_cmd_could_read_their_arguments in host::platform::windows::tests nvx-rs/src/host/platform/windows/tests.rs:443 nvx-rs/src/host/platform/windows/tests.rs:411
modified test fn the_attachments_that_openvmm_writes_are_read_in_their_order in host::snapshot::manifest::tests (moved, and it now compiles where feature = "host" and test hold, not where feature = "cli" and feature = "host" and test did) nvx-rs/src/host/snapshot/manifest/tests.rs:895 nvx-rs/src/host/snapshot/manifest/tests.rs:892
modified test fn run_guides_failed_cold_boots_on_cpus_that_no_built_in_profile_serves in cli (tests) nvx-rs/tests/cli.rs:2509 nvx-rs/tests/cli.rs:2509
modified test fn dry_runs_print_the_openvmm_command_and_launch_nothing in cli/dry_run (tests) nvx-rs/tests/cli/dry_run.rs:1271 nvx-rs/tests/cli/dry_run.rs:970
modified test fn one_shot_runs_exit_with_the_workload_or_openvmm_status in cli/one_shot (tests) nvx-rs/tests/cli/one_shot.rs:498 nvx-rs/tests/cli/one_shot.rs:331
removed helper fn terminal_console_snapshot in host::snapshot::manifest::tests nvx-rs/src/host/snapshot/manifest/tests.rs:916
removed helper fn with_snapshot in cli/dry_run (tests), impl DryRun`` nvx-rs/tests/cli/dry_run.rs:84
removed helper fn restore in cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:684
removed helper static TERMINAL_CONSOLE in cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:695
removed helper static DISCONNECTED_CONSOLE in cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:706
removed helper static NO_CONSOLE in cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:717
removed helper fn snapshot in cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:720
removed helper fn snapshot_manifest in cli/openvmm (tests) nvx-rs/tests/cli/openvmm.rs:733
removed helper class RunConsoleRestoreScenarioTests in scripts/test_microvm_tests.py scripts/test_microvm_tests.py:4039
removed helper def RunConsoleRestoreScenarioTests._run in scripts/test_microvm_tests.py scripts/test_microvm_tests.py:4043
removed test fn restore_policy_paths_attach_to_their_shares in host::cli::run::tests, proptest!`` nvx-rs/src/host/cli/run/tests.rs:603
removed test fn foreground_children_inherit_only_the_allowed_handles_besides_their_standard_ones in host::platform::windows::tests nvx-rs/src/host/platform/windows/tests.rs:412
removed test fn the_attachments_of_any_snapshot_are_read in host::snapshot::manifest::tests nvx-rs/src/host/snapshot/manifest/tests.rs:932
removed test fn the_attachments_of_unreadable_manifests_are_refused in host::snapshot::manifest::tests nvx-rs/src/host/snapshot/manifest/tests.rs:975
removed test fn restores_download_nothing_for_snapshots_that_they_cannot_read in cli/download (tests) nvx-rs/tests/cli/download.rs:195
removed test def RunConsoleRestoreScenarioTests.test_scenario_restores_a_terminal_console_through_nvx_rs_run in scripts/test_microvm_tests.py scripts/test_microvm_tests.py:4077
removed test def RunConsoleRestoreScenarioTests.test_scenario_requires_the_boot_console_of_the_restore in scripts/test_microvm_tests.py scripts/test_microvm_tests.py:4124
removed test def RunConsoleRestoreScenarioTests.test_scenario_requires_the_snapshot_unchanged in scripts/test_microvm_tests.py scripts/test_microvm_tests.py:4128
removed test def RunConsoleRestoreScenarioTests.test_runner_dispatches_the_scenario in scripts/test_microvm_tests.py scripts/test_microvm_tests.py:4132
removed test def NvxRsTests.test_run_leaves_openvmm_the_terminal_to_restore in scripts/test_nvx_tools.py scripts/test_nvx_tools.py:2550

End-to-end tests

Change What Before After
modified helper assignment MICROVM_TEST_SCENARIOS in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:99 scripts/nvx_tools/microvm_tests.py:99
modified helper def _run_scenarios in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:10894 scripts/nvx_tools/microvm_tests.py:10825
removed file scripts/nvx_tools/microvm_test_scripts/run-console-restore.sh scripts/nvx_tools/microvm_test_scripts/run-console-restore.sh
removed helper assignment RUN_CONSOLE_RESTORE_MARKER in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:397
removed helper def run_run_console_restore in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:8695

CI test wiring

Change What Before After
removed scenario run-console-restore on kvm scripts/nvx_tools/microvm_shards.json
removed scenario run-console-restore on mshv scripts/nvx_tools/microvm_shards.json
removed scenario run-console-restore on whp scripts/nvx_tools/microvm_shards.json

Relocated unchanged: 0; added: 11. Compared the merge base 64ad98afacca with ce25a7fa114a.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Changes are needed to prevent deletion of a state directory created by another process, correctly forward signal exit statuses, and migrate a terminal-restoration regression test that now fails in Linux CLI validation.

3 open issues
What changed in this PR

This PR implements SL3.8b of #1114, routing run --restore-snapshot through the managed named-snapshot restore path introduced by #1486 and #1487.

Changes:

  • Adds one-off restoration with shared sandbox options, workload-status forwarding, and cleanup.
  • Rejects raw snapshots and incompatible options, removing the obsolete raw-restore implementation.
  • Updates documentation and tests, and retires the raw console-restore scenario.
File Description
scripts/​test_microvm_tests.py Removes tests for the retired console-restore scenario.
scripts/​nvx_tools/​microvm_tests.py Removes the raw console-restore runner and dispatch.
scripts/​nvx_tools/​microvm_test_scripts/​run-console-restore.sh Deletes the retired guest script.
scripts/​nvx_tools/​microvm_shards.json Removes the scenario and rebalances shards.
nvx-rs/​tests/​cli/​validation.rs Tests named-restore requirements and refusals.
nvx-rs/​tests/​cli/​usage.rs Updates help and option-parsing expectations.
nvx-rs/​tests/​cli/​openvmm.rs Removes raw-snapshot fixtures and expectations.
nvx-rs/​tests/​cli/​one_shot.rs Removes obsolete raw-restore launch cases.
nvx-rs/​tests/​cli/​dry_run.rs Removes unsupported restore dry-run cases.
nvx-rs/​tests/​cli/​download.rs Checks rejected restores download nothing.
nvx-rs/​tests/​cli.rs Updates the failed-restore CPU-guidance case.
nvx-rs/​src/​host/​template/​mod.rs Removes the raw attachment-reader re-export.
nvx-rs/​src/​host/​snapshot/​manifest/​tests.rs Moves attachment checks to the retained manifest reader.
nvx-rs/​src/​host/​snapshot/​manifest.rs Removes the raw-snapshot attachment reader.
nvx-rs/​src/​host/​cli/​tests.rs Checks dynamically shared run options.
nvx-rs/​src/​host/​cli/​snapshots.rs Exposes restore options and store-boundary validation internally.
nvx-rs/​src/​host/​cli/​sandbox/​mod.rs Adds one-off restore orchestration and cleanup.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​start/​tests.rs Tests restore policy, NIC refusals, and status forwarding.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​start.rs Adds optional workload-status forwarding.
nvx-rs/​src/​host/​cli/​run/​tests.rs Tests shared restore parsing and network handling.
nvx-rs/​src/​host/​cli/​run.rs Replaces raw restores with named one-off restores.
nvx-rs/​src/​host/​cli/​network.rs Removes obsolete raw network-argument generation.
nvx-rs/​src/​host/​cli/​mod.rs Updates the run command’s module documentation.
nvx-rs/​README.md Documents named restores and revised CLI behavior.
doc/​usage.md Documents restore options, cleanup, and interruption recovery.
doc/​run.md Moves raw-restore examples to direct OpenVMM commands.

Review effort: Balanced

Comment thread nvx-rs/src/host/cli/run.rs Resolved
Comment thread nvx-rs/src/host/cli/run.rs Resolved
`run --restore-snapshot PATH` handed a raw OpenVMM snapshot directory to
OpenVMM with the options of a cold boot, outside the store, the locks,
and the restore driver of the named snapshots that `snapshot capture`
captures and `snapshot restore` restores. It now takes the name of a
snapshot of `--store-dir` and restores it once, through the path of
`snapshot restore`: `sandbox::restore_once` provisions a managed
sandbox in `--state-dir`, starts it from the snapshot under the
snapshot's lock, through the launch journal and the blocking restore
driver, which forwards the workload's output, stops the VM once the
workload has exited with status 0, deprovisions the sandbox once it is
provisioned, whatever happened, and ends with the workload's status.

The restore inputs are options of `snapshot restore`, taken from its
own option list and parsed by `sandbox` for both: `--state-dir`,
`--layer`, `--layer-format`, `--scratch`, `--upper`,
`--ram-upper-percent`, `--entrypoint`, `--arg`, `--hostname`,
`--workload-user`, and `--timeout`, so a restore of `run` takes none
that `snapshot restore` refuses, such as the exec-only `--wait`;
`--memory-mib` takes the sandbox's default for a restore. Both commands
prepare a restore through one function, into which `snapshot restore`'s
own preparation moves: it resolves the store, refuses a state directory
in the store, checks the workload as the exec request that each
restored start sends, which may thus hold whitespace and quotes, and
opens the snapshot, before anything is provisioned. The restore
links the snapshot's files into `--state-dir`, which must therefore be
on the store's filesystem, and outside the store, which SL3.8a
requires of every restored sandbox, at its provision, start, and
deprovision. Since the run removes it, it must not exist, and an empty
path, which names the working directory, is refused: a sandbox that it
names is never removed. A cold boot refuses `--store-dir`,
`--state-dir`, and the restore inputs, and a restore refuses the
options of a cold boot that the snapshot or its sandbox fixes:
`--guest`, `--memory-capacity-mib`, `--processors`, `--cpu-profile`,
`--net`, the share options, `--outcome-report`, `--cmdline`, and
`--dry-run`.

A raw OpenVMM snapshot is refused before anything is read or launched:
`--restore-snapshot` without `--store-dir`, or with a value that is no
snapshot name, such as a path, fails with a message that says that the
option names a snapshot of the store that `snapshot capture` captured.
`--restore-processors`, `--restore-memory-mib`, `--restore-ready-path`,
and `--restore-network-policy` only served the raw restore, and go: the
named restore takes its processors from the snapshot's key and its
memory from `--memory-mib`, owns its restore-ready endpoint, and always
installs the restore's own network policy.

A restored workload that does not exit with status 0 still fails the
start, which ends its VM and launch, but the start that `run` asks for
ends with the workload's status, which `run` exits with, as `sandbox
run` does: 128 plus the signal's number for a workload that a signal
ends. `snapshot restore`, and each later `sandbox start` of a restored
sandbox, keep their error, such as "the restored workload ended with
Exit status N", and status 1.

The snapshot fixes the NIC. The restore of a snapshot with a NIC takes
the NIC's address, and lowers the run's DNS server, egress, forwards,
and published listeners, which need `--network-profile`, into the
restore's own policy, as `sandbox` lowers them; that of a snapshot
without one refuses network options before it provisions anything.
`snapshot capture` gives its VM no NIC today, so only fake-backed tests
cover the policy that reaches the guest of a networked restore.

A `run` that dies before it removes its sandbox, as `kill -9` ends it,
leaves `--state-dir` and its VM, which keeps running until a command of
the sandbox ends it, and the snapshot's lock goes with the `run`. While
the workload runs, the start is unfinished, and `sandbox deprovision`
ends the VM, through the recovery of the start's launch journal, and
removes the directory; once the workload has exited with status 0,
`sandbox stop` must end the VM first, as it must after Ctrl-C while the
run stops the VM. usage.md says so.

The raw restore's code goes with it: the restore body and console
reattachment of `run`, the OpenVMM arguments of the network options,
the reader of a snapshot manifest's attachments that only it used, and
`sandbox::one_shot`, which has run OpenVMM on the terminal for it alone
since `sandbox run` runs its workload through the managed VM API, with
the Windows `spawn_foreground` and `standard_handle` that only
`one_shot` used. The `run-console-restore` microVM scenario, which
restored a raw snapshot with its boot console on the terminal, is
retired with its shard entries.

esaurez's partial implementation on esaurez/private-sl38b-0325-b2
(ef52ba6) restored blockless snapshots through unmerged
prerequisites. This keeps its refusal of raw snapshots and of the
options that a restore fixes, and restores through the layered
sandbox path of SL3.8a instead.

Evidence, on nvx-kvm-test (KVM, Azure Linux 3, kernel 6.6) and
nvx-mshv-test (MSHV, kernel 6.18), virtual machines, and on WHP on a
Windows 11 developer machine, with OpenVMM 14dc4116, the kernel of
release v0.1.0-dev.53ad09678bc1, and this tree's Alpine initramfs:
`snapshot capture` published `base`, and `run --restore-snapshot base`
restored the release's Ubuntu EROFS layer and a GPT image of its tree,
over a RAM upper and over a fresh scratch (GPT over a scratch on KVM
and MSHV), as nobody and as daemon (`--workload-user 1:1`), forwarded
the workload's output and its statuses 0, 7, and 1, and left no
directory, VM, or link to the snapshot's files behind, while `snapshot
restore` of /usr/bin/false still failed with its error and status 1.
The raw snapshot directory, with and without `--store-dir`, network
options for the snapshot without a NIC, a missing, existing, or
in-store `--state-dir`, and `--wait` were refused before launch. On
KVM and MSHV also: `--memory-mib 384` of a snapshot of 512 MiB capacity
(MemTotal 372564 kB, against 241492 kB), three concurrent one-offs of
one snapshot, SIGINT (status 130, the directory removed), and a state
directory of 89 bytes, whose socket fails the VM's creation and which
the run removes. A `run` that kill -9, or Stop-Process -Force on
Windows, ended while its workload ran left its VM running 120 s later,
while another restore of the snapshot succeeded; `sandbox deprovision`
alone ended that VM and removed the directory, as `sandbox stop`, which
then reports sandbox-not-running, and `deprovision` also did. A
restored sandbox whose workload had exited with status 0 refused
`deprovision` until `sandbox stop` ended it.

Tests: unit tests parse the restore options of `run` as those of
`snapshot restore`, give the restore of a snapshot the snapshot's NIC
with the policy of the run's options, and refuse network options for a
snapshot without a NIC; fake-backed restores install the run's network
options as the restore's own policy, with the DNS server and published
listeners of the activation record, and OpenVMM's egress, forward, and
host-loopback options without `--net`, refuse network options for a
snapshot without a NIC before they provision anything, and end with the
status of their workload, 3, or 143 for one that SIGTERM ends, once the
VM and its launch have ended, where `snapshot restore` keeps its error,
and take a workload with whitespace and quotes, as `snapshot restore`
does.
CLI tests cover the raw refusals, a missing store, the state
directory's refusals, a workload that is not a valid exec request, the
options that each kind of run refuses, the removed options and
`--wait`, and the help. The dry runs and launches
of raw restores, the Python test that a raw restore left OpenVMM the
terminal, which no `nvx-rs run` gives OpenVMM anymore, the proptest of
a restore's policy paths, the tests of the attachment reader, and the
test of the handles that `spawn_foreground` passes, with its case in
the test of refused batch files, are retired with the code that they
tested.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SL3.8b removes `nvx-rs run`'s restore of a raw OpenVMM snapshot and the
boot console that it reattached on the terminal, which the
`run-console-restore` microVM scenario checked, so the case matrix
gives that scenario the fate retired, and its summary of the microVM
scenarios counts 48 kept and 1 retired.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The post-workload stop/deprovision sequence lacks end-to-end fake-backed tests, leaving regressions that leak VMs, state directories, or snapshot hard links undetected.

1 open issue
3 resolved since last review

Review effort: Balanced

Comment thread nvx-rs/src/host/cli/sandbox/mod.rs Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants