[Pedro Henrique Penna (ppenna)]

Part of #1118 (SL5.1c, "Snapshots work end to end on real VMs"). Parent #1086, umbrella #1076.

This branch is three commits on feature/nvx-refactor 012123bf, which holds everything they need: SL5.1b (#1496), SL3.8a1 (#1486), SL3.8a2 (#1487) and SL3.8b (#1489). The first commit fixes a production defect that the rewritten scenarios found: the musl build of nvx-rs, which the releases ship, could not restore any named snapshot.

What and why

NVX deletes its runtime adapter in SL5.2 (#1119), once the tests that replace it pass. The microVM scenarios snapshot-core and snapshot-tiers drove raw OpenVMM snapshots: the guest captured itself with nvx-snapshot to a --snapshot-destination, and a raw --restore-snapshot restored it. The final CLI refuses raw restores (#1489). This PR rewrites both scenarios on the CLI's strict, named snapshots, in the existing real-VM lane, scripts/nvx.py test-microvm. Its one production change is the fix that these scenarios need on musl, described below.

nvx-rs snapshot capture captures a VM that nvx-init runs. nvx-rs snapshot restore and nvx-rs run --restore-snapshot open the snapshot with Snapshot::open and restore it under the CLI's per-snapshot lock, each with an image and an upper of its own. Each restore's workload is a probe that reports:

• the guest's clock and uptime;
• fresh random values and its machine ID;
• its host name and identity;
• its root's lower and upper directories;
• its block devices and network interfaces;
• the marker of its host share.

The probe is a script that /bin/sh -c runs. A restore runs its workload over the control protocol, as sandbox exec does, so the script goes as it is.

snapshot-core

• Capture. The capture must publish exactly key.bin, snapshot.json and openvmm/, each file with one link. It must leave no VM directory or OpenVMM process.

• Repeated restores. After a downtime longer than the capture, two run --restore-snapshot restores run: one in RAM as 65534:65534, and one over a scratch upper as 1:1, each with its own host name. Each guest must show:

  • the restore's identity and host name;
  • the layer as its root's lower directory, over its upper;
  • the snapshot's two block devices and no NIC;
  • a clock that reads the host's time, and an uptime that kept the snapshot's and counts the downtime.

  The restores must report different random bytes, kernel UUIDs, machine IDs and temporary file names.

• Failure attribution, as decided on nvx-rs slimming 4.1.1: public-API tests, and smoke tests on real VMs #1118 (A1). This PR checks two failures:

  • A managed snapshot restore over a scratch image of half the dormant scratch's length is refused before any VM starts. The error[unsupported] message names the lengths of the requested scratch and of the dormant one, which attributes the failure to the request. A failed start keeps its sandbox provisioned, and the refused restore's sandbox must hold config.json and sandbox.lock alone, with no trace of a launch: no view of the snapshot, OpenVMM log or outcome report. Deprovision then removes it.
  • A run --restore-snapshot over a zeroed scratch image of the right length, which the guest can't mount as its upper, fails inside the guest. The VM ends before the workload runs and frees the lock, which the restores that follow take. The run reports error[exited]: OpenVMM exited during startup: guest-exit status 125.

  That is all the attribution this PR covers. A2, a follow-up recorded on nvx-rs slimming 4.1.1: public-API tests, and smoke tests on real VMs #1118, covers the rest. nvx-init powers off every failed activation with status 125, so the library classifies the failure as Unknown, and the CLI never prints RestoreError::attribution. A2 gives nvx-init a status for each class of failure and has the CLI print the restore's attribution.

• Concurrent restores under the CLI's lock. Three snapshot restore restores start at once. Each has its own host name and read-only host share, and the last has a scratch upper. Each workload ends only once the scenario creates a release file in its share, which the guest sees live.

  • While the first restore holds the lock, a one-off restore that may wait one second for the lock fails with sandbox-busy before its workload runs.
  • While the first holds the lock, neither of the others may link its view of the snapshot, which a restore does only once it holds the lock.
  • After the scenario releases the first, one of the others takes the lock. The last one may link its view only after the scenario releases that one.

  The host observes this order directly, so the check needs no comparison of guest clocks. The three sandboxes then run at once:

  • Each guest reads its own share's marker.
  • A file that sandbox exec writes in one sandbox's /tmp appears in no other.
  • Each sandbox stops with a successful restore outcome and a complete teardown, and deprovision removes it.

• Snapshot integrity. No restore may change the snapshot, leave a link to its files, or leave an OpenVMM process.

snapshot-tiers captures a platform-tier snapshot, the one tier that snapshot capture captures. It restores it into a managed sandbox of each image format over each upper: the Ubuntu EROFS layer, and a GPT disk of its tree, written as sandbox-gpt-layer writes one, each over scratch and RAM. Each root must have its image as its lower directory, over its upper, and an exec must see the same root. Neither image nor the snapshot may change.

Networks, as decided on #1118 (N1). A snapshot that snapshot capture captures has no NIC, and a restore must match the snapshot's NIC presence. So no strict snapshot restores with a network today (comment on #1114). untiered-network-policy therefore stays as it is, on a raw snapshot. Its case-matrix row reads "kept until a strict snapshot can carry a NIC (SL3.7c)". Its coverage of a restore's DNS, egress, forwards and listener publication stays real until SL3.7c (#1114) adds snapshot capture --net and rewrites the scenario on a strict snapshot. The old snapshot-core and snapshot-tiers checked no network: snapshot-tiers gave its raw capture a NIC but checked nothing of it.

The musl restore-ready fix, the commit host: open the restore-ready socket with O_PATH on musl too. CI's Linux guest and sandbox jobs run the static musl nvx-rs. At 4093a941, in run 43137188812, the KVM and MSHV jobs both failed at snapshot-core's first restore: error[io]: cannot use the restore-ready endpoint: No such device or address (os error 6).

• Cause. set_ready_socket_mode (nvx-rs/src/host/platform/linux/endpoint.rs) opened the restore-ready socket with O_PATH | O_NOFOLLOW through std's OpenOptionsExt::custom_flags, which masks out the bits of O_ACCMODE. On musl, O_ACCMODE is 0o10000003 and includes O_PATH; on glibc, it is 0o3. So the musl build opened the socket itself, which the kernel refuses with ENXIO, and no named snapshot could be restored.
• Fix. The function opens the socket with nix::fcntl::open and O_PATH | O_NOFOLLOW | O_CLOEXEC, which passes the flags on as they are.
• Evidence, on nvx-kvm-test, whose kernel is the KVM runners':
  • CI's musl binary failed exactly as in CI, and a glibc build of the same commit passed.
  • On musl, ten existing listener tests fail without the fix and pass with it, also at this PR's head: host::platform::linux::tests::a_restore_ready_socket_lets_its_owner_connect_whatever_the_umask and nine host::restore::tests, among them a_different_real_process_cannot_supply_the_event, the_socket_is_owner_only_even_when_the_umask_clears_owner_write and the_receiver_fails_closed_without_original_peer_pidfds.
  • With the fix, the musl CLI passes CI's public sandbox step on KVM and MSHV, and CI's Linux jobs pass too.
• Why nothing caught it. CI runs the nvx-rs unit tests only on glibc.
• Scope. This PR's "Not in this PR" leaves out only production snapshot fixes unrelated to making these smoke tests pass, and these smoke tests can't pass on musl without this one. Three other places open a path with O_PATH through custom_flags: restore_private_file_mode and open_plain_dir in host/platform/linux/private.rs, and DirHandle::search in host/platform/linux/identity.rs. They, and a CI run of the unit tests on musl, are follow-ups.

CI. Both scenarios read the Ubuntu layer and the scratch template, so they become public sandbox scenarios (decided on #1118):

• Both "Run public managed sandbox acceptance" steps run them on KVM, MSHV and WHP.
• They leave the default suite's shards. microvm_shards.json is planned again with plan-microvm-shards.
• snapshot-core leaves the extended Ubuntu guest step on KVM, since only Alpine can act as sandbox control.
• snapshot-tiers leaves the debug-kernel jobs, since snapshot capture boots the installed kernel.

Adversarial campaign. The managed restores journal their OpenVMM process IDs. So the weekly snapshot-isolation campaign keeps its two cases with adversarial_broker.py unchanged. The snapshot-tier-boundary case's description and coverage now say what the scenario does. The campaign's targets now need the Ubuntu layer and the scratch template, which nvx-rs download installs with a release.

Docs.

• doc/run.md gains a "Named snapshots" section, and doc/ci.md and doc/usage.md follow.
• In the case matrix, the three scenarios get their fates. A new "Snapshots" section maps the group's smoke tests to the scenarios. The adapter's cases that named "SL5.1c's" tests now name the scenarios, and untiered-network-policy's cases name SL3.7c. The CI rows of the public acceptance steps and of the extended Ubuntu step say what they run now.

Raw-only checks that retire

snapshot-core:

• the guest's own capture with nvx-snapshot, and its terminal capture boundary;
• the snapshot request without a destination, which OpenVMM releases: no scenario sends one any longer. snapshot-rollback keeps only the checks of the source after a released request, which follow the capture that it rolls back (doc/ci.md now says so directly);
• the CPU clocks of a sleeping task, and an armed timer's wait, across a restore;
• the restore packet's entropy file and nvx-reseed. Fresh entropy stays checked: each restore's random bytes, kernel UUID and machine ID differ;
• the VM generation ID, and the time ABI's port access;
• its run on the Ubuntu guest.

snapshot-tiers:

• the workload-start and instance-checkpoint tiers, with their storage policies, since nvx-rs slimming workstream: make nvx a library of mechanisms and value types #1076 leaves paired tiers out of scope;
• a restore of the wrong tier, the restore's input gate and its gate timeout;
• an instance checkpoint's restore without --machine microvm, and its second claim;
• its run on the debug kernel, with its guests' nvx-time status after the tier checks.

Size

Measured against feature/nvx-refactor 012123bf:

• Core lines: 8, the fix's lines of code in endpoint.rs (+11/−6, with 3 lines of comment). The rest of the change is tests. If the workflow's four --scenario lines count as code, as nvx-rs: build every feature set, compile the documented examples, and give every old test a fate (SL5.1a) #1488 counted its CI lines, the total is 12.
• Counted files: 3, within the limit of 5. The estimate says 1: the workflow comes with moving the scenarios to the public job, as decided on nvx-rs slimming 4.1.1: public-API tests, and smoke tests on real VMs #1118, which matches SL5.1b, and endpoint.rs with the fix:
  • nvx-rs/src/host/platform/linux/endpoint.rs: +11/−6, the fix.
  • scripts/nvx_tools/microvm_tests.py holds the two scenarios: +1346/−771 lines, 1043 of them code. check_test_changes.py classifies the module as end-to-end tests, but it is a Python script, which nvx-rs slimming workstream: make nvx a library of mechanisms and value types #1076 counts.
  • .github/workflows/run-nvx-microvm-tests.yml: +4/−1.
• Not counted:
  • the tests in scripts/test_*.py;
  • the docs;
  • the generated microvm_shards.json;
  • the campaign's data, snapshot-isolation.json;
  • the two deleted guest scripts, snapshot-core.sh and snapshot-tier.sh.in.

Commits

1. host: open the restore-ready socket with O_PATH on musl too
2. tests: rewrite snapshot-core and snapshot-tiers on named snapshots
3. doc: give the snapshot scenarios their fates in the case matrix

Not in this PR

• A NIC for snapshot capture, and untiered-network-policy on a strict snapshot: SL3.7c (nvx-rs slimming 3.2.2: the CLI captures and restores named snapshots #1114).
• nvx-init's failure-class status and the CLI's report of a restore's attribution: A2, a follow-up on nvx-rs slimming 4.1.1: public-API tests, and smoke tests on real VMs #1118.
• Raw OpenVMM snapshot support, new tiers or storage policies, a library-owned restore driver or lock, and production changes other than the musl fix.
• The other three places that open a path with O_PATH through custom_flags, and a CI run of the unit tests on musl: follow-ups.
• Deleting the adapter, which SL5.2 (nvx-rs slimming 4.1.2: the test adapter is gone #1119) does.

Validation

Real VMs, at the head e0dc2374, on feature/nvx-refactor 012123bf. The runs used the guest artifacts of CI run 43131339763, the push of 14ca2671, whose guest code is the tip's, and the OpenVMM of CI run 43075697333, built from the same openvmm pin, 14dc4116. The artifacts were checked by their SHA-256 digests. On Linux, the CLI is the static musl release build, run as CI runs it, with TMPDIR unset.

Backend	Host	CLI	snapshot-core	snapshot-tiers	CI's public step: all eight public scenarios
Linux/KVM	nvx-kvm-test	musl release build	passed, 16.0 s	passed, 5.8 s	passed
Linux/MSHV	nvx-mshv-test	musl release build	passed, 15.4 s	passed, 4.8 s	passed
Windows/WHP	local	MSVC release build	passed, 19.9 s	passed, 6.4 s	passed in CI's WHP job

CI run 43141805757, at e0dc2374: 34 jobs passed and 9 were skipped. Among them, the guest and sandbox jobs of KVM, MSHV and WHP ran the public step, with the musl CLI on Linux.

At 4093a941, before the fix, a glibc CLI passed the same runs on KVM and MSHV, which is why the defect stayed hidden on the hosts. Before the rebase, on its temporary base, the branch passed the same runs on all three backends at 74e85514, and the snapshot scenarios on WHP at d606f767. There, a capture that timed out under --timeout 1 left no fixture and no OpenVMM process. The rebase changed one thing in the scenarios: the probe now goes to the restore's shell as it is, since a restore's workload now travels over the control protocol and may contain whitespace (#1487).

Rust, on nvx-kvm-test at e0dc2374:

Check	Result
The listener's unit tests on musl: cargo test --locked --lib --target x86_64-unknown-linux-musl -- host::platform::linux::tests::a_restore_ready_socket host::restore::tests	25 passed (1 ignored); with the fix reverted, 10 failed
cargo clippy --locked --all-targets --all-features -- -D warnings, for the host's glibc target and for x86_64-unknown-linux-musl	passed
cargo fmt --all --check	passed

CI's nvx-rs jobs ran the rest of the Rust checks, on Ubuntu with glibc and on Windows, and passed in that run.

Gates, from this branch's check-quality and validate-nvx actions, as local_gates.py --files selects them, at e0dc2374:

Check	Windows (local)	Linux (nvx-kvm-test)
ruff check and ruff format --check of scripts .github/specula	passed
pyright --pythonplatform Linux and --pythonplatform Windows	passed, 0 errors
check_markdown_links.py, check_test_layout.py	passed	check_test_layout.py passed
test_microvm_tests	353 tests, OK (9 skipped)	353 tests, OK (1 skipped)
test_nvx_tools	CiConfigurationTests, 94 tests, OK (6 skipped); CI's NVX CLI jobs ran the whole suite on Linux and Windows, and passed
test_adversarial	84 tests, OK (11 skipped)
nvx.py test-microvm --help	passed

Left to CI:

• shellcheck and shfmt (this PR deletes two shell scripts and changes no others);
• Specula's tests;
• the rest of the Rust suites, on glibc and on Windows, which doesn't build the fix;
• the guest binaries;
• the runtime adapter;
• the microVM lanes.

Changes to existing tests

python3 scripts/check_test_changes.py --base origin/feature/nvx-refactor lists 38 changes to existing tests and 85 added items, so this PR has the tests-modified label:

• Removed with the raw scenarios (each retired check is listed above):

  • the guest scripts snapshot-core.sh and snapshot-tier.sh.in;
  • the helpers _snapshot_tier_kinds, _snapshot_tier_script, _snapshot_storage_policy, _run_snapshot_tier and _parse_marker_pair, and the old run_snapshot_core and run_snapshot_tiers, which the new scenarios replace;
  • the unit tests of those helpers:
    • test_snapshot_core_script_handles_no_clocksource;
    • test_snapshot_core_checks_that_restored_processes_cannot_reach_the_ports;
    • test_snapshot_core_reads_the_entropy_that_nvx_time_kept;
    • test_snapshot_core_waits_for_no_destination_marker_line_before_exit;
    • test_snapshot_tier_scripts_preserve_tier_specific_policy;
    • test_restored_tier_guests_report_their_restore_checks_before_exiting;
    • test_snapshot_tier_entry_points_reject_unsupported_tiers;
    • test_snapshot_tier_runner_dispatches_all_tiers;
    • test_snapshot_storage_policy_selects_generation_and_materialization.

  PublicSnapshotTests replaces them with nine tests, which run both scenarios against a fake of the CLI, with its lock. They inject 26 regressions that the scenarios must detect, among them a restore that launches a VM before it refuses a short scratch. They also run two failed captures. In one, cleanup ends the capture's VM through the capture's launch journal. In the other, the VM survives cleanup, after which the fixture must stay for recovery.

• test_snapshot_marker_parsers_require_single_well_formed_values no longer checks _parse_marker_pair, which leaves with snapshot-core's markers.

• test_every_tcp_console_is_monitored_and_cold_boot_shells_are_queried no longer counts the raw snapshot-tiers' three TCP consoles (its capture, its restore and its gate-timeout restore) or its capture's time ABI monitor.

• test_debug_kernel_runs_the_same_host_restore_scenarios_on_vmlinux_debug drops snapshot-tiers from the debug-kernel scenarios.

• CiConfigurationTests.test_ci_runs_the_public_sandbox_acceptance_on_every_backend also requires --scenario snapshot-core and --scenario snapshot-tiers in both public acceptance steps.

• Scenario lists:

  • SANDBOX_CONTROL_SCENARIOS gains snapshot-core;
  • PUBLIC_SANDBOX_SCENARIOS gains both scenarios;
  • DEBUG_KERNEL_SCENARIOS loses snapshot-tiers;
  • _run_scenarios dispatches the new scenarios;
  • GptLayerAcceptance.write_disk now calls write_gpt_layer, which snapshot-tiers shares, unchanged.

• CI wiring: the extended Ubuntu step drops --scenario snapshot-core, and microvm_shards.json drops both scenarios from each backend's shards.

• Test data: snapshot-isolation.json's snapshot-tier-boundary case has a new description and coverage.

[github-actions]

This pull request changes existing tests

It modifies or removes 38 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change	What	Before	After
modified imports	the imports of scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:1	scripts/test_microvm_tests.py:1
modified test	def MicrovmTests.test_every_tcp_console_is_monitored_and_cold_boot_shells_are_queried in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:7685	scripts/test_microvm_tests.py:8581
modified test	def MicrovmTests.test_snapshot_marker_parsers_require_single_well_formed_values in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:8267	scripts/test_microvm_tests.py:9014
modified test	def MicrovmTests.test_debug_kernel_runs_the_same_host_restore_scenarios_on_vmlinux_debug in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:16186	scripts/test_microvm_tests.py:16868
modified test	def CiConfigurationTests.test_ci_runs_the_public_sandbox_acceptance_on_every_backend in scripts/test_nvx_tools.py	scripts/test_nvx_tools.py:12785	scripts/test_nvx_tools.py:12785
removed test	def MicrovmTests.test_snapshot_core_script_handles_no_clocksource in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:7724
removed test	def MicrovmTests.test_snapshot_core_checks_that_restored_processes_cannot_reach_the_ports in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:7737
removed test	def MicrovmTests.test_snapshot_core_reads_the_entropy_that_nvx_time_kept in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:7748
removed test	def MicrovmTests.test_snapshot_core_waits_for_no_destination_marker_line_before_exit in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:7857
removed test	def MicrovmTests.test_snapshot_tier_scripts_preserve_tier_specific_policy in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:8378
removed test	def MicrovmTests.test_restored_tier_guests_report_their_restore_checks_before_exiting in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:8400
removed test	def MicrovmTests.test_snapshot_tier_entry_points_reject_unsupported_tiers in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:8434
removed test	def MicrovmTests.test_snapshot_tier_runner_dispatches_all_tiers in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:8449
removed test	def MicrovmTests.test_snapshot_storage_policy_selects_generation_and_materialization in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:8466

Test data

Change	What	Before	After
modified file	scripts/nvx_tools/adversarial_cases/snapshot-isolation.json	scripts/nvx_tools/adversarial_cases/snapshot-isolation.json	scripts/nvx_tools/adversarial_cases/snapshot-isolation.json

End-to-end tests

Change	What	Before	After
modified helper	assignment SANDBOX_CONTROL_SCENARIOS in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:183	scripts/nvx_tools/microvm_tests.py:191
modified helper	assignment PUBLIC_SANDBOX_SCENARIOS in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:208	scripts/nvx_tools/microvm_tests.py:217
modified helper	assignment DEBUG_KERNEL_SCENARIOS in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:221	scripts/nvx_tools/microvm_tests.py:232
modified helper	assignment SNAPSHOT_CORE_CONTINUED_MARKER in scripts/nvx_tools/microvm_tests.py (renamed to SNAPSHOT_HOSTNAME)	scripts/nvx_tools/microvm_tests.py:381	scripts/nvx_tools/microvm_tests.py:2169
modified helper	assignment SNAPSHOT_CORE_COMPLETION_MARKER in scripts/nvx_tools/microvm_tests.py (renamed to SNAPSHOT_PROBE_PREFIX)	scripts/nvx_tools/microvm_tests.py:382	scripts/nvx_tools/microvm_tests.py:2087
modified helper	def _parse_marker_pair in scripts/nvx_tools/microvm_tests.py (renamed to named_snapshot_files)	scripts/nvx_tools/microvm_tests.py:1066	scripts/nvx_tools/microvm_tests.py:2255
modified helper	def GptLayerAcceptance.write_disk in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:1838	scripts/nvx_tools/microvm_tests.py:1885
modified helper	def run_snapshot_core in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:8754	scripts/nvx_tools/microvm_tests.py:3085
modified helper	def run_snapshot_tiers in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:11217	scripts/nvx_tools/microvm_tests.py:3226
modified helper	def _run_scenarios in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:11372	scripts/nvx_tools/microvm_tests.py:11955
removed file	scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh	scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh
removed file	scripts/nvx_tools/microvm_test_scripts/snapshot-tier.sh.in	scripts/nvx_tools/microvm_test_scripts/snapshot-tier.sh.in
removed helper	def _snapshot_tier_kinds in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:10805
removed helper	def _snapshot_tier_script in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:10811
removed helper	def _snapshot_storage_policy in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:10937
removed helper	def _run_snapshot_tier in scripts/nvx_tools/microvm_tests.py	scripts/nvx_tools/microvm_tests.py:10958

CI test wiring

Change	What	Before	After
removed line	--scenario snapshot-core	.github/workflows/run-nvx-microvm-tests.yml:334
removed scenario	snapshot-core on kvm	scripts/nvx_tools/microvm_shards.json
removed scenario	snapshot-core on mshv	scripts/nvx_tools/microvm_shards.json
removed scenario	snapshot-core on whp	scripts/nvx_tools/microvm_shards.json
removed scenario	snapshot-tiers on kvm	scripts/nvx_tools/microvm_shards.json
removed scenario	snapshot-tiers on mshv	scripts/nvx_tools/microvm_shards.json
removed scenario	snapshot-tiers on whp	scripts/nvx_tools/microvm_shards.json

Relocated unchanged: 0; added: 86. Compared the merge base 012123bf9ac6 with e0dc2374c98c.

[Copilot]

🟡 Changes recommended

The capture workflow never requests a snapshot, and indeterminate VM spawns can lose recovery context, allowing callers to release snapshot resources while OpenVMM still uses them.

3 open issues

• Capture workflow never sends a capture request · New
• Indeterminate spawn error omits recovery context · New
• Restore drops exit status for signal-terminated workloads · New

What changed in this PR

This PR moves real-VM snapshot acceptance from raw OpenVMM snapshots to the CLI’s named snapshots, preparing for runtime-adapter removal. The reviewed diff also includes temporary-base changes for named restores, cold-boot acceptance, and public-API examples.

Changes:

• Exercises named capture and restore, snapshot locking, guest isolation, failure handling, and image/upper combinations.
• Moves snapshot scenarios into public acceptance on KVM, MSHV, and WHP, retiring raw-only checks.
• Updates CLI coverage, workflow examples, and documentation for the replacement acceptance paths.

File	Description
scripts/​test-layout-allowlist.json	Removes obsolete test-hook allowances.
scripts/​test_nvx_tools.py	Requires new public acceptance scenarios in CI.
scripts/​nvx_tools/​microvm_test_scripts/​snapshot-tier.sh.in	Deletes the raw tier probe.
scripts/​nvx_tools/​microvm_test_scripts/​snapshot-core.sh	Deletes the raw snapshot probe.
scripts/​nvx_tools/​microvm_test_scripts/​run-console-restore.sh	Deletes the raw console-restore probe.
scripts/​nvx_tools/​microvm_shards.json	Rebalances shards after scenario removals and moves.
scripts/​nvx_tools/​adversarial_cases/​snapshot-isolation.json	Updates snapshot-tier campaign coverage descriptions.
README.md	Links the acceptance case matrix.
nvx-rs/​tests/​vm_lifecycle.rs	Adds real-VM public-API lifecycle acceptance.
nvx-rs/​tests/​cli/​validation.rs	Tests named-restore input refusals.
nvx-rs/​tests/​cli/​usage.rs	Updates restore option and help expectations.
nvx-rs/​tests/​cli/​openvmm.rs	Removes raw-restore fixtures and helpers.
nvx-rs/​tests/​cli/​one_shot.rs	Removes obsolete raw-restore launch cases.
nvx-rs/​tests/​cli/​dry_run.rs	Removes unsupported raw-restore dry runs.
nvx-rs/​tests/​cli/​download.rs	Checks invalid restores trigger no downloads.
nvx-rs/​tests/​cli.rs	Adapts restore failure coverage to named snapshots.
nvx-rs/​src/​host/​vm/​tests.rs	Adds a checked-launch substitution helper.
nvx-rs/​src/​host/​template/​mod.rs	Removes the raw attachment-reader re-export.
nvx-rs/​src/​host/​snapshot/​manifest/​tests.rs	Tests attachments through the retained manifest reader.
nvx-rs/​src/​host/​snapshot/​manifest.rs	Removes the raw snapshot attachment reader.
nvx-rs/​src/​host/​restore_driver.rs	Removes the now-unneeded dead-code allowance.
nvx-rs/​src/​host/​platform/​windows/​tests.rs	Removes obsolete foreground-spawn tests.
nvx-rs/​src/​host/​platform/​windows/​spawn.rs	Deletes unused foreground-spawn helpers.
nvx-rs/​src/​host/​platform/​windows/​mod.rs	Removes the foreground-spawn re-export.
nvx-rs/​src/​host/​cli/​tests.rs	Checks the assembled run option list.
nvx-rs/​src/​host/​cli/​snapshots/​tests.rs	Covers restore parsing and snapshot-view cleanup.
nvx-rs/​src/​host/​cli/​snapshots/​capture/​tests.rs	Adapts tests to capture-only argument parsing.
nvx-rs/​src/​host/​cli/​snapshots/​capture.rs	Moves snapshot subcommand dispatch out of capture.
nvx-rs/​src/​host/​cli/​snapshots.rs	Adds named restore dispatch and locked view cleanup.
nvx-rs/​src/​host/​cli/​sandbox/​state/​tests.rs	Tests complete restore-source records.
nvx-rs/​src/​host/​cli/​sandbox/​state.rs	Records snapshot store and restore workload.
nvx-rs/​src/​host/​cli/​sandbox/​one_shot.rs	Deletes the obsolete direct-launch helper.
nvx-rs/​src/​host/​cli/​sandbox/​mod.rs	Adds managed and one-off restore orchestration.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​stop.rs	Removes restored views under the snapshot lock.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​mod.rs	Threads snapshot sources through provisioning.
nvx-rs/​src/​host/​cli/​run/​tests.rs	Tests shared restore options and NIC policy.
nvx-rs/​src/​host/​cli/​network.rs	Removes obsolete direct argument lowering.
nvx-rs/​src/​host/​cli/​mod.rs	Wires snapshot restore into CLI dispatch.
nvx-rs/​src/​guest/​init/​boot/​tests.rs	Adds compile-only reference-space workflows.
nvx-rs/​examples/​host-workflows.rs	Adds documented host lifecycle examples.
nvx-rs/​examples/​guest-workflows.rs	Adds documented guest workflow examples.
nvx-rs/​Cargo.toml	Registers feature-gated workflow examples.
doc/​run.md	Documents named snapshot acceptance and raw OpenVMM usage.
doc/​project-structure.md	Lists new examples and acceptance tests.
doc/​contribute.md	Updates feature checks and test guidance.
doc/​ci.md	Describes replacement acceptance and scenario placement.
.github/​workflows/​run-nvx-microvm-tests.yml	Adds public acceptance scenarios and removes obsolete placement.
.github/​actions/​check-nvx-rs/​action.yml	Adds the standalone serde feature check.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The cleanup path can delete the fixture and its recovery journal even when OpenVMM termination is unconfirmed, leaving a potentially surviving VM without recovery data.

1 open issue

• Preserve recovery fixture when VM cleanup is uncertain · New

3 resolved since last review

• Indeterminate spawn error omits recovery context
• Capture workflow never sends a capture request
• Restore drops exit status for signal-terminated workloads

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Cleanup can miss an OpenVMM process with surviving threads and delete its fixture and recovery journal before exit is verified.

1 open issue

• Preserve fixtures until failed capture VMs are fully exited · New

1 resolved since last review

• Preserve recovery fixture when VM cleanup is uncertain

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the migration to named-snapshot acceptance workflows.

No open issues 🎉

1 resolved since last review

• Preserve fixtures until failed capture VMs are fully exited

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The short-scratch test does not prove rejection before VM launch, and the coverage matrix misleadingly implies that the deleted no-destination capture case still runs.

2 open issues

• Verify short-scratch requests are refused before VM launch · New
• Mark the no-destination snapshot case as no longer covered · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

An interruption during restore startup can block VM cleanup until the CLI subprocess times out; cancellable subprocesses and a regression test for startup interruption are still needed.

1 open issue

• Terminate stalled restore CLI subprocesses on interruption · New

2 resolved since last review

• Verify short-scratch requests are refused before VM launch
• Mark the no-destination snapshot case as no longer covered

_{Review effort: Balanced}