Repository navigation
cli: start restored sandboxes from their named snapshots (SL3.8a1) - #1486
Conversation
This pull request changes existing testsIt modifies or removes 3 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests. Test code
Relocated unchanged: 0; added: 39. Compared the merge base |
There was a problem hiding this comment.
🟡 Changes recommended
Unconditional VM-directory validation blocks deprovisioning older cold sandboxes on Windows whose permissions remain valid under the previous contract; validation should be limited to the restored-view branch.
1 open issue
What changed in this PR
This PR adds the managed-sandbox lifecycle support for named snapshot restores in #1114. It prepares the restored-start path used by the snapshot restore command in #1487, without adding that command here.
Changes:
- Restore, activate, and run recorded workloads through the existing launch journal and restore driver.
- Record snapshot sources and remove restored views under the snapshot’s lock.
- Add lifecycle and cleanup tests, and retire now-unneeded test-hook exemptions.
| File | Description |
|---|---|
scripts/test-layout-allowlist.json |
Removes exemptions for hooks now used in production. |
nvx-rs/src/host/vm/tests.rs |
Adds a fake-process launch helper. |
nvx-rs/src/host/restore_driver.rs |
Removes the unused-code exemption. |
nvx-rs/src/host/cli/snapshots/tests.rs |
Tests view cleanup and unsafe-path refusals. |
nvx-rs/src/host/cli/snapshots.rs |
Adds store reopening and locked view cleanup. |
nvx-rs/src/host/cli/sandbox/state/tests.rs |
Tests complete snapshot-source records and validation. |
nvx-rs/src/host/cli/sandbox/state.rs |
Records the source snapshot, store, and workload. |
nvx-rs/src/host/cli/sandbox/lifecycle/stop.rs |
Cleans restored views during deprovision. |
nvx-rs/src/host/cli/sandbox/lifecycle/start/tests.rs |
Covers restores, failures, recovery, restarts, and locking. |
nvx-rs/src/host/cli/sandbox/lifecycle/start.rs |
Integrates restored starts and workload completion handling. |
nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs |
Updates lifecycle documentation and serialization wiring. |
Review effort: Balanced
8011acd to
830970e
Compare
A sandbox's configuration can now name a snapshot of the CLI's named snapshot store: the snapshot, the store, and the workload that each restore runs, an entrypoint and its arguments. Each `sandbox start` of such a sandbox restores the snapshot in place of a cold boot, and `deprovision` removes the sandbox's view of the snapshot. SL3.8a2's `snapshot restore` provisions such sandboxes; until then `provision` records no snapshot. A start of a restored sandbox opens the store, takes the snapshot's exclusive lock, removes the view of an earlier start, and creates the VM with `host::vm::create` from the snapshot, with the snapshot's online processors, through the store's launch journal. The blocking restore driver then receives the restore-ready event of the verified process, activates the guest with the activation record of the start's own network policy, which gives the guest its DNS server and listener announcements, sends the sandbox's mapping table, and starts the workload. Closing the session that the driver returns at once can lose the workload's Start, as real KVM and MSHV restores showed, so the start keeps it until the workload ends, forwards its output, and fails unless it exits with status 0 within --timeout. The lock goes once the start ends, or once a failed start has verifiably ended the VM. Ctrl-C and --timeout cancel the driver and the wait. On Windows, OpenVMM opens a restore's scratch only while no other handle may write it, so the start closes the scratch claim's writable handle, and the VM inherits the claim's guard instead. A start whose command dies before it commits leaves its journal, whose recovery by the next command ends the VM. A spawn that cannot confirm that its VM ended has that recovery end the VM of a restore before the snapshot's lock goes; a start that cannot confirm that its VM ended at all leaves the journal to the next command. A closed control endpoint proves no exit, so, as for a cold start, OpenVMM then has the grace to exit by itself before the start ends it. A cold start takes the same launch path as before. Before it takes the lock, a start of a restored sandbox refuses a share of its snapshot store, as it refuses shares of the sandbox's own files, since the view of each restore links the snapshot's files, which the guest could change through such a share. A restore's guest accepts only a fresh upper, whose `upper` and `work` directories are missing or empty, so a later start of a restored sandbox with a scratch image refuses before it launches once the view of an earlier start shows that it gave the image to a guest. One with an upper in RAM restores again. `deprovision` removes a restored sandbox's `openvmm/` view, and on Linux its restore-ready socket, under the snapshot's lock, and refuses a view that is a link or holds anything else. Only that removal needs the state directory to be a VM directory, so deprovision still removes a cold sandbox whose directory the store accepts, as one that an earlier release made can be. Before a start or deprovision removes anything, it refuses a state directory in the snapshot store, once links are resolved, since the view there could be a snapshot's own files. A busy snapshot lock names the snapshot. The record is SL3.5a's typed configuration, format 9, which keeps `source_snapshot` and now records `source_store` and `workload` beside it, only for a restored sandbox. `Config::source` reads them back for start and deprovision, which ends the `expect(dead_code)` hook of the name's accessor, and `serialize_launch` takes the snapshot to record in place of `serialize_launch_with_snapshot`. The start is the production caller of SL4.3's restore driver, which ends its hook and its test-layout allowance. This is the first half of SL3.8a, which adapts the first implementation on esaurez/slim-3.2.2-sl3.8a-private-20261108 (e7d77fc), its restore start, source record, view removal under the lock, and restore-ready socket cleanup, to the CLI-owned lifecycle of SL3.5b, the typed configuration of SL3.5a, the restore creation of SL2.16, and the restore driver of SL4.3. Tests: fake-backed restores, whose checked launch spawns a fake OpenVMM and guest in place of OpenVMM, cover EROFS and GPT images with scratch and RAM uppers, the restore's own DNS server and published listeners in the activation record, the mapping table, and the workload; concurrent restores of one snapshot, the second of which spawns its VM only once the first one's has exited; failed restores, including a workload that exits with status 3, which end their VM and launch before the snapshot goes free, and a guest that hangs up after the workload's Start, whose VM exits by itself within its grace; a spawn that cannot confirm that its VM ended, whose VM the journal's recovery ends before the snapshot goes free; a later start, which restores again with an upper in RAM and refuses a scratch image that an earlier start gave to a guest; a command that dies before Ready, whose VM the next lock's recovery ends without any activation; deprovision, which waits for the snapshot's lock before it removes the view, and, on Windows, removes a cold sandbox whose state directory is no VM directory; the view removal's refusals, a VM directory in the store among them; a start that refuses a share of its snapshot store before it launches anything; and the record's source. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
830970e to
97a932d
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Coupled launch recovery, snapshot hard-link exclusion, and Windows scratch-handle inheritance warrant human confirmation of reported KVM/MSHV/WHP qualification, particularly killed-command recovery and scratch-claim exclusivity.
No open issues 🎉
2 resolved since last review
Review effort: Balanced
99855f7
into
feature/nvx-refactor


Part of #1114 (SL3.8a1, the first half of SL3.8a, "
snapshot restorestarts a sandbox from a named snapshot"). Parent #1084, umbrella #1076. SL3.8a2 (#1487), stacked on this PR, adds thesnapshot restorecommand.What and why
A managed sandbox whose configuration names a snapshot of the CLI's named snapshot store now starts by restoring that snapshot in place of a cold boot, and
deprovisionremoves its view of the snapshot under the snapshot's lock. Until SL3.8a2'ssnapshot restoremerges,provisionrecords no snapshot, so only the code paths change here.sandbox/lifecycle/start.rs).launch_lockedserves cold and restored starts.host::vm::createfrom the snapshot, with the snapshot's online processors and no boot, and journals and spawns it through the store's launch journal, as a cold start does.--timeout. Closing the session at once can lose the workload's Start, as real KVM and MSHV restores showed.--timeoutcancel the driver and the wait.lifecycle/stop.rs,snapshots.rs). Removes theopenvmm/view, and on Linux the restore-ready socket, under the snapshot's lock.sandbox/state.rs). SL3.5a's format-9 configuration keepssource_snapshotand recordssource_storeandworkloadbeside it, for a restored sandbox only.Config::source()accepts no source or a complete one: a contained name, an absolute store and a non-empty workload.serialize_launchtakes the source to record.restore_driver::restoreand of the record's source. That ends theirexpect(dead_code)hooks and the test-layout allowlist entries (check_test_layout.py --tighten).Not in this PR, per #1114:
run --restore-snapshot(SL3.8b).Snapshots that
snapshot capturepublishes have no NIC, so their restores take no--net. The fake-backed tests check the restore's DNS server and published listeners in the activation record; #1114 records the real-VM limitation.A restored workload runs as the sandbox's workload identity, which needs an account in the image, as a cold start of a layered root does.
--workload-user 1000:1000fails in the NVX Ubuntu image both cold and restored, at the base too: a pre-existing limitation, with the evidence in #1487.Takeover. This replays Enrique Saurez (@esaurez)'s first implementation (
esaurez/slim-3.2.2-sl3.8a-private-20261108,e7d77fcdda) with his authorship. It adapts the restored start, the source record, the view removal under the lock and the restore-ready socket cleanup to:Split and size. SL3.8a measures 541 core lines across its two PRs in 8 counted files, against its estimate of 250 lines in 3 files and its 500-line stop. Continuing past the stop was approved on Pedro Henrique Penna (@ppenna)'s behalf, up to the PR rules' hard limit of 600 core lines, because the command and the restored start are one coupled change. The 8 counted files were not: #1076 allows at most 7. No arrangement reached 7 without duplicating the
snapshotparser or carrying the source outsidelifecycle::provision. So SL3.8a is split in two:lifecycle/start.rs226,snapshots.rs77,state.rs30,lifecycle/stop.rs14 andlifecycle/mod.rs1.restore_driver.rsonly loses its hook, andsandbox/mod.rschanges only rustdoc.BOOT_CONTRACT_VERSION,CONTROL_CONTRACT_REVISIONand the public API are unchanged.Validation
The gates are those that
local_gates.pyselects for these files, run from this branch's action definitions at8c9847b7b09, on66fd43145ce.Rebases. This PR was rebased onto
d4edeeb9ab9(SL3.4b, #1490), which needed the resolution below, and since then cleanly ontof834bcb9672; its head is97a932d6c22. SL3.4b addedsandbox runtostart.rsand astart/tests.rsof its own.launch_lockedreports an early OpenVMM exit through SL3.4b'sexited_during_startup, and SL3.4b'srecord()passes no source toserialize_launch, since a run is a cold launch.start/tests.rsis unchanged but for amod restore;declaration, gated to Linux and Windows, ashost/vm/tests.rsgatesmod reattach;. This PR's restore tests are now instart/tests/restore.rs. SL3.4b's two tests stay ungated, so they still run wherever the lib's tests run, and still compile in the macOS check.state.rsandrestore_driver.rs; SL3.4b had dropped that ofplan.rs.Review fixes. Copilot's reviews found two gaps, both fixed in
97a932d6c22, each with a test that fails without the fix:deprovisionopened the VM directory of every sandbox. On Windows, that refuses directories thatStateDir::openaccepts, so a cold sandbox in such a directory could no longer be deprovisioned. It now opens it only to remove a restored sandbox's view (deprovision_removes_a_cold_sandbox_whose_state_directory_is_no_vm_directory, Windows).refuse_own_filesbefore it takes the snapshot's lock (a_restored_start_refuses_a_share_of_its_snapshot_store);doc/run.mddocuments it.A third finding, that a reader that stops reading the CLI's output blocks the start's writes past
--timeout, needs no change: those writes follow the CLI's output contract for such readers (host/cli/output.rs), as the reply on its thread explains.Rerun after the rebase onto SL3.4b, on Windows and on Linux (nvx-kvm-test), all passed:
cargo fmt --checkand the five Clippy variants with-D warnings;host::cli::sandbox,host::sandbox,host::cli::snapshotsandhost::restore*, among them SL3.4b's two run tests and the seven restore tests;cliintegration suite, with SL3.4b'ssandbox_runtests;check_test_layout.py;The table below is from before the rebase.
cargo fmt --check; Clippy-D warningswith--all-targets --all-features,--lib,--lib --no-default-features,--lib --bins --features cliand--lib --bins --no-default-features --features testingcargo test --all-features,cargo test, and--lib --no-default-featuresbuild and testcargo docwith-D warnings, MSRV 1.89, runtime adapter (protocol-only and host-only),public_api.py,check_test_layout.py,check_test_changes.py,check_markdown_links.pycargo test, and the--lib --no-default-featuresbuild and testpublic_api.py, guest binaries, runtime adapter, test layout, markdown links) passed on #1487's head.cargo test --all-featuresguest::tests, which fail in that host's isolated child processes (Operation not supported). They are exactly the 28 that fail at the base66fd43145ceon that host.nvx-rs, KVM: EROFS and GPT, scratch and RAM uppers; provision, start, exec, stop, start, exec, stop, deprovisionLeft to CI: validate-nvx's Linux CLI job, the guest tests that need working isolated child processes (they fail on nvx-kvm-test at the base too), and the microVM suites.
Changes to existing tests (
tests-modified)check_test_changes.py --base origin/feature/nvx-refactor: 3 changes, 39 added tests, none relocated.host::cli::sandbox::state::tests::restored_sandbox_records_round_trip_only_their_own_snapshot_nameis renamedrestored_sandbox_records_round_trip_only_their_own_source.serialize_launch.Config::source_snapshotis removed.Config::sourceis the production reader of the record's source.restore_driver::restoreis removed, and so is its allowlist entry (--tighten). The restored start is its production caller.