Skip to content

cli: start restored sandboxes from their named snapshots (SL3.8a1) - #1486

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-sl3-8a1-restored-start
Nov 11, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-sl3-8a1-restored-start

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1114 (SL3.8a1, the first half of SL3.8a, "snapshot restore starts a sandbox from a named snapshot"). Parent #1084, umbrella #1076. SL3.8a2 (#1487), stacked on this PR, adds the snapshot restore command.

What and why

A managed sandbox whose configuration names a snapshot of the CLI's named snapshot store now starts by restoring that snapshot in place of a cold boot, and deprovision removes its view of the snapshot under the snapshot's lock. Until SL3.8a2's snapshot restore merges, provision records no snapshot, so only the code paths change here.

  • Restored start (sandbox/lifecycle/start.rs). launch_locked serves cold and restored starts.
    • A restore opens the store, takes the snapshot's exclusive lock and removes the view of an earlier start.
    • It creates the VM with host::vm::create from the snapshot, with the snapshot's online processors and no boot, and journals and spawns it through the store's launch journal, as a cold start does.
    • SL4.3's blocking restore driver then takes the verified process's restore-ready event. It activates the guest with the activation record of the start's own network policy (its DNS server and listener announcements), sends the mapping table, and starts the workload.
    • The start keeps the session until the workload ends, forwards its output, and fails unless it exits with status 0 within --timeout. Closing the session at once can lose the workload's Start, as real KVM and MSHV restores showed.
    • The lock goes once the start ends, or once a failed start has verifiably ended the VM. Ctrl-C and --timeout cancel the driver and the wait.
  • Recovery. A start whose command dies before it commits leaves its journal, whose recovery by the next command ends the VM, as for an interrupted cold launch.
    • A spawn that cannot confirm that its VM ended runs that recovery before the snapshot's lock goes.
    • A closed control endpoint proves no exit, so OpenVMM gets the cold start's grace to exit by itself before the start ends it.
  • Scratch uppers. A restore's guest accepts only a fresh upper. So a later start of a restored sandbox with a scratch image refuses, once the view of an earlier start shows that it gave the image to a guest. A RAM upper restores again.
    • On Windows, the VM inherits the scratch claim's guard instead of its writable handle, since OpenVMM opens a restore's scratch only while no other handle may write it.
  • Deprovision (lifecycle/stop.rs, snapshots.rs). Removes the openvmm/ view, and on Linux the restore-ready socket, under the snapshot's lock.
    • It refuses a view that is a link or holds anything else.
    • It also refuses a state directory inside the snapshot store, where the view could be the snapshot's own files.
    • A busy snapshot lock names the snapshot.
  • Record (sandbox/state.rs). SL3.5a's format-9 configuration keeps source_snapshot and records source_store and workload beside it, for a restored sandbox only.
    • Config::source() accepts no source or a complete one: a contained name, an absolute store and a non-empty workload.
    • serialize_launch takes the source to record.
  • Hooks. The start is the production caller of SL4.3's restore_driver::restore and of the record's source. That ends their expect(dead_code) hooks and the test-layout allowlist entries (check_test_layout.py --tighten).

Not in this PR, per #1114:

  • durable in-flight restore records or a reconciler;
  • an index of restored VMs per snapshot;
  • shared locks or parallel startup from one snapshot;
  • resuming a restore whose command died;
  • run --restore-snapshot (SL3.8b).

Snapshots that snapshot capture publishes have no NIC, so their restores take no --net. The fake-backed tests check the restore's DNS server and published listeners in the activation record; #1114 records the real-VM limitation.

A restored workload runs as the sandbox's workload identity, which needs an account in the image, as a cold start of a layered root does. --workload-user 1000:1000 fails in the NVX Ubuntu image both cold and restored, at the base too: a pre-existing limitation, with the evidence in #1487.

Takeover. This replays Enrique Saurez (@esaurez)'s first implementation (esaurez/slim-3.2.2-sl3.8a-private-20261108, e7d77fcdda) with his authorship. It adapts the restored start, the source record, the view removal under the lock and the restore-ready socket cleanup to:

  • SL3.5b's CLI-owned lifecycle;
  • SL3.5a's typed configuration;
  • SL2.16's restore creation;
  • SL4.3's restore driver.

Split and size. SL3.8a measures 541 core lines across its two PRs in 8 counted files, against its estimate of 250 lines in 3 files and its 500-line stop. Continuing past the stop was approved on Pedro Henrique Penna (@ppenna)'s behalf, up to the PR rules' hard limit of 600 core lines, because the command and the restored start are one coupled change. The 8 counted files were not: #1076 allows at most 7. No arrangement reached 7 without duplicating the snapshot parser or carrying the source outside lifecycle::provision. So SL3.8a is split in two:

  • This PR: 348 core lines in 5 counted files: lifecycle/start.rs 226, snapshots.rs 77, state.rs 30, lifecycle/stop.rs 14 and lifecycle/mod.rs 1. restore_driver.rs only loses its hook, and sandbox/mod.rs changes only rustdoc.
  • SL3.8a2 (cli: restore managed sandboxes from named snapshots (SL3.8a2) #1487): the command, 193 core lines in 5 counted files.

BOOT_CONTRACT_VERSION, CONTROL_CONTRACT_REVISION and the public API are unchanged.

Validation

The gates are those that local_gates.py selects for these files, run from this branch's action definitions at 8c9847b7b09, on 66fd43145ce.

Rebases. This PR was rebased onto d4edeeb9ab9 (SL3.4b, #1490), which needed the resolution below, and since then cleanly onto f834bcb9672; its head is 97a932d6c22. SL3.4b added sandbox run to start.rs and a start/tests.rs of its own.

  • The doc comments and imports keep both sides. launch_locked reports an early OpenVMM exit through SL3.4b's exited_during_startup, and SL3.4b's record() passes no source to serialize_launch, since a run is a cold launch.
  • SL3.4b's start/tests.rs is unchanged but for a mod restore; declaration, gated to Linux and Windows, as host/vm/tests.rs gates mod reattach;. This PR's restore tests are now in start/tests/restore.rs. SL3.4b's two tests stay ungated, so they still run wherever the lib's tests run, and still compile in the macOS check.
  • The test-layout allowlist drops the entries of state.rs and restore_driver.rs; SL3.4b had dropped that of plan.rs.

Review fixes. Copilot's reviews found two gaps, both fixed in 97a932d6c22, each with a test that fails without the fix:

  • deprovision opened the VM directory of every sandbox. On Windows, that refuses directories that StateDir::open accepts, so a cold sandbox in such a directory could no longer be deprovisioned. It now opens it only to remove a restored sandbox's view (deprovision_removes_a_cold_sandbox_whose_state_directory_is_no_vm_directory, Windows).
  • A restored sandbox could share its snapshot store, whose files each restore's view links, and so let the guest change its snapshot. A start of a restored sandbox now refuses such a share with refuse_own_files before it takes the snapshot's lock (a_restored_start_refuses_a_share_of_its_snapshot_store); doc/run.md documents it.

A third finding, that a reader that stops reading the CLI's output blocks the start's writes past --timeout, needs no change: those writes follow the CLI's output contract for such readers (host/cli/output.rs), as the reply on its thread explains.

Rerun after the rebase onto SL3.4b, on Windows and on Linux (nvx-kvm-test), all passed:

  • cargo fmt --check and the five Clippy variants with -D warnings;
  • the lib tests of host::cli::sandbox, host::sandbox, host::cli::snapshots and host::restore*, among them SL3.4b's two run tests and the seven restore tests;
  • the whole cli integration suite, with SL3.4b's sandbox_run tests;
  • check_test_layout.py;
  • on Linux, the macOS check.

The table below is from before the rebase.

Check Result
Windows: cargo fmt --check; Clippy -D warnings with --all-targets --all-features, --lib, --lib --no-default-features, --lib --bins --features cli and --lib --bins --no-default-features --features testing Passed
Windows: cargo test --all-features, cargo test, and --lib --no-default-features build and test Passed
Windows: bundling, cargo doc with -D warnings, MSRV 1.89, runtime adapter (protocol-only and host-only), public_api.py, check_test_layout.py, check_test_changes.py, check_markdown_links.py Passed
Windows: validate-nvx's "Validate NVX CLI on Windows" step Passed on #1487's head, which contains this PR: 1834 tests OK (70 skipped), 84 OK (11 skipped), 4 OK
Linux (nvx-kvm-test): fmt, the five Clippy variants, doc, MSRV, cargo test, and the --lib --no-default-features build and test Passed. The rest of the Linux set (bundling, macOS check, public_api.py, guest binaries, runtime adapter, test layout, markdown links) passed on #1487's head.
Linux (nvx-kvm-test): cargo test --all-features Every target passes except the lib's 28 guest:: tests, which fail in that host's isolated child processes (Operation not supported). They are exactly the 28 that fail at the base 66fd43145ce on that host.
Cold sandbox lifecycle with this branch's nvx-rs, KVM: EROFS and GPT, scratch and RAM uppers; provision, start, exec, stop, start, exec, stop, deprovision Passed: scratch uppers keep their files across starts, RAM uppers start fresh
Restored starts, through SL3.8a2's command on top of this branch Passed on KVM, MSHV and WHP: see #1487

Left to CI: validate-nvx's Linux CLI job, the guest tests that need working isolated child processes (they fail on nvx-kvm-test at the base too), and the microVM suites.

Changes to existing tests (tests-modified)

check_test_changes.py --base origin/feature/nvx-refactor: 3 changes, 39 added tests, none relocated.

  • host::cli::sandbox::state::tests::restored_sandbox_records_round_trip_only_their_own_snapshot_name is renamed restored_sandbox_records_round_trip_only_their_own_source.
    • The record now holds the whole source (snapshot, store and workload), so the test round-trips each through serialize_launch.
    • It also refuses an escaping name, a missing part of a source, a relative store and an empty workload, beside the escaping name that it already refused.
  • The test hook of Config::source_snapshot is removed. Config::source is the production reader of the record's source.
  • The test hook of restore_driver::restore is removed, and so is its allowlist entry (--tighten). The restored start is its production caller.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 3 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified test fn restored_sandbox_records_round_trip_only_their_own_snapshot_name in host::cli::sandbox::state::tests (renamed to restored_sandbox_records_round_trip_only_their_own_source) nvx-rs/src/host/cli/sandbox/state/tests.rs:5 nvx-rs/src/host/cli/sandbox/state/tests.rs:7
removed hook the test hooks of fn source_snapshot in host::cli::sandbox::state (impl Config) nvx-rs/src/host/cli/sandbox/state.rs:296
removed hook the test hooks of fn restore in host::restore_driver nvx-rs/src/host/restore_driver.rs:118

Relocated unchanged: 0; added: 39. Compared the merge base f834bcb9672e with 97a932d6c22e.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unconditional VM-directory validation blocks deprovisioning older cold sandboxes on Windows whose permissions remain valid under the previous contract; validation should be limited to the restored-view branch.

1 open issue
What changed in this PR

This PR adds the managed-sandbox lifecycle support for named snapshot restores in #1114. It prepares the restored-start path used by the snapshot restore command in #1487, without adding that command here.

Changes:

  • Restore, activate, and run recorded workloads through the existing launch journal and restore driver.
  • Record snapshot sources and remove restored views under the snapshot’s lock.
  • Add lifecycle and cleanup tests, and retire now-unneeded test-hook exemptions.
File Description
scripts/​test-layout-allowlist.json Removes exemptions for hooks now used in production.
nvx-rs/​src/​host/​vm/​tests.rs Adds a fake-process launch helper.
nvx-rs/​src/​host/​restore_driver.rs Removes the unused-code exemption.
nvx-rs/​src/​host/​cli/​snapshots/​tests.rs Tests view cleanup and unsafe-path refusals.
nvx-rs/​src/​host/​cli/​snapshots.rs Adds store reopening and locked view cleanup.
nvx-rs/​src/​host/​cli/​sandbox/​state/​tests.rs Tests complete snapshot-source records and validation.
nvx-rs/​src/​host/​cli/​sandbox/​state.rs Records the source snapshot, store, and workload.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​stop.rs Cleans restored views during deprovision.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​start/​tests.rs Covers restores, failures, recovery, restarts, and locking.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​start.rs Integrates restored starts and workload completion handling.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​mod.rs Updates lifecycle documentation and serialization wiring.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues allow writable shares to corrupt snapshot-store files and blocked output pipes to prevent timeout enforcement, leaving the VM and locks held.

2 open issues
1 resolved since last review

Review effort: Balanced

A sandbox's configuration can now name a snapshot of the CLI's named
snapshot store: the snapshot, the store, and the workload that each
restore runs, an entrypoint and its arguments. Each `sandbox start` of
such a sandbox restores the snapshot in place of a cold boot, and
`deprovision` removes the sandbox's view of the snapshot. SL3.8a2's
`snapshot restore` provisions such sandboxes; until then `provision`
records no snapshot.

A start of a restored sandbox opens the store, takes the snapshot's
exclusive lock, removes the view of an earlier start, and creates the
VM with `host::vm::create` from the snapshot, with the snapshot's
online processors, through the store's launch journal. The blocking
restore driver then receives the restore-ready event of the verified
process, activates the guest with the activation record of the start's
own network policy, which gives the guest its DNS server and listener
announcements, sends the sandbox's mapping table, and starts the
workload. Closing the session that the driver returns at once can lose
the workload's Start, as real KVM and MSHV restores showed, so the
start keeps it until the workload ends, forwards its output, and fails
unless it exits with status 0 within --timeout. The lock goes once the
start ends, or once a failed start has verifiably ended the VM. Ctrl-C
and --timeout cancel the driver and the wait. On Windows, OpenVMM opens
a restore's scratch only while no other handle may write it, so the
start closes the scratch claim's writable handle, and the VM inherits
the claim's guard instead. A start whose command dies before it commits
leaves its journal, whose recovery by the next command ends the VM. A
spawn that cannot confirm that its VM ended has that recovery end the
VM of a restore before the snapshot's lock goes; a start that cannot
confirm that its VM ended at all leaves the journal to the next
command. A closed control endpoint proves no exit, so, as for a cold
start, OpenVMM then has the grace to exit by itself before the start
ends it. A cold start takes the same launch path as before.

Before it takes the lock, a start of a restored sandbox refuses a share
of its snapshot store, as it refuses shares of the sandbox's own files,
since the view of each restore links the snapshot's files, which the
guest could change through such a share.

A restore's guest accepts only a fresh upper, whose `upper` and `work`
directories are missing or empty, so a later start of a restored
sandbox with a scratch image refuses before it launches once the view
of an earlier start shows that it gave the image to a guest. One with
an upper in RAM restores again.

`deprovision` removes a restored sandbox's `openvmm/` view, and on
Linux its restore-ready socket, under the snapshot's lock, and refuses
a view that is a link or holds anything else. Only that removal needs
the state directory to be a VM directory, so deprovision still
removes a cold sandbox whose directory the store accepts, as one that
an earlier release made can be. Before a start or deprovision removes
anything, it refuses a state directory in the snapshot store, once links
are resolved, since the view there could be a snapshot's own files. A
busy snapshot lock names the snapshot.

The record is SL3.5a's typed configuration, format 9, which keeps
`source_snapshot` and now records `source_store` and `workload` beside
it, only for a restored sandbox. `Config::source` reads them back for
start and deprovision, which ends the `expect(dead_code)` hook of the
name's accessor, and `serialize_launch` takes the snapshot to record
in place of `serialize_launch_with_snapshot`. The start is the
production caller of SL4.3's restore driver, which ends its hook and
its test-layout allowance.

This is the first half of SL3.8a, which adapts the first implementation
on esaurez/slim-3.2.2-sl3.8a-private-20261108 (e7d77fc), its restore
start, source record, view removal under the lock, and restore-ready
socket cleanup, to the CLI-owned lifecycle of SL3.5b, the typed
configuration of SL3.5a, the restore creation of SL2.16, and the
restore driver of SL4.3.

Tests: fake-backed restores, whose checked launch spawns a fake OpenVMM
and guest in place of OpenVMM, cover EROFS and GPT images with scratch
and RAM uppers, the restore's own DNS server and published listeners in
the activation record, the mapping table, and the workload; concurrent
restores of one snapshot, the second of which spawns its VM only once
the first one's has exited; failed restores, including a workload that
exits with status 3, which end their VM and launch before the snapshot
goes free, and a guest that hangs up after the workload's Start, whose
VM exits by itself within its grace; a spawn that cannot confirm that
its VM ended, whose VM the journal's recovery ends before the snapshot
goes free; a later start, which restores again with an upper in RAM
and refuses a scratch image that an earlier start gave to a guest; a
command that dies before Ready, whose VM the next lock's recovery ends
without any activation; deprovision, which waits for the snapshot's
lock before it removes the view, and, on Windows, removes a cold
sandbox whose state directory is no VM directory; the view removal's
refusals, a VM directory in the store among them; a start that refuses
a share of its snapshot store before it launches anything; and the
record's source.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Coupled launch recovery, snapshot hard-link exclusion, and Windows scratch-handle inheritance warrant human confirmation of reported KVM/MSHV/WHP qualification, particularly killed-command recovery and scratch-claim exclusivity.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants