Repository navigation
Description
Parent: #1084, in #1076. Needs: #1102, #1108, #1109, #1110, #1113, #1117. Unblocks: #1112, #1118, #1131.
Goal. the CLI keeps named snapshots (#853), with one exclusive lock per snapshot for concurrent restores. snapshot capture creates one through host::vm::create with boot.capture, NVXC's capture request, Process::wait and snapshot::finish_capture, which takes the Exited proof and writes the snapshot's entries in the VM directory; the command then moves only key.bin, snapshot.json and openvmm/ into the named snapshot's private directory, and keeps them immutable; snapshot restore starts a sandbox from one, under the lock, through the private restore driver; and run --restore-snapshot restores a one-off VM from a strict snapshot.
Notes.
run --restore-snapshotchanges meaning. Today it passes a raw OpenVMM snapshot to OpenVMM; now it takes the CLI's strict snapshots, and refuses a raw one until NVX can snapshot blockless VMs.- An interrupted capture isn't resumed. The capture VM goes through the store's launch journal, so the next command's recovery ends its VM and removes its directory, and the user captures again.
- A capture that doesn't end in time is never finished. If the VMM hasn't exited by the wait's deadline, the command stops the VM.
- One lock per snapshot. A restore holds its snapshot's exclusive lock from
create, which adds the hard links, through the peer-verified restore-ready event or the VM's verified exit;deprovisionholds it while it removes a restored sandbox's view, which the sandbox's record names. A restore whose command dies can't complete: OpenVMM checks the snapshot's files before it connects the restore-ready endpoint, once, so the CLI keeps no durable record of its restores, and the next command's recovery ends the VM like an interrupted launch. - A named snapshot is captured before a layered root exists. The capture VM is blockless: its command line holds
nvx_sandbox=1, so the shell/inithands it to nvx-init, which assembles no root and waits for NVXC's capture request. A restored VM gets its image, upper and workload setup at activation; no snapshot holds a running layered workload.
PRs, in order. Each PR has one goal, leaves dev green, and names in its description each old test of its area as kept, rewritten or retired, with each change that check_test_changes.py reports. Its tests follow the repository's test layout: a module's unit tests, and the code that only they use, sit in its tests.rs or tests/ and move with it. The estimates count core lines (code, not comments, docs or tests) and counted files; each PR records its measured values. Each PR does what its Scope says and no more: a case, input or layout that its Scope doesn't name, and whatever "Not in this PR" lists, is out of scope. Address every review comment within the scope, and ask before acting on one that asks for more. A PR whose measured core lines pass the stop that its Estimate names (twice the estimate, at most 600) stops and asks before continuing.
-
SL3.7a: The CLI keeps named snapshots.
- Objective: The CLI needs a small persistent home for snapshots that users identify by name and can restore concurrently without changing a snapshot's files. This PR adds only the named snapshot directory and its per-snapshot exclusion.
- Contents: The snapshot store (nvx-rs: template and clone commands in
nvx-rs sandbox#853): named snapshots, each a private directory that holds exactly a snapshot's three entries, immutable and published whole by a rename, and one exclusive lock per snapshot, kept outside its directory. There is no index of restored VMs: each restored sandbox's record names its snapshot. - Scope:
- The private
host::cli::snapshotsstore and the sandbox record field that names a restored sandbox's snapshot. - One private directory per named snapshot, containing exactly
key.bin,snapshot.jsonandopenvmm/. - Whole publication by rename, immutable published entries, and one exclusive lock per snapshot stored outside that snapshot's directory.
- Snapshot-name and layout tests for the store's supported named snapshots.
- The private
- Not in this PR:
- Durable in-flight capture or restore records, or reconciliation after restart: the launch journal handles interrupted VMs and a capture is redone.
- An index from snapshots to restored VMs: each restored sandbox record names its own snapshot.
- Shared lock modes or lock upgrades: the CLI starts one restore per command and needs one exclusive lock.
snapshot list,snapshot remove, retention policy or garbage collection: no command in this workstream needs them.- Snapshot capture or restore commands: SL3.7b, SL3.8a and SL3.8b add those users.
- Approach: Use the existing private-directory and file-lock helpers from the CLI's store. Keep the layout fixed and publish a completed temporary directory with one rename; do not build a database, transaction log, restored-VM index, reconciliation pass or generalized lock manager.
- Done when:
- A valid snapshot name maps to a private directory and a separate exclusive-lock path.
- A published named snapshot contains exactly the three required entries and opens as a strict snapshot.
- Published snapshot entries cannot be modified through the store.
- A restored sandbox record round-trips the name of its source snapshot.
- Needs: SL3.2b, in nvx-rs slimming 3.1.3: the CLI keeps its own sandbox store #1108; SL2.12b, in nvx-rs slimming 2.3.2: snapshots have one strict format, and a capture finishes into it #1102.
- Estimate: 150 core lines, 2 counted files. Past 300 core lines, stop and ask before continuing.
-
SL3.7b:
snapshot capturecreates a named snapshot.- Objective: Users need
snapshot captureto create a strict named snapshot through the new VM and NVXC APIs. The command must publish only a completed capture and leave interrupted or timed-out attempts for normal launch recovery, never resumption. - Contents: The command, on
host::vm::createwithboot.capture, whose command line holdsnvx_sandbox=1, so that the shell/inithands the capture VM to nvx-init's capture entry (SL7.7a), NVXC's capture request (SL4.2),Process::waitandsnapshot::finish_capture: theExitedproof goes tofinish_capture; if the wait's deadline passes first, the command stops the VM and never finishes that capture. The capture VM goes through the store's launch journal, so after an interrupted capture the next command's recovery ends its VM and removes its directory; a capture is never resumed. Oncefinish_capturereturns, the command publishes the named snapshot: it moves onlykey.bin,snapshot.jsonandopenvmm/into the store, which keeps them immutable; the capture VM directory's other files stay outside the snapshot, and the command then removes that directory. - Scope:
- The CLI snapshot command, capture orchestration and fake-backed CLI tests.
- A blockless capture VM created with
boot.captureandnvx_sandbox=1, then driven byControlSession::capture. Process::waitsupplies theExitedproof tosnapshot::finish_capture; a missed wait deadline stops the VM and never finishes the capture.- After
finish_capture, onlykey.bin,snapshot.jsonandopenvmm/move into the named store, then the capture VM directory is removed. - The capture uses the ordinary launch journal, so the next command ends an interrupted VM, removes its directory and publishes no snapshot.
- Not in this PR:
- Resuming or finishing an interrupted capture: the user starts a new capture.
- A durable capture transaction beyond the existing launch journal: no separate in-flight record is kept.
- Capturing a running layered workload: the capture VM is blockless and has no layered root.
- Moving logs, capabilities or other VM-directory files into the snapshot: the strict format has exactly three entries.
- Restoring the named snapshot: SL3.8a and SL3.8b add restore commands.
- Approach: Compose
vm::create, the existing launch journal, NVXC capture,Process::waitandsnapshot::finish_capture. Publish through the small store from SL3.7a only after exit and validation; do not add resumable workflow state, a capture reconciler or a second snapshot format. - Done when:
- A capture publishes exactly
key.bin,snapshot.jsonandopenvmm/, opens strictly and removes its VM directory. - A VMM that misses the exit deadline is stopped and no capture is finished.
- Recovery ends an interrupted capture VM, removes its directory and publishes no snapshot.
- A capture publishes exactly
- Needs: SL3.7a, above; SL2.14, in nvx-rs slimming 2.3.2: snapshots have one strict format, and a capture finishes into it #1102; SL3.3b, in nvx-rs slimming 3.1.3: the CLI keeps its own sandbox store #1108; SL4.2, in nvx-rs slimming 3.2.1: NVXC asks for a capture and activates a restore, and a blocking driver restores #1113; SL7.7a, in nvx-rs slimming 3.3.3: nvx-init captures and restores, and boots the new launches' layered roots #1117; SL7.8a, in nvx-rs slimming 3.3.3: nvx-init captures and restores, and boots the new launches' layered roots #1117.
- Estimate: 220 core lines, 3 counted files. Past 440 core lines, stop and ask before continuing.
- Objective: Users need
-
SL3.8a:
snapshot restorestarts a sandbox from a named snapshot.- Objective: Users need
snapshot restoreto create a managed sandbox from a named strict snapshot without concurrent link changes corrupting restore startup. The restored sandbox remains recoverable through the ordinary launch journal and records which snapshot supplies its view. - Contents: The command, on
host::vm::createwithfrom: Someand the CLI's blocking restore driver besidehost::session, under the snapshot's exclusive lock, held fromcreatethrough the peer-verified restore-ready event or the VM's verified exit. The restored sandbox's record names its snapshot, anddeprovisionholds that snapshot's lock while it removes the sandbox's view. A restore whose command dies can't complete; the next command's recovery ends it, like an interrupted launch. Its tests check that a restore's own network policy reaches the guest: its DNS and its announcer. - Scope:
- The CLI's
snapshot restorecommand, restored-sandbox record anddeprovisionpath, with fake-backed CLI tests. vm::createwithfrom: Somefollowed by the CLI's blocking restore driver, using the restored sandbox's workload and mapping table.- The snapshot's exclusive lock from
createthrough peer-verified restore-ready or verified VM exit. - The sandbox record names its snapshot, and
deprovisionholds that snapshot's lock while removing the restore view. - The restore's own network policy reaches the guest, including its DNS and announcer.
- The CLI's
- Not in this PR:
- Durable in-flight restore records or reconciliation: a dead command cannot complete activation, and normal launch recovery ends its VM.
- An index of restored VMs per snapshot: the sandbox's own record is the only association.
- Shared restore locks or parallel startup from one snapshot: one exclusive lock serializes the link-sensitive interval.
- Resuming a restore whose command died: the next command cleans it up.
- One-off
run --restore-snapshot: SL3.8b adds that path.
- Approach: Reuse managed sandbox creation, its launch journal and recovery, then call the blocking restore driver while holding the snapshot store's single lock. Add the snapshot name to the existing sandbox record; do not create a restore ledger, background reconciler or snapshot-to-VM index.
- Done when:
- Named snapshots restore with each supported image format and upper.
- Concurrent restores of one snapshot wait for the same exclusive lock.
deprovisionholds the snapshot lock while it removes the restored sandbox's view.- A command killed before restore-ready never completes, and the next command's recovery ends its VM.
- The restored guest receives the restore's DNS and announcer policy.
- Needs: SL3.7b, above; SL3.5b, in nvx-rs slimming 3.1.5: the sandbox lifecycle runs on the new interface and nvx-init #1110; SL4.3, in nvx-rs slimming 3.2.1: NVXC asks for a capture and activates a restore, and a blocking driver restores #1113; SL7.7b, in nvx-rs slimming 3.3.3: nvx-init captures and restores, and boots the new launches' layered roots #1117.
- Estimate: 250 core lines, 3 counted files. Past 500 core lines, stop and ask before continuing.
- Objective: Users need
-
SL3.8b:
run --restore-snapshotrestores a one-off VM from a strict snapshot.- Objective:
run --restore-snapshotcurrently passes an arbitrary OpenVMM snapshot through to OpenVMM. This PR makes it restore the CLI's strict named-snapshot format through the same controlled path assnapshot restore, with the one-off run's own network policy. - Contents: The option, on the same path as
snapshot restore, which installs the network options as the restore's own network policy, with its DNS, egress, forwards and published listeners.rundrops--restore-network-policy, whichdevadded to select that for a raw snapshot, since it refuses raw snapshots. A raw OpenVMM snapshot is refused until NVX can snapshot blockless VMs. - Scope:
- The CLI's
runrestore option, its option validation and fake-backed CLI tests. - Strict snapshots opened through the CLI and restored through the same create, lock and blocking restore-driver path as
snapshot restore. - The run's DNS, egress, forwards and published listeners become the restore's own network policy.
- Removal of
--restore-network-policy, whose raw-snapshot selection role is no longer needed. - A clear refusal of raw OpenVMM snapshots.
- The CLI's
- Not in this PR:
- Continued raw snapshot pass-through or automatic conversion: strict snapshots are the only accepted input.
- Snapshots of blockless VMs: that capability is a later OpenVMM and NVX follow-up.
- A second restore implementation for
run: it uses the path added by SL3.8a. - Durable restore records, restored-VM indexes or restore reconciliation: the snapshot store has none.
- Changing network policy payloads in NVXC: OpenVMM installs the policy and activation carries the guest-applied state.
- Approach: Route the existing option into SL3.8a's strict restore path and reuse the normal network-option lowering. Delete the raw pass-through switch instead of preserving two meanings; do not probe arbitrary snapshot layouts, convert formats or add compatibility modes.
- Done when:
run --restore-snapshotrestores a strict snapshot with its DNS, egress, forwards and published listeners.- A raw OpenVMM snapshot is refused before launch.
- CLI tests that covered raw pass-through now cover the strict path and the removed
--restore-network-policyoption.
- Needs: SL3.8a, above; SL3.4a2, in nvx-rs slimming 3.1.4:
runuses the new interface and the CLI's own guest tokens #1109. - Estimate: 150 core lines, 3 counted files. Past 300 core lines, stop and ask before continuing.
- Objective:
Tests. A capture leaves exactly key.bin, snapshot.json and openvmm/ in its private named snapshot directory, which opens as a strict snapshot, and removes the capture VM's directory; then restores with each image format and upper. A capture whose VMM doesn't exit by the deadline is stopped and stays unfinished. An interrupted capture's VM is ended, and its directory removed, by the next command's recovery, and no snapshot is published. Concurrent restores of one snapshot wait for each other's lock, and deprovision of a restored sandbox holds its snapshot's lock while it removes the view. A restore whose command is killed before restore-ready never completes, and the next command's recovery ends its VM. run --restore-snapshot restores a strict snapshot and refuses a raw OpenVMM one; today's CLI tests of the raw pass-through are rewritten for that.
Done when: every PR above has merged, with dev green.
Supporting typed blockless snapshot work
The following are separate supporting changes for the strict one-off restore path, not an instruction to hide their library/guest changes inside SL3.8b's CLI delta:
- Typed strict snapshot support: represent genuine zero-block snapshots explicitly in capture configuration, authoritative strict metadata/key validation, compatibility checks and VM preparation. Determine mode from validated metadata/block inventory, never from a missing caller image, informational
snapshot.json, invented geometries or a fabricated layered activation record. Preserve the fixed strict entries, owner-private immutable inputs, layered checks and existing layered key encoding. - One shared restore driver: consume that typed mode in the existing checked host/session/guest restore path. Retain layered transaction equality, resource/root checks, MAPS/EXEC ordering, peer/process verification, cancellation, exit and cleanup. Blockless repair uses the existing untiered packet semantics; DNS and listener repair must finish before the native host-input gate releases. An interrupted repair must fail closed rather than repeat irreversible effects.
- Preserve the correct RESET/reconnect/capture-entry ordering, and the snapshot lock from create through verified restore-ready or verified VM exit, including view removal and interrupted-command recovery.
- Keep raw snapshot acceptance/conversion, a second restore implementation, a durable restore index/reconciler, and speculative native/protocol changes out. Final user-facing parser and
runintegration remain SL3.8b. Exercise malformed/mismatched modes, interrupted repair, network application, reconnect ordering, cancellation and verified exit while retaining layered behavior.
The checkpoint comment below identifies the two supporting branches independently. Their implementation checkpoints do not imply completed qualification.
Activity
[-]nvx-rs slimming 3.2.2: nvx-cli captures and restores named snapshots[/-][+]nvx-rs slimming 3.2.2: the CLI captures and restores named snapshots[/+]esaurez commentedon Nov 10, 2026
Workstream source/progress checkpoint - 2026-11-10 UTC
Checked boxes below mean merged scope, not merely an implemented first iteration. Implementation branches are source checkpoints, not automatic qualification, Ready or merge approval. Some starting inputs are frozen/anonymous commits rather than standalone branches; their exact SHAs and known merged producer refs are recorded explicitly.
SL3.7a - First implementation committed; not merged.
esaurez/slim-3.2.2-sl3.7a-20261108; exact head593a17d04f3a245ed8257b320b36943c79bc9c9d.esaurez/slim-3.1.3-sl3.2b-20261108; exact commitcf485b72c69d9374506f94d00bf82bcb8b5a7cad.esaurez/slim-2.3.2-sl2.12b-20261107at1a6f6ffef2839359307f23fdf015b4b98b9b26ff.SL3.7b - First implementation committed; not merged.
esaurez/slim-3.2.2-sl3.7b-private-20261108; exact headaa62dc1e546a61bf49906b3b6bbe6db773d52a10.esaurez/slim-3.1.3-sl3.3b-20261108; exact commit45d2c69d966efaf2ad4284b94c045483e4fe8c97.esaurez/slim-3.2.2-sl3.7a-20261108at593a17d04f3a245ed8257b320b36943c79bc9c9d;esaurez/slim-2.3.2-sl2.14-20261108atc14644df8a763272b10b0b611d13fd4d90da62e6; unnamed producer input atcfda8b6c45a4af0023c7d30a75cee7325f03fef4;esaurez/slim-3.3.3-sl7.7a-20261108atf61c8708e84074bbd42fef9aad73f28bef8ca84c.SL3.8a - First implementation committed; not merged.
esaurez/slim-3.2.2-sl3.8a-private-20261108; exact heade7d77fcdda1e136d8710f664585a295e291a9c95.ddc46d71864a4401651a4fda96f7e431d26fa140.esaurez/slim-3.2.2-sl3.7b-private-20261108ataa62dc1e546a61bf49906b3b6bbe6db773d52a10;esaurez/slim-3.2.1-sl4.3-private-20261108atb30be68869e256d74c55cf6a5933e20d8a6c6fb7.SL3.8b - WIP - Enrique Saurez (@esaurez) (partial implementation).
esaurez/private-sl38b-0325-b2; exact headef52ba6b5a85e981ed930d69bd0e2077dc001092.esaurez/private-blockless-driver-0325; exact commit4646569b665c45c851c51466af3fec1c795bb894.4646569b665c45c851c51466af3fec1c795bb894; the separately published driver branch contains later commits. No automatic producer update/rebase or inherited validation is implied.Blockless snapshot API prerequisite - WIP - Enrique Saurez (@esaurez) (implementation committed).
esaurez/private-blockless-snapshot-0325; exact head5b1c70656719e528d561bdc3af997488b5dcf62a.69df0ce842a0be7a9bed1789e1d682fe25cf8fee.esaurez/slim-3.2.2-sl3.8a-private-20261108ate7d77fcdda1e136d8710f664585a295e291a9c95;esaurez/slim-3.1.4-sl3.4a2-20261108atdbd78d9beea69232f25df8d089963b5397bbb36d.Shared blockless restore-driver prerequisite - WIP - Enrique Saurez (@esaurez) (implementation committed).
esaurez/private-blockless-driver-0325; exact head3c6f02b0c467b8cf3de02dd699486fc6017d5dba.esaurez/private-blockless-snapshot-0325; exact commit1a308dbf549c24b95e87555b52c21c7551ef9937.The original technical scope remains authoritative, supplemented only by any portable technical clarification in this issue. No machine-specific execution policy is part of this checkpoint.
esaurez commentedon Nov 11, 2026
Retirement record for the historical SL3.7a input aggregate #1444. It is closed unmerged; its branch
esaurez/issue1114-inputs-0c8e42ad-20261110and source4662351ccc22198ffe16b6935d157bee14a163c0are retained.The merged named-store consumer #1445 records the merged CLI-store/strict-reader inputs and its retargeting away from this input-only join. This reference preserves the aggregate's scope and source under the sequential-PR workflow. It does not complete this issue or its remaining capture/restore slices.
ppenna commentedon Nov 11, 2026
Gap found while preparing SL3.8a and SL3.8b: no strict snapshot can carry a NIC today.
snapshot capture(SL3.7b, merged in Add named BLOCKLESS snapshot capture (SL3.7b) #1464) creates its capture VM withnetwork: Noneand a disconnected console (nvx-rs/src/host/cli/snapshots/capture.rs, theVmConfigthat it builds).Snapshot's compatibility check refuses a restore whose NIC presence differs from the snapshot's (nvx-rs/src/host/snapshot.rs, "snapshot NIC presence differs from the requested VM's").So every named snapshot restores only without a NIC. The parts of this group that put a restore's own network policy in the guest can be checked only by fake-backed tests for now:
run --restore-snapshotrestores a strict snapshot with its DNS, egress, forwards and published listeners."SL5.1c's real-VM snapshot smoke tests (#1118) have the same limit for networked restores.
Earlier first implementations assumed a
snapshot capture --net, which no merged scope adds. In SL3.8a and SL3.8b, the network-policy lowering is covered by fake-backed tests, and a NIC-less snapshot refuses network options with a clear message. Their PRs record this limitation.Giving a capture VM a NIC, so that networked restores can run on real VMs, needs its own scoped follow-up. It isn't in any PR of this group as written.
ppenna commentedon Nov 11, 2026
SL3.8a is split into two stacked PRs. Its measured size was 518 core lines in 8 counted files, against an estimate of 250 lines in 3 files. The PR rules allow at most 7 counted files.
deprovisionholds the snapshot's lock while it removes the view, and the record names its source. 346 core lines, 5 counted files.snapshot restorecommand. 193 core lines, 5 counted files.The total, 539 core lines, stays under the PR rules' 600-line hard limit, which I approved for this slice. SL3.8b (#1489) is stacked on #1487.
ppenna commentedon Nov 11, 2026
Added slice: SL3.7c, "a named snapshot can carry a NIC". This closes the gap that I recorded above (#1114 (comment)).
network: None), andSnapshotrequires the restore's NIC presence to match the snapshot's, so no strict snapshot restores with a network.snapshot capture --net. The capture VM gets a NIC under a deny-all policy (no egress, forwards, DNS or listeners), and the snapshot records the captured NIC that the library already supports. Without--net, capture is unchanged.snapshot restoreandrun --restore-snapshotof a NIC snapshot take the restore's own DNS, egress, forwards and listeners through the existing lowering. The CLI keeps refusing network options for a NIC-less snapshot.untiered-network-policyis rewritten on strict snapshots, retiring only its raw parts. SL3.8a's "restored guest receives the restore's DNS and announcer policy" and SL3.8b's DNS, egress, forwards and listeners checks run on real VMs, not only fake-backed.nvxa library of mechanisms and value types #1076's "Not in scope"), and snapshots of blockless VMs' running workloads.untiered-network-policyraw until SL3.7c lands. SL5.1c's case matrix records that row as "kept until SL3.7c".doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix
ppenna commentedon Nov 12, 2026
Completed. Every slice of this group has merged into
feature/nvx-refactor:2e41db76523.snapshot capturecreates a named snapshot: Add named BLOCKLESS snapshot capture (SL3.7b) #1464. Its follow-up fix of the capture test's flaky timeout is cli: start a capture test's fake VMM before its deadline (SL3.7b follow-up) #1495.snapshot restorestarts a sandbox from a named snapshot, split as recorded above:99855f75e0e;snapshot restorecommand: cli: restore managed sandboxes from named snapshots (SL3.8a2) #1487,64ad98afacc.run --restore-snapshotrestores a one-off VM from a strict snapshot: cli: restore named snapshots once with run (SL3.8b) #1489,e1367882b22and1b902ac16a5.f110dcc7210andeb05744bd87.untiered-network-policynow runs on named snapshots on KVM, MSHV and WHP.Passing CI that contains all of them: the push run on
feature/nvx-refactorateb05744bd87, https://github.com/microsoft/nvx/actions/runs/43189211488, including the microVM jobs on KVM, MSHV and WHP.doc: give the snapshot scenarios their fates in the case matrix
doc: give the snapshot scenarios their fates in the case matrix