Skip to content

nvx-rs slimming 3.1.5: the sandbox lifecycle runs on the new interface and nvx-init #1110

Closed

Description

Parent: #1083, in #1076. Needs: #1108, #1109, #1117. Unblocks: #1111, #1112, #1114.

Goal. The sandbox lifecycle runs on the new interface. --memory-max and --pids-max go first: the VM is the limit. Then sandbox start launches through host::vm::{create, spawn}, journaled and recoverable, with the CLI's tokens for the managed mode and the workload identity, and the shell /init hands its layered root to nvx-init. In the same PR, exec, stop and deprovision take their lock, records, recovery and reattach from the CLI's store, with the runtime identity check: stop stops gracefully over NVXC when it can attach and forces the VM down once its grace period ends, and deprovision removes everything that a sandbox owned. start first builds the VmConfig from the record that provision writes today; then provision records a GPT or EROFS image, and a scratch or a RAM upper with its cap (50% by default), in the CLI's store, and start reads them back. A sandbox can't share its own files: a protected path is refused at provision and again at start (#818).

Notes.

  • sandbox start waits for nvx-init, whose layered boot comes from nvx-rs slimming 3.3.3: nvx-init captures and restores, and boots the new launches' layered roots #1117. Until then the old start stays, and its VMs keep the old layered init.
  • A launch's records and their readers switch together. The old launch writes the old store's records, which exec, stop and deprovision read, so the PR that switches start switches their lock, record lookup, recovery and reattach too: no build has a start whose VMs the other commands can't manage.
  • start switches before provision. The new start reads the record that provision writes today, so each PR leaves both commands working; provision then extends the record and start's reader together.

PRs, in order. Each PR has one goal, leaves dev green, and names in its description each old test of its area as kept, rewritten or retired, with each change that check_test_changes.py reports. Its tests follow the repository's test layout: a module's unit tests, and the code that only they use, sit in its tests.rs or tests/ and move with it. The estimates count core lines (code, not comments, docs or tests) and counted files; each PR records its measured values. Each PR does what its Scope says and no more: a case, input or layout that its Scope doesn't name, and whatever "Not in this PR" lists, is out of scope. Address every review comment within the scope, and ask before acting on one that asks for more. A PR whose measured core lines pass the stop that its Estimate names (twice the estimate, at most 600) stops and asks before continuing.

  1. SL3.5d: --memory-max and --pids-max go: the VM is the limit.

    • Objective: Sandbox workloads no longer advertise per-workload memory or process limits that the new guest lifecycle does not enforce. The VM's resources are the boundary.
    • Contents: provision and sandbox run no longer take --memory-max or --pids-max, and provision no longer records them, so the old launch sets no per-workload limit. resource_limits_are_bounded_ascii_decimal retires with them.
    • Scope:
      • nvx-rs/src/host/cli/sandbox/options.rs removes --memory-max and --pids-max from provision and sandbox run parsing and help.
      • nvx-rs/src/host/cli/sandbox/mod.rs stops carrying those values into a launch.
      • nvx-rs/src/host/cli/sandbox/state.rs stops writing the two limits, so the old launch sets no per-workload limit.
    • Not in this PR:
      • Adding replacement limit flags or another process-control mechanism: the VM is the limit.
      • Changing --memory-mib or the RAM upper's cap: those size the VM and its upper, not an individual workload.
      • Migrating old configuration records or preserving the removed CLI options: no compatibility reader or old-output compatibility is required.
      • Changing any other sandbox option, command or output.
    • Approach: Delete the two option specifications, fields and record writes along their existing straight-line path. Do not retain ignored aliases, record placeholders or a generalized limits object; retire resource_limits_are_bounded_ascii_decimal from the sandbox module's tests.rs with the options.
    • Done when:
      • provision and sandbox run reject --memory-max and --pids-max as unknown options.
      • New provision records contain neither per-workload limit.
      • resource_limits_are_bounded_ascii_decimal is retired with the removed options.
    • Needs: nothing.
    • Estimate: 40 core lines, 3 counted files. Past 80 core lines, stop and ask before continuing.
  2. SL3.5b: sandbox start launches through the new APIs, and the lifecycle commands manage its VM.

    • Objective: sandbox start launches a VM that every lifecycle command can recover, reattach, stop and remove through the CLI's store and the new VM interface. The layered root reaches nvx-init without an OpenVMM lifecycle option.
    • Contents: start on host::vm::{create, spawn}, journaled and recoverable, with the CLI's tokens for the managed mode and the workload identity. It builds the VmConfig from the record that provision writes today: the EROFS image, the scratch or RAM upper, the network and the export. Its command line holds nvx_sandbox=1 and no OpenVMM lifecycle token, so the shell /init hands its root to nvx-init (SL7.8a). The readers of its records switch with it: exec, stop and deprovision take the lock, the record lookup, the recovery and the reattach from the CLI's store, through host::vm::reattach with the runtime identity check; stop sends NVXC STOP when it can attach, and forces the VM down through host::vm::stop once its grace period ends; deprovision releases the log and scratch claims, and removes the VM directory and every record of the sandbox. A managed VM's outcome report stays readable under a restrictive umask, and the store and the supervisor are read from their split files.
    • Scope:
      • nvx-rs/src/host/cli/sandbox/lifecycle/ switches start, exec, stop and deprovision together to the CLI's lock, journal, runtime record, capability and claim files.
      • start builds VmConfig from today's provision record (its EROFS image, scratch or RAM upper, network and export), calls host::vm::create, journals the prepared capability and outputs, then calls host::vm::spawn.
      • start uses the CLI's managed-mode and workload-identity tokens, writes nvx_sandbox=1, and passes no OpenVMM lifecycle option.
      • exec and stop reattach with host::vm::reattach and the runtime identity check; stop requests NVXC STOP, then uses host::vm::stop after the grace period.
      • deprovision releases log and scratch claims and removes the VM directory plus every sandbox record; restrictive umasks do not hide the outcome report.
    • Not in this PR:
      • Writing the new GPT, image-format and RAM-cap provision record: SL3.5a follows after this PR.
      • Reading both old and new runtime stores or migrating running sandboxes: a launch and all of its readers switch together.
      • Changing exec admission: SL3.9 adds the separate per-sandbox exec lock.
      • New lifecycle commands, flags or output redesigns beyond the required launch switch.
      • Making the CLI helpers private: phase 4 owns those visibility changes after their library users leave.
    • Approach: Adapt the existing lifecycle flow in place to the CLI's store and host::vm; preserve today's provision-record reader until SL3.5a. Journal Prepared values before spawning, reuse the existing recovery and command tests, and switch all record readers in this PR rather than adding dual-store adapters.
    • Done when:
      • With today's provision record and each existing upper, provision, start, exec, stop, a second start and stop, and deprovision complete.
      • The launch journal exists before spawn, and a later command recovers an interrupted start while retaining an unverified live launch.
      • exec and stop check runtime identity; stop is graceful when attachment works and forced after its grace period.
      • deprovision removes every sandbox record, claim and VM file, and the outcome report remains readable under a restrictive umask.
      • start sends nvx_sandbox=1, sends no OpenVMM lifecycle option, and reads the split store and supervisor files.
    • Needs: SL3.5d, above; SL3.3b, in nvx-rs slimming 3.1.3: the CLI keeps its own sandbox store #1108; SL3.4a2, in nvx-rs slimming 3.1.4: run uses the new interface and the CLI's own guest tokens #1109; SL7.8a, in nvx-rs slimming 3.3.3: nvx-init captures and restores, and boots the new launches' layered roots #1117.
    • Estimate: 540 core lines, 5 counted files. Past 600 core lines, stop and ask before continuing.
  3. SL3.5a: sandbox provision records a GPT or EROFS image, and the RAM upper's cap, in the CLI's store.

    • Objective: sandbox provision records the root image and upper configuration that the new launcher needs. A later start can reproduce EROFS or GPT with scratch or a bounded RAM upper from the CLI's own record.
    • Contents: provision on host::cli::store, with a VmConfig built from its options: a GPT or EROFS image by path, a scratch or RAM upper, and the RAM upper's cap (50% by default). Its records name the format and the upper, and start reads them back (nvx-rs: GPT images, the RAM upper and protected paths in nvx-rs sandbox #818).
    • Scope:
      • nvx-rs/src/host/cli/sandbox/ builds a VmConfig from provision's image and upper options.
      • Provision accepts an EROFS or GPT image by path, and a scratch upper or RAM upper.
      • A RAM upper records an explicit cap from 1 through 90 percent, or 50 percent when omitted.
      • the CLI's configuration record names the image format, image path, upper kind and RAM cap, and start reads those fields back.
    • Not in this PR:
      • File-descriptor or final-path transfer for images: the new interface takes image paths.
      • Additional image formats, multiple lower images or upper kinds: the plan has EROFS, GPT, scratch and RAM only.
      • Migrating older records or writing two record formats: no compatibility reader is required.
      • New sandbox commands, unrelated flags or output redesigns.
      • Protected-path checks for shares: SL3.5c adds those after the record shape lands.
    • Approach: Extend the CLI's existing provision record and its start reader together, using RootConfig::Layered, the existing image option parsing and UpperConfig. Store the enum choices and cap directly; do not add format discovery, an image registry or a record-conversion layer.
    • Done when:
      • The lifecycle passes with EROFS and GPT, each using scratch and RAM uppers.
      • An explicit RAM cap and the 50 percent default are persisted and read back by start.
      • The record names the image format and upper kind that start places in VmConfig.
      • configurations_of_every_format_start_and_exec, lifecycle_of_a_sandbox_whose_upper_is_in_ram and the_upper_is_scratch_or_ram are kept or rewritten.
    • Needs: SL3.5b, above.
    • Estimate: 300 core lines, 4 counted files. Past 600 core lines, stop and ask before continuing.
  4. SL3.5c: A sandbox can't share its own files.

    • Objective: A sandbox cannot expose its own control or storage files back into its guest. Provision and start both reject such shares after resolving the relevant host paths.
    • Contents: A share whose host path equals, contains or lies inside the sandbox's state directory, scratch, images or artifacts is refused, by canonical path, at provision and again at start (nvx-rs: GPT images, the RAM upper and protected paths in nvx-rs sandbox #818). No alias matrix.
    • Scope:
      • nvx-rs/src/host/cli/sandbox/ checks each share's canonical host path against the sandbox state directory, scratch, images and artifacts.
      • A share is refused when its path equals a protected path, contains one, or lies inside one.
      • provision checks before recording, and start repeats the check before launching so changed path resolution cannot bypass it.
    • Not in this PR:
      • An alias matrix, inode graph, bind-mount discovery or continuous monitoring: the required rule is canonical-path containment.
      • Protecting arbitrary host paths beyond the sandbox's state, scratch, images and artifacts.
      • Changing share syntax, adding share flags or redesigning refusal output.
      • Checking sandbox run here: SL3.4b checks the protected paths that apply to its stateless launch.
    • Approach: Reuse the CLI's existing path resolution and containment helpers to compare canonical paths in both call sites. Keep one bounded list of protected roots; do not build a filesystem topology model or attempt to detect every possible alias.
    • Done when:
      • provision refuses a share equal to, containing or inside the state directory, scratch, an image or the artifacts.
      • start repeats the same refusals from the recorded configuration.
      • Ordinary shares outside the protected paths still provision and start.
    • Needs: SL3.5a, above.
    • Estimate: 120 core lines, 2 counted files. Past 240 core lines, stop and ask before continuing.

Tests. provision and sandbox run refuse --memory-max and --pids-max, and resource_limits_are_bounded_ascii_decimal is retired with them. At the launch's PR, with the record that provision still writes and each upper: provision, start, exec, stop, a second start and stop, and deprovision. start's command line holds nvx_sandbox=1 and no OpenVMM lifecycle token, so its root boots through nvx-init; its journal is written before the spawn, and a command after an interrupted start recovers it through the new journal and keeps an unverified live launch. exec and stop check the reattached VM's runtime identity; stop sends NVXC STOP when it can attach, and forces the VM down once its grace period ends; deprovision releases the log and scratch claims, and removes the VM directory and every record of the sandbox. After provision's new record, the same lifecycle with GPT and EROFS, each with a scratch and with a RAM upper: an explicit cap and the 50% default, the format and the upper persisted and read back by start. A share whose host path equals, contains or lies inside the sandbox's state directory, scratch, images or artifacts is refused, by canonical path, at provision and again at start. The CLI tests of these commands, such as lifecycles_of_every_configuration, lifecycle_of_a_sandbox_whose_upper_is_in_ram, configurations_of_every_format_start_and_exec and the_upper_is_scratch_or_ram, are kept or rewritten.

Done when: every PR above has merged, with dev green.

Activity

  1. esaurez commented on Nov 10, 2026

    @esaurez
    ContributorAuthor

    Workstream source/progress checkpoint - 2026-11-10 UTC

    Checked boxes below mean merged scope, not merely an implemented first iteration. Implementation branches are source checkpoints, not automatic qualification, Ready or merge approval. Some starting inputs are frozen/anonymous commits rather than standalone branches; their exact SHAs and known merged producer refs are recorded explicitly.

    The original technical scope remains authoritative, supplemented only by any portable technical clarification in this issue. No machine-specific execution policy is part of this checkpoint.

  2. ppenna commented on Nov 11, 2026

    @ppenna
    Contributor

    All four PRs of this group are on feature/nvx-refactor: SL3.5d (#1264, merged into dev, which the branch contains), SL3.5b (#1460), SL3.5a (#1466) and SL3.5c (#1485).

    The feature branch's post-merge CI run at f834bcb967, which contains all of them, passed: https://github.com/microsoft/nvx/actions/runs/43114171980. Closing this group as complete; dev receives the rest when feature/nvx-refactor lands there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions