From 9426169ab0602573c7888069caab6ca20f5d3f30 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 12 Nov 2026 07:43:59 -0800 Subject: [PATCH 1/3] doc: drop the deleted sandbox bootstrap from the build and guest docs SL7.8b (#1534) deleted the old layered init, nvx-init-agent, with the managed agent's sandbox mode, and SL7.8c (#1536) deleted the Alpine one-shot container helpers. The build guide still said that the Alpine initramfs includes "the sandbox PID-1 bootstrap and its container namespace scripts", that the kernel checks cover what "the sandbox bootstrap" needs, and that the APK manifest records the tools "used by the bootstrap". The Ubuntu guest still named "the Alpine control initramfs and its init agent" as the boot supervisor of the distro layer, and the cgroups and namespaces "that the sandbox uses". The project structure still credited the common scripts with shutdown and the time ABI, which the Rust guest binaries took over. Say instead what the code does now: - the Alpine initramfs is the one that nvx-rs sandbox boots, and with nvx_sandbox=1 the common /init hands PID 1 to nvx-init (exec /sbin/nvx-init in guest/common/init), which initializes the sandbox's layered root and serves its managed control session; - the APK manifest records every package of the image, among them the five that AlpineBuildConstants.PACKAGES adds to the minirootfs; - nvx-init supervises the Ubuntu distro layer, and the kernel provides the memory and process-count cgroups and the PID, mount, UTS and network namespaces for the guest's programs, without claiming that the sandbox uses them; - the common scripts mount the host share (nvx-hostmount), prepare the guest for a snapshot and repair its restores (nvx-snapshot), and probe for the tests and benchmarks; the Rust binaries provide PID 1, the time ABI's guest side, shutdown and the rest. The kernel inventory and its checks table keep the text that SL6.11b (#1542) gave them. Part of #1131 (SL6.10e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 11 +++++++---- doc/guests/ubuntu-guest.md | 6 +++--- doc/project-structure.md | 11 +++++++---- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/doc/build.md b/doc/build.md index 9601998b7..0d9c85830 100644 --- a/doc/build.md +++ b/doc/build.md @@ -206,8 +206,10 @@ Ubuntu, and Azure Linux; the kernel targets do not depend on that build. The Ubuntu determinism target instead rebuilds them from fresh targets in both attempts. Native image builds also rebuild them fresh. -The Alpine initramfs additionally includes the sandbox PID-1 bootstrap and -its container namespace scripts. `nvx-time-probe` measures TSC and +The Alpine initramfs is the one that `nvx-rs sandbox` boots: with +`nvx_sandbox=1` on the kernel command line, the common `/init` hands PID 1 to +`nvx-init`, which initializes the sandbox's layered root and serves its managed +control session. `nvx-time-probe` measures TSC and clock read latency, reports the guest's clock, clocksource, TSC flags, and hypervisor identity, and runs the cross-vCPU warp test that the [time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs @@ -231,8 +233,9 @@ predate the ABI. All ten executables link the musl supplied by the pinned Rust target. Every package manifest records the shared Rust source digest and each installed binary digest. The [kernel configuration checks](#kernel-configuration-checks) cover the kernel -features that the sandbox bootstrap needs. The APK manifest records the -`blkid` and `util-linux` tools used by the bootstrap along with every +features that sandboxes need. The APK manifest records every package of the +Alpine image, among them `blkid`, `busybox-extras`, `e2fsprogs`, `util-linux`, +and `util-linux-misc`, which the build adds to the minirootfs, along with every guest-binary digest. The Ubuntu initramfs uses Ubuntu userland with the NVX kernel. It is not an diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index 95b078899..417e3587d 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -16,7 +16,7 @@ describes how the pieces fit together and why. | Artifact | Used by | Boot supervisor | | --- | --- | --- | | `initramfs-ubuntu.cpio.gz` | `nvx-rs run --guest ubuntu` | The common NVX `/init` inside the Ubuntu initramfs | -| `ubuntu-distro.erofs` | `nvx-rs sandbox --layer distro,...` | The Alpine control initramfs and its init agent | +| `ubuntu-distro.erofs` | `nvx-rs sandbox --layer distro,...` | `nvx-init`, which the common `/init` of the Alpine initramfs starts | Alpine remains the default guest of `run` and the only control environment of `sandbox`. The sandbox launcher always boots the Alpine initramfs, and the @@ -38,8 +38,8 @@ what Ubuntu userland needs on this machine: - devtmpfs, procfs, sysfs, and tmpfs; - virtio-mmio block, network, console, and filesystem devices; - ext4, EROFS, and overlayfs; and -- the memory and process-count cgroups and the PID, mount, and UTS namespaces - that the sandbox uses, and network namespaces for guest services. +- the memory and process-count cgroups, and the PID, mount, UTS, and network + namespaces, which programs in the guest may use. Its [patches](../../kernel/patches) add the xe9 early and interactive consoles and the shared virtio-mmio interrupt status, and keep the HVC console from diff --git a/doc/project-structure.md b/doc/project-structure.md index a0be6cbf6..de3370de5 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -235,10 +235,13 @@ profiling, or the interactive shell. On an interactive boot, `init` prints a stable distribution boot marker for Alpine and Azure Linux, and `guest/ubuntu/nvx-bashrc` prints Ubuntu's. -The remaining common helpers handle shutdown, virtio-fs mounting, snapshot -preparation, and the guest side of the time ABI. The Rust guest binaries -provide shutdown, managed control, restore repair, device access, ingress -announcements, and diagnostic probes. +The other common scripts mount the live host share (`nvx-hostmount`) and +prepare the guest for a snapshot and repair its restores (`nvx-snapshot`); the +microVM tests and benchmarks run the rest as probes (`nvx-identity-probe`, +`nvx-sandbox-smoke`, and `nvx-virtio-restore-probe`). The Rust guest binaries +provide a layered root's PID 1, managed control, the guest side of the time +ABI, restore repair, shutdown, device access, ingress announcements, and +diagnostic probes. `ubuntu/packages.lock.json` pins the complete supplemental `.deb` closure installed without maintainer-script execution. From be94d41960d8e3bf3f8485f1777440c17ea7bc91 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 12 Nov 2026 07:44:36 -0800 Subject: [PATCH 2/3] doc: describe the nvx package's features, profiles, and binaries The build, distribution, and structure docs described the nvx crate only in passing: none listed its features, its Cargo profiles, or which of its binaries each build makes and where each ships; the guest docs spoke of an "aggregate nvx-rs source digest" without saying which files it covers; no document said how a guest that NVX does not build gets the guest tools; and the structure still credited the crate with artifact discovery, which SL6.4 (#1533) and SL6.6b made private to the CLI or deleted. Build gains two sections, traced to nvx-rs/Cargo.toml, Cargo.lock, scripts/nvx_tools/{guest_binaries,nvx_rs,release,build_constants}.py, ci.yml, package-platform.yml and the package-release action: - The nvx package: the one package, why it has its own lockfile, that NVX's builds pass --locked and the crate ships the lockfile; fixed tables of its five features and of its profiles, with the guest profile's settings, and of its binaries, with the feature, profile, target, builder and destination of each; its outputs; its caches: Cargo's target directories and registry, the guest routine's fresh target directory, which keeps RUSTC_WRAPPER, the sccache cache of the self-hosted runners, and CI's guest image keys on the guest inputs, which it lists as GuestRustBuildConstants selects them; its three provenance records; and its license inputs, nvx-rs/LICENSE, the lockfile and THIRD_PARTY_NOTICES.md, which the unit tests keep in step. - Release packages: the binary matrix of CI's three packages, the OpenVMM and nvx-rs build of each, the guest artifacts that each ships, the sandbox scratch template among them, and the archive, with a link to Continuous integration's Release packages section, which describes the package jobs. Distribution links them, leaves the package jobs' mechanics to that section of Continuous integration, adds the guest binaries and the crate's own license to its licensing paragraph, says what the crate holds, and documents installing the guest tools from the crate with cargo install nvx --version VERSION --locked --no-default-features --features guest --profile guest --target x86_64-unknown-linux-musl, with why each option is there. The structure lists the package's files, Cargo.toml, Cargo.lock, LICENSE and README.md, its top-level modules, with what is private to the crate since SL6.5a2, SL6.7b, SL6.8c, SL6.13b and SL6.14 (the CLI's export planner, egress policy compiler and sandbox launch profile, the error codes, the guest's space, the platform layer), and what each binary calls, guest_binaries.py and guest_artifact_reuse.py, and what build/, .cache/ and nvx-rs/target/ hold. The Ubuntu and Azure Linux guests name the package's guest inputs, link the release packages, and point to the scratch template. Evidence, on nvx-mshv-test (Azure Linux 3, Rust 1.95.0) at 04fbbd480bf: cargo build --no-default-features --features guest --profile guest finished the guest profile; cargo install --path nvx-rs and cargo install nvx --version 0.1.0-alpha.16 with the documented options installed nvx-ingress as a static-pie executable, which prints its usage and exits 2; without --target it is dynamic, and without --version cargo install finds no version; --bins installs all ten. build-guest-binaries wrote a manifest whose source_sha256 equals guest-binary-source-identity, and removed its work directory. Part of #1131 (SL6.10e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 160 ++++++++++++++++++++++++++++++++- doc/distribution.md | 89 ++++++++++++------ doc/guests/azurelinux-guest.md | 14 +-- doc/guests/ubuntu-guest.md | 13 +-- doc/project-structure.md | 74 +++++++++------ 5 files changed, 284 insertions(+), 66 deletions(-) diff --git a/doc/build.md b/doc/build.md index 0d9c85830..31ce6292e 100644 --- a/doc/build.md +++ b/doc/build.md @@ -43,9 +43,10 @@ The routine takes the exact toolchain from `rust-toolchain.toml`, the uses `--locked`, a fresh target directory, disabled incremental compilation, stable source and Cargo-home path remapping, and no linker build IDs. One Cargo invocation builds all ten programs, and every result must pass -`check-static-elf`. The output's `manifest.json` binds the exact build pins, -one digest over the `nvx` crate, the shared Python build recipe, and -`rust-toolchain.toml`, and each binary SHA-256. +`check-static-elf`. The output's `manifest.json` records the toolchain, the +target, and the profile, the SHA-256 of each binary, and one digest over the +guest inputs of the `nvx` package, which +[The nvx package](#the-nvx-package) lists. The native build keeps the Linux archive, the Ubuntu Base archive, and the supplemental Ubuntu packages under `downloads/` in the cache directory, @@ -510,6 +511,159 @@ and SHA-256. The debug kernel is a CI artifact and is never packaged. The `build-guest-artifacts` action caches and builds it only when its `components` input names `debug-kernel`. +## The nvx package + +[`nvx-rs/`](../nvx-rs/README.md) holds the repository's one Cargo package, +`nvx`. Its library is the crate that crates.io publishes, as +[The nvx crate](distribution.md#the-nvx-crate) describes, and its binaries are +NVX's own programs, each of which calls a `main` that `#[doc(hidden)]` hides in +the library. The package belongs to no workspace, since a workspace at the +repository root would interact with the nested `openvmm/` workspace, so it +keeps a lockfile of its own, `nvx-rs/Cargo.lock`, and builds without the +OpenVMM submodule. The builds that `nvx.py`, the setup scripts, and CI's +`nvx-rs` jobs run pass `--locked`, so they build the dependency versions that +the lockfile pins, and the crate ships the lockfile, which +`cargo install --locked` reads. NVX builds the package with the Rust release +that `rust-toolchain.toml` pins, and CI also checks it with the oldest release +that it supports, the manifest's `rust-version`; see +[Rust crates](ci.md#rust-crates). + +Its features select what it builds: + +| Feature | Builds | Turns on | +| --- | --- | --- | +| `host` (default) | The `host` module, the library's host side | The `serde` feature, and the optional `getrandom`, `serde_json`, `sha2`, and `nix` on Unix or `windows-sys` on Windows | +| `guest` | The `guest` module, on Linux, and the ten guest binaries | The optional `nix` and `getrandom` | +| `serde` | The serde forms of `Hypervisor` and `net::Ipv4Cidr` | The optional `serde` | +| `cli` | NVX's `nvx-rs` binary, which adds only its hidden entry point to the API | The `host` feature, and the optional `crossterm` | +| `testing` | NVX's `nvx-fake-openvmm` binary, a fake OpenVMM for tests, which adds only its hidden entry point to the API | The optional `getrandom`, `serde`, `serde_json`, and `nix` on Unix or `windows-sys` on Windows | + +Without default features, the library holds only what the host and the guest +share: `protocol`, `activation`, `contract`, `net`, `gpt`, `uuid`, +`Hypervisor`, and the crate's error types, such as `Error`. The manifest +defines one Cargo profile of its own, and leaves Cargo's built-in profiles at +their defaults. NVX's builds use these: + +| Profile | Settings | Builds | +| --- | --- | --- | +| `dev` and `test` | Cargo's defaults | The crate's checks and tests | +| `release` | Cargo's defaults | `nvx-rs`, with `--release` | +| `guest` | `release`'s, with `opt-level = "s"`, `lto = "fat"`, `codegen-units = 1`, `panic = "abort"`, `strip = true`, `debug = false`, and `incremental = false` | The guest binaries, small and stripped | + +Cargo applies a package's profiles only to a build whose root it is, such as +one in `nvx-rs/`, one with `--manifest-path nvx-rs/Cargo.toml`, or +`cargo install nvx`; a package that depends on `nvx` builds it with its own +profiles. The guest routine also drops inherited `RUSTFLAGS` and +`CARGO_PROFILE_*` overrides, so that the `guest` profile applies as the +manifest gives it. + +The package's binaries build with these features, profiles, and targets: + +| Binary | Feature | Profile and target | Built by | Ships in | +| --- | --- | --- | --- | --- | +| `nvx-rs` | `cli` | `release`, for the host's target in a checkout; for release packages, static `x86_64-unknown-linux-musl` on Linux and `x86_64-pc-windows-msvc` with a static C runtime on Windows | The [`nvx-rs` build command](usage.md#nvx-rs), which `nvx.py` and the setup scripts run, and CI's `nvx-rs` build jobs | `bin/nvx-rs[.exe]` of every [release package](#release-packages) | +| `nvx-fake-openvmm` | `testing` | The tests' | `cargo test` with `testing`, for the crate's tests, which run it as OpenVMM | No package | +| `nvx-console-pending`, `nvx-device-io`, `nvx-exit`, `nvx-ingress`, `nvx-init`, `nvx-managed-agent`, `nvx-mmio-write`, `nvx-reseed`, `nvx-time`, `nvx-time-probe` | `guest` | `guest`, static `x86_64-unknown-linux-musl` | [`build-guest-binaries`](usage.md#build-guest-binaries), for every guest image | `/sbin` of every initramfs; other guests [install them from the crate](distribution.md#the-guest-tools) | + +Its examples, `external-launch` and `host-workflows` with `host`, and +`guest-workflows` with `guest`, build with the crate's checks of all targets; +the [README](../nvx-rs/README.md#testing) says what each shows. + +The builds put their output here: + +| Output | Path | Written by | +| --- | --- | --- | +| `nvx-rs` | `nvx-rs/target/release/nvx-rs[.exe]`, where `nvx.py`, `package`, and the setup scripts look for it | `cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target`; CI's Linux job builds for `x86_64-unknown-linux-musl` and copies `nvx-rs/target/x86_64-unknown-linux-musl/release/nvx-rs` there | +| Its provenance | `build/nvx-rs.provenance.json` | `nvx.py` when it builds `nvx-rs`, or [`record-nvx-rs-provenance`](usage.md#record-nvx-rs-provenance) | +| The guest binaries, with their `manifest.json` | The `--output` directory of `build-guest-binaries`, or an image build's work directory, such as `build/initramfs-alpine-work/guest-binaries`, unless `NVX_GUEST_BINARIES` names a build to install | `build-guest-binaries`, or the image build, which runs the same routine | +| Cargo's other builds | `nvx-rs/target/`, in a directory per profile, such as `debug/` or `guest/`, under a directory per target when the build names one | Cargo | + +The package keeps no cache of its own. `nvx-rs/target/` keeps the checkout's +Cargo builds between runs, and `--target-dir nvx-rs/target` keeps `nvx-rs` +there even where `CARGO_TARGET_DIR` names another directory; Cargo keeps the +crates that the lockfile pins in its registry cache. The guest routine reuses +no earlier target tree: it builds in a fresh target directory beside its +output, without incremental compilation, and removes it afterwards. It keeps +the environment's `RUSTC_WRAPPER`, though, so where that names sccache, Cargo +runs rustc through sccache, which can return a compilation from its own cache. +CI restores no Cargo target tree of the package, but the services of the +self-hosted runners set `RUSTC_WRAPPER=sccache`, with a persistent cache in +`_work/_sccache`, so the jobs that compile the package there, such as the +microVM jobs whose `vm-lifecycle` and `managed-lifecycle` scenarios run its +tests with Cargo, compile it through that cache; see +[Compilation cache](ci.md#compilation-cache). The keys of the guest images' +cache entries end with the digest of the package's guest inputs, which +[`guest-binary-source-identity`](usage.md#guest-binary-source-identity) +prints, so that a change to the package rebuilds the images only when it +changes one of those inputs; see +[Guest artifact handoff](ci.md#guest-artifact-handoff). The guest inputs are: + +- `nvx-rs/Cargo.toml`, `nvx-rs/Cargo.lock`, and any Cargo configuration in + `nvx-rs/.cargo/`; +- every file under `nvx-rs/src/` but those of the host side, `src/host/`, + `src/testing/`, and the `nvx-rs` and `nvx-fake-openvmm` binaries, and those + of the test modules, the `tests.rs` files and the `tests` directories; and +- `rust-toolchain.toml`, and the two modules that define the build, + `scripts/nvx_tools/build_constants.py` and + `scripts/nvx_tools/guest_binaries.py`. + +The guest build fails when Cargo's dep-info lists a file outside them, other +than one that the build generates in its own target directory. + +Three records bind the package's binaries to their sources: + +- `build/nvx-rs.provenance.json` binds an `nvx-rs` executable's SHA-256 to the + digest of the sources that built it, the package's manifest, lockfile, and + `src/` tree and `rust-toolchain.toml`, and to the build's Rust release, + target, and features. `package` and the adversarial executor refuse an + `nvx-rs` that it does not describe as built from the checkout's sources; see + [nvx-rs](usage.md#nvx-rs). +- The guest binaries' `manifest.json` binds their SHA-256 digests to the digest + of the guest inputs, and every guest image's package manifest records that + digest and the SHA-256 of each binary that it installs. +- A release package's `SOURCE-MANIFEST.json` records the packaged `nvx-rs` and + its provenance in its `nvx_rs` section, and the guest binaries' Rust release + and target in its `guest_rust` section, as + [Release packages](#release-packages) describes. + +On crates.io, the crate's `.cargo_vcs_info.json` records the commit from which +CI packed it; see [crates.io releases](ci.md#cratesio-releases). + +The package is MIT licensed, as the `license` field of its manifest says, and +`nvx-rs/LICENSE` holds the license, which the crate ships with the README and +the lockfile. The binaries that NVX ships, `bin/nvx-rs[.exe]` and the guest +binaries, statically link the crates that `nvx-rs/Cargo.lock` pins by checksum, +and their target's runtime: musl and LLVM's `libunwind` on Linux, and the +Microsoft C runtime on Windows. +[`THIRD_PARTY_NOTICES.md`](../THIRD_PARTY_NOTICES.md) lists each of these with +its license, and the crates at the versions that the lockfile pins. Unit tests +in `scripts/test_nvx_tools.py` fail when the lockfile gains a crate that they +do not classify as linked into `bin/nvx-rs` or not, or when the notices lack a +linked crate at its pinned version or a runtime that the binaries link. Release +packages ship the notices, and +[Package and source delivery](distribution.md#source-inclusive-packages) +describes what each license asks of a distributor. + +## Release packages + +Each platform's release package ships the OpenVMM and `nvx-rs` executables of +its operating system, and the same guest artifacts. CI's package jobs, which +[Release packages](ci.md#release-packages) in Continuous integration +describes, stage the three binary packages from these builds: + +| File | `linux-kvm` | `linux-mshv` | `windows-whp` | +| --- | --- | --- | --- | +| `bin/openvmm[.exe]` | `x86_64-unknown-linux-gnu` | Static `x86_64-unknown-linux-musl` | `x86_64-pc-windows-msvc` | +| `bin/nvx-rs[.exe]` | Static `x86_64-unknown-linux-musl` | Static `x86_64-unknown-linux-musl` | `x86_64-pc-windows-msvc`, with a static C runtime | +| `guest/vmlinux` and its configuration, and the Alpine, Ubuntu, and Azure Linux initramfses with their package manifests | Shipped | Shipped | Shipped | +| `guest/ubuntu-distro.erofs`, the Ubuntu distro layer, with its manifest | Shipped | Shipped | Shipped | +| `guest/ubuntu-smoke-scratch.ext4`, the sandbox scratch template, which has no manifest | Shipped | Shipped | Shipped | +| Archive | `nvx-VERSION-linux-kvm.tar.gz` | `nvx-VERSION-linux-mshv.tar.gz` | `nvx-VERSION-windows-whp.zip` | + +`VERSION` is the content of the `VERSION` file. +[Package and source delivery](distribution.md) describes the package modes and +how CI publishes the packages. + Release packaging needs the OpenVMM binary, the guest artifacts in the layout below, and the OpenVMM, kernel, and Alpine initramfs provenance sidecars, so run `build-openvmm` and `build-guest --guest all` first. It also needs the release diff --git a/doc/distribution.md b/doc/distribution.md index 154837de1..81500c90e 100644 --- a/doc/distribution.md +++ b/doc/distribution.md @@ -1,10 +1,11 @@ # Package and source delivery `package` stages a release directory, `dist/` by default, from the -built OpenVMM binary, `nvx-rs` executable, and guest artifacts. [Build](build.md) -lists the artifacts and provenance it requires and the binary release layout. -Every package is either binary-only or source-inclusive, and you must choose -one. +built OpenVMM binary, `nvx-rs` executable, and guest artifacts. +[Release packages](build.md#release-packages) lists the artifacts and +provenance it requires, the executables of each platform's package, and the +binary release layout. Every package is either binary-only or +source-inclusive, and you must choose one. Every package ships the release build of `nvx-rs`, the Rust command-line interface that runs NVX's microVMs and sandboxes ([Usage](usage.md#nvx-rs)): @@ -94,11 +95,13 @@ OpenVMM is MIT licensed: retain its notice, but its source does not have to be published merely because it is aggregated with Linux. See `THIRD_PARTY_NOTICES.md`. -The `nvx-rs` executables are MIT licensed NVX code that statically links the -Rust standard library and permissively licensed crates, with musl and LLVM's -`libunwind` on Linux and the Microsoft C runtime on Windows. -`THIRD_PARTY_NOTICES.md` lists each with its license; `nvx-rs/Cargo.lock` -pins every crate by checksum. +The `nvx-rs` executables and the guest binaries are MIT licensed NVX code, +built from the `nvx` package, that statically links the Rust standard library +and permissively licensed crates, with musl and LLVM's `libunwind` on Linux and +the Microsoft C runtime on Windows. `THIRD_PARTY_NOTICES.md` lists each with +its license; `nvx-rs/Cargo.lock` pins every crate by checksum. See +[The nvx package](build.md#the-nvx-package) for the license inputs and the +tests that keep the notices in step with the lockfile. ## Release archives @@ -126,21 +129,16 @@ declares other OpenVMM contracts than the `nvx-rs` that installs it speaks, such as another microVM ABI, time ABI, restore packet, or control contract, and names each field of its manifest that differs. -CI follows the binary-only path. On a `dev` push that changes more than -documentation, on a ready pull request from a branch of this repository or a -merge group that changes an input of the platform or package jobs, and on -scheduled or manually dispatched full runs, a package job for each of the -`linux-kvm`, `linux-mshv`, and `windows-whp` platforms stages -`dist/nvx--` with `--binary-only` on a GitHub-hosted runner -of the platform's operating system, beside the platform job that benchmarks the -same OpenVMM build, then archives it as -`.tar.gz` on Linux and `.zip` on Windows through the -[`package-release`](../.github/actions/package-release/action.yml) action, -with the `nvx-rs` and provenance that the `nvx-rs` build job of that -operating system built and its NVX CLI job tested. Only `dev` pushes upload -the archives, so pull requests, merge groups, and full runs prove that their -packages build without publishing them. The development release job checks -that each package ships OpenVMM and `nvx-rs` before it publishes them. +CI follows the binary-only path. A package job for each of the `linux-kvm`, +`linux-mshv`, and `windows-whp` platforms stages +`dist/nvx--` with `--binary-only`, with the `nvx-rs` and +provenance that the `nvx-rs` build job of the platform's operating system built +and its NVX CLI job tested, and archives it as `.tar.gz` on Linux and `.zip` on +Windows; [Release packages](ci.md#release-packages) describes the jobs and +when they run. Only `dev` pushes upload the archives, so pull requests, merge +groups, and full runs prove that their packages build without publishing them. +The development release job checks that each package ships OpenVMM and +`nvx-rs` before it publishes them. [Continuous integration](ci.md) describes the jobs that gate and publish the resulting development release. @@ -158,7 +156,10 @@ crate, a preview whose versions are pre-releases, `0.1.0-alpha.N`, which Cargo selects only for a requirement that names a pre-release. The nightly CI run publishes a new version from `dev` whenever the crate changed, with the next number, as [crates.io releases](ci.md#cratesio-releases) describes, so a -change reaches crates.io within a day of its merge. +change reaches crates.io within a day of its merge. The crate holds the files +that Git tracks in `nvx-rs/`, among them the package's license, `LICENSE`, its +README, and its lockfile, `Cargo.lock`; its features, profiles, and binaries +are those that [The nvx package](build.md#the-nvx-package) lists. The version that `nvx-rs/Cargo.toml` declares is the lowest that CI publishes. A pull request changes it to start another series, such as @@ -186,3 +187,41 @@ that checked its files out with CRLF line endings before then keeps them until it checks them out again, and packs them: `0.1.0-alpha.2` came from such a clone, so its sources differ from the commit that it records. A fresh clone packs the bytes of its commit, which the `git ls-files` check confirms. + +### The guest tools + +The crate's `guest` feature builds NVX's guest programs, the ten `nvx-*` +binaries that NVX's guest images install under `/sbin`, and its `guest` +profile builds them small and stripped. A guest image that NVX does not build +installs them from the crate with `cargo install`, which builds the crate as +the root of its build, so that the crate's `guest` profile applies. On a Linux +host, with Rust 1.89 or later, the `x86_64-unknown-linux-musl` target, and a C +compiler driver, `cc`, this installs the announcer of the portable network, +`nvx-ingress`, as the static executable that NVX's images hold: + +```bash +rustup target add x86_64-unknown-linux-musl +cargo install nvx --version VERSION --locked \ + --no-default-features --features guest --profile guest \ + --target x86_64-unknown-linux-musl --bin nvx-ingress +``` + +- `--version VERSION` names a version that crates.io publishes, + `0.1.0-alpha.N`. Every version is a pre-release, which Cargo selects only for + a requirement that names one, so `cargo install nvx` without it finds none. +- `--no-default-features --features guest` builds the guest programs without + the default `host` feature, which no guest program uses. +- `--profile guest` builds them with the crate's `guest` profile, where + `cargo install` would otherwise use `release`. +- `--locked` builds the dependency versions that the crate's lockfile pins, as + NVX's own builds do. +- `--target x86_64-unknown-linux-musl` links them statically, so that they + need no C library in the guest. Without it, Cargo builds them for the host's + own target, such as `x86_64-unknown-linux-gnu`, whose executables need the + host's C library and dynamic loader. + +Repeat `--bin` for each program, or pass `--bins` for all ten. Cargo installs +them in its own `bin` directory, or in `DIRECTORY/bin` with +`--root DIRECTORY`. Each program's command line, exit statuses, and output are +compatibility contracts, which +[Guest binaries](../nvx-rs/README.md#guest-binaries) lists. diff --git a/doc/guests/azurelinux-guest.md b/doc/guests/azurelinux-guest.md index 22c69f013..26ec02b02 100644 --- a/doc/guests/azurelinux-guest.md +++ b/doc/guests/azurelinux-guest.md @@ -97,12 +97,12 @@ initramfs into a RAM filesystem capped at half of guest memory; see The package manifest `initramfs-azurelinux.cpio.gz.packages.json` records the release, architecture, base image, input digest, and artifact SHA-256. Each package entry names the RPM's version, release, architecture, license, and -source RPM. The manifest also records the shared aggregate Rust source digest -and each of the ten installed guest-binary digests. The input digest is a -domain-separated SHA-256 over the release, architecture, base image, and the -checkout files that define the build: the Dockerfile, package lock, build -modules, guest scripts, complete `nvx-rs` guest input set, and -`rust-toolchain.toml`. +source RPM. The manifest also records the digest of the `nvx` package's +[guest inputs](../build.md#the-nvx-package) and each of the ten installed +guest-binary digests. The input digest is a domain-separated SHA-256 over the +release, architecture, base image, and the checkout files that define the +build: the Dockerfile, package lock, build modules, guest scripts, and the +`nvx` package's guest inputs, `rust-toolchain.toml` among them. Release staging rejects a manifest that does not match the artifact, the base image, or the current inputs. Only binary-only packages carry the two Azure @@ -110,7 +110,7 @@ Linux files; the published platform releases are binary-only, and `nvx-rs download` installs them. `collect-sources` does not collect Azure Linux corresponding source, so `package --include-source` omits the guest; see [Package and source delivery](../distribution.md) and the -[release layout](../build.md). +[release layout](../build.md#release-packages). ## Validation diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index 417e3587d..205bc6ab0 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -89,7 +89,7 @@ The builder then: `/nonexistent`, a `nogroup` group, and usr-merged `/bin`, `/sbin`, `/lib`, and `/lib64` links; and 7. installs the NVX scripts from `guest/common` and `guest/ubuntu`, and the - ten Rust guest binaries from `nvx-rs`. + ten Rust guest binaries of the `nvx` package. The builder never executes a binary from the Ubuntu tree on the host. Both artifacts are packed with root ownership, fixed timestamps, and a stable file @@ -121,8 +121,8 @@ Alpine. It installs the same ten Rust guest binaries as Alpine and Azure Linux under their `/sbin/nvx-*` names. Docker copies all ten from one digest-pinned Rust builder, while native and determinism builds use the shared fresh-target -routine. The package manifest records one aggregate `nvx-rs` source digest -and each installed binary's digest. +routine. The package manifest records the digest of the `nvx` package's +[guest inputs](../build.md#the-nvx-package) and each installed binary's digest. ## Ubuntu sandbox layer @@ -146,7 +146,9 @@ proof. The layer runs under the Alpine control initramfs as the sandbox's read-only `distro` layer over a writable scratch disk, which is formatted separately -with `mkfs.ext4`; see the +with `mkfs.ext4`, such as a copy of the sandbox scratch template, +`ubuntu-smoke-scratch.ext4`, which the sandbox scenarios boot and +[`build-scratch-template`](../usage.md#build-scratch-template) formats; see the [experimental sandbox](../run.md#experimental-single-workload-sandbox). The sandbox's init, nvx-init, a static binary, runs each execution itself, as a plain process in the VM's root. Its executions share nvx-init's mount, PID, @@ -172,7 +174,8 @@ Base or the lock. Release staging rejects an Ubuntu manifest that does not match its artifact and the current inputs. Platform releases carry all four Ubuntu files under `guest/`, with the sandbox scratch template, `ubuntu-smoke-scratch.ext4`, -which has no manifest; `nvx-rs download` installs them all, and +which has no manifest, as [Release packages](../build.md#release-packages) +lists; `nvx-rs download` installs them all, and `run --guest ubuntu` fails when the initramfs is missing instead of falling back to Alpine. [Package and source delivery](../distribution.md) describes how diff --git a/doc/project-structure.md b/doc/project-structure.md index de3370de5..4ca410bbf 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -19,7 +19,7 @@ are build products or caches and are not part of the tracked source tree. The | `guest` | Common and distribution-specific guest scripts | | `ubuntu` | Pinned Ubuntu supplemental binary-package lock | | `azurelinux` | Checksum-pinned Azure Linux supplemental RPM lock | -| `nvx-rs` | Rust crate `nvx`: the NVX mechanics | +| `nvx-rs` | The `nvx` package: the library of NVX's mechanisms, with NVX's own programs as its binaries | | `openvmm` | Public OpenVMM Git submodule from `microsoft/openvmm`, tracking its `feature/microvm/dev` branch | | `data` | Tracked performance history and generated benchmark data | | `scripts/nvx_tools` | Implementation of the `nvx.py` commands | @@ -94,20 +94,28 @@ nvx/ | |-- config-microvm MicroVM kernel configuration, generated from the feature inventory | |-- config-microvm.derived Rule that sets each symbol of config-microvm beyond the inventory | `-- config-microvm-debug CI debug-kernel fragment (watchdogs on) -|-- nvx-rs/ Rust crate `nvx` for the NVX mechanics +|-- nvx-rs/ The `nvx` package: NVX's library, with NVX's own programs as its binaries | |-- src/ +| | |-- lib.rs The crate's root, its documentation, and its public modules | | |-- protocol/ Control-console codec, NVXS session, and the guest targets that live shares may not take, shared by the host and the guest | | |-- activation/ The activation record of a layered root, its canonical encoding, and what Ready reports about it, shared by the host, the guest, and every adapter's framing | | |-- contract.rs Control-contract versions and release compatibility | | |-- net/ CIDR parsing and subtraction, and hostname and guest-network forms | | |-- gpt/ Strict reading of GPT disks whose first partition configures an adapter and whose other partitions hold a root's read-only layers, shared by the host and the guest -| | |-- host/ Artifacts, launch, network and egress-policy lowering, sessions, VMs, the sandbox launch profile, and the `nvx-rs` CLI (feature `cli`) -| | |-- testing/ Fake OpenVMM (feature `testing`) -| | |-- guest/ Programs that run in Linux guests, the guest's NVXS connection, the libraries through which any guest init or agent opens the terminal on which it serves its session, starts a program, collects its children's statuses, settles its executions' cgroups, drives each execution, mounts host mappings beneath the descriptors of the roots that it names, assembles and tears down a layered root, and reads a GPT disk's partition table through its block device, and the typed calls through which it runs the time ABI's steps and the reseed (feature `guest`) -| | `-- bin/ nvx-rs CLI, nvx-fake-openvmm test double, and the guest binaries +| | |-- uuid.rs UUIDs in their canonical text and in GPT's layout, shared by the host and the guest +| | |-- hypervisor.rs The hypervisors that run the VMs: KVM, MSHV, and WHP +| | |-- error.rs `nvx::Error` and its `ErrorKind`, and the crate's private `ErrorCode` +| | |-- host/ Feature `host`: the runtime, VMs and their configuration, snapshots, restores, the transport, process references, outcome reports, and the lowering of network postures and host export plans; private to the crate, the platform layer and what only NVX's CLI uses, such as the export planner, the sandbox launch profile, and the egress policy compiler; with feature `cli`, NVX's hidden `nvx-rs` CLI +| | |-- testing/ Feature `testing`: NVX's hidden fake OpenVMM +| | |-- guest/ Feature `guest`, on Linux: the libraries on which a guest init builds, such as its NVXS session, terminal, host mappings, time ABI steps, reseed, snapshot capture, and activation; NVX's hidden guest programs; and NVX's guest space, private to the crate +| | `-- bin/ One file per binary, which calls its program's hidden `main`: nvx-rs, nvx-fake-openvmm, and the ten guest binaries | |-- examples/ The external-launch example, and the documented host and guest workflows, which only compile -| |-- tests/ Fake-OpenVMM behavior, process and VM spawn, CLI scenario, NVXS session, guest-binary, and hypervisor public-API VM lifecycle tests, the security checklist, sandbox configurations of every format, the cases of the export and root boot tokens, and the conformance vectors of the wire formats -| `-- public-api/ The crate's public API on Linux and on Windows, which CI checks +| |-- tests/ Integration tests of the CLI's scenarios, in cli/, the fake OpenVMM, the guest binaries, the NVXS session, in nvxs/, the layering of `nvx::host`, a VM's lifecycle on a hypervisor, the security checklist, and the copies of the OpenVMM submodule's data, with their fixtures and the conformance vectors of the wire formats +| |-- public-api/ The crate's public API on Linux and on Windows, which CI checks +| |-- Cargo.toml Package manifest: features, binaries, examples, and the `guest` profile +| |-- Cargo.lock The package's own lockfile +| |-- LICENSE The package's MIT license, which the crate ships +| `-- README.md The crate's guide: its modules, the nvx-rs CLI, its features, its guest binaries, and its tests |-- openvmm/ OpenVMM Git submodule |-- scripts/ Build, run, benchmark, and release tooling | |-- nvx_tools/ Python implementation behind the NVX CLI @@ -132,6 +140,8 @@ nvx/ | | |-- development_release.py Resumable development release publication | | |-- doctor.py Time ABI host qualification | | |-- elf.py Static-executable check of ELF binaries +| | |-- guest_artifact_reuse.py Reuse of a pull request's guest artifacts in its merge group and dev push +| | |-- guest_binaries.py Build, checks, and identity of the nvx package's guest binaries | | |-- guests.py Typed guest descriptors | | |-- host_telemetry.py Benchmark host provenance and Windows capture storage telemetry | | |-- host_time_probe.rs Host TSC probe that doctor builds and runs @@ -144,7 +154,7 @@ nvx/ | | |-- microvm_test_scripts/ Guest workloads for the microVM tests | | |-- microvm_tests.py NVX microVM correctness tests | | |-- nightly_ci.py Nightly CI failure report -| | |-- nvx_rs.py Finds and builds nvx-rs, and runs it for the microVM tests +| | |-- nvx_rs.py Finds and builds nvx-rs, records its provenance, and runs it for the microVM tests | | |-- openvmm_ci.py Check that OpenVMM's own CI passed the pinned revision | | |-- openvmm_process.py Foreground OpenVMM process control for tests | | |-- performance.py CI performance collection, persistence, and gates @@ -190,7 +200,10 @@ nvx/ ``` The ignored `.cache/` directory may also appear at the repository root. It -contains downloaded and prepared upstream source trees, including Linux. +contains the verified upstream archives and packages that the guest builds +download, under `downloads/`, and the prepared upstream source trees, including +Linux; `NVX_CACHE_DIR` moves the guest builds' cache elsewhere, as +[Build](build.md) describes. ## Source directories @@ -278,19 +291,25 @@ for other kernel-specific details. ### `nvx-rs/` -The Rust crate `nvx` holds the NVX mechanics: the -authenticated control protocol of the guest agent, artifact discovery, -OpenVMM's command line, the lowering of network and -filesystem policies, the compiler of the egress policy files that `nvx-rs run` -and `nvx-rs sandbox` accept, process and platform primitives for Linux and -Windows hosts, the creation, spawn, and stop of VMs, the sandbox launch profile -of `nvx-rs sandbox`, the `nvx-rs` command-line interface that runs NVX's -microVMs and sandboxes, downloads releases, compiles egress policies, and looks -up artifacts, and a fake OpenVMM for tests. Its -`guest` feature builds the ten static executables that every guest image -installs at `/sbin/nvx-*`. It is a standalone crate with its own `Cargo.lock`, -and builds independently of the OpenVMM submodule. See its -[README](../nvx-rs/README.md) and its [module diagrams](modules/README.md). +`nvx-rs/` holds the `nvx` package, the repository's one Cargo package. Its +library, `nvx`, holds the mechanisms of NVX's microVMs: the NVXS session and +the codecs that the host and the guest share, OpenVMM's command line, the +lowering of network postures and host export plans into OpenVMM's options, +verified references to OpenVMM's processes, the creation, spawn, reattachment, +and stop of VMs, their snapshots and restores, and the guest's side of a +snapshot. Its binaries are NVX's own programs: the `nvx-rs` command-line +interface (feature `cli`), which runs NVX's microVMs and sandboxes, downloads +releases, compiles egress policies, and looks up artifacts; a fake OpenVMM for +its tests (feature `testing`); and the ten static executables that every guest +image installs at `/sbin/nvx-*` (feature `guest`). The code that only these +programs use, such as the CLI's stores, export planner, and egress policy +compiler, the platform layer, and the guest's space, is private to the crate, +and each binary calls a `main` that `#[doc(hidden)]` hides, so the snapshots +in `public-api/` record the library's public API alone. The package has its +own `Cargo.lock`, and builds independently of the OpenVMM submodule; +[The nvx package](build.md#the-nvx-package) lists its features, profiles, +binaries, outputs, and license inputs. See its [README](../nvx-rs/README.md) +and its [module diagrams](modules/README.md). ### `openvmm/` @@ -315,6 +334,9 @@ module has no dependencies on the workflow or configuration modules. Standalone benchmark shell programs and parameterized guest templates live in `nvx_tools/benchmark_scripts/`, and the guest workloads of the microVM tests live in `nvx_tools/microvm_test_scripts/`. +`nvx_tools/guest_binaries.py` builds, checks, and identifies the guest binaries +of the `nvx` package, and `nvx_tools/nvx_rs.py` builds `nvx-rs` and records its +provenance; see [The nvx package](build.md#the-nvx-package). Source-collection scripts assemble corresponding-source archives for Linux, Alpine, and Ubuntu. The adversarial controller, typed broker, credential-free executor, watchdog, and tracked deterministic catalogs also live in @@ -373,14 +395,14 @@ draw the module diagrams that `doc/modules` publishes (see | Path | Contents | | --- | --- | -| `.cache/` | Downloaded, verified, and patched upstream source trees | -| `build/` | Kernels, initramfs and EROFS images, package manifests, and collected sources | +| `.cache/` | Downloaded and verified upstream archives and packages, and patched upstream source trees | +| `build/` | Kernels, initramfs and EROFS images, the sandbox scratch template, package manifests, the provenance of OpenVMM, the kernels, the Alpine initramfs, and `nvx-rs`, and collected sources | | `data/baseline/` | Base-branch histories staged by the performance gate | | `data/results/` | Collected p50 results for the current commit | | `data/runs/` | Raw benchmark logs and per-platform artifacts | | `dist/` | Staged binary and source release archives | | `openvmm/target/` | Rust build output produced inside the OpenVMM submodule | -| `nvx-rs/target/` | Rust build output of the `nvx` crate | +| `nvx-rs/target/` | Cargo's build output of the `nvx` package, among it `release/nvx-rs[.exe]` | | `.ruff_cache/` | Ruff's local lint cache | | `__pycache__/` | Python bytecode caches that may appear below Python source directories | From 435676be8d8248e929ba583aa09cd8689fbef279 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 12 Nov 2026 07:45:05 -0800 Subject: [PATCH 3/3] setup: describe the Python runner-setup package and the nvx-rs build The setup guide pointed to "Azure provisioning" without saying what it holds, and the structure did not list scripts/setup/azure at all, although nvx_azure, the standard-library Python package that the instances of NVX's CI scale sets run to provision themselves as runners, has replaced their shell bootstrap since October. The Azure guide described the package's modules, but not the commands that its cli.py takes or what runs each. And the guide's build checks said only that they validate "the command that NVX generates", although both setup scripts build nvx-rs from the nvx package, record its provenance, and dry-run it. - scripts/setup/README.md says what the package does, through which scripts, with which credential, and how build_package.py packages it, and that a build check builds nvx-rs with the command of Usage, records its provenance, and checks nvx-rs run --dry-run. - scripts/setup/azure/README.md gains a fixed table of the package's six commands, linux-bootstrap, linux-block-metadata, linux-refresh-metadata, linux-verify-mshv, windows-stage and windows-bootstrap, with the unit, task or shim that runs each, from build_package.py, linux_bootstrap.py, windows_stage.py and windows_system.py, and the step that each takes. - doc/project-structure.md lists scripts/setup/README.md and the azure/ tree, and describes the package beside the setup scripts. Evidence: python -c "...cli.main(['--help'])" lists exactly these six commands, and the guides' offline validation, test_setup_azure, test_setup_azure_linux, test_setup_azure_windows and test_tool_versions, passes on Windows, as does the link check. Part of #1131 (SL6.10e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/project-structure.md | 15 ++++++++++++++- scripts/setup/README.md | 27 +++++++++++++++++++-------- scripts/setup/azure/README.md | 11 +++++++++++ 3 files changed, 44 insertions(+), 9 deletions(-) diff --git a/doc/project-structure.md b/doc/project-structure.md index 4ca410bbf..d3ae54ec6 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -24,7 +24,7 @@ are build products or caches and are not part of the tracked source tree. The | `data` | Tracked performance history and generated benchmark data | | `scripts/nvx_tools` | Implementation of the `nvx.py` commands | | `scripts/nvx.py` | Canonical build, run, test, benchmark, and packaging CLI | -| `scripts/setup` | Development host and GitHub Actions runner bootstrap scripts, and the manifest of the tool versions that they install | +| `scripts/setup` | Development host and GitHub Actions runner bootstrap scripts, the manifest of the tool versions that they install, and the Python package that provisions NVX's CI runners on Azure | | `scripts/nvx_adversarial_executor.py` | Credential-free adversarial executor protocol entry point | | `.cache/linux` | Generated verified/patched Linux tree; ignored by Git | | `build/sources` | Generated Linux, Alpine, and Ubuntu release sources; ignored by Git | @@ -168,6 +168,14 @@ nvx/ | | |-- time_abi.py Time ABI harness checks and CPU profile guidance | | `-- ubuntu.py Verified Ubuntu rootfs and EROFS preparation | |-- setup/ Host and Actions runner bootstrap scripts +| | |-- azure/ Azure CI runner provisioning of existing scale sets +| | | |-- nvx_azure/ Standard-library Python package that scale-set instances run to become CI runners +| | | |-- linux/ Example Linux scale-set configuration +| | | |-- windows/ Example Windows configuration, Custom Script entry shim, and PowerShell adapters +| | | |-- build_package.py Offline packager of a scale set's bootstrap payload and model patch +| | | |-- requirements.txt The packager's Python requirement +| | | `-- README.md Azure runner provisioning guide +| | |-- README.md Development host and Actions runner bootstrap guide | | |-- setup-linux.sh Linux/KVM or Linux/MSHV development-host and Actions runner bootstrap | | |-- setup-windows-whp.ps1 Windows/WHP development-host and Actions runner bootstrap | | `-- tool-versions.conf Shared tool release and checksum pins @@ -366,6 +374,11 @@ hooks that `test-layout-allowlist.json` allows them, and The `setup/` scripts bootstrap development hosts and GitHub Actions runners using the [tool manifest](../scripts/setup/tool-versions.conf); see [Automated environment bootstrap](setup.md#automated-environment-bootstrap). +On Azure, `setup/azure/nvx_azure` is the standard-library Python package with +which the instances of NVX's CI scale sets provision themselves as runners, +through those scripts, and `setup/azure/build_package.py` packages it into a +scale set's custom data; see +[Azure CI runner provisioning](../scripts/setup/azure/README.md). `nvx-hosts.example.json` is the template for the [remote agent host](setup.md#remote-agent-hosts) inventory, and CI runs `publish_development_release.py` to publish development releases, diff --git a/scripts/setup/README.md b/scripts/setup/README.md index 15d6c620a..6e58e2535 100644 --- a/scripts/setup/README.md +++ b/scripts/setup/README.md @@ -11,8 +11,15 @@ Runner-only mode, which `--runner-only` on Linux and `-RunnerOnly` on Windows select, prepares a persistent GitHub Actions runner for NVX's CI jobs instead, and needs no checkout. -For Azure VMSS bootstrapping, see [Azure provisioning](azure/README.md). The -scripts in this directory remain usable on supported hosts outside Azure. +On Azure, [`azure/`](azure/README.md) provisions NVX's CI runners on existing +virtual machine scale sets with `nvx_azure`, a Python package that needs only +the standard library: each instance runs it from the scale set's custom data +when it is provisioned, and it checks the instance's identity, prepares its +data disk and backend, runs `setup-linux.sh` or `setup-windows-whp.ps1` in +runner-only mode, and registers the runner with a registration token that it +requests with a GitHub credential from Key Vault. `azure/build_package.py` +packages it, with these scripts and the tool manifest, into that custom data. +The scripts in this directory remain usable on supported hosts outside Azure. They do not create machines or initialize a checkout. Runner-only mode consumes a short-lived registration token from standard input and does not @@ -267,9 +274,11 @@ Validate dependencies without changing the host or requiring build artifacts: sh scripts/setup/setup-linux.sh --backend kvm --check-only --skip-build ``` -Omit `--skip-build` to also validate the built artifacts and the command that -NVX generates for the backend. Use `--workspace PATH` when the script is -outside the checkout. +Omit `--skip-build` to also validate the built artifacts, build `nvx-rs` from +the `nvx` package with the command of +[Usage](../../doc/usage.md#nvx-rs) and record its provenance, and check the +command that `nvx-rs run --dry-run` generates for the backend. Use +`--workspace PATH` when the script is outside the checkout. To produce a revision-bound guest bundle for a Windows build: @@ -313,9 +322,11 @@ Validate dependencies without changing the host or requiring build artifacts: .\scripts\setup\setup-windows-whp.ps1 -CheckOnly -SkipBuild ``` -Omit `-SkipBuild` to also validate the guest artifacts, OpenVMM binary, and -generated WHP command. Use `-Workspace PATH` when the script is outside the -checkout. +Omit `-SkipBuild` to also validate the guest artifacts and the OpenVMM binary, +build `nvx-rs` from the `nvx` package with the command of +[Usage](../../doc/usage.md#nvx-rs) and record its provenance, and check the +command that `nvx-rs run --dry-run` generates for WHP. Use `-Workspace PATH` +when the script is outside the checkout. ## Tool versions diff --git a/scripts/setup/azure/README.md b/scripts/setup/azure/README.md index 0fae3791c..13f6aa4de 100644 --- a/scripts/setup/azure/README.md +++ b/scripts/setup/azure/README.md @@ -25,6 +25,17 @@ Do not maintain a second copy of the runner setup or its tool versions here. | `windows/firewall.ps1`, `windows/openssh.ps1`, `windows/storage.ps1`, `windows/system.ps1` | Adapters that run the cmdlets a Python controller names, taking a JSON request and returning a JSON result. | | `build_package.py` | Validate configuration and generate deterministic guest payloads, hashes, and a partial VMSS model patch without accessing Azure or GitHub. | +`cli.py` takes one command, which names the step that it runs: + +| Command | Run by | Step | +| --- | --- | --- | +| `linux-bootstrap` | `setup-nvx-github-runner.service`, which cloud-init enables and starts after `cloud-final.service`, as root | Verify the instance identity and refresh the compute snapshot, prepare the data disk and the backend, run `setup-linux.sh --runner-only`, enable the metadata block, register the runner, and disable the unit once it has registered | +| `linux-block-metadata` | `nvx-runner-metadata-block.service`, and the runner service before each start | Block the runner account from Azure instance metadata | +| `linux-refresh-metadata` | The runner service before each start, as root | Refresh the compute snapshot alone | +| `linux-verify-mshv` | An administrator; `linux-bootstrap` runs the same check when `backend` is `mshv` and `kernelMode` is `azurelinux-mshv` | Report whether the VM booted as the MSHV root partition | +| `windows-stage` | `windows/vmss-bootstrap.ps1`, which the Custom Script extension runs | Verify and unpack the package, configure OpenSSH, and start the `Nvx-Setup-GitHub-Runner` task | +| `windows-bootstrap` | The `Nvx-Setup-GitHub-Runner` task, as SYSTEM, at every startup until registration succeeds | Block Network Service from instance metadata, verify the instance identity, prepare the data disk and the WHP and Hyper-V features, and register the runner | + Linux guests run every step as `/usr/bin/python3 -I -B /opt/nvx-bootstrap `. Cloud-init installs the distribution's `python3` and writes all guest files from the same payload