diff --git a/doc/build.md b/doc/build.md index 9601998b7..31ce6292e 100644 --- a/doc/build.md +++ b/doc/build.md @@ -43,9 +43,10 @@ The routine takes the exact toolchain from `rust-toolchain.toml`, the uses `--locked`, a fresh target directory, disabled incremental compilation, stable source and Cargo-home path remapping, and no linker build IDs. One Cargo invocation builds all ten programs, and every result must pass -`check-static-elf`. The output's `manifest.json` binds the exact build pins, -one digest over the `nvx` crate, the shared Python build recipe, and -`rust-toolchain.toml`, and each binary SHA-256. +`check-static-elf`. The output's `manifest.json` records the toolchain, the +target, and the profile, the SHA-256 of each binary, and one digest over the +guest inputs of the `nvx` package, which +[The nvx package](#the-nvx-package) lists. The native build keeps the Linux archive, the Ubuntu Base archive, and the supplemental Ubuntu packages under `downloads/` in the cache directory, @@ -206,8 +207,10 @@ Ubuntu, and Azure Linux; the kernel targets do not depend on that build. The Ubuntu determinism target instead rebuilds them from fresh targets in both attempts. Native image builds also rebuild them fresh. -The Alpine initramfs additionally includes the sandbox PID-1 bootstrap and -its container namespace scripts. `nvx-time-probe` measures TSC and +The Alpine initramfs is the one that `nvx-rs sandbox` boots: with +`nvx_sandbox=1` on the kernel command line, the common `/init` hands PID 1 to +`nvx-init`, which initializes the sandbox's layered root and serves its managed +control session. `nvx-time-probe` measures TSC and clock read latency, reports the guest's clock, clocksource, TSC flags, and hypervisor identity, and runs the cross-vCPU warp test that the [time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs @@ -231,8 +234,9 @@ predate the ABI. All ten executables link the musl supplied by the pinned Rust target. Every package manifest records the shared Rust source digest and each installed binary digest. The [kernel configuration checks](#kernel-configuration-checks) cover the kernel -features that the sandbox bootstrap needs. The APK manifest records the -`blkid` and `util-linux` tools used by the bootstrap along with every +features that sandboxes need. The APK manifest records every package of the +Alpine image, among them `blkid`, `busybox-extras`, `e2fsprogs`, `util-linux`, +and `util-linux-misc`, which the build adds to the minirootfs, along with every guest-binary digest. The Ubuntu initramfs uses Ubuntu userland with the NVX kernel. It is not an @@ -507,6 +511,159 @@ and SHA-256. The debug kernel is a CI artifact and is never packaged. The `build-guest-artifacts` action caches and builds it only when its `components` input names `debug-kernel`. +## The nvx package + +[`nvx-rs/`](../nvx-rs/README.md) holds the repository's one Cargo package, +`nvx`. Its library is the crate that crates.io publishes, as +[The nvx crate](distribution.md#the-nvx-crate) describes, and its binaries are +NVX's own programs, each of which calls a `main` that `#[doc(hidden)]` hides in +the library. The package belongs to no workspace, since a workspace at the +repository root would interact with the nested `openvmm/` workspace, so it +keeps a lockfile of its own, `nvx-rs/Cargo.lock`, and builds without the +OpenVMM submodule. The builds that `nvx.py`, the setup scripts, and CI's +`nvx-rs` jobs run pass `--locked`, so they build the dependency versions that +the lockfile pins, and the crate ships the lockfile, which +`cargo install --locked` reads. NVX builds the package with the Rust release +that `rust-toolchain.toml` pins, and CI also checks it with the oldest release +that it supports, the manifest's `rust-version`; see +[Rust crates](ci.md#rust-crates). + +Its features select what it builds: + +| Feature | Builds | Turns on | +| --- | --- | --- | +| `host` (default) | The `host` module, the library's host side | The `serde` feature, and the optional `getrandom`, `serde_json`, `sha2`, and `nix` on Unix or `windows-sys` on Windows | +| `guest` | The `guest` module, on Linux, and the ten guest binaries | The optional `nix` and `getrandom` | +| `serde` | The serde forms of `Hypervisor` and `net::Ipv4Cidr` | The optional `serde` | +| `cli` | NVX's `nvx-rs` binary, which adds only its hidden entry point to the API | The `host` feature, and the optional `crossterm` | +| `testing` | NVX's `nvx-fake-openvmm` binary, a fake OpenVMM for tests, which adds only its hidden entry point to the API | The optional `getrandom`, `serde`, `serde_json`, and `nix` on Unix or `windows-sys` on Windows | + +Without default features, the library holds only what the host and the guest +share: `protocol`, `activation`, `contract`, `net`, `gpt`, `uuid`, +`Hypervisor`, and the crate's error types, such as `Error`. The manifest +defines one Cargo profile of its own, and leaves Cargo's built-in profiles at +their defaults. NVX's builds use these: + +| Profile | Settings | Builds | +| --- | --- | --- | +| `dev` and `test` | Cargo's defaults | The crate's checks and tests | +| `release` | Cargo's defaults | `nvx-rs`, with `--release` | +| `guest` | `release`'s, with `opt-level = "s"`, `lto = "fat"`, `codegen-units = 1`, `panic = "abort"`, `strip = true`, `debug = false`, and `incremental = false` | The guest binaries, small and stripped | + +Cargo applies a package's profiles only to a build whose root it is, such as +one in `nvx-rs/`, one with `--manifest-path nvx-rs/Cargo.toml`, or +`cargo install nvx`; a package that depends on `nvx` builds it with its own +profiles. The guest routine also drops inherited `RUSTFLAGS` and +`CARGO_PROFILE_*` overrides, so that the `guest` profile applies as the +manifest gives it. + +The package's binaries build with these features, profiles, and targets: + +| Binary | Feature | Profile and target | Built by | Ships in | +| --- | --- | --- | --- | --- | +| `nvx-rs` | `cli` | `release`, for the host's target in a checkout; for release packages, static `x86_64-unknown-linux-musl` on Linux and `x86_64-pc-windows-msvc` with a static C runtime on Windows | The [`nvx-rs` build command](usage.md#nvx-rs), which `nvx.py` and the setup scripts run, and CI's `nvx-rs` build jobs | `bin/nvx-rs[.exe]` of every [release package](#release-packages) | +| `nvx-fake-openvmm` | `testing` | The tests' | `cargo test` with `testing`, for the crate's tests, which run it as OpenVMM | No package | +| `nvx-console-pending`, `nvx-device-io`, `nvx-exit`, `nvx-ingress`, `nvx-init`, `nvx-managed-agent`, `nvx-mmio-write`, `nvx-reseed`, `nvx-time`, `nvx-time-probe` | `guest` | `guest`, static `x86_64-unknown-linux-musl` | [`build-guest-binaries`](usage.md#build-guest-binaries), for every guest image | `/sbin` of every initramfs; other guests [install them from the crate](distribution.md#the-guest-tools) | + +Its examples, `external-launch` and `host-workflows` with `host`, and +`guest-workflows` with `guest`, build with the crate's checks of all targets; +the [README](../nvx-rs/README.md#testing) says what each shows. + +The builds put their output here: + +| Output | Path | Written by | +| --- | --- | --- | +| `nvx-rs` | `nvx-rs/target/release/nvx-rs[.exe]`, where `nvx.py`, `package`, and the setup scripts look for it | `cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target`; CI's Linux job builds for `x86_64-unknown-linux-musl` and copies `nvx-rs/target/x86_64-unknown-linux-musl/release/nvx-rs` there | +| Its provenance | `build/nvx-rs.provenance.json` | `nvx.py` when it builds `nvx-rs`, or [`record-nvx-rs-provenance`](usage.md#record-nvx-rs-provenance) | +| The guest binaries, with their `manifest.json` | The `--output` directory of `build-guest-binaries`, or an image build's work directory, such as `build/initramfs-alpine-work/guest-binaries`, unless `NVX_GUEST_BINARIES` names a build to install | `build-guest-binaries`, or the image build, which runs the same routine | +| Cargo's other builds | `nvx-rs/target/`, in a directory per profile, such as `debug/` or `guest/`, under a directory per target when the build names one | Cargo | + +The package keeps no cache of its own. `nvx-rs/target/` keeps the checkout's +Cargo builds between runs, and `--target-dir nvx-rs/target` keeps `nvx-rs` +there even where `CARGO_TARGET_DIR` names another directory; Cargo keeps the +crates that the lockfile pins in its registry cache. The guest routine reuses +no earlier target tree: it builds in a fresh target directory beside its +output, without incremental compilation, and removes it afterwards. It keeps +the environment's `RUSTC_WRAPPER`, though, so where that names sccache, Cargo +runs rustc through sccache, which can return a compilation from its own cache. +CI restores no Cargo target tree of the package, but the services of the +self-hosted runners set `RUSTC_WRAPPER=sccache`, with a persistent cache in +`_work/_sccache`, so the jobs that compile the package there, such as the +microVM jobs whose `vm-lifecycle` and `managed-lifecycle` scenarios run its +tests with Cargo, compile it through that cache; see +[Compilation cache](ci.md#compilation-cache). The keys of the guest images' +cache entries end with the digest of the package's guest inputs, which +[`guest-binary-source-identity`](usage.md#guest-binary-source-identity) +prints, so that a change to the package rebuilds the images only when it +changes one of those inputs; see +[Guest artifact handoff](ci.md#guest-artifact-handoff). The guest inputs are: + +- `nvx-rs/Cargo.toml`, `nvx-rs/Cargo.lock`, and any Cargo configuration in + `nvx-rs/.cargo/`; +- every file under `nvx-rs/src/` but those of the host side, `src/host/`, + `src/testing/`, and the `nvx-rs` and `nvx-fake-openvmm` binaries, and those + of the test modules, the `tests.rs` files and the `tests` directories; and +- `rust-toolchain.toml`, and the two modules that define the build, + `scripts/nvx_tools/build_constants.py` and + `scripts/nvx_tools/guest_binaries.py`. + +The guest build fails when Cargo's dep-info lists a file outside them, other +than one that the build generates in its own target directory. + +Three records bind the package's binaries to their sources: + +- `build/nvx-rs.provenance.json` binds an `nvx-rs` executable's SHA-256 to the + digest of the sources that built it, the package's manifest, lockfile, and + `src/` tree and `rust-toolchain.toml`, and to the build's Rust release, + target, and features. `package` and the adversarial executor refuse an + `nvx-rs` that it does not describe as built from the checkout's sources; see + [nvx-rs](usage.md#nvx-rs). +- The guest binaries' `manifest.json` binds their SHA-256 digests to the digest + of the guest inputs, and every guest image's package manifest records that + digest and the SHA-256 of each binary that it installs. +- A release package's `SOURCE-MANIFEST.json` records the packaged `nvx-rs` and + its provenance in its `nvx_rs` section, and the guest binaries' Rust release + and target in its `guest_rust` section, as + [Release packages](#release-packages) describes. + +On crates.io, the crate's `.cargo_vcs_info.json` records the commit from which +CI packed it; see [crates.io releases](ci.md#cratesio-releases). + +The package is MIT licensed, as the `license` field of its manifest says, and +`nvx-rs/LICENSE` holds the license, which the crate ships with the README and +the lockfile. The binaries that NVX ships, `bin/nvx-rs[.exe]` and the guest +binaries, statically link the crates that `nvx-rs/Cargo.lock` pins by checksum, +and their target's runtime: musl and LLVM's `libunwind` on Linux, and the +Microsoft C runtime on Windows. +[`THIRD_PARTY_NOTICES.md`](../THIRD_PARTY_NOTICES.md) lists each of these with +its license, and the crates at the versions that the lockfile pins. Unit tests +in `scripts/test_nvx_tools.py` fail when the lockfile gains a crate that they +do not classify as linked into `bin/nvx-rs` or not, or when the notices lack a +linked crate at its pinned version or a runtime that the binaries link. Release +packages ship the notices, and +[Package and source delivery](distribution.md#source-inclusive-packages) +describes what each license asks of a distributor. + +## Release packages + +Each platform's release package ships the OpenVMM and `nvx-rs` executables of +its operating system, and the same guest artifacts. CI's package jobs, which +[Release packages](ci.md#release-packages) in Continuous integration +describes, stage the three binary packages from these builds: + +| File | `linux-kvm` | `linux-mshv` | `windows-whp` | +| --- | --- | --- | --- | +| `bin/openvmm[.exe]` | `x86_64-unknown-linux-gnu` | Static `x86_64-unknown-linux-musl` | `x86_64-pc-windows-msvc` | +| `bin/nvx-rs[.exe]` | Static `x86_64-unknown-linux-musl` | Static `x86_64-unknown-linux-musl` | `x86_64-pc-windows-msvc`, with a static C runtime | +| `guest/vmlinux` and its configuration, and the Alpine, Ubuntu, and Azure Linux initramfses with their package manifests | Shipped | Shipped | Shipped | +| `guest/ubuntu-distro.erofs`, the Ubuntu distro layer, with its manifest | Shipped | Shipped | Shipped | +| `guest/ubuntu-smoke-scratch.ext4`, the sandbox scratch template, which has no manifest | Shipped | Shipped | Shipped | +| Archive | `nvx-VERSION-linux-kvm.tar.gz` | `nvx-VERSION-linux-mshv.tar.gz` | `nvx-VERSION-windows-whp.zip` | + +`VERSION` is the content of the `VERSION` file. +[Package and source delivery](distribution.md) describes the package modes and +how CI publishes the packages. + Release packaging needs the OpenVMM binary, the guest artifacts in the layout below, and the OpenVMM, kernel, and Alpine initramfs provenance sidecars, so run `build-openvmm` and `build-guest --guest all` first. It also needs the release diff --git a/doc/distribution.md b/doc/distribution.md index 154837de1..81500c90e 100644 --- a/doc/distribution.md +++ b/doc/distribution.md @@ -1,10 +1,11 @@ # Package and source delivery `package` stages a release directory, `dist/` by default, from the -built OpenVMM binary, `nvx-rs` executable, and guest artifacts. [Build](build.md) -lists the artifacts and provenance it requires and the binary release layout. -Every package is either binary-only or source-inclusive, and you must choose -one. +built OpenVMM binary, `nvx-rs` executable, and guest artifacts. +[Release packages](build.md#release-packages) lists the artifacts and +provenance it requires, the executables of each platform's package, and the +binary release layout. Every package is either binary-only or +source-inclusive, and you must choose one. Every package ships the release build of `nvx-rs`, the Rust command-line interface that runs NVX's microVMs and sandboxes ([Usage](usage.md#nvx-rs)): @@ -94,11 +95,13 @@ OpenVMM is MIT licensed: retain its notice, but its source does not have to be published merely because it is aggregated with Linux. See `THIRD_PARTY_NOTICES.md`. -The `nvx-rs` executables are MIT licensed NVX code that statically links the -Rust standard library and permissively licensed crates, with musl and LLVM's -`libunwind` on Linux and the Microsoft C runtime on Windows. -`THIRD_PARTY_NOTICES.md` lists each with its license; `nvx-rs/Cargo.lock` -pins every crate by checksum. +The `nvx-rs` executables and the guest binaries are MIT licensed NVX code, +built from the `nvx` package, that statically links the Rust standard library +and permissively licensed crates, with musl and LLVM's `libunwind` on Linux and +the Microsoft C runtime on Windows. `THIRD_PARTY_NOTICES.md` lists each with +its license; `nvx-rs/Cargo.lock` pins every crate by checksum. See +[The nvx package](build.md#the-nvx-package) for the license inputs and the +tests that keep the notices in step with the lockfile. ## Release archives @@ -126,21 +129,16 @@ declares other OpenVMM contracts than the `nvx-rs` that installs it speaks, such as another microVM ABI, time ABI, restore packet, or control contract, and names each field of its manifest that differs. -CI follows the binary-only path. On a `dev` push that changes more than -documentation, on a ready pull request from a branch of this repository or a -merge group that changes an input of the platform or package jobs, and on -scheduled or manually dispatched full runs, a package job for each of the -`linux-kvm`, `linux-mshv`, and `windows-whp` platforms stages -`dist/nvx--` with `--binary-only` on a GitHub-hosted runner -of the platform's operating system, beside the platform job that benchmarks the -same OpenVMM build, then archives it as -`.tar.gz` on Linux and `.zip` on Windows through the -[`package-release`](../.github/actions/package-release/action.yml) action, -with the `nvx-rs` and provenance that the `nvx-rs` build job of that -operating system built and its NVX CLI job tested. Only `dev` pushes upload -the archives, so pull requests, merge groups, and full runs prove that their -packages build without publishing them. The development release job checks -that each package ships OpenVMM and `nvx-rs` before it publishes them. +CI follows the binary-only path. A package job for each of the `linux-kvm`, +`linux-mshv`, and `windows-whp` platforms stages +`dist/nvx--` with `--binary-only`, with the `nvx-rs` and +provenance that the `nvx-rs` build job of the platform's operating system built +and its NVX CLI job tested, and archives it as `.tar.gz` on Linux and `.zip` on +Windows; [Release packages](ci.md#release-packages) describes the jobs and +when they run. Only `dev` pushes upload the archives, so pull requests, merge +groups, and full runs prove that their packages build without publishing them. +The development release job checks that each package ships OpenVMM and +`nvx-rs` before it publishes them. [Continuous integration](ci.md) describes the jobs that gate and publish the resulting development release. @@ -158,7 +156,10 @@ crate, a preview whose versions are pre-releases, `0.1.0-alpha.N`, which Cargo selects only for a requirement that names a pre-release. The nightly CI run publishes a new version from `dev` whenever the crate changed, with the next number, as [crates.io releases](ci.md#cratesio-releases) describes, so a -change reaches crates.io within a day of its merge. +change reaches crates.io within a day of its merge. The crate holds the files +that Git tracks in `nvx-rs/`, among them the package's license, `LICENSE`, its +README, and its lockfile, `Cargo.lock`; its features, profiles, and binaries +are those that [The nvx package](build.md#the-nvx-package) lists. The version that `nvx-rs/Cargo.toml` declares is the lowest that CI publishes. A pull request changes it to start another series, such as @@ -186,3 +187,41 @@ that checked its files out with CRLF line endings before then keeps them until it checks them out again, and packs them: `0.1.0-alpha.2` came from such a clone, so its sources differ from the commit that it records. A fresh clone packs the bytes of its commit, which the `git ls-files` check confirms. + +### The guest tools + +The crate's `guest` feature builds NVX's guest programs, the ten `nvx-*` +binaries that NVX's guest images install under `/sbin`, and its `guest` +profile builds them small and stripped. A guest image that NVX does not build +installs them from the crate with `cargo install`, which builds the crate as +the root of its build, so that the crate's `guest` profile applies. On a Linux +host, with Rust 1.89 or later, the `x86_64-unknown-linux-musl` target, and a C +compiler driver, `cc`, this installs the announcer of the portable network, +`nvx-ingress`, as the static executable that NVX's images hold: + +```bash +rustup target add x86_64-unknown-linux-musl +cargo install nvx --version VERSION --locked \ + --no-default-features --features guest --profile guest \ + --target x86_64-unknown-linux-musl --bin nvx-ingress +``` + +- `--version VERSION` names a version that crates.io publishes, + `0.1.0-alpha.N`. Every version is a pre-release, which Cargo selects only for + a requirement that names one, so `cargo install nvx` without it finds none. +- `--no-default-features --features guest` builds the guest programs without + the default `host` feature, which no guest program uses. +- `--profile guest` builds them with the crate's `guest` profile, where + `cargo install` would otherwise use `release`. +- `--locked` builds the dependency versions that the crate's lockfile pins, as + NVX's own builds do. +- `--target x86_64-unknown-linux-musl` links them statically, so that they + need no C library in the guest. Without it, Cargo builds them for the host's + own target, such as `x86_64-unknown-linux-gnu`, whose executables need the + host's C library and dynamic loader. + +Repeat `--bin` for each program, or pass `--bins` for all ten. Cargo installs +them in its own `bin` directory, or in `DIRECTORY/bin` with +`--root DIRECTORY`. Each program's command line, exit statuses, and output are +compatibility contracts, which +[Guest binaries](../nvx-rs/README.md#guest-binaries) lists. diff --git a/doc/guests/azurelinux-guest.md b/doc/guests/azurelinux-guest.md index 22c69f013..26ec02b02 100644 --- a/doc/guests/azurelinux-guest.md +++ b/doc/guests/azurelinux-guest.md @@ -97,12 +97,12 @@ initramfs into a RAM filesystem capped at half of guest memory; see The package manifest `initramfs-azurelinux.cpio.gz.packages.json` records the release, architecture, base image, input digest, and artifact SHA-256. Each package entry names the RPM's version, release, architecture, license, and -source RPM. The manifest also records the shared aggregate Rust source digest -and each of the ten installed guest-binary digests. The input digest is a -domain-separated SHA-256 over the release, architecture, base image, and the -checkout files that define the build: the Dockerfile, package lock, build -modules, guest scripts, complete `nvx-rs` guest input set, and -`rust-toolchain.toml`. +source RPM. The manifest also records the digest of the `nvx` package's +[guest inputs](../build.md#the-nvx-package) and each of the ten installed +guest-binary digests. The input digest is a domain-separated SHA-256 over the +release, architecture, base image, and the checkout files that define the +build: the Dockerfile, package lock, build modules, guest scripts, and the +`nvx` package's guest inputs, `rust-toolchain.toml` among them. Release staging rejects a manifest that does not match the artifact, the base image, or the current inputs. Only binary-only packages carry the two Azure @@ -110,7 +110,7 @@ Linux files; the published platform releases are binary-only, and `nvx-rs download` installs them. `collect-sources` does not collect Azure Linux corresponding source, so `package --include-source` omits the guest; see [Package and source delivery](../distribution.md) and the -[release layout](../build.md). +[release layout](../build.md#release-packages). ## Validation diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index 95b078899..205bc6ab0 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -16,7 +16,7 @@ describes how the pieces fit together and why. | Artifact | Used by | Boot supervisor | | --- | --- | --- | | `initramfs-ubuntu.cpio.gz` | `nvx-rs run --guest ubuntu` | The common NVX `/init` inside the Ubuntu initramfs | -| `ubuntu-distro.erofs` | `nvx-rs sandbox --layer distro,...` | The Alpine control initramfs and its init agent | +| `ubuntu-distro.erofs` | `nvx-rs sandbox --layer distro,...` | `nvx-init`, which the common `/init` of the Alpine initramfs starts | Alpine remains the default guest of `run` and the only control environment of `sandbox`. The sandbox launcher always boots the Alpine initramfs, and the @@ -38,8 +38,8 @@ what Ubuntu userland needs on this machine: - devtmpfs, procfs, sysfs, and tmpfs; - virtio-mmio block, network, console, and filesystem devices; - ext4, EROFS, and overlayfs; and -- the memory and process-count cgroups and the PID, mount, and UTS namespaces - that the sandbox uses, and network namespaces for guest services. +- the memory and process-count cgroups, and the PID, mount, UTS, and network + namespaces, which programs in the guest may use. Its [patches](../../kernel/patches) add the xe9 early and interactive consoles and the shared virtio-mmio interrupt status, and keep the HVC console from @@ -89,7 +89,7 @@ The builder then: `/nonexistent`, a `nogroup` group, and usr-merged `/bin`, `/sbin`, `/lib`, and `/lib64` links; and 7. installs the NVX scripts from `guest/common` and `guest/ubuntu`, and the - ten Rust guest binaries from `nvx-rs`. + ten Rust guest binaries of the `nvx` package. The builder never executes a binary from the Ubuntu tree on the host. Both artifacts are packed with root ownership, fixed timestamps, and a stable file @@ -121,8 +121,8 @@ Alpine. It installs the same ten Rust guest binaries as Alpine and Azure Linux under their `/sbin/nvx-*` names. Docker copies all ten from one digest-pinned Rust builder, while native and determinism builds use the shared fresh-target -routine. The package manifest records one aggregate `nvx-rs` source digest -and each installed binary's digest. +routine. The package manifest records the digest of the `nvx` package's +[guest inputs](../build.md#the-nvx-package) and each installed binary's digest. ## Ubuntu sandbox layer @@ -146,7 +146,9 @@ proof. The layer runs under the Alpine control initramfs as the sandbox's read-only `distro` layer over a writable scratch disk, which is formatted separately -with `mkfs.ext4`; see the +with `mkfs.ext4`, such as a copy of the sandbox scratch template, +`ubuntu-smoke-scratch.ext4`, which the sandbox scenarios boot and +[`build-scratch-template`](../usage.md#build-scratch-template) formats; see the [experimental sandbox](../run.md#experimental-single-workload-sandbox). The sandbox's init, nvx-init, a static binary, runs each execution itself, as a plain process in the VM's root. Its executions share nvx-init's mount, PID, @@ -172,7 +174,8 @@ Base or the lock. Release staging rejects an Ubuntu manifest that does not match its artifact and the current inputs. Platform releases carry all four Ubuntu files under `guest/`, with the sandbox scratch template, `ubuntu-smoke-scratch.ext4`, -which has no manifest; `nvx-rs download` installs them all, and +which has no manifest, as [Release packages](../build.md#release-packages) +lists; `nvx-rs download` installs them all, and `run --guest ubuntu` fails when the initramfs is missing instead of falling back to Alpine. [Package and source delivery](../distribution.md) describes how diff --git a/doc/project-structure.md b/doc/project-structure.md index a0be6cbf6..d3ae54ec6 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -19,12 +19,12 @@ are build products or caches and are not part of the tracked source tree. The | `guest` | Common and distribution-specific guest scripts | | `ubuntu` | Pinned Ubuntu supplemental binary-package lock | | `azurelinux` | Checksum-pinned Azure Linux supplemental RPM lock | -| `nvx-rs` | Rust crate `nvx`: the NVX mechanics | +| `nvx-rs` | The `nvx` package: the library of NVX's mechanisms, with NVX's own programs as its binaries | | `openvmm` | Public OpenVMM Git submodule from `microsoft/openvmm`, tracking its `feature/microvm/dev` branch | | `data` | Tracked performance history and generated benchmark data | | `scripts/nvx_tools` | Implementation of the `nvx.py` commands | | `scripts/nvx.py` | Canonical build, run, test, benchmark, and packaging CLI | -| `scripts/setup` | Development host and GitHub Actions runner bootstrap scripts, and the manifest of the tool versions that they install | +| `scripts/setup` | Development host and GitHub Actions runner bootstrap scripts, the manifest of the tool versions that they install, and the Python package that provisions NVX's CI runners on Azure | | `scripts/nvx_adversarial_executor.py` | Credential-free adversarial executor protocol entry point | | `.cache/linux` | Generated verified/patched Linux tree; ignored by Git | | `build/sources` | Generated Linux, Alpine, and Ubuntu release sources; ignored by Git | @@ -94,20 +94,28 @@ nvx/ | |-- config-microvm MicroVM kernel configuration, generated from the feature inventory | |-- config-microvm.derived Rule that sets each symbol of config-microvm beyond the inventory | `-- config-microvm-debug CI debug-kernel fragment (watchdogs on) -|-- nvx-rs/ Rust crate `nvx` for the NVX mechanics +|-- nvx-rs/ The `nvx` package: NVX's library, with NVX's own programs as its binaries | |-- src/ +| | |-- lib.rs The crate's root, its documentation, and its public modules | | |-- protocol/ Control-console codec, NVXS session, and the guest targets that live shares may not take, shared by the host and the guest | | |-- activation/ The activation record of a layered root, its canonical encoding, and what Ready reports about it, shared by the host, the guest, and every adapter's framing | | |-- contract.rs Control-contract versions and release compatibility | | |-- net/ CIDR parsing and subtraction, and hostname and guest-network forms | | |-- gpt/ Strict reading of GPT disks whose first partition configures an adapter and whose other partitions hold a root's read-only layers, shared by the host and the guest -| | |-- host/ Artifacts, launch, network and egress-policy lowering, sessions, VMs, the sandbox launch profile, and the `nvx-rs` CLI (feature `cli`) -| | |-- testing/ Fake OpenVMM (feature `testing`) -| | |-- guest/ Programs that run in Linux guests, the guest's NVXS connection, the libraries through which any guest init or agent opens the terminal on which it serves its session, starts a program, collects its children's statuses, settles its executions' cgroups, drives each execution, mounts host mappings beneath the descriptors of the roots that it names, assembles and tears down a layered root, and reads a GPT disk's partition table through its block device, and the typed calls through which it runs the time ABI's steps and the reseed (feature `guest`) -| | `-- bin/ nvx-rs CLI, nvx-fake-openvmm test double, and the guest binaries +| | |-- uuid.rs UUIDs in their canonical text and in GPT's layout, shared by the host and the guest +| | |-- hypervisor.rs The hypervisors that run the VMs: KVM, MSHV, and WHP +| | |-- error.rs `nvx::Error` and its `ErrorKind`, and the crate's private `ErrorCode` +| | |-- host/ Feature `host`: the runtime, VMs and their configuration, snapshots, restores, the transport, process references, outcome reports, and the lowering of network postures and host export plans; private to the crate, the platform layer and what only NVX's CLI uses, such as the export planner, the sandbox launch profile, and the egress policy compiler; with feature `cli`, NVX's hidden `nvx-rs` CLI +| | |-- testing/ Feature `testing`: NVX's hidden fake OpenVMM +| | |-- guest/ Feature `guest`, on Linux: the libraries on which a guest init builds, such as its NVXS session, terminal, host mappings, time ABI steps, reseed, snapshot capture, and activation; NVX's hidden guest programs; and NVX's guest space, private to the crate +| | `-- bin/ One file per binary, which calls its program's hidden `main`: nvx-rs, nvx-fake-openvmm, and the ten guest binaries | |-- examples/ The external-launch example, and the documented host and guest workflows, which only compile -| |-- tests/ Fake-OpenVMM behavior, process and VM spawn, CLI scenario, NVXS session, guest-binary, and hypervisor public-API VM lifecycle tests, the security checklist, sandbox configurations of every format, the cases of the export and root boot tokens, and the conformance vectors of the wire formats -| `-- public-api/ The crate's public API on Linux and on Windows, which CI checks +| |-- tests/ Integration tests of the CLI's scenarios, in cli/, the fake OpenVMM, the guest binaries, the NVXS session, in nvxs/, the layering of `nvx::host`, a VM's lifecycle on a hypervisor, the security checklist, and the copies of the OpenVMM submodule's data, with their fixtures and the conformance vectors of the wire formats +| |-- public-api/ The crate's public API on Linux and on Windows, which CI checks +| |-- Cargo.toml Package manifest: features, binaries, examples, and the `guest` profile +| |-- Cargo.lock The package's own lockfile +| |-- LICENSE The package's MIT license, which the crate ships +| `-- README.md The crate's guide: its modules, the nvx-rs CLI, its features, its guest binaries, and its tests |-- openvmm/ OpenVMM Git submodule |-- scripts/ Build, run, benchmark, and release tooling | |-- nvx_tools/ Python implementation behind the NVX CLI @@ -132,6 +140,8 @@ nvx/ | | |-- development_release.py Resumable development release publication | | |-- doctor.py Time ABI host qualification | | |-- elf.py Static-executable check of ELF binaries +| | |-- guest_artifact_reuse.py Reuse of a pull request's guest artifacts in its merge group and dev push +| | |-- guest_binaries.py Build, checks, and identity of the nvx package's guest binaries | | |-- guests.py Typed guest descriptors | | |-- host_telemetry.py Benchmark host provenance and Windows capture storage telemetry | | |-- host_time_probe.rs Host TSC probe that doctor builds and runs @@ -144,7 +154,7 @@ nvx/ | | |-- microvm_test_scripts/ Guest workloads for the microVM tests | | |-- microvm_tests.py NVX microVM correctness tests | | |-- nightly_ci.py Nightly CI failure report -| | |-- nvx_rs.py Finds and builds nvx-rs, and runs it for the microVM tests +| | |-- nvx_rs.py Finds and builds nvx-rs, records its provenance, and runs it for the microVM tests | | |-- openvmm_ci.py Check that OpenVMM's own CI passed the pinned revision | | |-- openvmm_process.py Foreground OpenVMM process control for tests | | |-- performance.py CI performance collection, persistence, and gates @@ -158,6 +168,14 @@ nvx/ | | |-- time_abi.py Time ABI harness checks and CPU profile guidance | | `-- ubuntu.py Verified Ubuntu rootfs and EROFS preparation | |-- setup/ Host and Actions runner bootstrap scripts +| | |-- azure/ Azure CI runner provisioning of existing scale sets +| | | |-- nvx_azure/ Standard-library Python package that scale-set instances run to become CI runners +| | | |-- linux/ Example Linux scale-set configuration +| | | |-- windows/ Example Windows configuration, Custom Script entry shim, and PowerShell adapters +| | | |-- build_package.py Offline packager of a scale set's bootstrap payload and model patch +| | | |-- requirements.txt The packager's Python requirement +| | | `-- README.md Azure runner provisioning guide +| | |-- README.md Development host and Actions runner bootstrap guide | | |-- setup-linux.sh Linux/KVM or Linux/MSHV development-host and Actions runner bootstrap | | |-- setup-windows-whp.ps1 Windows/WHP development-host and Actions runner bootstrap | | `-- tool-versions.conf Shared tool release and checksum pins @@ -190,7 +208,10 @@ nvx/ ``` The ignored `.cache/` directory may also appear at the repository root. It -contains downloaded and prepared upstream source trees, including Linux. +contains the verified upstream archives and packages that the guest builds +download, under `downloads/`, and the prepared upstream source trees, including +Linux; `NVX_CACHE_DIR` moves the guest builds' cache elsewhere, as +[Build](build.md) describes. ## Source directories @@ -235,10 +256,13 @@ profiling, or the interactive shell. On an interactive boot, `init` prints a stable distribution boot marker for Alpine and Azure Linux, and `guest/ubuntu/nvx-bashrc` prints Ubuntu's. -The remaining common helpers handle shutdown, virtio-fs mounting, snapshot -preparation, and the guest side of the time ABI. The Rust guest binaries -provide shutdown, managed control, restore repair, device access, ingress -announcements, and diagnostic probes. +The other common scripts mount the live host share (`nvx-hostmount`) and +prepare the guest for a snapshot and repair its restores (`nvx-snapshot`); the +microVM tests and benchmarks run the rest as probes (`nvx-identity-probe`, +`nvx-sandbox-smoke`, and `nvx-virtio-restore-probe`). The Rust guest binaries +provide a layered root's PID 1, managed control, the guest side of the time +ABI, restore repair, shutdown, device access, ingress announcements, and +diagnostic probes. `ubuntu/packages.lock.json` pins the complete supplemental `.deb` closure installed without maintainer-script execution. @@ -275,19 +299,25 @@ for other kernel-specific details. ### `nvx-rs/` -The Rust crate `nvx` holds the NVX mechanics: the -authenticated control protocol of the guest agent, artifact discovery, -OpenVMM's command line, the lowering of network and -filesystem policies, the compiler of the egress policy files that `nvx-rs run` -and `nvx-rs sandbox` accept, process and platform primitives for Linux and -Windows hosts, the creation, spawn, and stop of VMs, the sandbox launch profile -of `nvx-rs sandbox`, the `nvx-rs` command-line interface that runs NVX's -microVMs and sandboxes, downloads releases, compiles egress policies, and looks -up artifacts, and a fake OpenVMM for tests. Its -`guest` feature builds the ten static executables that every guest image -installs at `/sbin/nvx-*`. It is a standalone crate with its own `Cargo.lock`, -and builds independently of the OpenVMM submodule. See its -[README](../nvx-rs/README.md) and its [module diagrams](modules/README.md). +`nvx-rs/` holds the `nvx` package, the repository's one Cargo package. Its +library, `nvx`, holds the mechanisms of NVX's microVMs: the NVXS session and +the codecs that the host and the guest share, OpenVMM's command line, the +lowering of network postures and host export plans into OpenVMM's options, +verified references to OpenVMM's processes, the creation, spawn, reattachment, +and stop of VMs, their snapshots and restores, and the guest's side of a +snapshot. Its binaries are NVX's own programs: the `nvx-rs` command-line +interface (feature `cli`), which runs NVX's microVMs and sandboxes, downloads +releases, compiles egress policies, and looks up artifacts; a fake OpenVMM for +its tests (feature `testing`); and the ten static executables that every guest +image installs at `/sbin/nvx-*` (feature `guest`). The code that only these +programs use, such as the CLI's stores, export planner, and egress policy +compiler, the platform layer, and the guest's space, is private to the crate, +and each binary calls a `main` that `#[doc(hidden)]` hides, so the snapshots +in `public-api/` record the library's public API alone. The package has its +own `Cargo.lock`, and builds independently of the OpenVMM submodule; +[The nvx package](build.md#the-nvx-package) lists its features, profiles, +binaries, outputs, and license inputs. See its [README](../nvx-rs/README.md) +and its [module diagrams](modules/README.md). ### `openvmm/` @@ -312,6 +342,9 @@ module has no dependencies on the workflow or configuration modules. Standalone benchmark shell programs and parameterized guest templates live in `nvx_tools/benchmark_scripts/`, and the guest workloads of the microVM tests live in `nvx_tools/microvm_test_scripts/`. +`nvx_tools/guest_binaries.py` builds, checks, and identifies the guest binaries +of the `nvx` package, and `nvx_tools/nvx_rs.py` builds `nvx-rs` and records its +provenance; see [The nvx package](build.md#the-nvx-package). Source-collection scripts assemble corresponding-source archives for Linux, Alpine, and Ubuntu. The adversarial controller, typed broker, credential-free executor, watchdog, and tracked deterministic catalogs also live in @@ -341,6 +374,11 @@ hooks that `test-layout-allowlist.json` allows them, and The `setup/` scripts bootstrap development hosts and GitHub Actions runners using the [tool manifest](../scripts/setup/tool-versions.conf); see [Automated environment bootstrap](setup.md#automated-environment-bootstrap). +On Azure, `setup/azure/nvx_azure` is the standard-library Python package with +which the instances of NVX's CI scale sets provision themselves as runners, +through those scripts, and `setup/azure/build_package.py` packages it into a +scale set's custom data; see +[Azure CI runner provisioning](../scripts/setup/azure/README.md). `nvx-hosts.example.json` is the template for the [remote agent host](setup.md#remote-agent-hosts) inventory, and CI runs `publish_development_release.py` to publish development releases, @@ -370,14 +408,14 @@ draw the module diagrams that `doc/modules` publishes (see | Path | Contents | | --- | --- | -| `.cache/` | Downloaded, verified, and patched upstream source trees | -| `build/` | Kernels, initramfs and EROFS images, package manifests, and collected sources | +| `.cache/` | Downloaded and verified upstream archives and packages, and patched upstream source trees | +| `build/` | Kernels, initramfs and EROFS images, the sandbox scratch template, package manifests, the provenance of OpenVMM, the kernels, the Alpine initramfs, and `nvx-rs`, and collected sources | | `data/baseline/` | Base-branch histories staged by the performance gate | | `data/results/` | Collected p50 results for the current commit | | `data/runs/` | Raw benchmark logs and per-platform artifacts | | `dist/` | Staged binary and source release archives | | `openvmm/target/` | Rust build output produced inside the OpenVMM submodule | -| `nvx-rs/target/` | Rust build output of the `nvx` crate | +| `nvx-rs/target/` | Cargo's build output of the `nvx` package, among it `release/nvx-rs[.exe]` | | `.ruff_cache/` | Ruff's local lint cache | | `__pycache__/` | Python bytecode caches that may appear below Python source directories | diff --git a/scripts/setup/README.md b/scripts/setup/README.md index 15d6c620a..6e58e2535 100644 --- a/scripts/setup/README.md +++ b/scripts/setup/README.md @@ -11,8 +11,15 @@ Runner-only mode, which `--runner-only` on Linux and `-RunnerOnly` on Windows select, prepares a persistent GitHub Actions runner for NVX's CI jobs instead, and needs no checkout. -For Azure VMSS bootstrapping, see [Azure provisioning](azure/README.md). The -scripts in this directory remain usable on supported hosts outside Azure. +On Azure, [`azure/`](azure/README.md) provisions NVX's CI runners on existing +virtual machine scale sets with `nvx_azure`, a Python package that needs only +the standard library: each instance runs it from the scale set's custom data +when it is provisioned, and it checks the instance's identity, prepares its +data disk and backend, runs `setup-linux.sh` or `setup-windows-whp.ps1` in +runner-only mode, and registers the runner with a registration token that it +requests with a GitHub credential from Key Vault. `azure/build_package.py` +packages it, with these scripts and the tool manifest, into that custom data. +The scripts in this directory remain usable on supported hosts outside Azure. They do not create machines or initialize a checkout. Runner-only mode consumes a short-lived registration token from standard input and does not @@ -267,9 +274,11 @@ Validate dependencies without changing the host or requiring build artifacts: sh scripts/setup/setup-linux.sh --backend kvm --check-only --skip-build ``` -Omit `--skip-build` to also validate the built artifacts and the command that -NVX generates for the backend. Use `--workspace PATH` when the script is -outside the checkout. +Omit `--skip-build` to also validate the built artifacts, build `nvx-rs` from +the `nvx` package with the command of +[Usage](../../doc/usage.md#nvx-rs) and record its provenance, and check the +command that `nvx-rs run --dry-run` generates for the backend. Use +`--workspace PATH` when the script is outside the checkout. To produce a revision-bound guest bundle for a Windows build: @@ -313,9 +322,11 @@ Validate dependencies without changing the host or requiring build artifacts: .\scripts\setup\setup-windows-whp.ps1 -CheckOnly -SkipBuild ``` -Omit `-SkipBuild` to also validate the guest artifacts, OpenVMM binary, and -generated WHP command. Use `-Workspace PATH` when the script is outside the -checkout. +Omit `-SkipBuild` to also validate the guest artifacts and the OpenVMM binary, +build `nvx-rs` from the `nvx` package with the command of +[Usage](../../doc/usage.md#nvx-rs) and record its provenance, and check the +command that `nvx-rs run --dry-run` generates for WHP. Use `-Workspace PATH` +when the script is outside the checkout. ## Tool versions diff --git a/scripts/setup/azure/README.md b/scripts/setup/azure/README.md index 0fae3791c..13f6aa4de 100644 --- a/scripts/setup/azure/README.md +++ b/scripts/setup/azure/README.md @@ -25,6 +25,17 @@ Do not maintain a second copy of the runner setup or its tool versions here. | `windows/firewall.ps1`, `windows/openssh.ps1`, `windows/storage.ps1`, `windows/system.ps1` | Adapters that run the cmdlets a Python controller names, taking a JSON request and returning a JSON result. | | `build_package.py` | Validate configuration and generate deterministic guest payloads, hashes, and a partial VMSS model patch without accessing Azure or GitHub. | +`cli.py` takes one command, which names the step that it runs: + +| Command | Run by | Step | +| --- | --- | --- | +| `linux-bootstrap` | `setup-nvx-github-runner.service`, which cloud-init enables and starts after `cloud-final.service`, as root | Verify the instance identity and refresh the compute snapshot, prepare the data disk and the backend, run `setup-linux.sh --runner-only`, enable the metadata block, register the runner, and disable the unit once it has registered | +| `linux-block-metadata` | `nvx-runner-metadata-block.service`, and the runner service before each start | Block the runner account from Azure instance metadata | +| `linux-refresh-metadata` | The runner service before each start, as root | Refresh the compute snapshot alone | +| `linux-verify-mshv` | An administrator; `linux-bootstrap` runs the same check when `backend` is `mshv` and `kernelMode` is `azurelinux-mshv` | Report whether the VM booted as the MSHV root partition | +| `windows-stage` | `windows/vmss-bootstrap.ps1`, which the Custom Script extension runs | Verify and unpack the package, configure OpenSSH, and start the `Nvx-Setup-GitHub-Runner` task | +| `windows-bootstrap` | The `Nvx-Setup-GitHub-Runner` task, as SYSTEM, at every startup until registration succeeds | Block Network Service from instance metadata, verify the instance identity, prepare the data disk and the WHP and Hyper-V features, and register the runner | + Linux guests run every step as `/usr/bin/python3 -I -B /opt/nvx-bootstrap `. Cloud-init installs the distribution's `python3` and writes all guest files from the same payload