doc: describe the final gates, test owners, packaging and publishing (SL6.10f)

.github/copilot-instructions.md

@@ -29,7 +29,10 @@
   run the applicable commands from those actions.
 - For changes under `nvx-rs/`, run the commands in
   [check-nvx-rs](./actions/check-nvx-rs/action.yml).
-  They need no hypervisor.
+  They need no hypervisor. Its public-API check compares the public API of
+  `nvx` with the snapshots in `nvx-rs/public-api/`: a change to the public API
+  updates them with `python3 scripts/public_api.py --update` (see
+  [Rust crates](../doc/ci.md#rust-crates)).
 - OpenVMM source changes also require the package-scoped checks prescribed by the
   nested repository. Do not claim unavailable hardware or cross-platform gates
   passed; report the missing prerequisite and rely on CI when appropriate.
@@ -39,7 +42,8 @@
 - Put a Rust module's unit tests in its `tests` module, in files of their own
   (`src/a/b.rs` declares `#[cfg(test)] mod tests;` and keeps them in
   `src/a/b/tests.rs`), never inline or as test-only items in production files.
-  Python unit tests are `test_*.py` modules in `scripts/`. Run
+  Python unit tests are the `test_*.py` modules of `scripts/`,
+  `.github/specula/tests/`, and `.github/skills/*/scripts/`. Run
   `python3 scripts/check_test_layout.py`; a test hook that cannot move needs an
   entry in `scripts/test-layout-allowlist.json`, and moving test code out needs
   `--tighten`. See [Test layout](../doc/contribute.md#test-layout).
@@ -96,8 +100,9 @@
   `cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml`,
   and again after changing `nvx-rs/`. The microVM scenarios run it too, and build
   it there the first time that they need it, unless `NVX_RS` names another build
-  (see [Usage](../doc/usage.md#nvx-rs)). The `managed-lifecycle` scenario runs an
-  `nvx-rs` test with Cargo. In a checkout that holds no artifacts, `run` and
+  (see [Usage](../doc/usage.md#nvx-rs)). The `managed-lifecycle` and
+  `vm-lifecycle` scenarios run `nvx-rs` tests with Cargo. In a checkout that
+  holds no artifacts, `run` and
   `sandbox` first download those of the release of `HEAD` or of its nearest
   released ancestor, as `nvx-rs download` does; build the artifacts that a change
   affects instead of testing a release's.
@@ -157,8 +162,8 @@
   not manually edit `data/*.csv` unless the task explicitly concerns baseline
   maintenance, and change an expected benchmark host, `data/*-host.json`, only
   to accept a new host together with a reset of its series' history.
-- Full runs of `dev`, scheduled or dispatched, publish the `nvx` crate to
-  crates.io when `nvx-rs/` changed; see
+- Full runs of `dev`, scheduled or dispatched, publish the `nvx` crate, the
+  only crate that NVX publishes, to crates.io when `nvx-rs/` changed; see
   [crates.io releases](../doc/ci.md#cratesio-releases). Do not publish it by
   hand, and change the version that `nvx-rs/Cargo.toml` declares only to start
   another series, as [The nvx crate](../doc/distribution.md#the-nvx-crate)

.github/skills/nvx-ci-investigation/SKILL.md

@@ -40,6 +40,10 @@ cancels an older run when a newer head is synchronized.
   behind earlier pushes instead of being replaced, so its cancellation is never a
   supersession. Under earlier workflows, a newer `dev` push replaced the one waiting
   run.
+- A merge group's run that a `Cancel superseded merge group runs` job cancelled
+  belongs to a group that the merge queue replaced or that its pull request left.
+  Check the pull request's queue outcome before treating it as a failure; see
+  [Superseded merge groups](../../../doc/ci.md#superseded-merge-groups).
 - A run started while a pull request was Ready remains active after conversion to
   Draft because `converted_to_draft` is not a workflow trigger. Do not diagnose this
   as the draft guard being ignored.
@@ -79,9 +83,13 @@ Reproduce the earliest actionable failure with the narrowest equivalent command.
 Match the run's revision, platform, tools, environment, feature flags, matrix inputs,
 and generated artifacts as closely as practical.
 
-- For quality or CLI failures, run the focused command from
-  [check-quality](../../actions/check-quality/action.yml) or
-  [validate-nvx](../../actions/validate-nvx/action.yml).
+- For quality, CLI, or `nvx-rs` failures, run the focused command from
+  [check-quality](../../actions/check-quality/action.yml),
+  [validate-nvx](../../actions/validate-nvx/action.yml), or
+  [check-nvx-rs](../../actions/check-nvx-rs/action.yml). A failed public-API check
+  prints how the public API of `nvx` differs from its snapshots in
+  `nvx-rs/public-api/`; a deliberate change updates them with
+  `python3 scripts/public_api.py --update`.
 - For an NVX boot, microVM, snapshot, guest-console, or benchmark acceptance failure,
   load [nvx-debug](../nvx-debug/SKILL.md). When the matching backend is remote, first
   load [nvx-host-connect](../nvx-host-connect/SKILL.md) and use a configured profile
@@ -97,6 +105,13 @@ and generated artifacts as closely as practical.
   only on request, as
   [Expected benchmark hosts](../../../doc/benchmarks.md#expected-benchmark-hosts)
   describes.
+- A failed `OpenVMM CI of the pinned revision` job (`openvmm-ci`) means that
+  OpenVMM's own CI has not passed a job that NVX requires for the pinned revision,
+  as its annotations say, unless it could not read OpenVMM's runs;
+  `python3 scripts/nvx.py check-openvmm-ci` checks the pin again. Classify it as
+  OpenVMM's CI result, not an NVX defect: rerun a failed OpenVMM job in OpenVMM,
+  then the `openvmm-ci` job once OpenVMM's CI has passed the pin, as
+  [OpenVMM CI](../../../doc/ci.md#openvmm-ci) describes.
 
 Do not alter the current checkout merely to reach the run SHA. Use immutable `git show`
 content for inspection and report when exact-SHA execution is unavailable. Do not

doc/ci.md

@@ -16,8 +16,9 @@ platform job also checks that its runner is still the host that the series'
 history comes from (see
 [Expected benchmark hosts](benchmarks.md#expected-benchmark-hosts)). CI runs
 none of OpenVMM's own tests: OpenVMM's checkin gates test the pinned revision
-of `feature/microvm/dev`, on each pull request into the branch and on each push
-to it, on OpenVMM's own runners, and the GitHub-hosted `openvmm-ci` job checks
+of `feature/microvm/dev`, on the pull requests into the branch that are ready
+for review and on the pushes to it, on OpenVMM's own runners, and the
+GitHub-hosted `openvmm-ci` job checks
 that they passed the jobs that cover NVX's use of OpenVMM; see
 [OpenVMM CI](#openvmm-ci).
 Two jobs per backend exercise Linux, SMP, virtio, sandbox, and snapshot
@@ -555,8 +556,9 @@ backend also runs the public `managed-exec-config`, `sandbox-lifecycle`,
 of `test-microvm`, which drive `nvx-rs sandbox run`, `provision`, `start`,
 `exec`, `stop`, and `deprovision` as subprocesses (see
 [Run](run.md#managed-lifecycle)): request-scoped working directories and
-environments, timeouts, nonzero exits, stdout and stderr, bounded output and
-outcome reports, warm-guest state across requests and restarts, identity
+environments, timeouts, nonzero exits, stdout and stderr, output that `exec`
+streams whole past the 1 MiB that version 1 of NVXC forwarded, outcome
+reports, warm-guest state across requests and restarts, identity
 refusal, fail-closed lifecycle transitions, a crashed OpenVMM's stale runtime
 state, teardown that leaves no OpenVMM process, control endpoint,
 capability, or runtime record while the layer and scratch artifacts survive,
@@ -650,8 +652,8 @@ groups, or create a device node. When the test process also passes
 `CAP_SETUID` and `CAP_SETGID` to OpenVMM, as on a privileged developer host,
 the scenario requires the foreign identity to own the files it creates.
 Linux/KVM runs the
-broader Ubuntu SMP, managed lifecycle, network snapshot, blockless snapshot,
-and workload-identity set.
+broader Ubuntu SMP, managed lifecycle, network snapshot, and workload-identity
+set.
 
 OpenVMM release executables and provenance are built once by the independently
 addressable `build-openvmm-linux-gnu`, `build-openvmm-linux-musl`, and
@@ -732,14 +734,16 @@ of those jobs.
 `smp-lapic` repeats `smp` with the counting-LAPIC facts asserted, for explicit
 local use only; no CI job runs it. [Benchmarks](benchmarks.md#ci-collection)
 lists the jobs that consume, gate, publish, or persist the results. The workflow uses the read-only OpenVMM deploy key stored in the
-`OPENVMM_DEPLOY_KEY` Actions secret to fetch the private submodule at its pinned
+`OPENVMM_DEPLOY_KEY` Actions secret to fetch the OpenVMM submodule at its pinned
 commit. Shared guest binaries and development release packages move through
 short-lived workflow artifacts alongside the OpenVMM handoff and benchmark
 results. Caches only accelerate reproducible build inputs and outputs; consumers
 do not depend on them as a handoff.
 Pull requests and merge groups gate regressions against recent matching-platform
-history, and successful pushes to `dev` append their p50 values under `data/`. A
-pull request or merge group that changes no input of the platform or package
+history, and successful pushes to `dev` record their p50 values, which a full
+run of `dev` then appends under `data/` (see
+[Development releases and performance history](#development-releases-and-performance-history)).
+A pull request or merge group that changes no input of the platform or package
 jobs skips those jobs and the gate (see
 [Change classification](#change-classification)). Metadata-only
 performance jobs use GitHub-hosted Ubuntu runners. Provisioning instructions
@@ -779,8 +783,9 @@ Beside each platform job, a package job on a GitHub-hosted runner of the
 platform's operating system, `package-kvm`, `package-mshv`, or `package-whp`,
 packages the platform's release once `artifacts`, `distro-artifacts`, the
 platform's OpenVMM producer, and the `nvx-rs` build of its operating system
-end. The microVM guest and sandbox jobs also wait for `distro-artifacts`, and
-the debug-kernel jobs for `debug-kernel`. `Performance regression gate`,
+end (see [Release packages](#release-packages)). The microVM guest and sandbox
+jobs also wait for `distro-artifacts`, and the debug-kernel jobs for
+`debug-kernel`. `Performance regression gate`,
 `Publish development release`, `Record performance results`, and `Required
 status check` wait for the jobs whose results they check, and in a full run of
 `dev`, `Package nvx crate` waits for `Required status check` and
@@ -817,7 +822,8 @@ operating system hands it, as described above.
 Each also runs the ignored `nvx-rs` test `tests/openvmm_vectors.rs`, which
 compares the submodule's control-session protocol vectors, and the lines of its
 platform allowlist of kernel command-line tokens, with the copies that the
-protocol and template tests read in the `nvx-rs` job, which cannot check out
+tests of `nvx::protocol::nvxs` and `nvx::host::config` read in the `nvx-rs`
+job, which cannot check out
 OpenVMM, and the versions of OpenVMM's snapshot format, saved-state schema,
 time ABI, and restore packets with those of `nvx::contract`;
 see [OpenVMM's vectors](../nvx-rs/tests/vectors/README.md#openvmms-vectors).
@@ -1639,7 +1645,8 @@ NVX pins a revision of `feature/microvm/dev` in `microsoft/openvmm` (see the
 [OpenVMM upstream roadmap](openvmm-upstream-roadmap.md)), whose own checkin
 gates test it: since OpenVMM's `c2722fbcd` of October 21, 2026, the OpenVMM PR
 workflow runs them on each pull request into the branch that is ready for
-review, and the OpenVMM CI workflow on each push to it. On OpenVMM's own
+review, and the OpenVMM CI workflow on each push to it but those that change
+only its `Guide/` and `petri/logview/`. On OpenVMM's own
 runners, they run OpenVMM's unit and documentation tests on x86-64 and AArch64
 Linux, Linux musl, and Windows, and its VMM tests on Windows Intel and AMD
 hosts, where the WHP tests run, on Linux KVM and MSHV hosts, and on TDX,
@@ -1891,6 +1898,36 @@ expired, seven days after the run uploaded them, as lost. Until a full run
 persists them, the performance regression gate compares a pull request or a
 merge group with a history that lacks up to a day of merges.
 
+## Release packages
+
+Each package job, `package-kvm`, `package-mshv`, or `package-whp`, calls the
+reusable [`package-platform.yml`](../.github/workflows/package-platform.yml)
+workflow, which packages one platform's release on the runner that the job
+names, `ubuntu-latest` or `windows-latest`. The job passes it the platform's
+suffix, `linux-kvm`, `linux-mshv`, or `windows-whp`, the base name of the
+platform's OpenVMM artifacts, and that of the `nvx-rs` artifacts of its
+operating system, and, like the other jobs that need the OpenVMM deploy key,
+runs only for pushes, merge groups, full runs, and same-repository pull
+requests. The workflow checks out NVX and OpenVMM, whose license the package
+ships, checks the source pins with [`verify`](usage.md#verify), and downloads
+the guest kernel, every guest image, the sandbox scratch template, and the
+OpenVMM executable and its provenance (see
+[Guest artifact handoff](#guest-artifact-handoff)). Its
+[`package-release`](../.github/actions/package-release/action.yml) action then
+downloads `nvx-rs` and its provenance, stages the binary package with
+[`package --binary-only`](usage.md#package), on Windows after it recreates the
+kernel's provenance inputs with
+[`materialize-kernel-provenance-inputs`](usage.md#materialize-kernel-provenance-inputs),
+and archives it with [`archive-release`](usage.md#archive-release) as
+`nvx-<VERSION>-<platform>.tar.gz`, or `.zip` on Windows.
+
+Every run that runs the package jobs packages and archives the releases, which
+proves that they build, but only a `dev` push uploads them, as the one-day
+`release-<platform>` artifacts that `Publish development release` publishes
+(see [Development release publication](#development-release-publication)).
+Pull requests, merge groups, full runs, and pushes to `feature/nvx-refactor`
+upload no package.
+
 ## Development release publication
 
 `Publish development release` runs
@@ -2087,13 +2124,19 @@ flagging script shares to read Rust.
 ## crates.io releases
 
 A scheduled or manually dispatched full run of `dev` publishes the
-[`nvx` crate](../nvx-rs/README.md) to crates.io once every job of the run has
-passed, which `Required status check` confirms. Two jobs on GitHub-hosted
+[`nvx` crate](../nvx-rs/README.md) to crates.io once every job of the run but
+`Persist performance baselines`, which runs apart, has passed, which
+`Required status check` confirms. Two jobs on GitHub-hosted
 Ubuntu runners, whose checkouts have the LF line endings of the commit, run
 [`publish_nvx_crate.py`](../scripts/publish_nvx_crate.py): `crate-package`,
 `Package nvx crate`, waits for that check, and `crate-release`,
 `Publish nvx crate`, waits for `crate-package`. Pushes, pull requests, and
-merge groups never publish the crate. A run publishes a new version only when `nvx-rs/`
+merge groups never publish the crate; a `dev` push publishes its development
+release instead (see
+[Development release publication](#development-release-publication)). `nvx` is
+the only crate that NVX publishes: NVX's own programs, such as the `nvx-rs`
+CLI, nvx-init, and the other guest binaries, are binaries of that package, not
+crates of their own. A run publishes a new version only when `nvx-rs/`
 changed since the commit from which the newest version on crates.io was
 packed, which that version's `.cargo_vcs_info.json` records. A night without
 changes to the crate publishes nothing, and neither does a run of a commit
@@ -2289,10 +2332,22 @@ through the [`check-nvx-rs`](../.github/actions/check-nvx-rs/action.yml)
 action, on the Rust release that `rust-toolchain.toml` pins. It runs rustfmt,
 Clippy with warnings denied for all targets and features, for the library with
 and without default features, and for the `serde`, `cli` and `testing`
-features alone, the unit, protocol
-conformance-vector and property, and fake-OpenVMM instance tests, a test that
-the direct children of `nvx::host` depend on one another without cycles, rustdoc
-with warnings denied, the minimum Rust version check, and the macOS cross-check.
+features alone, the crate's tests with every feature and with the default
+features, and the library's without default features, rustdoc with warnings
+denied, the minimum Rust version check, and the macOS cross-check. The tests
+are the unit tests of each module, among them the protocol's conformance-vector
+and property tests and the process tests of `nvx::host::process`, and the
+integration tests of `nvx-rs/tests/`: the CLI's scenario tests, which run
+`nvx-rs` with the fake OpenVMM; the fake OpenVMM's own behavior; on Linux, the
+command lines, exit statuses, and output of the guest binaries; the NVXS
+session, whose host and guest roles exchange messages through a broker that
+relays records as OpenVMM's does; the
+[security checklist](../nvx-rs/tests/security_checklist.rs), which checks
+through the public API alone the security requirements that the host's
+documentation states for a VM's directory and outputs and for a snapshot's
+directory; and a test that the direct children of `nvx::host` depend on one
+another without cycles. The crate's [Testing](../nvx-rs/README.md#testing)
+section describes them, and the ignored tests that other jobs run.
 Clippy checks `serde`, `cli` and `testing` alone so that the compiler checks the
 code that each of them builds, which the code of other features may
 otherwise use; the musl check of the guest binaries checks `guest` alone.
@@ -2311,8 +2366,9 @@ rustup. rustdoc needs only a target's standard library, not its linker, so the
 Linux runner checks both snapshots, and any host lists both platforms alike.
 Each snapshot lists one public item a line, sorted by path: its kind, its path,
 and `(hidden)` if `#[doc(hidden)]` hides it or an item that contains it. The
-list follows re-exports, and leaves trait implementations out. The check
-prints how the API differs from a snapshot, and fails; a pull request that
+list follows re-exports, and leaves trait implementations out. The check,
+which `python3 scripts/public_api.py` runs on any host, prints how the API
+differs from a snapshot, and fails; a pull request that
 changes the public API runs `python3 scripts/public_api.py --update` and
 commits the snapshots, so that its diff shows the change. rustdoc's JSON is
 unstable, so the script reads only the format version of the pinned release,