[Pedro Henrique Penna (ppenna)]

Part of #1112 (SL3.11a, "scripts/nvx.py drives the CLI, and its checks pass"). Parent #1083, umbrella #1076.

What and why

SL3.11a's goal is that scripts/nvx.py drives the existing nvx-rs binary through NVX_RS, and that its checks of the CLI assert behavior instead of the old output. Most of its Scope was already true on feature/nvx-refactor b7350086, through these pull requests:

Scope	Already done by
scripts/nvx_tools/nvx_rs.py builds nvx-rs from nvx-rs/Cargo.toml into nvx-rs/target (--target-dir nvx-rs/target). It runs the executable that NVX_RS names, else nvx-rs/target/release/nvx-rs[.exe], else that one after it builds it.	#597
microvm_tests.py and its scenario drivers (managed_exec_tests.py, sandbox_lifecycle_tests.py, sandbox_proxy_tests.py, sandbox_ram_upper_tests.py) run every nvx-rs command through nvx_rs.command_line, and so through that selection.	#902
CI's NVX CLI jobs run the Python checks with NVX_RS naming the build that CI made at nvx-rs/target/release/. The microVM jobs put the same build there.	#597, #1028
A default test-microvm run runs the public sandbox scenarios.	#1047
The public sandbox scenarios are now nine. sandbox-gpt-layer, snapshot-core, snapshot-tiers and untiered-network-policy also run nvx-rs through nvx_rs.command_line, the last three on the CLI's named snapshots. vm-lifecycle is a Cargo test of the library, which runs no CLI.	#1496 (SL5.1b), #1500 (SL5.1c), #1502 (SL3.7c)
The checks tell nvx-rs's errors apart by their stable codes. The CLI's own tests replaced the transcripts that scripts recorded from nvx.py.	#812, #830 (#728)
The scenario drivers' expectations of the CLI's state, records and refusals follow the CLI as this workstream changes them.	SL3.5a #1466, SL3.5b #1460, SL3.4b #1490, SL3.9 #1469 and SL3.10c #1501, which records format 10, updated them, and SL3.8b #1489 retired run-console-restore.

What remained were two checks that compared the output of the nvx-rs that NVX_RS names with wording that it inherited from nvx.py. A reworded message or help failed them, while a refusal with another code, or a guest booted with another memory default, passed them. This PR makes them assert behavior:

• NvxRsTests.test_the_real_nvx_rs_prints_what_nvx_py_reads required the refusal of an egress policy that isn't an object to read exactly egress policy must be an object, the message of the Python code that nvx-rs replaced in Make nvx-rs the backend of nvx.py sandbox, egress, and artifact lookup #597. It now requires the refusal's stable code, unsupported, and a message of one line without that code, as nvx_rs.query reads it.
• NvxRsTests.test_run_help_states_each_guest_memory_default required the help of nvx-rs run to state alpine 128 and the other guests' defaults, as the help of nvx.py run did before nvx.py: retire the run and sandbox commands in favor of nvx-rs #902. test_run_boots_each_guest_with_its_default_memory replaces it. For each guest, it runs nvx-rs run --guest NAME --dry-run through nvx_rs.command_line, on placeholder artifacts. It then requires the --memory of the OpenVMM command that a run would launch to be the memory that guests.py gives test-microvm.
• The docstring of nvx_rs.query no longer says that nvx.py prints the message "as it printed the message of the Python code that nvx-rs replaced".

Nothing else needed a change. The scenario drivers match the CLI's error codes and fragments of its current messages, such as option, file and snapshot names, and none compares the old output. The outputs that the checks keep, the native rules of the egress policies that they compile, still hold, so none needed recording again. No scenario driver changes, so there is no scenario-driver change to flag.

The case matrix (doc/acceptance-case-matrix.md) lists neither test, and its validate-nvx row, which runs them, stays kept.

Size

Measured against feature/nvx-refactor b7350086, as against 14ca2671 before the rebase:

• Core lines: 0. The change is tests and a docstring.
• Counted files: 1, scripts/nvx_tools/nvx_rs.py, whose docstring changes. scripts/test_nvx_tools.py holds tests.

Commits

1. tests: check the CLI's behavior instead of its old output (SL3.11a)

Not in this PR

• Retiring scripts/nvx.py, or changing NVX_RS's precedence, the build into nvx-rs/target, or packaging.
• Changes to nvx-rs: no command, flag or output changes.
• test_usage_documents_every_option_of_nvx_rs, which checks that doc/usage.md documents each option that the help of nvx-rs run and sandbox lists. It compares option names, which are stable, not wording.
• The microVM scenarios, which don't change.

Validation

On this head, 7d143782, the same commit rebased onto b7350086 (git range-diff shows no content change), with nvx-rs built from it and NVX_RS naming the build:

Check	Windows (local)	Linux (nvx-kvm-test)
python -m unittest scripts.test_nvx_tools.NvxRsTests	21 tests, OK	21 tests, OK, with the static x86_64-unknown-linux-musl build that CI's NVX CLI job on Linux uses
validate-nvx's Python suite (python -m unittest scripts/test_performance.py ... scripts/test_public_api.py)	1833 tests, OK (70 skipped)	test_nvx_tools and test_microvm_tests, with a GNU build: 927 tests, OK (12 skipped)
test_adversarial.py, test_hosts.py, and validate-nvx's nvx.py ... --help commands	passed
ruff check and ruff format --check of scripts .github/specula; pyright --pythonplatform Linux and Windows	passed
check_test_layout.py, check_markdown_links.py, and the case matrix's check (check_matrix.py against 99d2022085f)	passed

Before the rebase, on 5b5b3339, with nvx-rs built from it (cargo build --release --locked --features cli --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target) and NVX_RS naming the build:

Check	Windows (local)	Linux (nvx-kvm-test)
python -m unittest scripts.test_nvx_tools.NvxRsTests	21 tests, OK	21 tests, OK
The three checks that run nvx-rs, with a static x86_64-unknown-linux-musl build, as CI's NVX CLI job on Linux uses		OK
validate-nvx's Python suite (python -m unittest scripts/test_performance.py ... scripts/test_public_api.py)	1829 tests, OK (69 skipped)	test_nvx_tools and test_microvm_tests: 923 tests, OK (12 skipped)
test_adversarial.py, test_hosts.py, and validate-nvx's nvx.py ... --help commands	passed
ruff check and ruff format --check of scripts .github/specula; pyright --pythonplatform Linux and Windows	passed
check_test_layout.py, check_markdown_links.py	passed

The rewritten checks test behavior. On Windows, I rebuilt nvx-rs twice with temporary edits, which I then reverted:

nvx-rs built with	Base checks	Rewritten checks
The refusal reworded to has to be a JSON object and the help to alpine: 128 MiB	both failed	both passed
The refusal's code changed to io, and Alpine's default memory to 256 MiB, with the message and the help unchanged	both passed	both failed ('io' != 'unsupported'; ['256M'] != ['128M'])

No microVM ran: no scenario changes. CI's guest-and-sandbox jobs run the public sandbox scenarios with CI's build. On the base's push run, 43131339763 at 14ca2671, they passed on KVM, MSHV and WHP, and its NVX CLI jobs ran all 21 NvxRsTests with NVX_RS set, none skipped.

Left to CI: shellcheck and Specula's tests (no shell or Specula change), the nvx-rs checks (no Rust change), and the microVM, platform and package lanes.

Changes to existing tests

python3 scripts/check_test_changes.py --base origin/feature/nvx-refactor lists 2 changes to existing tests and 1 added test, so this PR has the tests-modified label:

• NvxRsTests.test_the_real_nvx_rs_prints_what_nvx_py_reads, in scripts/test_nvx_tools.py, is rewritten: it checks the refusal's code and the message's form instead of the old message.
• NvxRsTests.test_run_help_states_each_guest_memory_default is removed and rewritten as the added NvxRsTests.test_run_boots_each_guest_with_its_default_memory. It checks the memory with which nvx-rs run would boot each guest instead of the help's words.

[github-actions]

This pull request changes existing tests

It modifies or removes 2 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change	What	Before	After
modified test	def NvxRsTests.test_the_real_nvx_rs_prints_what_nvx_py_reads in scripts/test_nvx_tools.py	scripts/test_nvx_tools.py:2501	scripts/test_nvx_tools.py:2529
removed test	def NvxRsTests.test_run_help_states_each_guest_memory_default in scripts/test_nvx_tools.py	scripts/test_nvx_tools.py:2455

Relocated unchanged: 0; added: 1. Compared the merge base b73500866e03 with 7d14378255ab.

[Copilot]

🟢 Approval recommended

No active review issues block approval of the test and docstring updates.

No open issues 🎉

What changed in this PR

This PR advances SL3.11a by making Python’s checks of the selected nvx-rs CLI verify behavior rather than legacy wording. It changes tests and a docstring only; CLI behavior remains unchanged.

Changes:

• Replace help-text checks with dry-run assertions that each guest’s OpenVMM memory argument matches its Python-defined default.
• Check invalid egress policies by stable error code and single-line message format instead of exact wording.
• Clarify that callers should distinguish failures by code because messages may change.

File	Description
scripts/​test_nvx_tools.py	Rewrites guest-memory and egress-error checks to assert behavior.
scripts/​nvx_tools/​nvx_rs.py	Updates the query docstring to emphasize stable error codes.

_{Review effort: Balanced}