Skip to content

tests: wait for the fake OpenVMM's whole command line (flaky state read) - #1511

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-cli-state-race
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-cli-state-race

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Part of #1076. Fixes a flaky test of the CLI's suite (cli-state-race) that #1460 (SL3.5b) introduced.

What and why

Symptom. Run 43131964028, attempt 1, of #1496, which doesn't touch this code, failed in the job nvx-rs / windows-latest, step "Check nvx-rs":

test start_opens_the_stable_caller_paths_that_provision_checked ... FAILED
thread 'start_opens_the_stable_caller_paths_that_provision_checked' (2888) panicked at tests\cli\harness.rs:538:33:
C:\Users\runneradmin\AppData\Local\Temp\nvx-cli-p3Sm4N\build\fake-openvmm-state.json: EOF while parsing a value at line 1 column 0:

The test passes in other runs, and 24 parallel local runs didn't reproduce it.

Root cause. The test starts nvx-rs sandbox start in the background and, to look at the VM while it starts, waits with wait_for until the fake OpenVMM's record of its command line, build/fake-openvmm-state.json, exists. It then parses the record with Root::openvmm_arguments, which panics on a parse error. The fake writes the record first thing in its main, with fs::write, which creates the file and then writes its content, so a test that reads it in between finds it empty. wait_for_json, beside wait_for in the same file, says exactly this ("the fake OpenVMM writes its dumps with fs::write, which creates a file before it writes the file's content"), and the next test already waits with it for the fake's record of its block devices.

Origin. #1460 (SL3.5b), in 597ec1bfbf1 ("Test the approved stable caller-path ownership contract"), turned start_hands_openvmm_the_images_that_it_checked into this test and added the parse right after the wait_for, which #752 had added only to know that OpenVMM had started.

Fix. Only the test changes:

  • It takes the arguments from wait_for_json, which returns once the record holds a whole JSON document. The record is a JSON array, so no prefix of it parses.
  • It checks the same things in the same order: the record of the block devices still doesn't exist then, since the fake opens its images 1.5 s (fake_block_delay_ms=1500) after it starts, and the record's arguments still name the stable caller paths.
  • wait_for's comment now sends a test that reads a file's content to wait_for_json.

No sleep and no timeout changes.

Why the fake keeps fs::write. No other reader of the fake's records waits for one to exist and then parses it (see below), so this reader's fix covers them all. Writing the records through a temporary file and a rename would spare future readers the wait, but on Windows a rename fails while another process, such as a scanner, holds the file that it replaces open without sharing its deletion, as rename_away_and_back in checked_handoff.rs allows for. The fake ignores its write errors, so the second start of a sandbox that a lifecycle test starts twice could then leave the first start's record in place.

Other readers of the same kind. In nvx-rs/tests/ and the fake OpenVMM:

  • The fake's records (fake-openvmm-TOKEN.json, -blocks.json, -maps.json): tests/cli/{lifecycle,formats,sandbox_run,one_shot,openvmm}.rs, tests/cli.rs and tests/template_capture.rs read them once the command that launched the fake has returned, which a start does only after the guest's handshake and mapping table that the fake serves after it wrote them, or once the fake has exited, as tests/fake_openvmm_behavior.rs does. an_interrupted_start_keeps_its_claim_until_recovery_confirms_exit already waits with wait_for_json. None waits for a record to exist and then parses it, so this PR fixes the only one.
  • The other waits for a file's existence read nothing in it (the touch markers of tests/cli/exec_lock.rs, of the interrupts tests and of a_run_passes_openvmm_no_handle_of_its_caller, and fake-openvmm-.json in interrupting_a_cold_boot_ends_it_without_guidance), or wait for files written through a temporary file and a rename (the fake's foreground gate's phase.json, the fixtures' reports in checked_handoff.rs and supervisor.rs, the CLI's launch.json).
  • Not fixed here, the same shape on other files, neither seen failing:
    1. tests/template_capture.rs, crash_adapter: it polls until the fake's capture marker active exists and the journal records the VMM, kills the adapter, then parses the process ID in the marker with active_pid. The fake creates the marker with File::create_new and then writes the ID (Active::begin in src/testing/fake_openvmm.rs), so the marker is empty for a moment.
    2. Outside nvx-rs/tests/, in the Linux guest unit tests of src/guest/managed_agent/dispatch/tests.rs: a_reset_does_not_wait_for_output_from_an_escaped_direct_descendant and direct_endings_do_not_wait_for_an_escaped_descendant_to_close_output wait until escaped exists, and later parse the process ID that a shell wrote there with echo $$ > "$0". A third test in that file already waits for the file to be non-empty.

Size

0 core lines in 0 counted files: the change is in nvx-rs/tests/cli.rs alone, with 5 lines added and 3 removed.

Changes to existing tests

python scripts\check_test_changes.py --base origin/feature/nvx-refactor lists 1 change to an existing test, hence tests-modified:

  • start_opens_the_stable_caller_paths_that_provision_checked waits for the fake OpenVMM's whole record of its command line instead of its existence, and parses the arguments from it. It keeps every assertion, in the same order.

doc/acceptance-case-matrix.md lists no test of nvx-rs/tests/cli.rs, so no row changes.

Not in this PR

The fake OpenVMM's writes, and the two readers listed under "Not fixed here".

Validation

Reproduction. A temporary, uncommitted edit of the fake widens the window between the record's creation and its content to 300 ms:

-    let _ = fs::write(
-        &options.args_dump,
-        serde_json::to_vec(&arguments).unwrap_or_default(),
-    );
+    let _ = fs::File::create(&options.args_dump).and_then(|mut file| {
+        thread::sleep(Duration::from_millis(300));
+        file.write_all(&serde_json::to_vec(&arguments).unwrap_or_default())
+    });
Run of the test with the widened window, at 14ca2671205 Windows 11 workstation (Intel i9-12900H, 20 logical processors) nvx-kvm-test (Azure Linux 3 VM, KVM)
The base test failed 1 of 1, with the CI symptom (harness.rs:538:33, EOF while parsing a value at line 1 column 0) failed 1 of 1, with the same symptom
This change passed 6 of 6 passed 1 of 1

Checks, in nvx-rs/ with NVX_BUNDLE=skip, without the widened window. On Windows they ran at 3ab9d37df43, on 14ca2671205; on nvx-kvm-test, in a worktree of 14ca2671205 with this patch applied, the same tree. The head then moved only by a rebase onto 012123bf9ac, whose three commits don't touch nvx-rs/tests/cli.rs, nvx-rs/tests/cli/ or the fake, with no change to the patch (git range-diff), and the test passed again there on Windows. The head, 1c8c07e5990, differs from that rebase only by its commit message.

Check Windows 11 workstation nvx-kvm-test
start_opens_the_stable_caller_paths_that_provision_checked, 50 runs in a row 50 of 50 passed 50 of 50 passed
The whole cli suite (cargo test --locked --all-features --test cli), 3 runs 110 passed, 2 ignored, each 119 passed, each
cargo fmt --all --check passed passed
cargo clippy --locked --all-targets --all-features -- -D warnings passed passed
cargo clippy --locked --lib --no-default-features -- -D warnings passed passed
cargo +1.89 check --locked --all-targets --all-features passed not run
python scripts\check_test_layout.py passed not run
python scripts\check_test_changes.py --base origin/feature/nvx-refactor 1 change, above not run

Left to CI: the full Linux and Windows nvx-rs suites at this head, the microVM scenarios, and the gates that a change to one integration test can't affect (artifact bundling, rustdoc, the runtime adapter, nvx-cli and Markdown links).

@ppenna Pedro Henrique Penna (ppenna) added tests-modified Changes existing tests, test data, or test wiring feat-nvx-refactor labels Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 1 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified test fn start_opens_the_stable_caller_paths_that_provision_checked in cli (tests) nvx-rs/tests/cli.rs:1922 nvx-rs/tests/cli.rs:1923

Relocated unchanged: 0; added: 0. Compared the merge base 5b3fab2c45e6 with 08ba44719a1c.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the test-only change to wait for complete CLI argument JSON.

No open issues 🎉

What changed in this PR

This test-only change fixes a flaky CLI state read by waiting for the fake OpenVMM’s complete JSON command-line record rather than merely waiting for the file to exist.

Changes:

  • Uses wait_for_json to obtain the arguments, preserving existing assertions and their order.
  • Clarifies the wait_for documentation to distinguish file existence from readable JSON content.
  • Leaves production code, sleeps, and timeouts unchanged.
File Description
nvx-rs/​tests/​cli.rs Waits for complete argument JSON in the stable caller-path test and clarifies the wait helper’s documentation.

Review effort: Balanced

The Windows job of CI run 43131964028, on #1496, which doesn't touch
this code, failed in the CLI test
start_opens_the_stable_caller_paths_that_provision_checked:

    tests\cli\harness.rs:538:33: ...\build\fake-openvmm-state.json:
    EOF while parsing a value at line 1 column 0

The test starts a sandbox in the background, waits until the fake
OpenVMM's record of its command line, fake-openvmm-state.json, exists,
and then parses it. The fake writes the record with fs::write, which
creates the file before it writes its content, so a test that reads it
in between finds it empty. #1460 added the parse right after that wait,
which until then only told the test that OpenVMM had started.
wait_for_json, which the same file already uses for the fake's record
of its block devices, waits for exactly this.

The test now takes the arguments from wait_for_json, which returns once
the record holds a whole JSON document, and checks the same things in
the same order: the record of the block devices still doesn't exist
then, since the fake opens its images 1.5 s after it starts. wait_for's
comment now sends a test that reads a file's content to wait_for_json.

The fake keeps writing its records with fs::write. Its other readers
run once the command that launched it has returned, or once it has
exited, so after it wrote them. A temporary file and a rename would
spare future readers the wait, but on Windows a rename fails while
another process, such as a scanner, holds the record that it replaces
open without sharing its deletion, and the fake ignores its write
errors, so a second launch of the same sandbox could leave the first
one's record in place.

Evidence, at 14ca267, with a fake that waits 300 ms between
creating the record and writing it: the base test failed with the CI
symptom on a Windows 11 workstation and on nvx-kvm-test (Linux, KVM),
and this change passed on both. Without that wait, this change passed
50 of 50 runs of the test and 3 of 3 runs of the CLI suite on each.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants