Skip to content

test-microvm: have the control console interrupt with input (port of #1410) - #1527

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-port-1410
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/nvx-refactorfrom
agents/slim-port-1410

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Part of #1076 (port-1410: the port of #1410 to feature/nvx-refactor). Refs #1400.

What and why

feature/nvx-refactor forked from dev at 53ad09678bc on 2026-11-06, three days before #1410 fixed the flaky interrupt-map scenario of #1400, so every slice's CI still risks that flake. On this branch, the scenario failed with NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0, and OpenVMM exited with status 37, in NVX microVM tests / Linux / MSHV / Shard 1 of 2:

Both runners have an Intel Xeon Platinum 8573C.

This PR holds one commit: dev's 64975689e14, the whole of #1410, cherry-picked with git cherry-pick -x, which keeps its author and message and records its origin. It applied without conflicts. The guest script, interrupt-map.sh.in, hasn't changed on either branch since the fork, and this branch's changes to the other three files, by SL5.1b, SL5.1c, SL5.2, SL3.10c and earlier slices, don't reach #1410's hunks. The patch is #1410's, unchanged: both commits have the same git patch-id --stable.

What #1410 found and changed, in short (its description has the evidence):

  • Root cause. No interrupt is lost: the device never sends one. Linux's console driver waits for each write by polling the used ring of the console's output, and takes the buffer back as soon as the device returns it, moving its used event past the buffer. With VIRTIO_F_EVENT_IDX, the device notifies the driver only if the used event still names the buffer when the device reads it after publishing the buffer (virtio 1.2, 2.7.10). So a write interrupts only when the device reads the event first, a race that the driver can win at every write, and the scenario's 16 one-byte writes to the control console could all lose it.
  • Fix. The control console now interrupts with input, which nothing polls. The guest sends the console's control-session broker the bootstrap GuestAttach record, OpenVMM's GUEST_ATTACH vector, with /dev/hvc2 held open, raw and silent. The device then writes the broker's answer, a Reset record, into a buffer of the console's input. The check is unchanged: IRQ 23 must fire with the console's own traffic. run_interrupt_map renders the guest script through the new _interrupt_map_script, which substitutes the record's bytes into the script's printf as octal escapes, and doc/ci.md describes the new traffic.

Nothing else from dev is needed.

  • OpenVMM: this branch pins 14dc4116ec7, where dev pinned 881619911d1 when test-microvm: have the control console interrupt with input #1410 merged. At 14dc4116ec7, the broker already answers a GuestAttach with a Reset in its first state (vm/devices/virtio/virtio_console/src/control_session_broker.rs, which differs between the two pins only in one test's formatting). --microvm-control-console none, which the scenario passes, still runs a broker, with a random capability (openvmm/openvmm_entry/src/microvm/console.rs).
  • The unit test reads nvx-rs/tests/vectors/openvmm/control_session_protocol_vectors.txt, which this branch has unchanged since the fork. The guest's stty raw -echo already serves console-snapshot.

Case matrix. interrupt-map stays kept in doc/acceptance-case-matrix.md: it runs as before, without the adapter, and checks the same interrupts. Only the control console's traffic changes. Nothing else in the matrix names what this PR changes, and SL5.2's matrix check passes.

Size. 14 core lines in 2 counted files, if the scenario's harness and guest script count as code rather than tests: scripts/nvx_tools/microvm_tests.py 12 (the record, _interrupt_map_script and its two calls) and interrupt-map.sh.in 2 (the new step); otherwise 0. The unit test and doc/ci.md don't count. nvx-rs, its public API, BOOT_CONTRACT_VERSION and CONTROL_CONTRACT_REVISION don't change.

Validation

Real VMs. Each host ran the scenario (nvx.py test-microvm --backend <backend> --scenario interrupt-map) in temporary worktrees, alternating this head, 4486e511cf5, and the base, c642f09e5ed. Every run used the artifacts of this branch's push run 43172681016 at the base, which passed: the kernel, the Alpine initramfs, OpenVMM 14dc4116ec7 (openvmm-linux-gnu and openvmm-windows-msvc) and nvx-rs, each checked by SHA-256 on the host, with NVX_AUTO_DOWNLOAD=0.

Profile Backend Host type CPU This head Base
nvx-mshv-test MSHV, Linux 6.18.34.mshv3 virtual machine Intel Xeon Platinum 8573C 100 of 100 pass 100 of 100 pass
nvx-kvm-test KVM, Linux 6.6.159 virtual machine Intel Xeon Platinum 8573C 50 of 50 pass 50 of 50 pass
nvx-whp-test WHP, Windows Server 2025 virtual machine Intel Xeon Platinum 8370C 25 of 25 pass 25 of 25 pass

In all 175 runs of this head, IRQ 23 counted 0 after the boot and 1 after the traffic: the broker's answer, the one interrupt that the new step waits for.

The base passed on these hosts too, but the race is there. In 100 more runs of the base on nvx-mshv-test, with its guest script instrumented in a scratch worktree to report how many one-byte writes the control console needed before IRQ 23 fired, the first write interrupted in 92 boots. The other 8 needed 2 (3 boots), 3, 5, 6, 7 and 9 writes. The old step fails when the driver wins all 16 races of one boot, as it did on the CI runners above.

Local, on Windows, from this worktree's action definitions, for the gates that local_gates.py selects (test-changes, markdown-links, test-layout, quality-code, nvx-cli):

  • python -m ruff check scripts .github/specula and python -m ruff format --check scripts .github/specula (0.15.22): passed.
  • python -m pyright --pythonplatform Linux and --pythonplatform Windows (1.1.411): 0 errors, 0 warnings.
  • python scripts/check_markdown_links.py and python scripts/check_test_layout.py: passed.
  • validate-nvx's Windows step, without its pip install: python -m unittest of its 22 files ran 1,825 tests, OK (73 skipped). python scripts\test_adversarial.py -v had 1 error in test_contained_guest_runner_timeout_kills_descendants, a PermissionError reading the test's temporary child.pid; this change touches nothing that the test runs, and 3 reruns passed. test_hosts.py and the --help checks passed.
  • The new test, test_interrupt_map_attaches_the_control_console_to_its_broker, passes, and fails with this branch's previous guest script (AssertionError: 0 != 1).
  • SL5.2's case matrix check (check_matrix.py <worktree> 99d2022085f) passed.

Linux, on nvx-kvm-test (Python 3.12.16), in the same worktree:

  • ruff 0.15.22, from PyPI's manylinux wheel (SHA-256 checked against PyPI): check and format --check passed.
  • pyright 1.1.411 on Node.js 22.23.5, both checked by digest and unpacked in a scratch directory: --pythonplatform Linux and --pythonplatform Windows, 0 errors, 0 warnings.
  • check_markdown_links.py, check_test_layout.py: passed.
  • validate-nvx's Linux step, without its pip install: the 22-file unittest run (1,825 tests, OK, 40 skipped), test_adversarial.py (84 tests, OK), test_hosts.py and the --help checks passed.

Left to CI: shellcheck and shfmt, whose file lists don't include this change's files, and the microVM shards on the CI runners.

python scripts/check_test_changes.py --base origin/feature/nvx-refactor lists the two changes to existing tests that #1410 made, hence tests-modified:

  • scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in, the scenario's guest script: the control console's traffic step sends the GuestAttach record, with /dev/hvc2 held open, raw and silent, instead of up to 16 one-byte writes, for the reasons above. The comments on the traffic and on the boot console's writes say why. The checks and their failure codes don't change.
  • run_interrupt_map in scripts/nvx_tools/microvm_tests.py: it renders the script through _interrupt_map_script, which also substitutes the record's bytes, as octal escapes, into the script's printf. The cold boot and the snapshot's capture render the same template as before.

The openvmm gitlink doesn't move.

@ppenna Pedro Henrique Penna (ppenna) added tests-modified Changes existing tests, test data, or test wiring feat-nvx-refactor labels Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 2 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

End-to-end tests

Change What Before After
modified file scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in
modified helper def run_interrupt_map in scripts/nvx_tools/microvm_tests.py scripts/nvx_tools/microvm_tests.py:7125 scripts/nvx_tools/microvm_tests.py:7147

Relocated unchanged: 0; added: 3. Compared the merge base baf560d104f6 with f3f2cef27ba0.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the control-console interrupt-check update.

No open issues 🎉

What changed in this PR

Ports #1410’s interrupt-map fix to feature/nvx-refactor, addressing the flaky control-console interrupt check described in #1400 without changing OpenVMM or NVX’s public API.

Changes:

  • Replaces repeated control-console writes with a bootstrap GuestAttach request, prompting broker input while holding the terminal open, raw, and silent.
  • Centralizes script rendering and adds a regression test that checks the request bytes against OpenVMM’s protocol vector.
  • Documents the revised traffic while preserving the existing per-device interrupt checks.
File Description
scripts/​test_microvm_tests.py Adds regression coverage for handshake bytes and terminal setup.
scripts/​nvx_tools/​microvm_tests.py Defines the bootstrap record and renders it into both script variants.
scripts/​nvx_tools/​microvm_test_scripts/​interrupt-map.sh.in Requests broker input to trigger the control-console interrupt.
doc/​ci.md Explains the output race and revised control-console traffic.

Review effort: Balanced

The interrupt-map scenario's control console sometimes never raised its
interrupt: the guest wrote 16 one-byte buffers to /dev/hvc2, IRQ 23
never fired, and OpenVMM exited with status 37 after
NVX-INTERRUPT-MAP-QUIET. It failed in about one run in ten on
GitHub-hosted runners, on KVM and WHP and on AMD and Intel CPUs, and in
3 merge groups on NVX CI's Intel MSHV runners.

No interrupt was lost: the device never sent one. The console's driver
waits for each write by polling the used ring of the console's output,
takes the buffer back as soon as the device returns it, and moves its
used event past the buffer. With VIRTIO_F_EVENT_IDX, the device
notifies the driver of a buffer only if the used event still names the
buffer when the device reads the event after publishing it, as virtio
1.2 (2.7.10) requires, so a write interrupts only when the device reads
the event first. Runs of an instrumented OpenVMM recorded the event
that the device read at each completion: at every skipped notification
of either console, the driver had already moved the event to the next
buffer. How often the driver wins this race varies by boot. In 320
boots on AMD runners under KVM, the control console's first write
interrupted in 205, while 22 boots used up the scenario's 16 writes (5
of 140 on EPYC 9V74 runners, 17 of 60 on EPYC 9V45, and none of 120 on
EPYC 7763), 2 of them with none of 43 writes interrupting. The boot
console's driver won at most 15% of its races, so its 16 writes still
suffice.

So the control console now interrupts with input, which nothing waits
for: the guest sends the console's control-session broker the bootstrap
GuestAttach record, and the device writes the broker's answer, a Reset
record, into a buffer of the console's input, which the driver takes
back from its interrupt. The guest holds the console open, raw, and
silent until the interrupt, so that the terminal sends the record
unchanged and echoes no answer back, at which the broker would stop as
at any malformed record, never completing the echo's write. The
scenario's check is unchanged: IRQ 23 must fire with the console's own
traffic.

On GitHub-hosted runners, with the release of 7714de1 and each job
alternating the two scenarios, the previous one failed in 64 of 640
runs (31 of 320 on KVM, 33 of 320 on WHP) and this one in 1 of 640, a
WHP run on an Intel Xeon 8573C runner that left no evidence of its
cause. With an instrumented OpenVMM, this one passed 960 runs on KVM
and 960 on WHP.

A unit test checks that the script writes OpenVMM's GUEST_ATTACH vector
to /dev/hvc2, once, held open, raw, and silent; it fails with the
previous script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 6497568)
@ppenna
Pedro Henrique Penna (ppenna) marked this pull request as ready for review November 12, 2026 08:14
@ppenna
Pedro Henrique Penna (ppenna) merged commit d58c687 into feature/nvx-refactor Nov 12, 2026
81 of 84 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/slim-port-1410 branch November 12, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants