Repository navigation
test-microvm: have the control console interrupt with input (port of #1410) - #1527
Merged
Pedro Henrique Penna (ppenna) merged 1 commit intoNov 12, 2026
Merged
Conversation
Contributor
This pull request changes existing testsIt modifies or removes 2 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests. End-to-end tests
Relocated unchanged: 0; added: 3. Compared the merge base |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the control-console interrupt-check update.
No open issues 🎉
What changed in this PR
Ports #1410’s interrupt-map fix to feature/nvx-refactor, addressing the flaky control-console interrupt check described in #1400 without changing OpenVMM or NVX’s public API.
Changes:
- Replaces repeated control-console writes with a bootstrap
GuestAttachrequest, prompting broker input while holding the terminal open, raw, and silent. - Centralizes script rendering and adds a regression test that checks the request bytes against OpenVMM’s protocol vector.
- Documents the revised traffic while preserving the existing per-device interrupt checks.
| File | Description |
|---|---|
scripts/test_microvm_tests.py |
Adds regression coverage for handshake bytes and terminal setup. |
scripts/nvx_tools/microvm_tests.py |
Defines the bootstrap record and renders it into both script variants. |
scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in |
Requests broker input to trigger the control-console interrupt. |
doc/ci.md |
Explains the output race and revised control-console traffic. |
Review effort: Balanced
The interrupt-map scenario's control console sometimes never raised its interrupt: the guest wrote 16 one-byte buffers to /dev/hvc2, IRQ 23 never fired, and OpenVMM exited with status 37 after NVX-INTERRUPT-MAP-QUIET. It failed in about one run in ten on GitHub-hosted runners, on KVM and WHP and on AMD and Intel CPUs, and in 3 merge groups on NVX CI's Intel MSHV runners. No interrupt was lost: the device never sent one. The console's driver waits for each write by polling the used ring of the console's output, takes the buffer back as soon as the device returns it, and moves its used event past the buffer. With VIRTIO_F_EVENT_IDX, the device notifies the driver of a buffer only if the used event still names the buffer when the device reads the event after publishing it, as virtio 1.2 (2.7.10) requires, so a write interrupts only when the device reads the event first. Runs of an instrumented OpenVMM recorded the event that the device read at each completion: at every skipped notification of either console, the driver had already moved the event to the next buffer. How often the driver wins this race varies by boot. In 320 boots on AMD runners under KVM, the control console's first write interrupted in 205, while 22 boots used up the scenario's 16 writes (5 of 140 on EPYC 9V74 runners, 17 of 60 on EPYC 9V45, and none of 120 on EPYC 7763), 2 of them with none of 43 writes interrupting. The boot console's driver won at most 15% of its races, so its 16 writes still suffice. So the control console now interrupts with input, which nothing waits for: the guest sends the console's control-session broker the bootstrap GuestAttach record, and the device writes the broker's answer, a Reset record, into a buffer of the console's input, which the driver takes back from its interrupt. The guest holds the console open, raw, and silent until the interrupt, so that the terminal sends the record unchanged and echoes no answer back, at which the broker would stop as at any malformed record, never completing the echo's write. The scenario's check is unchanged: IRQ 23 must fire with the console's own traffic. On GitHub-hosted runners, with the release of 7714de1 and each job alternating the two scenarios, the previous one failed in 64 of 640 runs (31 of 320 on KVM, 33 of 320 on WHP) and this one in 1 of 640, a WHP run on an Intel Xeon 8573C runner that left no evidence of its cause. With an instrumented OpenVMM, this one passed 960 runs on KVM and 960 on WHP. A unit test checks that the script writes OpenVMM's GUEST_ATTACH vector to /dev/hvc2, once, held open, raw, and silent; it fails with the previous script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 6497568)
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/slim-port-1410
branch
from
November 12, 2026 08:14
4486e51 to
f3f2cef
Compare
Pedro Henrique Penna (ppenna)
marked this pull request as ready for review
November 12, 2026 08:14
Pedro Henrique Penna (ppenna)
merged commit Nov 12, 2026
d58c687
into
feature/nvx-refactor
81 of 84 checks passed
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1076 (port-1410: the port of #1410 to
feature/nvx-refactor). Refs #1400.What and why
feature/nvx-refactorforked fromdevat53ad09678bcon 2026-11-06, three days before #1410 fixed the flakyinterrupt-mapscenario of #1400, so every slice's CI still risks that flake. On this branch, the scenario failed withNVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0, and OpenVMM exited with status 37, inNVX microVM tests / Linux / MSHV / Shard 1 of 2:azure-mshv-3;29fca59317f, on runnerazure-mshv-5.Both runners have an Intel Xeon Platinum 8573C.
This PR holds one commit:
dev's64975689e14, the whole of #1410, cherry-picked withgit cherry-pick -x, which keeps its author and message and records its origin. It applied without conflicts. The guest script,interrupt-map.sh.in, hasn't changed on either branch since the fork, and this branch's changes to the other three files, by SL5.1b, SL5.1c, SL5.2, SL3.10c and earlier slices, don't reach #1410's hunks. The patch is #1410's, unchanged: both commits have the samegit patch-id --stable.What #1410 found and changed, in short (its description has the evidence):
VIRTIO_F_EVENT_IDX, the device notifies the driver only if the used event still names the buffer when the device reads it after publishing the buffer (virtio 1.2, 2.7.10). So a write interrupts only when the device reads the event first, a race that the driver can win at every write, and the scenario's 16 one-byte writes to the control console could all lose it.GuestAttachrecord, OpenVMM'sGUEST_ATTACHvector, with/dev/hvc2held open, raw and silent. The device then writes the broker's answer, aResetrecord, into a buffer of the console's input. The check is unchanged: IRQ 23 must fire with the console's own traffic.run_interrupt_maprenders the guest script through the new_interrupt_map_script, which substitutes the record's bytes into the script'sprintfas octal escapes, anddoc/ci.mddescribes the new traffic.Nothing else from
devis needed.14dc4116ec7, wheredevpinned881619911d1when test-microvm: have the control console interrupt with input #1410 merged. At14dc4116ec7, the broker already answers aGuestAttachwith aResetin its first state (vm/devices/virtio/virtio_console/src/control_session_broker.rs, which differs between the two pins only in one test's formatting).--microvm-control-console none, which the scenario passes, still runs a broker, with a random capability (openvmm/openvmm_entry/src/microvm/console.rs).nvx-rs/tests/vectors/openvmm/control_session_protocol_vectors.txt, which this branch has unchanged since the fork. The guest'sstty raw -echoalready servesconsole-snapshot.Case matrix.
interrupt-mapstays kept indoc/acceptance-case-matrix.md: it runs as before, without the adapter, and checks the same interrupts. Only the control console's traffic changes. Nothing else in the matrix names what this PR changes, and SL5.2's matrix check passes.Size. 14 core lines in 2 counted files, if the scenario's harness and guest script count as code rather than tests:
scripts/nvx_tools/microvm_tests.py12 (the record,_interrupt_map_scriptand its two calls) andinterrupt-map.sh.in2 (the new step); otherwise 0. The unit test anddoc/ci.mddon't count.nvx-rs, its public API,BOOT_CONTRACT_VERSIONandCONTROL_CONTRACT_REVISIONdon't change.Validation
Real VMs. Each host ran the scenario (
nvx.py test-microvm --backend <backend> --scenario interrupt-map) in temporary worktrees, alternating this head,4486e511cf5, and the base,c642f09e5ed. Every run used the artifacts of this branch's push run 43172681016 at the base, which passed: the kernel, the Alpine initramfs, OpenVMM14dc4116ec7(openvmm-linux-gnuandopenvmm-windows-msvc) andnvx-rs, each checked by SHA-256 on the host, withNVX_AUTO_DOWNLOAD=0.In all 175 runs of this head, IRQ 23 counted 0 after the boot and 1 after the traffic: the broker's answer, the one interrupt that the new step waits for.
The base passed on these hosts too, but the race is there. In 100 more runs of the base on nvx-mshv-test, with its guest script instrumented in a scratch worktree to report how many one-byte writes the control console needed before IRQ 23 fired, the first write interrupted in 92 boots. The other 8 needed 2 (3 boots), 3, 5, 6, 7 and 9 writes. The old step fails when the driver wins all 16 races of one boot, as it did on the CI runners above.
Local, on Windows, from this worktree's action definitions, for the gates that
local_gates.pyselects (test-changes,markdown-links,test-layout,quality-code,nvx-cli):python -m ruff check scripts .github/speculaandpython -m ruff format --check scripts .github/specula(0.15.22): passed.python -m pyright --pythonplatform Linuxand--pythonplatform Windows(1.1.411): 0 errors, 0 warnings.python scripts/check_markdown_links.pyandpython scripts/check_test_layout.py: passed.validate-nvx's Windows step, without itspip install:python -m unittestof its 22 files ran 1,825 tests, OK (73 skipped).python scripts\test_adversarial.py -vhad 1 error intest_contained_guest_runner_timeout_kills_descendants, aPermissionErrorreading the test's temporarychild.pid; this change touches nothing that the test runs, and 3 reruns passed.test_hosts.pyand the--helpchecks passed.test_interrupt_map_attaches_the_control_console_to_its_broker, passes, and fails with this branch's previous guest script (AssertionError: 0 != 1).check_matrix.py <worktree> 99d2022085f) passed.Linux, on nvx-kvm-test (Python 3.12.16), in the same worktree:
checkandformat --checkpassed.--pythonplatform Linuxand--pythonplatform Windows, 0 errors, 0 warnings.check_markdown_links.py,check_test_layout.py: passed.validate-nvx's Linux step, without itspip install: the 22-fileunittestrun (1,825 tests, OK, 40 skipped),test_adversarial.py(84 tests, OK),test_hosts.pyand the--helpchecks passed.Left to CI: shellcheck and shfmt, whose file lists don't include this change's files, and the microVM shards on the CI runners.
python scripts/check_test_changes.py --base origin/feature/nvx-refactorlists the two changes to existing tests that #1410 made, hencetests-modified:scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in, the scenario's guest script: the control console's traffic step sends theGuestAttachrecord, with/dev/hvc2held open, raw and silent, instead of up to 16 one-byte writes, for the reasons above. The comments on the traffic and on the boot console's writes say why. The checks and their failure codes don't change.run_interrupt_mapinscripts/nvx_tools/microvm_tests.py: it renders the script through_interrupt_map_script, which also substitutes the record's bytes, as octal escapes, into the script'sprintf. The cold boot and the snapshot's capture render the same template as before.The
openvmmgitlink doesn't move.