Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions doc/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -1322,8 +1322,10 @@ Runs the consolidated suite of the [runtime adapter](../tests/runtime-adapter/RE
the [stage scenarios](../tests/runtime-adapter/README.md#stage-scenarios) that
`tests/runtime-adapter/scenarios.txt` names, against a real hypervisor. Each
line of that file names one of the adapter's test targets, then one of its
tests. The command first checks that the lines name exactly the tests of their
targets, and that neither the adapter's library nor another target has an
tests, optionally followed by `linux` or `windows` to select that host platform.
Unqualified entries run on both. The command first checks that the selected
lines name exactly the tests of their targets, and that neither the adapter's
library nor another target has an
ignored test, as the adapter's VM scenarios are, so that a scenario that is
renamed, added, or gone fails it rather than drop out of the suite. It then
runs each target's scenarios by
Expand All @@ -1333,8 +1335,10 @@ directory that `NVX_ARTIFACTS_DIR` names, from the current directory if it is
relative, or this checkout unless it names one, and the adapter's guest agent
from `--guest-agent` (default
`tests/runtime-adapter/target/x86_64-unknown-linux-musl/release/runtime-adapter-guest`),
and fails unless each of them ran and passed. `--check-only` checks the names
alone, and needs neither a hypervisor nor the guest agent.
and fails unless each of them ran and passed. Execution and counts use the same
platform-selected entries as the inventory check. `--check-only` compiles and
lists the tests but runs no scenarios, and needs neither a hypervisor nor the
guest agent.

### `test-adversarial`

Expand Down
32 changes: 20 additions & 12 deletions scripts/nvx_tools/runtime_adapter_tests.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
"""The runtime adapter's consolidated suite, run by name on a real hypervisor.

`tests/runtime-adapter/scenarios.txt` names the suite's scenarios, each a test of one of
the adapter's test targets. The command checks that the scenarios name exactly the tests
of their targets, and that neither the library nor another target has an ignored test,
as the scenarios that boot VMs are, so that a scenario that is renamed, added, or gone
the adapter's test targets, optionally limited to Linux or Windows. The command checks
that the host's scenarios name exactly the tests of their targets, and that neither the
library nor another target has an ignored test, as the scenarios that boot VMs are, so
that a scenario that is renamed, added, or gone
fails the command rather than drop out of it. It then runs each target's scenarios by
name, the ignored ones with the others, and checks that each of them ran and passed.
"""
Expand Down Expand Up @@ -77,25 +78,32 @@ def configure_parser(parser: argparse.ArgumentParser) -> None:


def read_scenarios(path: Path = SCENARIOS) -> list[Target]:
"""Returns the targets that the scenarios of `path` name, in the order of their
first lines, each with its tests in the order of their lines.
"""Returns the host's scenarios of `path`, grouped by target in the order of their
first selected lines, each with its tests in the order of their lines.

A line names a target, then one of its tests; `#` starts a comment. Raises
`ScriptError` for another line, and a scenario that two lines name.
A line names a target, then one of its tests, optionally followed by `linux` or
`windows`; `#` starts a comment. Raises `ScriptError` for another line, an unknown
platform, and a scenario that two lines name, even on different platforms.
"""
host_platform = "windows" if sys.platform == "win32" else sys.platform
tests: dict[str, list[str]] = {}
seen: set[tuple[str, str]] = set()
for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
fields = line.split("#", 1)[0].split()
if not fields:
continue
where = f"{path}:{number}"
if len(fields) != 2:
if len(fields) not in (2, 3):
raise ScriptError(f"{where}: expected a target and a test, not {line!r}")
target, test = fields
named = tests.setdefault(target, [])
if test in named:
target, test = fields[:2]
platform = fields[2] if len(fields) == 3 else None
if platform not in (None, "linux", "windows"):
raise ScriptError(f"{where}: unknown scenario platform {platform!r}")
if (target, test) in seen:
raise ScriptError(f"{where}: {target} {test} is named twice")
named.append(test)
seen.add((target, test))
if platform is None or platform == host_platform:
tests.setdefault(target, []).append(test)
if not tests:
raise ScriptError(f"{path} names no scenario")
return [Target(name, tuple(named)) for name, named in tests.items()]
Expand Down
144 changes: 131 additions & 13 deletions scripts/test_nvx_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -21038,30 +21038,86 @@ def test_scenarios_name_targets_and_tests_in_the_order_of_their_lines(self):
def test_scenarios_refuse_other_lines(self):
for text, problem in (
("vm_a\n", r":1: expected a target and a test, not 'vm_a'"),
("unit x\nvm_a a b\n", r":2: expected a target and a test"),
("unit x\nvm_a a linux extra\n", r":2: expected a target and a test"),
("unit x\nvm_a a b\n", r":2: unknown scenario platform 'b'"),
("vm_a a Linux\n", r":1: unknown scenario platform 'Linux'"),
("vm_a a\nvm_a a\n", r":2: vm_a a is named twice"),
("vm_a a linux\nvm_a a windows\n", r":2: vm_a a is named twice"),
("vm_a a linux\nvm_a a\n", r":2: vm_a a is named twice"),
("vm_a a windows\nvm_a a windows\n", r":2: vm_a a is named twice"),
("# Nothing yet.\n", r"names no scenario"),
):
for platform in ("linux", "win32"):
with (
self.subTest(text=text, platform=platform),
patch.object(sys, "platform", platform),
self.assertRaisesRegex(common.ScriptError, problem),
):
runtime_adapter_tests.read_scenarios(self._scenarios(text))

def test_scenarios_select_the_host_platform_in_line_order(self):
path = self._scenarios(
"vm_a a windows\nunit x\nvm_a b linux # Linux only\nvm_a common\n"
)
for platform, expected in (
(
"linux",
[
runtime_adapter_tests.Target("unit", ("x",)),
runtime_adapter_tests.Target("vm_a", ("b", "common")),
],
),
(
"win32",
[
runtime_adapter_tests.Target("vm_a", ("a", "common")),
runtime_adapter_tests.Target("unit", ("x",)),
],
),
):
with (
self.subTest(text=text),
self.assertRaisesRegex(common.ScriptError, problem),
self.subTest(platform=platform),
patch.object(sys, "platform", platform),
):
runtime_adapter_tests.read_scenarios(self._scenarios(text))
self.assertEqual(runtime_adapter_tests.read_scenarios(path), expected)

with (
patch.object(sys, "platform", "win32"),
self.assertRaisesRegex(common.ScriptError, "names no scenario"),
):
runtime_adapter_tests.read_scenarios(self._scenarios("vm_a a linux\n"))

def test_the_adapter_s_readme_lists_its_scenarios(self):
targets = runtime_adapter_tests.read_scenarios()
named = [(target.name, test) for target in targets for test in target.tests]
self.assertLessEqual(
{"vm_clone", "vm_template", "vm_layered", "template_operations"},
{target.name for target in targets},
)
readme = (runtime_adapter_tests.ADAPTER_DIRECTORY / "README.md").read_text(
encoding="utf-8"
)
section = re.split(r"\n#+ ", readme.split("\n### Stage scenarios\n", 1)[1])[0]
rows = re.findall(r"^\| `(\w+)` \| `(\w+)` \|", section, re.MULTILINE)

self.assertEqual(rows, named)
rows = re.findall(
r"^\| `(\w+)` \| `([^`]+)`(?: \((linux|windows)\))? \|",
section,
re.MULTILINE,
)
for platform, host in (("linux", "linux"), ("win32", "windows")):
with (
self.subTest(platform=platform),
patch.object(sys, "platform", platform),
):
targets = runtime_adapter_tests.read_scenarios()
named = [
(target.name, test) for target in targets for test in target.tests
]
self.assertLessEqual(
{"vm_clone", "vm_template", "vm_layered", "template_operations"},
{target.name for target in targets},
)
self.assertEqual(
[
(target, test)
for target, test, on in rows
if not on or on == host
],
named,
)

def test_libtest_s_terse_list_names_the_tests(self):
self.assertEqual(
Expand Down Expand Up @@ -21189,6 +21245,68 @@ def test_the_handler_takes_a_relative_directory_of_artifacts_from_the_caller(sel
self.assertEqual(status, 0, stderr)
self.assertEqual(Path(runs[0][1]["NVX_ARTIFACTS_DIR"]), expected)

def test_the_handler_checks_and_runs_the_same_platform_scenarios(self):
path = self._scenarios(
"vm_a common\nvm_a linux_case linux\nunit windows_case windows\n"
)
for platform, selected in (
("linux", [("vm_a", ("common", "linux_case"))]),
("win32", [("vm_a", ("common",)), ("unit", ("windows_case",))]),
):
listings = {
(target, "--list"): "".join(f"{test}: test\n" for test in tests)
for target, tests in selected
}
listings.update(
((target, "--ignored"), "") for target in ("lib", "unit", "other")
)
for check_only in (False, True):
with (
self.subTest(platform=platform, check_only=check_only),
patch.object(sys, "platform", platform),
):
status, runs, stdout, stderr = self._run(
path, listings, check_only=check_only
)
self.assertEqual(status, 0, stderr)
expected = [
runtime_adapter_tests.cargo_test(
"cargo", target, "--include-ignored", "--exact", *tests
)
for target, tests in selected
]
self.assertEqual(
[command for command, _ in runs], [] if check_only else expected
)
if check_only:
self.assertIn(
f"2 scenarios name exactly the tests of {len(selected)} targets",
stdout,
)
else:
self.assertIn(
f"the 2 scenarios of {len(selected)} targets passed", stdout
)

def test_platform_selection_keeps_unlisted_and_ignored_test_errors(self):
path = self._scenarios(
"vm_a common\nvm_a linux_case linux\nother hidden linux\n"
)
listings = {
("vm_a", "--list"): "common: test\nlinux_case: test\n",
("lib", "--ignored"): "",
("unit", "--ignored"): "",
("other", "--ignored"): "hidden: test\n",
}
with patch.object(sys, "platform", "win32"):
status, runs, _, stderr = self._run(path, listings)

self.assertEqual((status, runs), (1, []))
self.assertIn("vm_a has the test linux_case, which no scenario names", stderr)
self.assertIn(
"other has the ignored test hidden, which no scenario names", stderr
)

def test_the_handler_runs_nothing_once_the_names_are_not_the_tests(self):
path = self._scenarios("vm_a a\nvm_a gone\ngone x\n")
listings = {
Expand Down
12 changes: 9 additions & 3 deletions tests/runtime-adapter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3050,16 +3050,21 @@ the template, whose VM might still run.

The template and clone stage's scenarios run as one suite, which
[`scenarios.txt`](scenarios.txt) names: each line names a test target of
`tests/`, then one of its tests. CI's runtime adapter step runs the suite on
each backend, KVM, MSHV, and WHP, with
`tests/`, then one of its tests, optionally followed by `linux` or `windows`.
Unqualified entries run on both platforms; qualified entries run only on the
named host platform. Duplicate target/test pairs, even with different
qualifiers, and unknown qualifiers are errors. CI's runtime adapter step runs
the suite on each backend, KVM, MSHV, and WHP, with
`python3 scripts/nvx.py test-runtime-adapter --backend BACKEND`, in the
host-only configuration: it runs each target's scenarios by name, the VM
scenarios, which are ignored, with the others, and checks that each of them ran
and passed. It fails once a line names a test that its target does not have, a
target that a line names has a test that no line names, the library or another
target has an ignored test, or a named test does not run and pass, so that a
scenario that is renamed, added, or gone fails it rather than drop out of the
suite. With `--check-only`, it checks the names alone, and runs nothing.
suite. Inventory checks, execution and scenario counts all use the same
platform-selected entries. With `--check-only`, it compiles and lists the
tests to check the names, but runs no scenarios and needs no guest artifacts.

The suite holds each operation's outcomes, from `create` to `run`, without a
hypervisor and in VMs, and every scenario of the stage's VMs: launches of
Expand Down Expand Up @@ -3222,6 +3227,7 @@ layered roots, captures of templates, and the starts and runs of their clones.
| `vm_clone` | `one_generation_serves_clones_of_either_format_over_either_upper` | A template of a GPT placeholder serves EROFS and GPT clones of other layouts, shorter than the placeholder, at its length, over either upper, those in RAM with no scratch file, and a first partition over its bound ends a clone with status 54 |
| `vm_clone` | `one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address` | One template with a NIC serves the clones of every network policy at its guest's address, whose restores install their own policies on its NIC, which lets through what each allows, publishes the guest's listeners, through the clone's announcer, and forwards ports where it says, and NVX's plan refuses another address, or no NIC, with status 8 before any VMM starts |
| `vm_clone` | `openvmm_reads_none_of_the_host_s_input_until_the_repair_acknowledges_the_restore` | OpenVMM reads none of the host's attachment until the repair of a clone, whose template delays it, acknowledges the restore: the broker answers it only after the delay, and on Linux, where the host writes it a byte at a time, `SIOCOUTQ` rises with each byte and stays as the write left it until then, while a reattach's is answered at once |
| `vm_clone` | `restore_exclusion::hard_linked_restore_views_obey_the_callers_exclusion` (linux) | Caller-locked concurrent hard-linked restores succeed, an unlocked link change during startup fails the restore, a controller killed before ready prevents completion, and link changes after the peer-verified ready connection do not fail it |
| `vm_clone` | `the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store` | The lease of a start that ended before it recorded its clone is released only through its own template store, which `--template` names |

### Running the VM tests
Expand Down
4 changes: 3 additions & 1 deletion tests/runtime-adapter/scenarios.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# The runtime adapter's consolidated suite: every scenario of the template and clone stage, which
# CI's runtime adapter step runs by name on each backend, with
# `python3 scripts/nvx.py test-runtime-adapter`. Each line names a test target of `tests/`, then
# one of its tests, which the step runs whether it is ignored or not. The step fails once a line
# one of its tests, optionally followed by linux or windows to select only that host platform.
# Unqualified lines run on both platforms, whether ignored or not. The step fails once a line
# names a test that its target does not have, a target that a line names has a test that no line
# names, the library or a target that no line names has an ignored test, or a named test does not
# run and pass. The README's "Stage scenarios" says what each scenario proves.
Expand Down Expand Up @@ -171,4 +172,5 @@ vm_clone one_generation_serves_clones_of_each_upper_and_memory_target
vm_clone one_generation_serves_clones_of_either_format_over_either_upper
vm_clone one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address
vm_clone openvmm_reads_none_of_the_host_s_input_until_the_repair_acknowledges_the_restore
vm_clone restore_exclusion::hard_linked_restore_views_obey_the_callers_exclusion linux
vm_clone the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store
Loading
Loading