Introduce conformance test CLI - #346
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
d07c9e3 to
9482ae7
Compare
jserv
left a comment
jserv
left a comment
There was a problem hiding this comment.
Introduce re-usable shell scripts as common library.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
9482ae7 to
40f41ac
Compare
Extracted the SSH part out. |
417be2a to
f303d4b
Compare
jserv
left a comment
jserv
left a comment
There was a problem hiding this comment.
Rebase latest main branch and resolve conflicts.
| run: | | ||
| [ "$DISCOVER" = success ] | ||
| [ "$HARNESS" = success ] | ||
| if [ "$SUITES" = '[]' ]; then |
There was a problem hiding this comment.
The registry is empty on this branch, so suites returns [], the payload/qemu/elfuse jobs all skip, and this branch asserts that skip as success. The required check is green while nothing runs, and it stays green if the last provider is ever dropped from the registry.
Once the first suite lands this needs a floor: assert the discovered list is non-empty, or compare it against a checked-in expected inventory.
| out: Dict[str, List[str]] = {} | ||
| for (suite, group), members in sorted(by_group.items()): | ||
| actions = {wanted.get(m) for m in members} | ||
| if whole_groups and len(actions) == 1 and None not in actions and len(members) > 1: |
There was a problem hiding this comment.
whole_groups is set for any run whose scope is not cases, so a --scope pr run collapses to suite:group/* even when it covered only part of the group (a pr subset, or an entry with only filters). Two failing cases out of a larger group seed s:g/*.
The untouched siblings then inherit expect_failure, and the next full run reports them as unexpected_pass. Collapse only when the run actually covered the whole group.
| ``` | ||
|
|
||
| Selectors and expectation matchers use shell globs across the complete ID. | ||
| A bare group selector also selects its cases. An unmatched selector is an |
There was a problem hiding this comment.
True for --case selectors, which selection.resolve_ids expands to pattern plus pattern/*, but not for expectation matchers: expectations.resolve calls fnmatch directly, and ids.matches("s:g", "s:g/c") is False.
So a matcher naming a bare group silently covers nothing, and only a full-scope run's stale() reports it. Either split this sentence per consumer, or give resolve the same expansion.
| if lost: | ||
| return Invocation(execution="transport", wall_us=inv.wall_us, | ||
| stdout=str(out_path), stderr=inv.stderr) | ||
| timed_out = rc == 137 and inv.wall_us >= timeout_s * 1_000_000 |
There was a problem hiding this comment.
The tiebreaker compares the guest's exit status against the host's total ssh wall time, which also covers connect, auth and teardown, so it can pass timeout_s while the guest command itself ran for a fraction of that. A case killed by SIGKILL for an unrelated reason (guest OOM, the suite killing its own child) then records TIMEOUT instead of CRASH.
Both statuses are red and both seed skip, so the cost is the recorded status rather than the verdict. An explicit marker printed by the remote script when timeout fires would make it exact.
| try: | ||
| # The lock lives in a shared namespace, so another uid may own it. | ||
| self.lock_file.parent.mkdir(parents=True, exist_ok=True) | ||
| fd = os.open(self.lock_file, os.O_RDWR | os.O_CREAT, 0o600) |
There was a problem hiding this comment.
ElfuseBackend.lock_file is /tmp/elfuse-conformance-<uid>.lock, a predictable name in a shared directory, and this open follows symlinks, so another local user can pre-create it as a symlink and redirect the flock.
os.O_NOFOLLOW closes it. The comment just above already notes the namespace is shared.
qemu-runner.sh and test-matrix.sh each spelled the ssh option list for the test VM, and the copies had drifted: the runner kept the peer alive for 10 s x 6 and the matrix for 15 s x 4. tests/lib/qemu-ssh.sh now holds the one list. qemu_ssh_opts fills QEMU_SSH_OPTS at call time so a caller can still wrap its ssh in timeout(1), which cannot wrap a shell function. Both lanes keep the 60 s dead-peer budget.
The parser owns one capability-first command tree used by Make and CI. Suite registration cannot add commands. run writes results.json before the backend stops, so a failing teardown cannot lose a completed lane, and records the argv it parsed. report is red for a red lane in every format. Exit 77 is for a prerequisite reported absent before anything starts; a backend that fails to start is red. qemu-runner.sh stores VM state across start and stop commands. A command that starts a VM owns its cleanup trap, preserving the trap installed by test-matrix.sh. Start rejects an unusable guest /tmp, and reports the serial console and qemu's own output before the run directory holding them goes, since that text is all the caller sees of a VM that did not boot. Stop verifies the recorded process still names its pidfile before sending a signal. tests/test-qemu-runner.sh pins both against stand-ins, so no VM boots: a recycled pid, a process whose argv names the pidfile, and a start that never opens the port.
Every elfuse cleanup is an atexit hook, so a guest killed on timeout, Ctrl-C, or a dead harness leaves its scratch, fork children, and SysV objects behind, and the run leaves the QEMU VM with them. make clean deletes build/conformance/qemu.state, the only record of that VM. scripts/conformance clean sweeps in dependency order: the VM through qemu-runner.sh stop, disowned VMs whose 9p share names this checkout, the harness's detached guest session leaders and --fork-child orphans, then the runtime scratch, and last a chmod u+rwx over the results tree so make clean can remove the mode-0 directories gVisor leaves. Only the exact scratch templates the runtime uses are swept, because review checkouts share the elfuse- prefix. SysV objects are reported, never removed. The guest key reaches the host unchanged, so an elfuse object is indistinguishable from another of the user's, and outliving its creator is ordinary SysV lifecycle. It keeps the lock files, since unlinking a held lock lets two sessions run; the results; another checkout's VMs and guests; and any elfuse run from a terminal. While any elfuse of the user is alive both the scratch sweep and the SysV report are skipped, since every elfuse shares the /tmp names. A held lock makes it refuse and name the pid holding it. --dry-run prefixes every action it would take with would. make clean-conformance is the alias and joins the goals the build-flavor guard skips.
clean sees only that a segment's creator is dead, which any long-lived segment of the user's looks like, so it reports them all and removes none. A lane knows more: run records the segments that appeared while it held the backend, by id, key and creator pid, and clean removes those and reports the rest. Shared memory only. A segment's CPID is its creator and never changes, while ipcs gives a queue its last sender and receiver.
docs/testing.md lists the public commands with their defaults and the Make aliases. docs/conformance.md defines the result and extension contracts and what clean-conformance sweeps, reports, and keeps.
f303d4b to
ffa54f4
Compare
|
Closing in favor of smaller PR. |
2 participants
Build extensible CLI to host gvisor and LTP (and potentially much more) conformance test suites
Summary by cubic
Introduces a conformance test harness that runs registered Linux suites (gvisor, LTP, and more) against elfuse and a QEMU reference, judging results against JSONC expectations. The
scripts/conformanceCLI has a fixed capability-first command tree that suite registration cannot extend.New Features
scripts/conformancecommands for listing suites, enumerating cases, running lanes, seeding expectations, writing reports, and cleaning up after interrupted runs.Other Changes
/tmpsetup fails, arm its cleanup per CLI command, and verify a recorded pid still belongs to the VM before stopping; the shared ssh options now live intests/lib/qemu-ssh.sh.make clean-conformanceto sweep leftover VMs, guest processes, and scratch after interrupted runs; lanes record the shared memory segments they create so cleanup removes those and reports the rest.make checkvia selftests and addsdocs/conformance.mdplus the command reference indocs/testing.md.Written for commit ffa54f4. Summary will update on new commits.