tests: replace keystone with llvm-mc firmware build pipeline#57
Draft
jeromehaxhiaj-qti wants to merge 3 commits into
Draft
tests: replace keystone with llvm-mc firmware build pipeline#57jeromehaxhiaj-qti wants to merge 3 commits into
jeromehaxhiaj-qti wants to merge 3 commits into
Conversation
6dac2fc to
17dc52a
Compare
Add llvm and lld to install_dependencies.sh's dependency lists (Ubuntu apt, macOS brew, msys2 pacboy) so the script covers everything tests/qbox will need once the keystone-to-llvm-mc migration that follows lands. On macOS Homebrew ships ld.lld in the separate `lld` formula, not `llvm`, so both formulae are required there. Signed-off-by: Jerome Haxhiaj <jhaxhiaj@qti.qualcomm.com>
17dc52a to
6377fa3
Compare
- aarch64-dmi-test-reset:num-cpu=(4|32) segfaults intermittently on
Ubuntu in both SINGLE and MULTI threading (across the
ubuntu-{22,24}.04 × {gcc,gcc-debug,clang,clang-debug,clang-lto}
matrix, num-cpu=4 and num-cpu=32 each fail in some runs while
num-cpu=1 and num-cpu=2 always pass). Mark all num-cpu>2 cells as
flaky regardless of threading mode or sync-policy.
- hexagon-smmu-stress-test-v2 already skipped num-cpu>1 for COROUTINE
and SINGLE as "takes too long". Extend the same skip to MULTI so
num-cpu>1 is uniformly skipped — the macOS run hit a 100s ctest
timeout on multithread-quantum:num-cpu=4:MULTI while the same
policy at num-cpu=1 finishes in well under a second.
Signed-off-by: Jerome Haxhiaj <jhaxhiaj@qti.qualcomm.com>
Retire the keystone runtime assembler from Qbox CPU tests in favour of
firmware binaries built at CMake-configure time with llvm-mc + ld.lld
+ llvm-objcopy. Each test ships a paired .S file (with .quad/.word
placeholders for runtime-patched constants like MMIO_ADDR / NUM_WRITES)
and the new CpuTestBenchBase::load_firmware_binary() helper patches
those placeholders and ptr_load()s the result into m_mem at runtime.
Deletes patch/keystone/ and the keystone CPM fetch; LLVM tools are
now a hard build requirement (covered by the install_dependencies.sh
update in the previous commit).
Build-side helpers in tests/qbox/CMakeLists.txt:
- qbox_build_firmware_bin(<name> <arch> <src_dir> [EXTRA_LD_ARGS ...])
assembles <src_dir>/<name>.S, links with ld.lld using
<src_dir>/<name>.ld (or the per-directory firmware.ld fallback),
and objcopies to a raw .bin via a <name>-firmware custom target.
- qbox_attach_firmware(<target> <fw_name>) wires the build dependency
and defines FIRMWARE_BIN_PATH for the test executable.
- qbox_add_cpu_test(<target> <timeout_s> <sources>...
[ARCH <arch>] [FIRMWARE <name>]
[FIRMWARE_SRC_DIR <dir>] [EXTRA_LD_ARGS ...])
collapses the build + attach + add_test boilerplate into one call.
By default the firmware base name is the stem (NAME_WE) of the
first source file; FIRMWARE overrides it.
- qbox_add_cpu_<arch>_test(<target> <timeout_s> <source>.cc) per-arch
thin wrappers in cpu/<arch>/CMakeLists.txt that pin ARCH=<arch>.
This is what call sites use, so each test reads as one line.
Portability:
- On MSYS2/Windows, ld.lld defaults to the lld-link/PE-COFF driver and
silently ignores -flavor gnu, leaving GNU-style options like -T
unrecognised. We pass the correct ELF emulation per arch
(aarch64elf / hexagonelf / elf32lriscv / elf64lriscv) via -m on
WIN32; older Linux LLDs (e.g. Ubuntu's stock llvm-18) don't know
every emulation name, so we leave -m off elsewhere.
- hexagon-smmu-stress-test-v2.s uses Hexagon V73 ops (tlbw, dmwait,
dmlink) that older LLVM Hexagon backends don't recognise. macOS
via Homebrew's llvm and MSYS2/Windows both ship a recent-enough
backend; the firmware (and its test) are gated behind APPLE OR
WIN32 (QQVPQSP-714).
- tests/CMakeLists.txt gates each tests/qbox/cpu/<arch>/ subdirectory
on the corresponding architecture being present in LIBQEMU_TARGETS.
Naming convention applied across all cpu-test sources:
- Filenames use dashes only, never underscores; the hexagon dir uses
hexagon- consistently rather than the previous hex- shortening.
- Each test owns its firmware: aarch64 reset-test was previously
shared by reset-test-system and reset-test-cpu; the .S is now
duplicated so each has its own. hexagon-smmu-firmware was renamed
to hexagon-smmu-stress-test-v2 to match its driver.
- After these renames every qbox_add_cpu_<arch>_test call has
firmware == source-stem, so qbox_add_cpu_test derives the firmware
base from the first source's NAME_WE and call sites just read
(target, timeout, source).
Covers all aarch64, hexagon, and riscv32 CPU tests previously using
set_firmware() (including the multi-block smmu-router-stress-test-v2
layout via .org), and the standalone hexagon load-store-test harness.
Signed-off-by: Jerome Haxhiaj <jhaxhiaj@qti.qualcomm.com>
6377fa3 to
c1d29d0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retire the keystone runtime assembler from Qbox CPU tests and replace it
with a configure-time firmware build pipeline based on the LLVM toolchain
(
llvm-mc+ld.lld+llvm-objcopy)..S(and optional.ld) file that isassembled, linked, and objcopied to a raw
.binat CMake configuretime. The test loads the
.binat runtime via the newCpuTestBenchBase::load_firmware_binary()helper, which patches atrailing window of
.quad/.wordplaceholders for runtimeconstants like
MMIO_ADDRandNUM_WRITES.keystoneCPM fetch and the patches underpatch/keystone/.LLVM tools are now a hard build requirement on all platforms; the
install-dependencies script is updated to cover them.
tests/qbox/CMakeLists.txt(
qbox_build_firmware_bin,qbox_attach_firmware,qbox_add_cpu_test)plus per-arch wrappers
qbox_add_cpu_<arch>_testso each test readsas a single line at the call site.
What's in this PR
scripts: install llvm/lld for the firmware build pipeline(67abcce)Adds
llvmandlldtoinstall_dependencies.sh's package lists forUbuntu (
apt), macOS (brew), and MSYS2 (pacboy). On macOS Homebrewships
ld.lldin the separatelldformula, so both formulae arerequired.
tests: replace keystone with llvm-mc firmware build pipeline(54cf583)The bulk of the change. Highlights:
New CMake helpers (in
tests/qbox/CMakeLists.txt):qbox_build_firmware_bin(<name> <arch> <src_dir> [EXTRA_LD_ARGS ...])—assembles
<src_dir>/<name>.S, links withld.lldusing<src_dir>/<name>.ld(or per-directoryfirmware.ldfallback),objcopies to a raw
.binvia a<name>-firmwarecustom target.qbox_attach_firmware(<target> <fw_name>)— wires the build dependencyand defines
FIRMWARE_BIN_PATHfor the test executable.qbox_add_cpu_test(<target> <timeout_s> <sources>... [ARCH <arch>] [FIRMWARE <name>] [FIRMWARE_SRC_DIR <dir>] [EXTRA_LD_ARGS ...])—collapses the build + attach + add_test boilerplate into one call.
By default the firmware base name is the stem (
NAME_WE) of the firstsource file;
FIRMWAREoverrides it.qbox_add_cpu_<arch>_test(<target> <timeout_s> <source>.cc)— per-archthin wrappers in
cpu/<arch>/CMakeLists.txtpinningARCH=<arch>.This is what call sites use, so each test is one line.
Portability:
ld.llddefaults to thelld-link/PE-COFF driverand silently ignores
-flavor gnu, leaving GNU-style options like-Tunrecognised. Pass the correct ELF emulation per arch(
aarch64elf/hexagonelf/elf32lriscv/elf64lriscv) via-monWIN32; older Linux LLDs (e.g. Ubuntu's stock llvm-18)don't know every emulation name, so leave
-moff elsewhere.hexagon-smmu-stress-test-v2.suses Hexagon V73 ops (tlbw,dmwait,dmlink) that older LLVM Hexagon backends don't recognise.macOS via Homebrew's
llvmand MSYS2/Windows both ship a recentenough backend; the firmware (and its test) are gated behind
APPLE OR WIN32(QQVPQSP-714).tests/CMakeLists.txtgates eachtests/qbox/cpu/<arch>/subdirectory on the corresponding arch being present in
LIBQEMU_TARGETS.Naming convention applied uniformly across cpu-test sources:
hexagon-consistently rather than the previoushex-shortening.reset-testwas previouslyshared by
reset-test-systemandreset-test-cpu; the.Sis nowduplicated so each test has its own.
hexagon-smmu-firmwarewasrenamed to
hexagon-smmu-stress-test-v2to match its driver.qbox_add_cpu_<arch>_testcall hasfirmware == source-stem, soqbox_add_cpu_testderives thefirmware base from the first source's
NAME_WEand call sites justread
(target, timeout, source).Coverage: all aarch64, hexagon, and riscv32 CPU tests previously
using
set_firmware()(including the multi-blocksmmu-router-stress-test-v2layout via.org), and the standalonehexagon-load-store-testharness.tests: extend SKIP_TESTS for known-flaky and slow cpu-test configs(7778beb)Two SKIP_TESTS entries surfaced by CI on this branch (both pre-existing
issues, unrelated to the keystone migration):
aarch64-dmi-test-reset:num-cpu=(4|32):threading=SINGLEsegfaultsintermittently on Ubuntu (across the
ubuntu-{22,24}.04 × {gcc,gcc-debug,clang,clang-debug,clang-lto}matrix,
num-cpu=4andnum-cpu=32each fail in roughly half ofruns while
num-cpu=1andnum-cpu=2always pass). Marked flaky.hexagon-smmu-stress-test-v2already skippednum-cpu>1forCOROUTINEandSINGLEas "takes too long". Extends the same skipto
MULTIsonum-cpu>1is uniformly skipped — macOS hit a 100sctest timeout on
multithread-quantum:num-cpu=4:MULTIwhile thesame policy at
num-cpu=1finishes in well under a second.Test plan
cmake --build build --target testpasses on macOS (apart fromthe
hexagon-smmu-stress-test-v2:multithread-quantum:num-cpu=4timeout, now skipped) — verified locally
aarch64-dmi-test-reset SEGFAULTs are pre-existing, now skipped)
<arch>-<name>.{cc,S,ld}) round-tripvia
qbox_build_firmware_binand load at runtime — verified byend-to-end runs of
aarch64-simple-write-test,aarch64-reset-test-system,aarch64-reset-test-cpu,hexagon-ld-st-mmio-test,riscv32-reset-test,hexagon-smmu-stress-test-v2(all pass)if(NOT TARGET ...)guard insideqbox_add_cpu_testmeansshared firmware (when used) is built once — no longer needed
after this PR (every test owns its firmware) but the guard
stays for safety
hexagon-load-store-test(which doesn't gothrough
qbox_add_cpu_test) still builds and runsMigration notes for downstream
If you depend on the now-removed
set_firmware()runtime API or on thekeystone library itself, you'll need to migrate to the
qbox_add_cpu_<arch>_testhelpers (seetests/qbox/cpu/*/CMakeLists.txtfor examples). The C++-side
load_firmware_binary()helper expects afirmware that ends with
.quad(aarch64) or.word(riscv32/hexagon) placeholders patched by initializer-list values at
the tail of the binary; see
tests/qbox/include/test/cpu.h.