From 51c51448d7d28cf2df75721903a7f0ce65161ead Mon Sep 17 00:00:00 2001 From: hmeiland Date: Wed, 22 Jul 2026 20:26:37 +0200 Subject: [PATCH] cores: add SpaceMiT X60 and SiFive U74 ISA & codegen notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add reference docs for two RISC-V cores used across the benchmark suite: - cores/x60: SpaceMiT X60 (K1) — rv64imafdcv + 30 exts, RVV 1.0 ratified VLEN=256; documents why GCC 14 has no native detection and the exact EESSI -march that matches the kernel ISA. - cores/u74: SiFive U74 (JH7110) — scalar rv64gc, no vector; in-order dual-issue single-FP-pipe tuning flags (-mtune=sifive-u74). Grounded in measured board evidence (Orange Pi RV2, VisionFive 2). --- cores/u74/README.md | 124 ++++++++++++++++++++++++++++++++++++++++++ cores/x60/README.md | 130 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 254 insertions(+) create mode 100644 cores/u74/README.md create mode 100644 cores/x60/README.md diff --git a/cores/u74/README.md b/cores/u74/README.md new file mode 100644 index 0000000..6378a28 --- /dev/null +++ b/cores/u74/README.md @@ -0,0 +1,124 @@ +# SiFive U74 — ISA & codegen notes + +Reference notes on the **SiFive U74** RISC-V core (SoC: StarFive **JH7110**; +board: **StarFive VisionFive 2**) — specifically **what ISA it implements and +how to make a compiler target it correctly**. Companion to +[`../x60/`](../x60); the two cores sit at opposite ends of the RISC-V spectrum +(U74 = scalar RVA20-class, X60 = RVV 1.0 vector), so the codegen story differs +sharply. + +## TL;DR + +- **4× SiFive U74 @ 1.5 GHz**, in-order, dual-issue. +- **`rv64gc` — scalar. There is NO vector extension** (no `v`, no `zve*`), + and on the JH7110 **no `zba`/`zbb`** bit-manip and **no `Zicbop`** prefetch. +- `rv64gc` == `rv64imafdc` (the `g` shorthand = `imafd` + `zicsr` + `zifencei`). +- Same RISC-V GCC rule as every RISC-V core: **no `-march=native`** — pass an + explicit `-march=rv64…` string. +- For a **tuned** OpenBLAS build the target flags are + `-march=rv64imafdc_zba_zbb -mabi=lp64d -mtune=sifive-u74` — but note the + `_zba_zbb` there is a *tuning convenience for the kernel that assembles + cleanly*; the **hand-written asm microkernel is pure `rv64gc`** (no `zba`/`zbb`, + no `.option arch`) precisely because the JH7110 U74 lacks those extensions. + +## Microarchitecture + +| | | +|---|---| +| SoC | StarFive JH7110 | +| Cores | 4× SiFive U74 @ 1.5 GHz | +| Vector | **none** (scalar `rv64gc`) | +| Pipeline | dual-issue **in-order**, **single FP pipe** | +| FP | `fmadd.d` fully pipelined, ~7-cycle latency, 1/cycle throughput | +| Memory | **one load port**; L1D 32 KB / 64 B line; **2 MB shared L2** | +| Board (this repo) | StarFive VisionFive 2, 8 GB, Ubuntu 24.04 | +| Toolchain | EESSI 2025.06, **GCC 14.3.0** | + +Peak DP ≈ **3.0 GFLOP/s/core** (12 GF board). Empirical ceilings measured on +the VF2: a load co-issues *free* with an FMA (distinct operands); the FP-pipe +peak is ≈ **2.90 GFLOP/s/core**; and the streaming DGEMM plateau is +**memory-latency-bound**, not FP-bound (one load port + shared L2). + +## The ISA string + +``` +rv64gc == rv64imafdc (+ implied zicsr, zifencei) +``` + +- **`i m a f d c`** — base integer, mul/div, atomics, single + double FP, + compressed. `d` ⇒ `-mabi=lp64d`. +- **`c`** — compressed 16-bit encodings. +- **No `v`** — *no vector unit at all.* Any RVV kernel is irrelevant here (this + is why the X60's RVV `gemv_n` NaN bug — see [`../x60/`](../x60) — **cannot** + affect the U74; the scalar U74 has no RVV kernels to be buggy). +- **No `zba`/`zbb`** on the JH7110 U74, and **no `Zicbop`** prefetch. Code that + emits address-generation (`zba`) or basic bit-manip (`zbb`) instructions will + fault. The DGEMM asm microkernel is therefore deliberately written to + assemble under **plain `-march=rv64gc`** (no `.option arch` bumping the ISA). + +## RISC-V GCC has no `-march=native` + +Same contract as every RISC-V core (see [`../x60/`](../x60) for the on-board +proof): the GCC 14.3.0 RISC-V back-end has **no native detection** — `-march=native` +errors (`ISA string must begin with rv32 or rv64`) and `-mcpu=native` errors +(`unknown CPU`). You must give an explicit string. There is, however, a valid +**`-mtune`**: `-mtune=sifive-u74` (scheduling only — it changes instruction +*scheduling*, never which instructions are legal, which is governed solely by +`-march`). + +## Build flags + +**Generic / stock** (what a default `rv64gc` build uses): + +``` +-march=rv64gc -mabi=lp64d +``` + +**U74-tuned** (the OpenBLAS `U74` target added upstream — see below): + +``` +-march=rv64imafdc_zba_zbb -mabi=lp64d -mtune=sifive-u74 +``` + +`rv64imafdc` is exactly `rv64gc`. The `_zba_zbb` appears in the OpenBLAS +`Makefile.prebuild` `TARGET_FLAGS` / `Makefile.riscv64` `CCOMMON_OPT` for the +compiler-emitted C kernels; the **critical hand-asm microkernel stays pure +`rv64gc`** so it never emits a `zba`/`zbb` instruction the hardware lacks. + +## The OpenBLAS `U74` target (measured wins) + +Adding a dedicated `U74` target + a hand-written 4×4 DGEMM asm microkernel +(software-pipelined, one-load-port aware) beat the generic `rv64gc` build on +the VisionFive 2: + +| Metric | Stock (`rv64gc`) | U74-tuned | Gain | +|---|---|---|---| +| HPL, N=10000, 4 cores | 3.08–3.13 GFLOP/s | **5.28 GFLOP/s** | **1.69×** | +| DGEMM, single core | ~1.4 GFLOP/s | **1.77 GFLOP/s** | ~1.26× | +| DGEMM, 4 cores | ~3 GFLOP/s | **6.31 GFLOP/s** | ~2× | + +HPL trails raw DGEMM because of its non-GEMM work (panel factorization, +pivoting, comms). Build: `make TARGET=U74 DYNAMIC_ARCH=0 -j4`. + +Upstreamed: + +- OpenBLAS source PR **OpenMathLib/OpenBLAS#5903** — + +- EasyBuild easyconfig PR **easybuilders/easybuild-easyconfigs#26436** — + + +Files touched by the target (all in +`OpenBLAS-0.3.30_add-sifive-u74-target.patch`): `TargetList.txt`, `getarch.c` +(`FORCE_U74` block after `FORCE_RISCV64_GENERIC`), `cpuid_riscv64.c`, `param.h` +(U74 params, `Q=256`), `Makefile.prebuild`, `Makefile.riscv64`, +`kernel/riscv64/KERNEL.U74`, and the kernels `kern_u74.S` / +`gemmkernel_4x4_u74.c`. + +## Sources + +- `riscv-u74.md` — full U74 / OpenBLAS / EESSI working notes (measured on the + VisionFive 2, JH7110, 4× U74 @ 1.5 GHz). +- `u74-openblas-gemm-optimization.md` / `.pdf` — the full design paper. +- `blog-hpl-u74-eessi.md` / `.pdf` — fresh-install HPL reproduction walkthrough. +- RISC-V unprivileged ISA (`rv64gc` = `rv64imafdc` + `zicsr`/`zifencei`); + SiFive U74 core complex manual. diff --git a/cores/x60/README.md b/cores/x60/README.md new file mode 100644 index 0000000..666ee35 --- /dev/null +++ b/cores/x60/README.md @@ -0,0 +1,130 @@ +# SpaceMiT X60 (K1) — ISA & codegen notes + +Reference notes on the **SpaceMiT X60** RISC-V core (SoC: SpaceMiT **K1** / +"M1"; boards: **Orange Pi RV2**, **Banana Pi BPI-F3**) — specifically **what +ISA it implements and how to make a compiler target it correctly**. These are +the facts you need before building any vectorized HPC kernel for this board. + +## TL;DR + +- **8× SpaceMiT X60 @ 1.6 GHz**, in-order. +- **RVV 1.0 — ratified, `v` in the ISA string, `VLEN=256` (`zvl256b`)**. + *Not* the 0.7.1 draft. Full 32-register `v` with `zve*`/`zvl*` sub-extensions. +- Kernel-reported ISA is `rv64imafdcv` + ~26 sub-extensions (`Zicbom`, `Zba`, + `Zbb`, `Zbc`, `Zbs`, `Zfh`, `Zvfh`, `Zvkt`, …). +- **RISC-V GCC has *no* native CPU detection** — `-march=native`, + `-mcpu=native`, and `-mcpu=spacemit-x60` all **fail** on this toolchain. You + **must** pass an explicit `-march=rv64…` string. EESSI already does, and its + string is X60-correct. + +## Microarchitecture + +| | | +|---|---| +| SoC | SpaceMiT K1 (a.k.a. M1) | +| Cores | 8× X60 @ 1.6 GHz | +| Vector | **RVV 1.0**, `VLEN=256` bits, `ELEN=64` | +| Board (this repo) | Orange Pi RV2, 8–16 GB LPDDR4 | +| Toolchain | EESSI 2025.06 / `dev.eessi.io` riscv overlay, **GCC 14.3.0** | + +## The ISA string + +The kernel advertises the following in `/proc/cpuinfo` (`isa :` line). All 30 +tokens, canonical order: + +``` +rv64imafdcv + zicbom zicboz zicntr zicond zicsr zifencei zihintpause zihpm + zfh zfhmin + zca zcd + zba zbb zbc zbs + zkt + zve32f zve32x zve64d zve64f zve64x + zvfh zvfhmin zvkt + sscofpmf sstc svinval svnapot svpbmt +``` + +What matters for **code generation**: + +- **`v`** — full RVV 1.0 vector (implies the `zve*` embedded-vector subsets and + `zvl*` up to the machine `VLEN=256`). +- **`f d`** — hardware single + double FP (so `-mabi=lp64d`). +- **`zba zbb zbc zbs`** — bit-manip (address-gen, basic, carry-less, single-bit). +- **`zfh zfhmin` / `zvfh zvfhmin`** — scalar + vector half-precision FP. +- **`zicbom zicboz`** — cache-block management / zero (only `zicbom` is the + block-op set; **there is no `zicbop` prefetch** on this core). +- **`zkt`, `zvkt`** — data-independent (constant-time) execution latency + guarantees (crypto), not perf-relevant for GEMM/FFT. +- **`ss*` / `sv*`** (`sscofpmf`, `sstc`, `svinval`, `svnapot`, `svpbmt`) are + **supervisor / OS-facing** (counter-overflow, stimecmp, TLB-invalidate, + NAPOT page, page-based memory types). The kernel uses them; they are **not + code-generation-relevant** for a user-space compute kernel. + +## RISC-V GCC has no `-march=native` — verified on the board + +Unlike x86 (which resolves `-march=native` via `CPUID`) and AArch64 (via +`HWCAP` / MIDR), **the RISC-V GCC 14.3.0 back-end has no runtime native +detection at all.** All three "just detect it" spellings fail on the X60: + +``` +$ gcc -mcpu=native ... + → error: unknown CPU 'native' +$ gcc -march=native ... + → error: ISA string must begin with rv32 or rv64 +$ gcc -mcpu=spacemit-x60 ... + → error: unknown CPU 'spacemit-x60' # not in this GCC build's core DB +``` + +**Consequence:** the RISC-V GCC contract is an **explicit `-march=` string** — +there is no autodetection fallback. Anything relying on `native` (e.g. a build +system, or **Kokkos's `ARCH_NATIVE`**) will either error out or silently emit +scalar `rv64gc`. + +## EESSI's `-march` is already X60-correct + +EESSI compiles with an explicit `-march` that **matches the kernel's X60 ISA +exactly** (same 30 extensions, only reordered), plus the explicit `zvl` tokens: + +``` +-march=rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_ + zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_ + zvfh_zvfhmin_zvkt_zvl128b_zvl32b_zvl64b_sscofpmf_sstc_svinval_svnapot_svpbmt + -mabi=lp64d -misa-spec=20191213 +``` + +So for HPC builds on this board you do **not** need to change `-march`: the +EESSI default already targets the X60 correctly. The only place "native" bites +is a build system that *asks* for it — see the Kokkos note below. + +### Practical build lever (Kokkos / LAMMPS, seen in this repo's stack) + +Kokkos defaults to `Kokkos_ARCH_NATIVE=ON`, which makes it append +`-mcpu=native -mtune=native` → **fails on RISC-V GCC** (per above). The fix is +to route Kokkos to its no-op generic arch so it adds **no** `-m` flags and +inherits the EESSI `-march`: + +```python +# in the LAMMPS/Kokkos easyconfig +kokkos_arch = 'EASYBUILD_GENERIC' # -> -DKokkos_ARCH_EASYBUILD_GENERIC=yes, a no-op target +``` + +(Setting `-DKokkos_ARCH_NATIVE=OFF` via `configopts` alone does **not** work — +the LAMMPS easyblock appends its own `Kokkos_ARCH_*=yes` *after* your +`configopts`, so it wins. The `kokkos_arch` easyconfig parameter is the +sanctioned lever.) + +## Where the real X60 wins are + +The vector FP (`v`, VLEN=256) is genuine but, on the whole-application +benchmarks in this repo, drop-in vectorized BLAS/FFT libraries **do not move a +real DFT/MD run much**. The distinctive X60 win is its **IME** integer-matrix +extension (`smt.vmadot`, an `Xsmtvdot`-style vendor op) for int8/int4 GEMM — +see [`../../ime/`](../../ime) and [`../../onnx/`](../../onnx). + +## Sources + +- Verified on-board (`/proc/cpuinfo`, `gcc -mcpu=native` / `-march=native` / + `-mcpu=spacemit-x60` probes) on the Orange Pi RV2, EESSI GCC 14.3.0. +- `riscv-x60.md` — full X60 / OpenBLAS / EESSI working notes. +- SpaceMiT K1/X60 documentation; RISC-V RVV 1.0 spec; RISC-V `Zvl*` / `Zve*` / + bit-manip (`Zb*`) extension specs.