From a692ef1671ea51ea79db94fcd3fe5a1faedc62b0 Mon Sep 17 00:00:00 2001 From: Evgeni Burovski Date: Tue, 17 Nov 2026 19:15:41 +0000 Subject: [PATCH 1/4] Add the numpy-revisions example: two spin-built revisions, one asv benchmark UC-03 for NumPy. Two revisions are built with `spin`, each in a git worktree of its own, and one of NumPy's own asv benchmarks (bench_function_base.Mean.time_mean, unchanged) is run on each, alternated round by round, then compared with `bx compare --run --profile revisions`. The sides are selected by a per-side PYTHONPATH, one `bx bench` call at a time, and compare.sh checks that each side imports its own build: the asv adapter does not put the target on the benchmark's sys.path, and benchx does not record PYTHONPATH. The README says so. Co-Authored-By: Claude Sonnet 5.5 --- examples/numpy-revisions/.gitignore | 2 + examples/numpy-revisions/README.md | 119 ++++++++++++++++++++++ examples/numpy-revisions/benchx.conf.json | 52 ++++++++++ examples/numpy-revisions/compare.sh | 65 ++++++++++++ examples/numpy-revisions/environment.yml | 26 +++++ examples/numpy-revisions/numpy_facts.py | 39 +++++++ 6 files changed, 303 insertions(+) create mode 100644 examples/numpy-revisions/.gitignore create mode 100644 examples/numpy-revisions/README.md create mode 100644 examples/numpy-revisions/benchx.conf.json create mode 100755 examples/numpy-revisions/compare.sh create mode 100644 examples/numpy-revisions/environment.yml create mode 100644 examples/numpy-revisions/numpy_facts.py diff --git a/examples/numpy-revisions/.gitignore b/examples/numpy-revisions/.gitignore new file mode 100644 index 0000000..ea93687 --- /dev/null +++ b/examples/numpy-revisions/.gitignore @@ -0,0 +1,2 @@ +out/ +__pycache__/ diff --git a/examples/numpy-revisions/README.md b/examples/numpy-revisions/README.md new file mode 100644 index 0000000..7669c4a --- /dev/null +++ b/examples/numpy-revisions/README.md @@ -0,0 +1,119 @@ +# numpy-revisions: two NumPy revisions, one of NumPy's own asv benchmarks + +This is [UC-03](https://github.com/benchx-dev/benchx/blob/main/docs/use-cases/UC-03-compare.md), +"run a benchmark comparison across two revisions", for NumPy. Two revisions are built with +`spin`, each in a git worktree of its own so both builds exist at once. A benchmark that is +already in NumPy, written in [asv](https://asv.readthedocs.io)'s format, is run on each, +alternately, round by round, and `bx compare` says whether the second revision differs from +the first. + +The benchmark is `Mean.time_mean` in NumPy's `benchmarks/benchmarks/bench_function_base.py`, +run unchanged. benchx reads asv's format as a language (discovery, `params`, `setup`) and times it +with pyperf: neither asv nor asv_runner is installed. + +```python +class Mean(Benchmark): + param_names = ['size'] + params = [[1, 10, 100_000]] + + def setup(self, size): + self.array = np.arange(2 * size).reshape(2, size) + + def time_mean(self, size): + np.mean(self.array) +``` + +One benchmark with three sizes is three cases. The file is the same in `v2.4.0` and `v2.4.3`, the +default pair, so both sides run the same benchmark. + +## Run it + +You need a git checkout of NumPy (it is never built in: the builds are in worktrees next to it), +network access the first time (for NumPy's submodules), and an environment with NumPy's build +requirements, `spin`, pyperf and `bx`: + +``` +$ micromamba create -f environment.yml +$ micromamba activate numpy-revisions +$ pip install benchx +$ NUMPY=~/repos/numpy ./compare.sh v2.4.0 v2.4.3 +``` + +The two revisions are any tags, branches or commits of `$NUMPY`. The first run builds both (a few +minutes each on a many-core machine); the worktrees, named after the revision and its commit, such +as `v2.4.0-c5ab79c14c`, stay under `$WORKTREES` (by default `$NUMPY-worktrees`), and later runs reuse +them. The measuring took about a minute on a 64-core machine. `ROUNDS=3 ./compare.sh` changes the number of rounds, and +`benchx.conf.json` holds the precision (5 processes x 3 values per case). + +Everything that is generated is in `out/`, which is git-ignored; the worktrees are outside this +directory. Remove one with `git -C $NUMPY worktree remove`. + +## What `compare.sh` does + +1. **Builds both revisions.** For each, a worktree of `$NUMPY` and `spin build` in it. Then it checks + that `python`, with that build's `site-packages` as `PYTHONPATH`, imports the `numpy` of that + build, and stops if not. +2. **Alternates the sides.** For each of the 5 rounds, one `bx bench` call per side: + + ``` + PYTHONPATH= bx bench --filter 'bench_function_base\.Mean\.time_mean\(' \ + --run-key numpy-v2.4.0-vs-v2.4.3 --round 0 --slot 0 --out out/numpy-v2.4.0-vs-v2.4.3 + ``` + + The target is the worktree. `adapter.suite` in `benchx.conf.json`, `benchmarks/benchmarks`, is the + asv suite inside it, and the worktree's revision is what the result records. `--filter` is asv's + `--bench`: a regex searched in `name(params)`, so `\(` keeps `time_mean_axis` out. `PYTHONPATH` + is what makes the benchmark's processes import that revision's build. The calls share a run key; + `--round` and `--slot` place each one in the alternation, slots counting the calls. +3. **Compares.** `bx compare --run KEY --profile revisions --baseline `. + +## What comes out + +The comparison (your numbers will differ): + +``` +run numpy-v2.4.0-vs-v2.4.3 profile revisions baseline edd9d7c22ba2 contender 63d7b31efb1a + differs: benchmark.revision.key + caveats: baseline: setup-per-call-unbatched, contender: setup-per-call-unbatched + +NO CHANGE DETECTED (3) + bench_function_base.Mean.time_mean [size=1] wall-time effect +1.84% noise 3.47% rounds=5 + bench_function_base.Mean.time_mean [size=10] wall-time effect +0.17% noise 2.26% rounds=5 + bench_function_base.Mean.time_mean [size=100000] wall-time effect -0.38% noise 1.15% rounds=5 +``` + +A case is a change when its effect, the mean relative difference of the paired rounds, is more than +`k` (3) times its noise. These calls take microseconds, so a run to run spread of a few percent is +normal, and a pair of revisions that differ in nothing can show a small "improvement" or +"regression" now and then. `differs: benchmark.revision.key` says that each side's benchmark is the +copy in its own revision. `out/compare.txt` keeps this text and `out/store.parquet` the results (try +`bx head --table`). benchx 0.1.0, from PyPI, prints the three rows without the `[size=...]`. + +## Notes + +- **`setup-per-call-unbatched`.** `Mean` has a `setup`, so, as in asv_runner >= 0.3.0, every timed call + sees fresh state: one call per loop, with the setup re-run between calls outside the timer. For a + call this short that timing includes some timer overhead, and the result is flagged. +- **Why a loop, and not one `bx compare A B`.** With a target of its own, `bx compare` runs both sides + in one command, as the pyperf adapter does by putting each build first on `sys.path`. The asv + adapter does not do that yet: it runs the `python` on the `PATH`, and the target only says which + tree is recorded. So the two builds are selected by `PYTHONPATH`, one `bx bench` call at a time. + benchx does not record `PYTHONPATH`, which is why step 1 checks it. +- **The suite is each revision's own.** An asv suite outside the checkout being measured does not work + with `bx bench` yet. Each result records asv's hash of its benchmark's code in its protocol, so a + benchmark that was edited between the two revisions starts a new series. +- **Skipped module.** `bench_lib.py` imports `asv_runner`, which is not installed, so benchx skips + that module. The note, and the table of each call, are in `out/bench.log`. +- **Python and the build must match.** `python` on the `PATH` runs the benchmark, so it must be the + Python `spin` built with, here 3.12; the environment provides both. +- Tried with Python 3.12, `spin` 0.15, OpenBLAS 0.3.34 and GCC 15.3 on Linux, with benchx 0.1.0 and + with a checkout of benchx. + +## What is in this directory + +| File | Role | +|---|---| +| `compare.sh` | the driver: builds, the loop, the comparison | +| `benchx.conf.json` | the project file: the asv adapter, its suite and precision, the collector, the `revisions` profile, the paths | +| `numpy_facts.py` | a context collector: how the numpy this Python imports was built (BLAS, LAPACK, compiler, SIMD). benchx matches these facts between the sides, so they agree only if both builds were configured alike. There is no numpy version among them, on purpose: it would differ. | +| `environment.yml` | an environment to build and run in | diff --git a/examples/numpy-revisions/benchx.conf.json b/examples/numpy-revisions/benchx.conf.json new file mode 100644 index 0000000..b483eb4 --- /dev/null +++ b/examples/numpy-revisions/benchx.conf.json @@ -0,0 +1,52 @@ +{ + "schema_version": "benchx/config/0.1.0", + "project": "numpy", + "source": { + "uri": "https://github.com/numpy/numpy.git" + }, + "adapter": { + "name": "asv", + "suite": "benchmarks/benchmarks", + "quantities": [ + "wall-time" + ], + "precision": { + "repetitions": { + "mode": "fixed", + "levels": [ + { + "unit": "process", + "n": 5 + }, + { + "unit": "value", + "n": 3 + } + ] + }, + "calibration": { + "mode": "adaptive", + "minimum_sample_seconds": 0.1 + }, + "warmup": { + "mode": "count", + "n_warmup": 1 + } + } + }, + "compare": { + "profile": "revisions" + }, + "collectors": { + "numpy-build": { + "command": [ + "python", + "{root}/numpy_facts.py" + ] + } + }, + "paths": { + "store": "out/store.parquet", + "results": "out" + } +} diff --git a/examples/numpy-revisions/compare.sh b/examples/numpy-revisions/compare.sh new file mode 100755 index 0000000..1801847 --- /dev/null +++ b/examples/numpy-revisions/compare.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# Compare two numpy revisions on one benchmark of numpy's own asv suite, with `bx`. +# +# usage: ./compare.sh [BASELINE [CONTENDER]] (tags, branches or commits of $NUMPY) +# +# Each revision is built by `spin` in a git worktree of its own, so both builds exist at once. +# The benchmark is numpy's `bench_function_base.Mean.time_mean`, unchanged, from each +# revision's own benchmarks/ directory. The sides alternate round by round, one `bx bench` +# call per side, and `bx compare --run` compares them. See README.md. +set -euo pipefail +cd "$(dirname "$0")" + +NUMPY=${NUMPY:-~/repos/numpy} # a git checkout of numpy; it is never built in +WORKTREES=${WORKTREES:-$NUMPY-worktrees} # one worktree per revision, named after it and its commit +BASELINE=${1:-v2.4.0} +CONTENDER=${2:-v2.4.3} +ROUNDS=${ROUNDS:-5} +CASES='bench_function_base\.Mean\.time_mean\(' # a regex, searched in `name(params)`: time_mean(1), ..., not time_mean_axis +RUN_KEY=numpy-$BASELINE-vs-$CONTENDER +unset PYTHONPATH # each call below sets its own + +# prepare REV: a worktree of $NUMPY at REV, built by `spin`. Sets COMMIT, TREE and SITE. +# `spin build` is incremental, so a worktree that is already built costs almost nothing. +prepare() { + COMMIT=$(git -C "$NUMPY" rev-parse --verify --quiet "$1^{commit}") || { + echo "compare.sh: $1 is not a revision of $NUMPY (set NUMPY to a git checkout of numpy)" >&2; exit 1; } + TREE=$WORKTREES/${1//[^A-Za-z0-9._-]/-}-${COMMIT:0:10} + if [ ! -d "$TREE" ]; then + git -C "$NUMPY" worktree add --detach "$TREE" "$COMMIT" >&2 + git -C "$TREE" submodule update --init >&2 + fi + (cd "$TREE" && spin build) >&2 + # The directory that holds the built `numpy` package: what the benchmark must import from. + SITE=$(dirname "$(ls -d "$TREE"/build-install/usr/lib/python3*/site-packages/numpy | head -n 1)") + # benchx does not record PYTHONPATH, so check here that this side imports its own build. + local imported + imported=$(PYTHONPATH=$SITE python -c 'import numpy; print(numpy.__file__)') + case $imported in + "$TREE"/build-install/*) echo " $1: $imported" ;; + *) echo "compare.sh: with PYTHONPATH=$SITE, python imports $imported, not the build of $1" >&2; exit 1 ;; + esac +} + +echo "== 1. build both revisions with spin, each in its own worktree" +mkdir -p "$WORKTREES" +prepare "$BASELINE"; BASE_COMMIT=$COMMIT; BASE_TREE=$TREE; BASE_SITE=$SITE +prepare "$CONTENDER"; CONT_TREE=$TREE; CONT_SITE=$SITE + +echo "== 2. alternate the sides, round by round: one bx bench call per side and round" +rm -rf out; mkdir -p out +# bx bench prints a table and some notes for every call; they go to out/bench.log, shown if a call fails. +side() { # side TREE SITE ROUND SLOT: one bx bench call + # PYTHONPATH selects the numpy that the benchmark imports. The target is the worktree: its + # benchmarks/benchmarks directory (adapter.suite) holds the suite, and its revision is recorded. + PYTHONPATH=$2 bx bench "$1" --filter "$CASES" --run-key "$RUN_KEY" --round "$3" --slot "$4" \ + --out "out/$RUN_KEY" >> out/bench.log 2>&1 || { tail -n 30 out/bench.log >&2; exit 1; } +} +for ROUND in $(seq 0 $((ROUNDS - 1))); do + side "$BASE_TREE" "$BASE_SITE" "$ROUND" $((2 * ROUND)) + side "$CONT_TREE" "$CONT_SITE" "$ROUND" $((2 * ROUND + 1)) + echo " round $ROUND: $BASELINE, then $CONTENDER" +done + +echo "== 3. compare: the revisions profile; the baseline is named by its commit" +bx compare --run "$RUN_KEY" --profile revisions --baseline "${BASE_COMMIT:0:10}" | tee out/compare.txt diff --git a/examples/numpy-revisions/environment.yml b/examples/numpy-revisions/environment.yml new file mode 100644 index 0000000..e32e247 --- /dev/null +++ b/examples/numpy-revisions/environment.yml @@ -0,0 +1,26 @@ +# To use: +# +# $ micromamba create -f environment.yml # `conda env create` works too +# $ micromamba activate numpy-revisions +# $ pip install benchx # `bx` +# +# NumPy's own build requirements (its environment.yml, without the docs and lint +# tools), plus what benchx needs: pyperf for the benchmark, pyarrow for the store. +name: numpy-revisions +channels: + - conda-forge +dependencies: + - python=3.12 # spin's builds and the benchmark must use the same Python + - cython>=3.0 + - compilers + - openblas + - nomkl + - ninja + - pkg-config + - meson-python + - spin==0.15 + - ccache + - git + - pyperf + - pyarrow + - pip diff --git a/examples/numpy-revisions/numpy_facts.py b/examples/numpy-revisions/numpy_facts.py new file mode 100644 index 0000000..18d9deb --- /dev/null +++ b/examples/numpy-revisions/numpy_facts.py @@ -0,0 +1,39 @@ +"""Print how the numpy that this Python imports was built, as one JSON object. + +benchx runs this as the collector in benchx.conf.json, once per `bx bench` call, in the +environment of the call (the `PYTHONPATH` that compare.sh sets selects the numpy it imports). + + blas, lapack name and version of the library numpy was built against + c_compiler name and version of the C compiler + simd_baseline, simd_found the SIMD extensions numpy was built for, and detected + summary the same, for the results + +There is no numpy version here, on purpose. benchx matches a collector's facts between +the two sides of a revisions comparison, which may differ in nothing but the revision, +so facts that name the version would make every comparison fail. Here the sides agree +only if both builds were configured alike, which is what a comparison of two revisions +wants to know. +""" + +import json + +import numpy + +config = numpy.show_config(mode="dicts") +dependencies = config.get("Build Dependencies", {}) +compilers = config.get("Compilers", {}) +simd = config.get("SIMD Extensions", {}) + + +def named(entry): + return {key: entry.get(key) for key in ("name", "version")} + + +facts = { + "blas": named(dependencies.get("blas", {})), + "lapack": named(dependencies.get("lapack", {})), + "c_compiler": named(compilers.get("c", {})), + "simd_baseline": simd.get("baseline"), + "simd_found": simd.get("found"), +} +print(json.dumps({**facts, "summary": facts}, sort_keys=True)) From eab903e861b2c01619f3f5dcbed5800b42dd8bbf Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Wed, 18 Nov 2026 09:06:10 +0000 Subject: [PATCH 2/4] numpy-revisions: build in named directories, show asv's tables, one conf for asv and pyperf compare.sh no longer makes a git worktree per revision. Each revision is checked out (with its submodules) in the NumPy checkout and built there by `spin build -C`: the checked-out code in `build`, any other commit in `build-`. The checkout is restored when the script ends, and the script refuses a tree with uncommitted changes. It takes `[-t TEST]... [BASELINE [CONTENDER]]` as `spin bench --compare` does, with no default benchmark, and prints which numpy each call imports. The comparison ends with `bx compare --tables`: a table of each commit and a table of the changes, as `asv continuous` prints them. The script passes `--adapter asv --suite benchmarks/benchmarks`, and benchx.conf.json has `adapter.name: pyperf` and only the timing, so the same file serves a pyperf suite. This needs a benchx newer than 0.1.0: `--tables`, and a pyperf conf serving `--adapter asv`, are not in the PyPI release. Co-Authored-By: Claude Sonnet 5.5 --- examples/numpy-revisions/README.md | 162 ++++++++++++++++------ examples/numpy-revisions/benchx.conf.json | 3 +- examples/numpy-revisions/compare.sh | 89 +++++------- 3 files changed, 157 insertions(+), 97 deletions(-) diff --git a/examples/numpy-revisions/README.md b/examples/numpy-revisions/README.md index 7669c4a..f84cea0 100644 --- a/examples/numpy-revisions/README.md +++ b/examples/numpy-revisions/README.md @@ -2,14 +2,15 @@ This is [UC-03](https://github.com/benchx-dev/benchx/blob/main/docs/use-cases/UC-03-compare.md), "run a benchmark comparison across two revisions", for NumPy. Two revisions are built with -`spin`, each in a git worktree of its own so both builds exist at once. A benchmark that is -already in NumPy, written in [asv](https://asv.readthedocs.io)'s format, is run on each, -alternately, round by round, and `bx compare` says whether the second revision differs from -the first. +`spin`, each in a build directory of its own in the NumPy checkout, so both builds exist at once. +A benchmark that is already in NumPy, written in [asv](https://asv.readthedocs.io)'s format, is +run on each, alternately, round by round, and `bx compare` says whether the second revision +differs from the first. -The benchmark is `Mean.time_mean` in NumPy's `benchmarks/benchmarks/bench_function_base.py`, -run unchanged. benchx reads asv's format as a language (discovery, `params`, `setup`) and times it -with pyperf: neither asv nor asv_runner is installed. +`compare.sh` is used like `spin bench --compare`: two revisions, and `-t` to choose benchmarks of +NumPy's own asv suite, `benchmarks/benchmarks`, which are run unchanged. The example below is +`Mean.time_mean`, in `bench_function_base.py`. benchx reads asv's format as a language (discovery, +`params`, `setup`) and times it with pyperf: neither asv nor asv_runner is installed. ```python class Mean(Benchmark): @@ -24,70 +25,144 @@ class Mean(Benchmark): ``` One benchmark with three sizes is three cases. The file is the same in `v2.4.0` and `v2.4.3`, the -default pair, so both sides run the same benchmark. +pair used below, so both sides run the same benchmark. ## Run it -You need a git checkout of NumPy (it is never built in: the builds are in worktrees next to it), -network access the first time (for NumPy's submodules), and an environment with NumPy's build -requirements, `spin`, pyperf and `bx`: +You need a git checkout of NumPy with no uncommitted changes (the revisions are checked out in it, +see below), network access the first time (for NumPy's submodules), and an environment with NumPy's +build requirements, `spin`, pyperf and `bx`: ``` $ micromamba create -f environment.yml $ micromamba activate numpy-revisions $ pip install benchx -$ NUMPY=~/repos/numpy ./compare.sh v2.4.0 v2.4.3 +$ NUMPY=~/repos/numpy ./compare.sh -t 'Mean\.time_mean\(' v2.4.0 v2.4.3 ``` -The two revisions are any tags, branches or commits of `$NUMPY`. The first run builds both (a few -minutes each on a many-core machine); the worktrees, named after the revision and its commit, such -as `v2.4.0-c5ab79c14c`, stay under `$WORKTREES` (by default `$NUMPY-worktrees`), and later runs reuse -them. The measuring took about a minute on a 64-core machine. `ROUNDS=3 ./compare.sh` changes the number of rounds, and -`benchx.conf.json` holds the precision (5 processes x 3 values per case). - -Everything that is generated is in `out/`, which is git-ignored; the worktrees are outside this -directory. Remove one with `git -C $NUMPY worktree remove`. +The usage is `./compare.sh [-t TEST]... [BASELINE [CONTENDER]]`, and everything is optional, as it is +for `spin bench --compare`: + +- **Revisions.** Anything that git resolves in `$NUMPY`: tags, branches, `HEAD`, `HEAD^`, commits. They + default to `main` and the checked-out code, and `./compare.sh v2.4.0` compares `v2.4.0` with it. The + first is the baseline and the second the contender. +- **`-t TEST`.** A regular expression searched in asv's name of a benchmark, `file.Class.method` + (the file is under `benchmarks/benchmarks/`, without `.py`), and in `name(params)` for each parameter + combination, as `spin bench -t` does. So `-t bench_function_base` is a file, `-t Mean` a class, and + `-t Mean.time_mean` a method, and also `time_mean_axis`, since it only searches. The `\(` above ends the + name, so that only `time_mean` is selected. Give `-t` more than once to select more. Without `-t` + every benchmark runs, which takes hours. A `-t` that selects nothing is refused, and the refusal lists + the benchmarks that exist. + +**Builds.** `spin build` builds each revision in `$NUMPY`, in a build directory of its own, so both builds +exist at once: a revision in `build-` (installed in `build--install`), the checked-out +code in `build`, where your own `spin build` puts it. It runs every time, and is incremental: a revision +takes a few minutes the first time on a many-core machine, and less after. NumPy git-ignores these +directories. `` is the short id (10 hex digits) of the commit the revision names now, so `main` is built in +the directory of the commit `main` is at (`build-c5ab79c14c` for `v2.4.0`). + +| You run | Baseline | Contender | +|---|---|---| +| `./compare.sh` | `main`, built in `build-` | the checked-out code, built in `build` | +| `./compare.sh SHA1` | `SHA1`, built in `build-` | the checked-out code, built in `build` | +| `./compare.sh SHA1 SHA2` | `SHA1`, built in `build-` | `SHA2`, built in `build-` | + +**The checkout.** `bx` measures the tree it is given, and tells the sides apart by their trees, so each +call has its commit checked out first: ten checkouts for 5 rounds. A checkout includes NumPy's +submodules (`git submodule update --init`): releases pin other commits of `vendored-meson`, `svml` and +others than `main` does, and without the update a build is made of the wrong sources and benchx tags the +tree dirty. When the script ends, also when it fails or is interrupted, the checkout is +put back, on its branch or at its commit. That is why it refuses to start when `$NUMPY` has +uncommitted changes, which would otherwise go into a build: commit them first. Untracked files are left +alone. Checking out another revision touches the files that differ, so your next `spin build` in +`$NUMPY` recompiles those. + +The measuring took about a minute on a 64-core machine. `benchx.conf.json` holds the precision +(5 processes x 3 values per case), and the loop in `compare.sh` the number of rounds (5). + +Everything that is generated here is in `out/`, which is git-ignored. ## What `compare.sh` does -1. **Builds both revisions.** For each, a worktree of `$NUMPY` and `spin build` in it. Then it checks - that `python`, with that build's `site-packages` as `PYTHONPATH`, imports the `numpy` of that - build, and stops if not. -2. **Alternates the sides.** For each of the 5 rounds, one `bx bench` call per side: +1. **Builds both revisions.** Each is resolved to a commit with `git rev-parse` before any checkout moves + `HEAD` (`bx compare` takes a commit id, and a tag is not one: `^{commit}`). For each, the checkout of + the commit and `spin build -C build-`, or `spin build -C build` for the checked-out code. +2. **Alternates the sides.** For each of the 5 rounds, one `bx bench` call per side, after checking out + its commit and printing which numpy it imports (version and path): ``` - PYTHONPATH= bx bench --filter 'bench_function_base\.Mean\.time_mean\(' \ - --run-key numpy-v2.4.0-vs-v2.4.3 --round 0 --slot 0 --out out/numpy-v2.4.0-vs-v2.4.3 + PYTHONPATH= bx bench $NUMPY --adapter asv --suite benchmarks/benchmarks \ + --filter 'Mean\.time_mean\(' --run-key numpy-revisions --round 0 --slot 0 ``` - The target is the worktree. `adapter.suite` in `benchx.conf.json`, `benchmarks/benchmarks`, is the - asv suite inside it, and the worktree's revision is what the result records. `--filter` is asv's - `--bench`: a regex searched in `name(params)`, so `\(` keeps `time_mean_axis` out. `PYTHONPATH` - is what makes the benchmark's processes import that revision's build. The calls share a run key; - `--round` and `--slot` place each one in the alternation, slots counting the calls. -3. **Compares.** `bx compare --run KEY --profile revisions --baseline `. + The target is the checkout, `--suite` is the asv suite inside it, and the checkout's revision is what + the result records. `--adapter asv` and `--suite` are in the script: the `adapter` block of + `benchx.conf.json` names pyperf and holds only the timing, and asv, which is timed by pyperf, takes + it too, so the same file serves a pyperf suite. `--filter` is asv's `--bench`, the `-t` options + joined by `|` (without `-t` there is none). `PYTHONPATH` is what makes + the benchmark's processes import that revision's build. The calls share a run key; `--round` and + `--slot` place each one in the alternation, slots counting the calls. +3. **Compares.** `bx compare --run numpy-revisions --baseline --tables`; `compare.profile` in + `benchx.conf.json` makes it the `revisions` profile, and `--tables` adds what `spin bench --compare` + prints: a table of each commit, then one of the changes (only the changed rows). Then the checkout is + put back. ## What comes out The comparison (your numbers will differ): ``` -run numpy-v2.4.0-vs-v2.4.3 profile revisions baseline edd9d7c22ba2 contender 63d7b31efb1a +run numpy-revisions profile revisions baseline edd9d7c22ba2 contender 63d7b31efb1a differs: benchmark.revision.key caveats: baseline: setup-per-call-unbatched, contender: setup-per-call-unbatched NO CHANGE DETECTED (3) - bench_function_base.Mean.time_mean [size=1] wall-time effect +1.84% noise 3.47% rounds=5 - bench_function_base.Mean.time_mean [size=10] wall-time effect +0.17% noise 2.26% rounds=5 - bench_function_base.Mean.time_mean [size=100000] wall-time effect -0.38% noise 1.15% rounds=5 + bench_function_base.Mean.time_mean [size=1] wall-time effect -0.41% noise 1.59% rounds=5 + bench_function_base.Mean.time_mean [size=10] wall-time effect +0.58% noise 0.58% rounds=5 + bench_function_base.Mean.time_mean [size=100000] wall-time effect -6.34% noise 6.65% rounds=5 + +For numpy commit c5ab79c1 (5 rounds): +bench_function_base.Mean.time_mean ok +======== ============= + size +-------- ------------- + 1 8.21±0.2μs + 10 8.34±0.08μs + 100000 162±0.3μs +======== ============= + +For numpy commit 8bcb2e72 (5 rounds): +bench_function_base.Mean.time_mean ok +======== ============= + size +-------- ------------- + 1 8.12±0.2μs + 10 8.36±0.08μs + 100000 162±20μs +======== ============= + +No benchmark changed (the verdict is benchx's; --all lists every row). +``` + +After the summary, `--tables` prints what `spin bench --compare` prints, laid out as asv lays it out: a +table of each commit (a value `8.21±0.2μs` per size, the `±` being the spread of the per-round +medians), then a table of the changes, which has only the changed rows: here there are none. `--all` +lists every row, with the noise and the verdict repeated; run by hand, +`bx compare --run numpy-revisions --baseline c5ab79c14c --tables --all` ends with + +``` +| Change | Before [c5ab79c1] | After [8bcb2e72] | Ratio | Noise | Verdict | Benchmark (Parameter) | +|----------|---------------------|--------------------|---------|---------|--------------------|--------------------------------------------| +| | 8.21±0.2μs | 8.12±0.2μs | 0.99 | 1.59% | no change detected | bench_function_base.Mean.time_mean(1) | +| | 8.34±0.08μs | 8.36±0.08μs | 1.00 | 0.58% | no change detected | bench_function_base.Mean.time_mean(10) | +| | 162±0.3μs | 162±20μs | 1.00 | 6.65% | no change detected | bench_function_base.Mean.time_mean(100000) | ``` A case is a change when its effect, the mean relative difference of the paired rounds, is more than `k` (3) times its noise. These calls take microseconds, so a run to run spread of a few percent is normal, and a pair of revisions that differ in nothing can show a small "improvement" or "regression" now and then. `differs: benchmark.revision.key` says that each side's benchmark is the -copy in its own revision. `out/compare.txt` keeps this text and `out/store.parquet` the results (try -`bx head --table`). benchx 0.1.0, from PyPI, prints the three rows without the `[size=...]`. +copy in its own revision. `out/store.parquet` holds the results (try `bx head --table`). benchx 0.1.0, from PyPI, prints the three rows without the `[size=...]`. ## Notes @@ -98,12 +173,17 @@ copy in its own revision. `out/compare.txt` keeps this text and `out/store.parqu in one command, as the pyperf adapter does by putting each build first on `sys.path`. The asv adapter does not do that yet: it runs the `python` on the `PATH`, and the target only says which tree is recorded. So the two builds are selected by `PYTHONPATH`, one `bx bench` call at a time. - benchx does not record `PYTHONPATH`, which is why step 1 checks it. + benchx does not record `PYTHONPATH`, which is why `compare.sh` prints the numpy each call imports. - **The suite is each revision's own.** An asv suite outside the checkout being measured does not work with `bx bench` yet. Each result records asv's hash of its benchmark's code in its protocol, so a benchmark that was edited between the two revisions starts a new series. +- **BLAS threads.** A benchmark that calls BLAS, such as `bench_linalg.Eindot.time_matmul_a_b`, uses + all the cores by default, and is very noisy on a many-core machine: here it gave an effect of +91% + against a noise of 117%. `OPENBLAS_NUM_THREADS=1 ./compare.sh -t bench_linalg.Eindot.time_matmul_a_b v2.4.0 v2.4.3` + pins it: the workers inherit the variable, benchx records it, and the same pair then gave +0.10% + against 0.56%. - **Skipped module.** `bench_lib.py` imports `asv_runner`, which is not installed, so benchx skips - that module. The note, and the table of each call, are in `out/bench.log`. + that module, and says so on every `bx bench` call. - **Python and the build must match.** `python` on the `PATH` runs the benchmark, so it must be the Python `spin` built with, here 3.12; the environment provides both. - Tried with Python 3.12, `spin` 0.15, OpenBLAS 0.3.34 and GCC 15.3 on Linux, with benchx 0.1.0 and @@ -114,6 +194,6 @@ copy in its own revision. `out/compare.txt` keeps this text and `out/store.parqu | File | Role | |---|---| | `compare.sh` | the driver: builds, the loop, the comparison | -| `benchx.conf.json` | the project file: the asv adapter, its suite and precision, the collector, the `revisions` profile, the paths | +| `benchx.conf.json` | the project file: the timing (precision, as pyperf takes it; asv takes it too), the collector, the `revisions` profile, the paths. Nothing asv's: `compare.sh` passes `--adapter asv --suite ...` | | `numpy_facts.py` | a context collector: how the numpy this Python imports was built (BLAS, LAPACK, compiler, SIMD). benchx matches these facts between the sides, so they agree only if both builds were configured alike. There is no numpy version among them, on purpose: it would differ. | | `environment.yml` | an environment to build and run in | diff --git a/examples/numpy-revisions/benchx.conf.json b/examples/numpy-revisions/benchx.conf.json index b483eb4..2ad4765 100644 --- a/examples/numpy-revisions/benchx.conf.json +++ b/examples/numpy-revisions/benchx.conf.json @@ -5,8 +5,7 @@ "uri": "https://github.com/numpy/numpy.git" }, "adapter": { - "name": "asv", - "suite": "benchmarks/benchmarks", + "name": "pyperf", "quantities": [ "wall-time" ], diff --git a/examples/numpy-revisions/compare.sh b/examples/numpy-revisions/compare.sh index 1801847..4156b57 100755 --- a/examples/numpy-revisions/compare.sh +++ b/examples/numpy-revisions/compare.sh @@ -1,65 +1,46 @@ #!/usr/bin/env bash -# Compare two numpy revisions on one benchmark of numpy's own asv suite, with `bx`. +# Compare two numpy revisions on numpy's own asv benchmarks, as `spin bench --compare` does. +# See README.md. # -# usage: ./compare.sh [BASELINE [CONTENDER]] (tags, branches or commits of $NUMPY) +# NUMPY=~/repos/numpy ./compare.sh [-t TEST]... [BASELINE [CONTENDER]] # -# Each revision is built by `spin` in a git worktree of its own, so both builds exist at once. -# The benchmark is numpy's `bench_function_base.Mean.time_mean`, unchanged, from each -# revision's own benchmarks/ directory. The sides alternate round by round, one `bx bench` -# call per side, and `bx compare --run` compares them. See README.md. +# BASELINE and CONTENDER default to main and the checked-out code. Each is checked out in $NUMPY, which +# must have no uncommitted changes, and built there: a revision in build-, the checked-out code +# in build. TEST is a regular expression for asv's benchmark names, as spin's -t is; without any, +# every benchmark runs, which takes hours. set -euo pipefail cd "$(dirname "$0")" -NUMPY=${NUMPY:-~/repos/numpy} # a git checkout of numpy; it is never built in -WORKTREES=${WORKTREES:-$NUMPY-worktrees} # one worktree per revision, named after it and its commit -BASELINE=${1:-v2.4.0} -CONTENDER=${2:-v2.4.3} -ROUNDS=${ROUNDS:-5} -CASES='bench_function_base\.Mean\.time_mean\(' # a regex, searched in `name(params)`: time_mean(1), ..., not time_mean_axis -RUN_KEY=numpy-$BASELINE-vs-$CONTENDER -unset PYTHONPATH # each call below sets its own +while getopts t: opt; do [ "$opt" = t ] || exit 2; FILTER=${FILTER:+$FILTER|}$OPTARG; done +shift $((OPTIND - 1)) -# prepare REV: a worktree of $NUMPY at REV, built by `spin`. Sets COMMIT, TREE and SITE. -# `spin build` is incremental, so a worktree that is already built costs almost nothing. -prepare() { - COMMIT=$(git -C "$NUMPY" rev-parse --verify --quiet "$1^{commit}") || { - echo "compare.sh: $1 is not a revision of $NUMPY (set NUMPY to a git checkout of numpy)" >&2; exit 1; } - TREE=$WORKTREES/${1//[^A-Za-z0-9._-]/-}-${COMMIT:0:10} - if [ ! -d "$TREE" ]; then - git -C "$NUMPY" worktree add --detach "$TREE" "$COMMIT" >&2 - git -C "$TREE" submodule update --init >&2 - fi - (cd "$TREE" && spin build) >&2 - # The directory that holds the built `numpy` package: what the benchmark must import from. - SITE=$(dirname "$(ls -d "$TREE"/build-install/usr/lib/python3*/site-packages/numpy | head -n 1)") - # benchx does not record PYTHONPATH, so check here that this side imports its own build. - local imported - imported=$(PYTHONPATH=$SITE python -c 'import numpy; print(numpy.__file__)') - case $imported in - "$TREE"/build-install/*) echo " $1: $imported" ;; - *) echo "compare.sh: with PYTHONPATH=$SITE, python imports $imported, not the build of $1" >&2; exit 1 ;; - esac -} +NUMPY=${NUMPY:-~/repos/numpy} +COMMITS=($(git -C "$NUMPY" rev-parse "${1:-main}^{commit}" "${2:-HEAD}^{commit}" | cut -c1-10)) # before the checkouts move HEAD +BUILDS=(build-${COMMITS[0]} build-${COMMITS[1]}) +[ $# -ge 2 ] || BUILDS[1]=build -echo "== 1. build both revisions with spin, each in its own worktree" -mkdir -p "$WORKTREES" -prepare "$BASELINE"; BASE_COMMIT=$COMMIT; BASE_TREE=$TREE; BASE_SITE=$SITE -prepare "$CONTENDER"; CONT_TREE=$TREE; CONT_SITE=$SITE +git -C "$NUMPY" diff --quiet HEAD || { echo "$NUMPY has uncommitted changes" >&2; exit 1; } +START=$(git -C "$NUMPY" symbolic-ref -q --short HEAD || git -C "$NUMPY" rev-parse HEAD) +# A checkout includes the submodules, else a build is made of the wrong sources and benchx tags the tree dirty. +checkout() { git -C "$NUMPY" checkout -q "$1"; git -C "$NUMPY" submodule update --init -q; } +trap 'checkout "$START"' EXIT -echo "== 2. alternate the sides, round by round: one bx bench call per side and round" -rm -rf out; mkdir -p out -# bx bench prints a table and some notes for every call; they go to out/bench.log, shown if a call fails. -side() { # side TREE SITE ROUND SLOT: one bx bench call - # PYTHONPATH selects the numpy that the benchmark imports. The target is the worktree: its - # benchmarks/benchmarks directory (adapter.suite) holds the suite, and its revision is recorded. - PYTHONPATH=$2 bx bench "$1" --filter "$CASES" --run-key "$RUN_KEY" --round "$3" --slot "$4" \ - --out "out/$RUN_KEY" >> out/bench.log 2>&1 || { tail -n 30 out/bench.log >&2; exit 1; } -} -for ROUND in $(seq 0 $((ROUNDS - 1))); do - side "$BASE_TREE" "$BASE_SITE" "$ROUND" $((2 * ROUND)) - side "$CONT_TREE" "$CONT_SITE" "$ROUND" $((2 * ROUND + 1)) - echo " round $ROUND: $BASELINE, then $CONTENDER" +for i in 0 1; do + checkout "${COMMITS[i]}" + (cd "$NUMPY" && spin build -C "${BUILDS[i]}") done -echo "== 3. compare: the revisions profile; the baseline is named by its commit" -bx compare --run "$RUN_KEY" --profile revisions --baseline "${BASE_COMMIT:0:10}" | tee out/compare.txt +# The sides alternate, round by round: each call has its commit checked out, and imports its build, +# the version of which (it has the commit in it) is printed. +rm -rf out +for round in 0 1 2 3 4; do + for i in 0 1; do + checkout "${COMMITS[i]}" + export PYTHONPATH=$(echo "$NUMPY/${BUILDS[i]}"-install/usr/lib/python*/site-packages) + python -c 'import numpy; print(numpy.__version__, numpy.__file__)' # which numpy this side measures + bx bench "$NUMPY" --adapter asv --suite benchmarks/benchmarks ${FILTER:+--filter "$FILTER"} \ + --run-key numpy-revisions --round $round --slot $((2 * round + i)) + done +done + +bx compare --run numpy-revisions --baseline "${COMMITS[0]}" --tables From 031cb867c0ec98c19076cdb28205298546c203bd Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Wed, 18 Nov 2026 09:18:03 +0000 Subject: [PATCH 3/4] numpy-revisions: README sample with the indented tables The sample output is regenerated from the same real v2.4.0 vs v2.4.3 run, with the per-benchmark tables indented by two spaces as `bx compare --tables` now prints them. Co-Authored-By: Claude Sonnet 5.5 --- examples/numpy-revisions/README.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/examples/numpy-revisions/README.md b/examples/numpy-revisions/README.md index f84cea0..f3cb0d1 100644 --- a/examples/numpy-revisions/README.md +++ b/examples/numpy-revisions/README.md @@ -123,23 +123,23 @@ NO CHANGE DETECTED (3) For numpy commit c5ab79c1 (5 rounds): bench_function_base.Mean.time_mean ok -======== ============= - size --------- ------------- - 1 8.21±0.2μs - 10 8.34±0.08μs - 100000 162±0.3μs -======== ============= + ======== ============= + size + -------- ------------- + 1 8.21±0.2μs + 10 8.34±0.08μs + 100000 162±0.3μs + ======== ============= For numpy commit 8bcb2e72 (5 rounds): bench_function_base.Mean.time_mean ok -======== ============= - size --------- ------------- - 1 8.12±0.2μs - 10 8.36±0.08μs - 100000 162±20μs -======== ============= + ======== ============= + size + -------- ------------- + 1 8.12±0.2μs + 10 8.36±0.08μs + 100000 162±20μs + ======== ============= No benchmark changed (the verdict is benchx's; --all lists every row). ``` From d1aa460fb2fc43b44ceec94f48c61ffa1e69a43e Mon Sep 17 00:00:00 2001 From: ev-br's Claude Code Agent Date: Wed, 18 Nov 2026 13:44:28 +0000 Subject: [PATCH 4/4] numpy-revisions: README sample from a new run, and how to pick one parameter combination - The sample output (the comparison, the table of each commit and the `--all` table) is from a new run of `./compare.sh -t 'Mean\.time_mean\(' v2.4.0 v2.4.3` with the current benchx: both builds from scratch, 5m35s with the measuring about a minute. - `-t` and a parametrized benchmark: a case is searched as `name(p0, p1, ...)`, the arguments in the order of `param_names`, each as its repr. A worked example for `Partition.time_argpartition`, which was run end to end (one case selected, the same parameters in `v2.4.0` and `v2.4.3`), and what the refusal says when the arguments are in another order or the name is misspelled. Co-Authored-By: Claude Sonnet 5.5 --- examples/numpy-revisions/README.md | 52 +++++++++++++++++++----------- 1 file changed, 34 insertions(+), 18 deletions(-) diff --git a/examples/numpy-revisions/README.md b/examples/numpy-revisions/README.md index f3cb0d1..15acb72 100644 --- a/examples/numpy-revisions/README.md +++ b/examples/numpy-revisions/README.md @@ -52,7 +52,23 @@ for `spin bench --compare`: `-t Mean.time_mean` a method, and also `time_mean_axis`, since it only searches. The `\(` above ends the name, so that only `time_mean` is selected. Give `-t` more than once to select more. Without `-t` every benchmark runs, which takes hours. A `-t` that selects nothing is refused, and the refusal lists - the benchmarks that exist. + the benchmarks whose names are closest to it (a misspelled class or method finds its benchmarks), or the first few that exist. + +**One combination of parameters.** A parametrized benchmark is searched once per combination, as +`name(p0, p1, ...)`: the arguments in the order of its `param_names`, each as its Python `repr`, so a string +is quoted and a tuple keeps its parentheses. `Partition.time_argpartition` takes `dtype, array_type, k`, so +this selects one case of it, the same in `v2.4.0` and `v2.4.3`: + +``` +$ NUMPY=~/repos/numpy ./compare.sh -t "Partition\.time_argpartition\('int16', \('sorted_block', 1000\), 10\)" v2.4.0 v2.4.3 +``` + +The parentheses are escaped because the filter is a regular expression, and the double quotes keep the shell +from reading the single ones. The arguments in another order select nothing. When a `-t` names a +parametrized benchmark and none of its combinations, the refusal says so, gives the order of the parameters +and one case as it is searched: +`bench_function_base.Partition.time_argpartition(dtype, array_type, k) is searched as +bench_function_base.Partition.time_argpartition('float64', ('random',), 10)`. **Builds.** `spin build` builds each revision in `$NUMPY`, in a build directory of its own, so both builds exist at once: a revision in `build-` (installed in `build--install`), the checked-out @@ -117,35 +133,35 @@ run numpy-revisions profile revisions baseline edd9d7c22ba2 contender 63d7b31 caveats: baseline: setup-per-call-unbatched, contender: setup-per-call-unbatched NO CHANGE DETECTED (3) - bench_function_base.Mean.time_mean [size=1] wall-time effect -0.41% noise 1.59% rounds=5 - bench_function_base.Mean.time_mean [size=10] wall-time effect +0.58% noise 0.58% rounds=5 - bench_function_base.Mean.time_mean [size=100000] wall-time effect -6.34% noise 6.65% rounds=5 + bench_function_base.Mean.time_mean [size=1] wall-time effect -0.18% noise 0.97% rounds=5 + bench_function_base.Mean.time_mean [size=10] wall-time effect -0.24% noise 0.79% rounds=5 + bench_function_base.Mean.time_mean [size=100000] wall-time effect +0.88% noise 0.83% rounds=5 For numpy commit c5ab79c1 (5 rounds): bench_function_base.Mean.time_mean ok - ======== ============= - size - -------- ------------- - 1 8.21±0.2μs - 10 8.34±0.08μs - 100000 162±0.3μs - ======== ============= + ======== ============ + size + -------- ------------ + 1 8.26±0.2μs + 10 8.33±0.1μs + 100000 162±2μs + ======== ============ For numpy commit 8bcb2e72 (5 rounds): bench_function_base.Mean.time_mean ok ======== ============= size -------- ------------- - 1 8.12±0.2μs - 10 8.36±0.08μs - 100000 162±20μs + 1 8.26±0.09μs + 10 8.41±0.09μs + 100000 164±2μs ======== ============= No benchmark changed (the verdict is benchx's; --all lists every row). ``` After the summary, `--tables` prints what `spin bench --compare` prints, laid out as asv lays it out: a -table of each commit (a value `8.21±0.2μs` per size, the `±` being the spread of the per-round +table of each commit (a value `8.26±0.2μs` per size, the `±` being the spread of the per-round medians), then a table of the changes, which has only the changed rows: here there are none. `--all` lists every row, with the noise and the verdict repeated; run by hand, `bx compare --run numpy-revisions --baseline c5ab79c14c --tables --all` ends with @@ -153,9 +169,9 @@ lists every row, with the noise and the verdict repeated; run by hand, ``` | Change | Before [c5ab79c1] | After [8bcb2e72] | Ratio | Noise | Verdict | Benchmark (Parameter) | |----------|---------------------|--------------------|---------|---------|--------------------|--------------------------------------------| -| | 8.21±0.2μs | 8.12±0.2μs | 0.99 | 1.59% | no change detected | bench_function_base.Mean.time_mean(1) | -| | 8.34±0.08μs | 8.36±0.08μs | 1.00 | 0.58% | no change detected | bench_function_base.Mean.time_mean(10) | -| | 162±0.3μs | 162±20μs | 1.00 | 6.65% | no change detected | bench_function_base.Mean.time_mean(100000) | +| | 8.26±0.2μs | 8.26±0.09μs | 1.00 | 0.97% | no change detected | bench_function_base.Mean.time_mean(1) | +| | 8.33±0.1μs | 8.41±0.09μs | 1.01 | 0.79% | no change detected | bench_function_base.Mean.time_mean(10) | +| | 162±2μs | 164±2μs | 1.01 | 0.83% | no change detected | bench_function_base.Mean.time_mean(100000) | ``` A case is a change when its effect, the mean relative difference of the paired rounds, is more than