Skip to content

WIP: dogfood the comparisons output - #18

Open
ev-br wants to merge 3 commits into
benchx-dev:mainfrom
ev-br:numpy-revisions-asv
Open

ev-br wants to merge 3 commits into
benchx-dev:mainfrom
ev-br:numpy-revisions-asv

Conversation

@ev-br

@ev-br ev-br commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

ev-br and others added 3 commits November 17, 2026 19:15
…nchmark

UC-03 for NumPy. Two revisions are built with `spin`, each in a git worktree of
its own, and one of NumPy's own asv benchmarks (bench_function_base.Mean.time_mean,
unchanged) is run on each, alternated round by round, then compared with
`bx compare --run --profile revisions`.

The sides are selected by a per-side PYTHONPATH, one `bx bench` call at a time,
and compare.sh checks that each side imports its own build: the asv adapter does
not put the target on the benchmark's sys.path, and benchx does not record
PYTHONPATH. The README says so.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…onf for asv and pyperf

compare.sh no longer makes a git worktree per revision. Each revision is checked out
(with its submodules) in the NumPy checkout and built there by `spin build -C`: the
checked-out code in `build`, any other commit in `build-<short sha>`. The checkout is
restored when the script ends, and the script refuses a tree with uncommitted changes.
It takes `[-t TEST]... [BASELINE [CONTENDER]]` as `spin bench --compare` does, with no
default benchmark, and prints which numpy each call imports.

The comparison ends with `bx compare --tables`: a table of each commit and a table of
the changes, as `asv continuous` prints them.

The script passes `--adapter asv --suite benchmarks/benchmarks`, and benchx.conf.json
has `adapter.name: pyperf` and only the timing, so the same file serves a pyperf suite.

This needs a benchx newer than 0.1.0: `--tables`, and a pyperf conf serving
`--adapter asv`, are not in the PyPI release.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The sample output is regenerated from the same real v2.4.0 vs v2.4.3 run, with the
per-benchmark tables indented by two spaces as `bx compare --tables` now prints them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant