Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or request
Description
I tried this with SciPy, comparing HEAD and main without any performance-relevant changes.
After plumbing everything in with spin and pixi, running:
pixi run bench --compare main HEAD --case linalg.Bench.time_solve
yields output with multiple problems, shown here (full output in the foldout at the end:
differs: benchmark.revision.key
caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched
...
NO CHANGE DETECTED (18)
linalg.Bench.time_solve wall-time effect -0.28% noise 1.57% rounds=3
linalg.Bench.time_solve wall-time effect -2.38% noise 3.54% rounds=3
...
INDETERMINATE (6)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
...
excluded (36)
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
Issues:
- Every case is indentified as
linalg.Bench.time_solve, so it's not clear which parametrization is reported on. - The dirty tree determination is too aggressive and too verbose. It should check for changes to files only, not for uncommitted files.
- It's unclear why there are
rounds=0results - probably those were skipped, but how does that differ from the "excluded" results below it? - The
excludedoutput is pure noise, I suggest leaving that out completely.
Details
48 result(s) -> /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96 (order sha256:2d7d43b68483)
/workspace/benchmarks/results/benchx/store.parquet: ingested 48, duplicate 0
Switched to branch 'benchx'
run bench-20261117T105803-01560d96 profile revisions baseline 976c3b526bdf contender e2f901ecec94 [local-only]
976c3b526bdf5c70720e3f770237e776b2111247: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
e2f901ecec946749d5d13df403e9fdd711649ae7: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
differs: benchmark.revision.key
caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched
NO CHANGE DETECTED (18)
linalg.Bench.time_solve wall-time effect -0.28% noise 1.57% rounds=3
linalg.Bench.time_solve wall-time effect -2.38% noise 3.54% rounds=3
linalg.Bench.time_solve wall-time effect -0.31% noise 2.64% rounds=3
linalg.Bench.time_solve wall-time effect +0.85% noise 0.54% rounds=3
linalg.Bench.time_solve wall-time effect +3.44% noise 3.01% rounds=3
linalg.Bench.time_solve wall-time effect -0.40% noise 0.78% rounds=3
linalg.Bench.time_solve wall-time effect -0.51% noise 0.42% rounds=3
linalg.Bench.time_solve wall-time effect -3.07% noise 5.44% rounds=3
linalg.Bench.time_solve wall-time effect +4.54% noise 1.75% rounds=3
linalg.Bench.time_solve wall-time effect -0.38% noise 1.74% rounds=3
linalg.Bench.time_solve wall-time effect -1.99% noise 3.81% rounds=3
linalg.Bench.time_solve wall-time effect +0.59% noise 0.66% rounds=3
linalg.Bench.time_solve wall-time effect +1.82% noise 1.58% rounds=3
linalg.Bench.time_solve wall-time effect +1.53% noise 4.39% rounds=3
linalg.Bench.time_solve wall-time effect -0.41% noise 0.38% rounds=3
linalg.Bench.time_solve wall-time effect -0.13% noise 0.40% rounds=3
linalg.Bench.time_solve wall-time effect -0.39% noise 5.34% rounds=3
linalg.Bench.time_solve wall-time effect -0.76% noise 1.06% rounds=3
INDETERMINATE (6)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
linalg.Bench.time_solve wall-time effect ? noise ? rounds=0 (insufficient-rounds)
excluded (36)
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
{'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
Results: /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96
Store: /workspace/benchmarks/results/benchx/store.parquet
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
ev-br commentedon Nov 18, 2026
Thanks Ralf!
Could you try #118? It adds the
--tablesoption to emit asv-style tables.Also, could you show how you drive
bx? I seem to be getting different outputs with benchx-dev/benchx_examples#18 (which runs a real numpy benchmark, builds numpy with spin, but the driver is plumbed out-of-tree).In particular,
The too-verbose output is probably best covered by #85, where most of the verbosity goes into a log file with controllable logging levels.
ev-br commentedon Nov 18, 2026
Also,
AFAIU, the main objection is that untracked files make the tree "dirty"? That's on purpose actually, because an untracked
bench_new.py,conftest.pyor.socan change what runs.The duplicate "dirty" warning is because numpy/scipy benchmarks are in-tree, and benchx checks both the subject and benchmarks (which can be out of tree as well).