Skip to content

--compare output needs improvement #117

Open

Description

@rgommers

I tried this with SciPy, comparing HEAD and main without any performance-relevant changes.

After plumbing everything in with spin and pixi, running:

pixi run bench --compare main HEAD --case linalg.Bench.time_solve

yields output with multiple problems, shown here (full output in the foldout at the end:

  differs: benchmark.revision.key
  caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched
...
NO CHANGE DETECTED (18)
  linalg.Bench.time_solve          wall-time  effect   -0.28%  noise   1.57%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -2.38%  noise   3.54%  rounds=3
...

INDETERMINATE (6)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
...

excluded (36)
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}

Issues:

  1. Every case is indentified as linalg.Bench.time_solve, so it's not clear which parametrization is reported on.
  2. The dirty tree determination is too aggressive and too verbose. It should check for changes to files only, not for uncommitted files.
  3. It's unclear why there are rounds=0 results - probably those were skipped, but how does that differ from the "excluded" results below it?
  4. The excluded output is pure noise, I suggest leaving that out completely.
Details
48 result(s) -> /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96  (order sha256:2d7d43b68483)
/workspace/benchmarks/results/benchx/store.parquet: ingested 48, duplicate 0
Switched to branch 'benchx'
run bench-20261117T105803-01560d96  profile revisions  baseline 976c3b526bdf  contender e2f901ecec94  [local-only]
  976c3b526bdf5c70720e3f770237e776b2111247: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
  e2f901ecec946749d5d13df403e9fdd711649ae7: observed env: MKL_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1
  differs: benchmark.revision.key
  caveats: baseline: dirty-benchmark-tree, baseline: dirty-subject-tree, baseline: setup-per-call-unbatched, contender: dirty-benchmark-tree, contender: dirty-subject-tree, contender: setup-per-call-unbatched

NO CHANGE DETECTED (18)
  linalg.Bench.time_solve          wall-time  effect   -0.28%  noise   1.57%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -2.38%  noise   3.54%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.31%  noise   2.64%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +0.85%  noise   0.54%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +3.44%  noise   3.01%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.40%  noise   0.78%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.51%  noise   0.42%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -3.07%  noise   5.44%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +4.54%  noise   1.75%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.38%  noise   1.74%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -1.99%  noise   3.81%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +0.59%  noise   0.66%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +1.82%  noise   1.58%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   +1.53%  noise   4.39%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.41%  noise   0.38%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.13%  noise   0.40%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.39%  noise   5.34%  rounds=3
  linalg.Bench.time_solve          wall-time  effect   -0.76%  noise   1.06%  rounds=3

INDETERMINATE (6)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)
  linalg.Bench.time_solve          wall-time  effect        ?  noise       ?  rounds=0  (insufficient-rounds)

excluded (36)
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27contig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27fcontig%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%221000%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-0'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-2'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-4'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-1'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-3'}, 'reason': 'skipped: harness.skipped'}
  {'result': {'producer': 'benchx/asv-adapter', 'ingest_key': 'bench-20261117T105803-01560d96:linalg.Bench.time_solve/%7B%22contiguous%22%3A%22%27nocont%27%22%2C%22module%22%3A%22%27numpy%27%22%2C%22size%22%3A%22500%22%7D:wall-time:slot-5'}, 'reason': 'skipped: harness.skipped'}
Results: /workspace/benchmarks/results/benchx/bench-20261117T105803-01560d96
Store: /workspace/benchmarks/results/benchx/store.parquet

Activity

  1. ev-br commented on Nov 18, 2026

    @ev-br
    Collaborator

    Thanks Ralf!

    Could you try #118? It adds the --tables option to emit asv-style tables.

    Also, could you show how you drive bx? I seem to be getting different outputs with benchx-dev/benchx_examples#18 (which runs a real numpy benchmark, builds numpy with spin, but the driver is plumbed out-of-tree).

    In particular,

    • for your point 1, I do get the parameter values,
    12 result(s) -> /home/ev-br/proj/quansight/benchx_user/examples/numpy-revisions/out/numpy-revisions  (order sha256:9accb515aad0)
    /home/ev-br/proj/quansight/benchx_user/examples/numpy-revisions/out/store.parquet: ingested 12, duplicate 0
    run numpy-revisions  profile revisions  baseline fdcebd8854a7  contender f72cb64e8b4f
      differs: benchmark.revision.key
      caveats: baseline: setup-per-call-unbatched, contender: setup-per-call-unbatched
    
    NO CHANGE DETECTED (6)
      bench_function_base.Mean.time_mean [size=1] wall-time  effect   +2.46%  noise   1.00%  rounds=5
      bench_function_base.Mean.time_mean [size=10] wall-time  effect   +0.14%  noise   0.44%  rounds=5
      bench_function_base.Mean.time_mean [size=100000] wall-time  effect   -0.32%  noise   0.78%  rounds=5
      bench_function_base.Mean.time_mean_axis [size=1] wall-time  effect   +0.14%  noise   0.89%  rounds=5
      bench_function_base.Mean.time_mean_axis [size=10] wall-time  effect   -0.43%  noise   0.56%  rounds=5
      bench_function_base.Mean.time_mean_axis [size=100000] wall-time  effect   -0.06%  noise   1.37%  rounds=5
    

    The too-verbose output is probably best covered by #85, where most of the verbosity goes into a log file with controllable logging levels.

  2. ev-br commented on Nov 18, 2026

    @ev-br
    Collaborator

    Also,

    1. The dirty tree determination is too aggressive and too verbose. It should check for changes to files only, not for uncommitted files.

    AFAIU, the main objection is that untracked files make the tree "dirty"? That's on purpose actually, because an untracked bench_new.py, conftest.py or .so can change what runs.

    The duplicate "dirty" warning is because numpy/scipy benchmarks are in-tree, and benchx checks both the subject and benchmarks (which can be out of tree as well).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions