Skip to content

UI: add asv-stype table for comparisons - #118

Open
ev-br wants to merge 4 commits into
benchx-dev:mainfrom
ev-br:asv-style-tables
Open

ev-br wants to merge 4 commits into
benchx-dev:mainfrom
ev-br:asv-style-tables

Conversation

@ev-br

@ev-br ev-br commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Addresses a part of gh-117; a driver is at benchx-dev/benchx_examples#18

Add asv-style tables for the output of bx --compare: with the new option, --table it additionally emits:

$ NUMPY=~/repos/numpy ./compare.sh -t Mean.time_mean HEAD^

....

For numpy commit f9f1772c (5 rounds):
bench_function_base.Mean.time_mean  ok
======== =============
  size                
-------- -------------
   1      6.91±0.03μs 
   10     7.10±0.06μs 
 100000     106±1μs   
======== =============
bench_function_base.Mean.time_mean_axis  ok
======== =============
  size                
-------- -------------
   1      7.55±0.09μs 
   10     7.74±0.04μs 
 100000     107±2μs   
======== =============

For numpy commit 6e1aca7e (5 rounds):
bench_function_base.Mean.time_mean  ok
======== =============
  size                
-------- -------------
   1       7.06±0.1μs 
   10     7.07±0.08μs 
 100000    106±0.7μs  
======== =============
bench_function_base.Mean.time_mean_axis  ok
======== =============
  size                
-------- -------------
   1       7.53±0.1μs 
   10     7.68±0.06μs 
 100000     107±2μs   
======== =============

| Change   | Before [f9f1772c]   | After [6e1aca7e]   |   Ratio |   Noise | Verdict            | Benchmark (Parameter)                           |
|----------|---------------------|--------------------|---------|---------|--------------------|-------------------------------------------------|
|          | 6.91±0.03μs         | 7.06±0.1μs         |    1.02 |   1.00% | no change detected | bench_function_base.Mean.time_mean(1)           |
|          | 7.10±0.06μs         | 7.07±0.08μs        |    1.00 |   0.44% | no change detected | bench_function_base.Mean.time_mean(10)          |
|          | 106±1μs             | 106±0.7μs          |    1.00 |   0.78% | no change detected | bench_function_base.Mean.time_mean(100000)      |
|          | 7.55±0.09μs         | 7.53±0.1μs         |    1.00 |   0.89% | no change detected | bench_function_base.Mean.time_mean_axis(1)      |
|          | 7.74±0.04μs         | 7.68±0.06μs        |    0.99 |   0.56% | no change detected | bench_function_base.Mean.time_mean_axis(10)     |
|          | 107±2μs             | 107±2μs            |    1.00 |   1.37% | no change detected | bench_function_base.Mean.time_mean_axis(100000) |

@MaanasArora two asks:

  • could please take a quick peek? You're working on comparisons, and while this is mainly about the output, I'd rather make sure I'm not messing with your work!

  • One larger item is that to construct the comparison documents do not carry enough information to construct these tables from the store. This PR recomputes the values at display time, but maybe it's useful to add them to the documents themselves? That would require updating the schema though. Claude-generated discussion is under the fold below.

Details

Discussion: recomputing at display time, or putting the numbers in the comparison document

What the tables need that the comparison document does not carry: each side's value and ±, its
unit, its status when it has no value, and its commit. The document has reference_estimate for the
baseline only. (effect, noise and the verdict are in it, so the Noise and Verdict columns need no
recomputation.) Two ways to get the rest.

A. Recompute at display time (chosen for now, decisions 1 and 6). tables.summarize(doc, documents)
reads the run's results from the store, rebuilds the projection from doc["varying"], assigns each
result to a side, groups by workload variant and quantity, and takes the median and sd of the per-round
medians.

  • For: nothing else changes. The schema, its examples, its docs and the JSON output stay as they are, no
    version question arises, and the feature can be dropped or reworked without a migration. It is a
    text view and stays one.
  • Against:
    • It repeats part of compare(): matching a result to a side by axis[0], the workload-variant key,
      the sort order (reused through compare._unit_order and _workload). A change to the comparator's
      grouping has to be made twice, or the tables drift from the verdicts.
    • It rebuilds the projection from the dotted varying string (provenance.labels.NAME,
      coordinates.workload.parameters.NAME), which is lossy if a name ever contains a ..
    • It needs the stored results as well as the document: --tables reads the store in measure mode, and
      tables.render cannot be given a document alone (for example one read from --json output or a CI
      artifact).
    • The numbers are not the comparator's. A side's value uses its own successful rounds; the
      comparator's reference_estimate uses only the rounds both sides completed. They agree when no
      round failed, and can differ in the third decimal when one did. Ratio is After over Before of the
      printed values, effect the mean of the per-round ratios, so they differ slightly too (documented).
    • JSON consumers (a CI gate, another renderer) do not get the per-side numbers.

B. Put them in the document (the alternative; what I first recommended). _unit already holds each
side's per-round medians and the pairs, so it would add side_estimates: {baseline, contender}, each
{estimate, spread, rounds, status}, plus the quantity's unit, and sides.<role>.revision for the
commit. tables.render(doc) would then be a pure function of the document.

  • For: one place decides which results count and how they are summarized, so verdicts and tables agree
    by construction; the numbers are in --json; the renderer needs no store, no projection and no
    summarize, and is testable from a document alone. It keeps the comparator's contract
    (comparator.md §5): documents are derived and recomputed on demand from the recorded inputs.
  • Against:
    • runUnit has unevaluatedProperties: false, and sides is closed too, so it is a schema change:
      optional properties in schema.json, the run.json example (the history unit is a separate
      definition), docs/schemas/comparison-document.md, and tests in test_comparator.py.
    • The version. runUnit and sides are closed, so a document with the new fields does not validate
      against 0.1.0: this cannot be an amendment of 0.1.0, it is a new version. The scheme the docs state
      (docs/design/runner-schema.md §7: semver in the directory path, minor versions only add optional
      fields) is written for the runner's schemas and the result schema, not for the comparison document,
      which has the same layout. Read that way, this is 0.2.0: a new directory with the schema and
      examples, a new COMPARISON_DOCUMENT version string, and every consumer that checks the string sees
      a new value. Whether the comparison document follows that scheme is for the project to say.
    • Once published, the fields are a contract: their meaning (median of medians, sd) has to be
      kept, or versioned, even though ± is deliberately benchx's own and not asv's.

Why A for now. The tables are a view, the numbers' definitions (decision 2) are still the
author's to change after seeing them used, and a view should not be what forces a schema version
decision. A costs a duplicated grouping and a read of the store; B costs a new schema version.

When to move to B. When a second consumer needs the per-side numbers (JSON, a CI gate, another
renderer); or when the tables and the verdicts are seen to disagree (a drift of the kind above); or when
the grouping in compare() changes. Then the steps are: add the fields in _unit and sides; settle the
version question (a new minor version, if the comparison document follows §7); update schema.json, the examples, docs/schemas/comparison-document.md
and test_comparator.py; change tables.render to take the document alone; delete summarize,
_side and _projection; and drop the documents argument in cli._show. The golden tests of the
tables stay as they are, since the output does not change.

ev-br's Claude Code Agent and others added 3 commits November 18, 2026 08:57
asv is timed by pyperf and takes the same precision, but `bx bench --adapter asv`
was refused when the file's `adapter.name` was `pyperf`, so one benchx.conf.json
could not hold the timing for both kinds of suite.

Harnesses now have a precision family: pyperf and asv (which subclasses it) are
one, google-benchmark is another. `--adapter` may name any adapter of the file's
block family. Across families it is still refused, with a message that says so.

No schema change: `adapter.name` stays required, and only its description is
corrected (it did not mention asv).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`bx compare --tables` prints, after the usual summary, what `spin bench --compare`
(asv) prints: a table of each side's results, then a table of the changes with
Ratio, Noise and the comparator's verdict. As with `asv continuous`, only the
changed rows are listed unless `--all` is given.

The layout and the number formatting follow asv's (`format_text_table`: every
cell centered; `tabulate`'s github format; `human_time` and `human_float` for
values). The `±` is benchx's own: the standard deviation of the per-round
medians, and a side's value is their median.

The per-side numbers are recomputed from the run's results for display, so the
comparison document and its schema are unchanged. `--tables` is a text view:
it is refused together with `--json`.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spaces

The grids sit two spaces in under the benchmark's title line. The title, the
one-line value of a benchmark without parameters, and the change table are
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… compared

`bx compare` returned no verdict at all when the matched facts of one benchmark
disagreed between the sides. The `sides-agree` check is about one unit, but any
disagreement made `compare()` return a document with no units, so a benchmark that
raised on one side (it reports no harness version or runtime) stopped `time_mean`
from being compared, though it ran fine on both sides. `equal-attempts` already
dropped only its unit, and the design says a failed invariant leaves "no verdicts for
the units it affects".

- `sides-agree` now drops only the unit that disagrees: it is recorded in
  `failed_invariants` with its workload, and the other units are compared. The
  invariants about the whole run (the sides, their tree ids, the alternation, the
  benchmark file) still leave no unit compared.
- The matched facts are those of the attempts that succeeded. A failed attempt reports
  none of them: it is excluded and listed, and its benchmark is `indeterminate`
  (`insufficient-rounds`). A benchmark edited between the revisions, so that its
  benchmark version differs on two successful sides, still fails `sides-agree`, for
  itself only.
- A unit's `unmatched_differences` (the `differs:` line) also counts successful
  attempts only, so the facts a failed attempt did not report are not differences.

No schema change. `comparator.md` and `prototype-design.md` say which invariants affect
the run and which one unit.

An example output:

```
bench_function_base.Mean.time_mean_axis  error
  ======== =======
    size
  -------- -------
     1      error
     10     error
   100000   error
  ======== =======

| Change   | Before [6e1aca7e]   | After [7b3b54e0]   | Ratio   | Noise   | Verdict                             | Benchmark (Parameter)                           |
|----------|---------------------|--------------------|---------|---------|-------------------------------------|-------------------------------------------------|
|          | 7.04±0.1μs          | 7.01±0.1μs         | 1.00    | 1.33%   | no change detected                  | bench_function_base.Mean.time_mean(1)           |
|          | 7.21±0.1μs          | 7.21±0.2μs         | 1.00    | 1.06%   | no change detected                  | bench_function_base.Mean.time_mean(10)          |
|          | 106±0.8μs           | 106±0.3μs          | 1.00    | 0.34%   | no change detected                  | bench_function_base.Mean.time_mean(100000)      |
| ?        | 7.71±0.05μs         | error              | n/a     | n/a     | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(1)      |
| ?        | 7.91±0.1μs          | error              | n/a     | n/a     | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(10)     |
| ?        | 108±1μs             | error              | n/a     | n/a     | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(100000) |
```

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br

ev-br commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

One other change: a failing benchmark does not fail whole the comparison: in benchx-dev/benchx_examples#18, $ NUMPY=~/repos/numpy ./compare.sh -t Mean.time_mean main selects two benchmarks, time_mean and time_mean_axis. If I edit the second one to raise an error, and I expect the comparison to tell me something about the first one.

With ba72add, the output is

bench_function_base.Mean.time_mean_axis  error
  ======== =======
    size
  -------- -------
     1      error
     10     error
   100000   error
  ======== =======

| Change   | Before [6e1aca7e]   | After [7b3b54e0]   | Ratio   | Noise   | Verdict                             | Benchmark (Parameter)                           |
|----------|---------------------|--------------------|---------|---------|-------------------------------------|-------------------------------------------------|
|          | 7.04±0.1μs          | 7.01±0.1μs         | 1.00    | 1.33%   | no change detected                  | bench_function_base.Mean.time_mean(1)           |
|          | 7.21±0.1μs          | 7.21±0.2μs         | 1.00    | 1.06%   | no change detected                  | bench_function_base.Mean.time_mean(10)          |
|          | 106±0.8μs           | 106±0.3μs          | 1.00    | 0.34%   | no change detected                  | bench_function_base.Mean.time_mean(100000)      |
| ?        | 7.71±0.05μs         | error              | n/a     | n/a     | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(1)      |
| ?        | 7.91±0.1μs          | error              | n/a     | n/a     | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(10)     |
| ?        | 108±1μs             | error              | n/a     | n/a     | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(100000) |

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant