Repository navigation
Conversation
asv is timed by pyperf and takes the same precision, but `bx bench --adapter asv` was refused when the file's `adapter.name` was `pyperf`, so one benchx.conf.json could not hold the timing for both kinds of suite. Harnesses now have a precision family: pyperf and asv (which subclasses it) are one, google-benchmark is another. `--adapter` may name any adapter of the file's block family. Across families it is still refused, with a message that says so. No schema change: `adapter.name` stays required, and only its description is corrected (it did not mention asv). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`bx compare --tables` prints, after the usual summary, what `spin bench --compare` (asv) prints: a table of each side's results, then a table of the changes with Ratio, Noise and the comparator's verdict. As with `asv continuous`, only the changed rows are listed unless `--all` is given. The layout and the number formatting follow asv's (`format_text_table`: every cell centered; `tabulate`'s github format; `human_time` and `human_float` for values). The `±` is benchx's own: the standard deviation of the per-round medians, and a side's value is their median. The per-side numbers are recomputed from the run's results for display, so the comparison document and its schema are unchanged. `--tables` is a text view: it is refused together with `--json`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spaces The grids sit two spaces in under the benchmark's title line. The title, the one-line value of a benchmark without parameters, and the change table are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… compared
`bx compare` returned no verdict at all when the matched facts of one benchmark
disagreed between the sides. The `sides-agree` check is about one unit, but any
disagreement made `compare()` return a document with no units, so a benchmark that
raised on one side (it reports no harness version or runtime) stopped `time_mean`
from being compared, though it ran fine on both sides. `equal-attempts` already
dropped only its unit, and the design says a failed invariant leaves "no verdicts for
the units it affects".
- `sides-agree` now drops only the unit that disagrees: it is recorded in
`failed_invariants` with its workload, and the other units are compared. The
invariants about the whole run (the sides, their tree ids, the alternation, the
benchmark file) still leave no unit compared.
- The matched facts are those of the attempts that succeeded. A failed attempt reports
none of them: it is excluded and listed, and its benchmark is `indeterminate`
(`insufficient-rounds`). A benchmark edited between the revisions, so that its
benchmark version differs on two successful sides, still fails `sides-agree`, for
itself only.
- A unit's `unmatched_differences` (the `differs:` line) also counts successful
attempts only, so the facts a failed attempt did not report are not differences.
No schema change. `comparator.md` and `prototype-design.md` say which invariants affect
the run and which one unit.
An example output:
```
bench_function_base.Mean.time_mean_axis error
======== =======
size
-------- -------
1 error
10 error
100000 error
======== =======
| Change | Before [6e1aca7e] | After [7b3b54e0] | Ratio | Noise | Verdict | Benchmark (Parameter) |
|----------|---------------------|--------------------|---------|---------|-------------------------------------|-------------------------------------------------|
| | 7.04±0.1μs | 7.01±0.1μs | 1.00 | 1.33% | no change detected | bench_function_base.Mean.time_mean(1) |
| | 7.21±0.1μs | 7.21±0.2μs | 1.00 | 1.06% | no change detected | bench_function_base.Mean.time_mean(10) |
| | 106±0.8μs | 106±0.3μs | 1.00 | 0.34% | no change detected | bench_function_base.Mean.time_mean(100000) |
| ? | 7.71±0.05μs | error | n/a | n/a | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(1) |
| ? | 7.91±0.1μs | error | n/a | n/a | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(10) |
| ? | 108±1μs | error | n/a | n/a | indeterminate (insufficient-rounds) | bench_function_base.Mean.time_mean_axis(100000) |
```
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Collaborator
Author
|
One other change: a failing benchmark does not fail whole the comparison: in benchx-dev/benchx_examples#18, With ba72add, the output is |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses a part of gh-117; a driver is at benchx-dev/benchx_examples#18
Add
asv-style tables for the output ofbx --compare: with the new option,--tableit additionally emits:@MaanasArora two asks:
could please take a quick peek? You're working on comparisons, and while this is mainly about the output, I'd rather make sure I'm not messing with your work!
One larger item is that to construct the comparison documents do not carry enough information to construct these tables from the store. This PR recomputes the values at display time, but maybe it's useful to add them to the documents themselves? That would require updating the schema though. Claude-generated discussion is under the fold below.
Details
Discussion: recomputing at display time, or putting the numbers in the comparison document
What the tables need that the comparison document does not carry: each side's value and
±, itsunit, its status when it has no value, and its commit. The document has
reference_estimatefor thebaseline only. (
effect,noiseand the verdict are in it, so the Noise and Verdict columns need norecomputation.) Two ways to get the rest.
A. Recompute at display time (chosen for now, decisions 1 and 6).
tables.summarize(doc, documents)reads the run's results from the store, rebuilds the projection from
doc["varying"], assigns eachresult to a side, groups by workload variant and quantity, and takes the median and sd of the per-round
medians.
version question arises, and the feature can be dropped or reworked without a migration. It is a
text view and stays one.
compare(): matching a result to a side byaxis[0], the workload-variant key,the sort order (reused through
compare._unit_orderand_workload). A change to the comparator'sgrouping has to be made twice, or the tables drift from the verdicts.
varyingstring (provenance.labels.NAME,coordinates.workload.parameters.NAME), which is lossy if a name ever contains a..--tablesreads the store in measure mode, andtables.rendercannot be given a document alone (for example one read from--jsonoutput or a CIartifact).
comparator's
reference_estimateuses only the rounds both sides completed. They agree when noround failed, and can differ in the third decimal when one did.
Ratiois After over Before of theprinted values,
effectthe mean of the per-round ratios, so they differ slightly too (documented).B. Put them in the document (the alternative; what I first recommended).
_unitalready holds eachside's per-round medians and the pairs, so it would add
side_estimates: {baseline, contender}, each{estimate, spread, rounds, status}, plus the quantity'sunit, andsides.<role>.revisionfor thecommit.
tables.render(doc)would then be a pure function of the document.by construction; the numbers are in
--json; the renderer needs no store, no projection and nosummarize, and is testable from a document alone. It keeps the comparator's contract(
comparator.md§5): documents are derived and recomputed on demand from the recorded inputs.runUnithasunevaluatedProperties: false, andsidesis closed too, so it is a schema change:optional properties in
schema.json, therun.jsonexample (the history unit is a separatedefinition),
docs/schemas/comparison-document.md, and tests intest_comparator.py.runUnitandsidesare closed, so a document with the new fields does not validateagainst 0.1.0: this cannot be an amendment of 0.1.0, it is a new version. The scheme the docs state
(
docs/design/runner-schema.md§7: semver in the directory path, minor versions only add optionalfields) is written for the runner's schemas and the result schema, not for the comparison document,
which has the same layout. Read that way, this is
0.2.0: a new directory with the schema andexamples, a new
COMPARISON_DOCUMENTversion string, and every consumer that checks the string seesa new value. Whether the comparison document follows that scheme is for the project to say.
kept, or versioned, even though
±is deliberately benchx's own and not asv's.Why A for now. The tables are a view, the numbers' definitions (decision 2) are still the
author's to change after seeing them used, and a view should not be what forces a schema version
decision. A costs a duplicated grouping and a read of the store; B costs a new schema version.
When to move to B. When a second consumer needs the per-side numbers (JSON, a CI gate, another
renderer); or when the tables and the verdicts are seen to disagree (a drift of the kind above); or when
the grouping in
compare()changes. Then the steps are: add the fields in_unitandsides; settle theversion question (a new minor version, if the comparison document follows §7); update
schema.json, the examples,docs/schemas/comparison-document.mdand
test_comparator.py; changetables.renderto take the document alone; deletesummarize,_sideand_projection; and drop thedocumentsargument incli._show. The golden tests of thetables stay as they are, since the output does not change.