Skip to content
20 changes: 17 additions & 3 deletions docs/benchx.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,15 @@ bx compare BASELINE CONTENDER --profile revisions|environments
--suite NAME [--filter REGEX] [--quantity wall-time]
--rounds R [-e NAME=VALUE ...] [--repetitions N] [--min-time S]
[--project P] [--label NAME=BASE,CONTENDER]
[--run-key KEY] [--out DIR] [--no-build] [--k 3] [--json]
[--run-key KEY] [--out DIR] [--no-build] [--k 3] [--json | --tables [--all]]

# measure mode, one target: alternate the two values of a benchmark parameter
bx compare TARGET --profile variants --adapter pyperf|asv --suite NAME --rounds R
[--vary NAME] [--baseline VALUE] [--filter REGEX] [...] # the options above, except --label

# read mode: compare results already measured
bx compare --run KEY --profile revisions|environments|variants --baseline VALUE
[--label NAME] [--vary NAME] [--results DIR] [--k 3] [--json]
[--label NAME] [--vary NAME] [--results DIR] [--k 3] [--json | --tables [--all]]

# low-level command: run a handwritten work oreder
bx run <order.json> [--out DIR] [--no-ingest]
Expand Down Expand Up @@ -183,7 +183,12 @@ neither asv nor asv_runner is used. PATH is a directory holding `asv.conf.json`
`benchmark_dir` is the suite) or the benchmark directory itself, and its checkout is
the target (`bx bench . --adapter asv --suite benchmarks` is the same order).
`--filter` is asv's `--bench`: searched in the name, and in `name(p0, p1)` for each
parameter combination; a CASE is an exact name, `name(p0, p1)` for a combination. `adapters/asv_runtime.py` discovers
parameter combination, the arguments in the order of `param_names`, each as its `repr`; a CASE is an exact name,
`name(p0, p1)` for a combination. A filter that names a parametrized benchmark and none of its combinations is
refused with that order and one case as it is searched. The runner records the order in
`provenance.info.parameter_names` (the store keeps a document with its keys sorted), and `bx bench`, `bx head --table`
and `bx compare` show the parameters in it, so what they print can be given to `--filter`; results made before have
no such list and show them sorted. `adapters/asv_runtime.py` discovers
the benchmarks as asv does (names, `params`, `setup`/`teardown`/`setup_cache`, skips,
and asv's `version` hash all agree with asv_runner's; a test checks it). The
benchmarks run in the `python` first on `PATH`, which must import pyperf and the
Expand Down Expand Up @@ -263,6 +268,15 @@ every result is pyperf. Results also record the installed project's version
(`side-failed`). `--results` compares result files through a throwaway
store: the ad hoc path. Condition measurements (`measurement.kind:
condition`) are not compared, and deterministic quantities are deferred.
- **The tables** (`--tables`, `benchx/tables.py`) print, after the usual summary, what
`asv continuous` prints, laid out and rounded as asv does: a table of each side (a value
`4.71±0.08μs` per parameter combination, every cell centered), then one of the changes
(Change, Before, After, Ratio, Noise, Verdict, Benchmark), only the changed rows unless `--all`.
They are recomputed for display from the run's results; the comparison document is the
same. A side's value is the median of its per-round medians and its `±` their standard
deviation (not asv's `±`); `Change` and `Verdict` are the comparator's verdict (`+`
regressed, `-` improved), `Noise` the one it compares the effect with (a change is more than *k*
times it), and `Ratio` is After over Before. A text view: not with `--json`.

## Run it

Expand Down
14 changes: 9 additions & 5 deletions docs/design/comparator.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,11 +136,15 @@ side are equal, and sides alternate by `procedure.slot`; under `revisions`,
that each side has exactly one tree id and the two differ; under `revisions`
and `variants`, that a separately recorded benchmark has one `benchmark_tree`
across both sides and all rounds (`one-benchmark-tree`, schema §5.5). A side that is not clean does
not fail the run; it makes the comparison local-only (UC-03 §10). A run that
fails an invariant is not compared: the comparison document records the failed invariant
and contains no verdicts for the units it affects.

Excluded points and their reasons remain visible in the output. Changes in
not fail the run; it makes the comparison local-only (UC-03 §10). An invariant of the
run (the sides, their tree ids, the alternation, the benchmark file) that fails leaves no unit compared. One of a
single unit (the matched facts agree, attempt counts are equal) takes that unit out and the others are compared: the
document records the failed invariant with its workload, and has no verdict for that unit. The matched facts are
those of the attempts that succeeded; a failed attempt reports none of them, and is excluded and listed.

Excluded points and their reasons remain visible in the output: the reason of an excluded result ends with what the
attempt said when it said something (a failed pyperf case's exception line, an asv benchmark's skip reason), and
the rendered listing is one line per reason with a count. Changes in
facts the projection ignores do not split a series by themselves, but differences between
the target and its reference are carried as caveats so that a kernel, load, or
other recorded condition is not hidden from the reader.
Expand Down
12 changes: 8 additions & 4 deletions docs/design/prototype-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ bx series [--workload … --quantity …] # list series
bx history <series> # one series' points (a query)
bx head [-n 10] [--project P] [--workload W] [--json|--table] # the latest results, newest first
bx compare --run KEY --profile revisions|environments|variants --baseline VALUE
[--label NAME] [--vary NAME] [--results DIR] [--k 3] [--min-rounds 3] [--json]
[--label NAME] [--vary NAME] [--results DIR] [--k 3] [--min-rounds 3] [--json | --tables [--all]]
# read mode: compare results already measured
bx target describe|prepare TARGET # a target's description (a provider's answer, or a build directory's sidecar)
bx compare BASELINE CONTENDER --profile revisions|environments
Expand All @@ -33,7 +33,7 @@ bx compare BASELINE CONTENDER --profile revisions|environments
[--min-time S]
[--project P | --no-project] [--label NAME=BASE,CONTENDER]
[--run-key KEY] [--out DIR] [--no-build] [--source-uri URI]
[--k 3] [--min-rounds 3] [--json]
[--k 3] [--min-rounds 3] [--json | --tables [--all]]
# measure mode: run both sides, then compare
bx compare TARGET --profile variants [--vary NAME] [--baseline VALUE]
--suite NAME --rounds R [--filter REGEX] [...] # the options above, except --label
Expand Down Expand Up @@ -270,7 +270,10 @@ criterion 1 holds by construction.

`--json` emits this; the default output renders it as the reviewer table
(regressed, improved, no change detected, and indeterminate sections, then
the missing, failed-invariant, and excluded listings with counts). In any
the missing, failed-invariant, and excluded listings with counts, in words: a failed invariant names the unit
it affects, units that failed alike are one line, and results excluded for the same reason are one line with
their number, the reason carrying what the failed attempt said, such as `error: harness.error: ValueError: …`;
which results they are is in `--json`). In any
profile, a comparison in which either side is not `clean` has `local_only:
true` (schema §5.5), is headed `[local-only]`, and names the sides that are not
clean on the `caveats:` line; the `revisions` profile fails a side with no
Expand Down Expand Up @@ -514,7 +517,8 @@ matched facts agree (`sides-agree`) and the invariants the profile names over
the run: exactly two sides, which for `environments` differ in more than their
label, equal attempt counts per side, and sides alternating by
`procedure.slot`. Results the order did not ask for are not compared. A failed invariant is recorded and the affected units get no
verdict; under `revisions` the same happens to a side with no tree id (the
verdict (`sides-agree` and `equal-attempts` affect one unit, and the others are still compared; the rest
affect every unit; `sides-agree` looks at the attempts that succeeded); under `revisions` the same happens to a side with no tree id (the
`sides` invariant), to a tree edited during the run (a third side: `two-sides`),
and to a benchmark recorded apart from the subject that is not one file on
both sides and in every round (`one-benchmark-tree`, schema §5.5), as it does
Expand Down
2 changes: 1 addition & 1 deletion src/benchx/adapters/asv.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ def list_cases(runnable, case_filter, env, timeout=None) -> list[str]:
for case, index in benchx_params.expand(info["name"], info["params"], case_filter)]
if not selected:
if benchx_params.matches_name_only(info["name"], info["params"], case_filter):
unmatched.append(info["name"])
unmatched.append(info)
continue
if info["type"] != "time":
other[info["type"]] = other.get(info["type"], 0) + 1
Expand Down
63 changes: 48 additions & 15 deletions src/benchx/adapters/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,15 @@
attempt. For a harness that repeats at several levels.
"""

import difflib
import os
import re
import signal
import subprocess
import textwrap

from ..core import Refusal, Rejected
from .benchx_params import combination
from .benchx_params import combination, search_name


class Unsupported(Rejected):
Expand All @@ -97,25 +100,55 @@ def parametrized_workload(case, runnable) -> tuple[str, dict]:
return name, dict(zip(info["param_names"], combination(info["params"], index)))


def name_only_note(names) -> str:
"""For a refusal: the filter matched parametrized benchmarks by name, but not their
arguments (see `benchx_params.matches_name_only`)."""
return (f"the filter matches the name of a parametrized benchmark ({', '.join(names[:3])}) but "
"not its arguments, and those are matched as name(p0, p1)")


def no_cases(where, case_filter, available=(), notes=(), limit=10) -> Unsupported:
def name_only_note(found) -> str:
"""For a refusal: the filter is about parametrized benchmarks (`found`, their listing entries)
but matches none of their arguments (see `benchx_params.matches_name_only`). Says how a case
is searched, with the first benchmark's parameters, in order, and one case as it is searched."""
first = found[0]
names = ", ".join(info["name"] for info in found[:3])
return "\n".join([
f"the filter matches the name of a parametrized benchmark ({names}) but not its arguments.",
"They are searched as name(p0, p1), each as its repr, in the order of param_names:",
f" {first['name']}({', '.join(first['param_names'])})",
"for example:",
f" {search_name(first['name'], first['params'], 0)}"])


def similar(case_filter, names, limit=5, cutoff=0.75) -> list[str]:
"""The names a filter may have meant: those that have, as a dotted part (module, class, method), the
most of its words, or words close to them, then the closest first. A word is a run of letters, digits
and `_` of three characters or more, so an escape such as `\\d` says nothing, and the case of a letter
is not a difference. A filter that names a module and misspells its class thus finds that class's
benchmarks, and not the whole module."""
words = [w.lower() for w in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", case_filter or "") if len(w) >= 3]
scored = []
for name in names:
parts = name.lower().split(".")
best = (max(difflib.SequenceMatcher(None, word, part).ratio() for part in parts) for word in words)
hits = [score for score in best if score >= cutoff]
if hits:
scored.append((-len(hits), -sum(hits), name))
scored.sort()
return [name for count, _, name in scored[:limit] if count == scored[0][0]]


def no_cases(where, case_filter, available=(), notes=(), limit=5) -> Unsupported:
"""The refusal for a plan with no case in it. Nothing would run, and a filter with a
typo must not end in `0 result(s)` and a success status. `available` are the names
that could have been selected (the first `limit` are shown) and `notes` say what else
the adapter knows about why the plan is empty."""
head = f"filter {case_filter!r} selects no case in {where}" if case_filter else f"no case to run in {where}"
parts = [head]
the adapter knows about why the plan is empty, each on its own lines (a note may have several).
The last line is the names that are similar to the filter (`similar`), else the first available.
The filter is shown as given, not as a Python string, so its backslashes are the ones typed."""
head = f"filter '{case_filter}' selects no case in {where}" if case_filter else f"no case to run in {where}"
lines = [head] + [textwrap.indent(note, " ") for note in notes]
names = sorted(available)
if names:
close = similar(case_filter, names, limit)
if close:
lines.append(f" similar: {', '.join(close)}")
elif names:
more = f", … ({len(names) - limit} more)" if len(names) > limit else ""
parts.append(f"available: {', '.join(names[:limit])}{more}")
return Unsupported("; ".join(parts + list(notes)))
lines.append(f" available: {', '.join(names[:limit])}{more}")
return Unsupported("\n".join(lines))


def run_bounded(args, *, timeout, what, **kwargs) -> subprocess.CompletedProcess:
Expand Down
16 changes: 13 additions & 3 deletions src/benchx/adapters/benchx_params.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,16 @@ def expand(name, ids, case_filter):


def matches_name_only(name, ids, case_filter) -> bool:
"""Whether a filter that selected none of a benchmark's cases matches its name alone:
a parametrized benchmark is searched in `name(p0, p1)`, so `name$` is a likely slip."""
return bool(ids and case_filter and re.search(case_filter, name))
"""Whether a filter that selected none of a benchmark's cases is about this benchmark, though
not about any of its combinations: it matches the name alone (`name$` is a likely slip), or the
part of it before its first `\\(` does, so what it gives after the name, the arguments, is
what fails (in the wrong order, or a value the benchmark does not have)."""
if not (ids and case_filter):
return False
named = case_filter.split(r"\(")[0]
if not named.strip("^"): # it starts with its arguments: no name to go by
return False
try:
return bool(re.search(named, name))
except re.error:
return False
31 changes: 22 additions & 9 deletions src/benchx/adapters/pyperf.py
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ def list_cases(runnable, case_filter, env, timeout=None) -> list[str]:
selected = benchx_params.expand(name, info["params"], case_filter)
if not selected:
if benchx_params.matches_name_only(name, info["params"], case_filter):
unmatched.append(name)
unmatched.append(info)
continue
if "error" in info:
raise Unsupported(f"cannot read the params of {name} in {where}: {info['error']}")
Expand All @@ -214,6 +214,8 @@ def list_cases(runnable, case_filter, env, timeout=None) -> list[str]:
workload = parametrized_workload # the function, and its parameters as value strings


# The combination index is bound to a name before it is tested: `0 is not None` is a SyntaxWarning, which
# Python prints on the stderr of every worker.
DRIVER = '''\
# Generated by benchx. Runs one function under pyperf: one case, which for a function
# that declares params is one combination of them, passed as its arguments.
Expand Down Expand Up @@ -247,10 +249,11 @@ def list_cases(runnable, case_filter, env, timeout=None) -> list[str]:
sys.modules["benchx_suite"] = module
spec.loader.exec_module(module)
func = getattr(module, {function!r})
index = {index!r}
args = ()
if {index!r} is not None:
if index is not None:
values, _, _ = params.normalize(getattr(func, "params", []), getattr(func, "param_names", []))
args = params.combination(values, {index!r})
args = params.combination(values, index)
runner.bench_func({case!r}, func, *args)
'''

Expand Down Expand Up @@ -305,12 +308,22 @@ def run_case(runnable, case, flags, env, timeout, native_path, env_names=None) -


def _failure_message(stderr: str) -> str:
"""The worker's exception line. The manager then raises its own
"<python> failed with exit code N" after a second traceback; that wrapper
only counts when nothing else was said."""
lines = [l for l in stderr.splitlines() if l and not l[0].isspace() and not l.startswith("Traceback")]
own = [l for l in lines if "failed with exit code" not in l]
return (own or lines or [stderr.strip().splitlines()[-1]])[0]
"""The worker's exception line: the line that ends a traceback (the first unindented line after
"Traceback"), the last one when exceptions were chained. A warning printed before is not it, and
neither is the manager's own "<python> failed with exit code N" after a second traceback: that wrapper
only counts when nothing else was said. Without a traceback (a killed process), the first line said."""
lines = [line for line in stderr.splitlines() if line.strip()]
ended, in_traceback = [], False
for line in lines:
if line.startswith("Traceback"):
in_traceback = True
elif in_traceback and not line[0].isspace():
ended.append(line)
in_traceback = False
own = [line for line in ended if "failed with exit code" not in line]
if own or ended:
return (own or ended)[-1]
return next((line for line in lines if not line[0].isspace()), lines[-1] if lines else stderr.strip())


def attempted(applied: dict) -> int:
Expand Down
20 changes: 18 additions & 2 deletions src/benchx/bench.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

from . import core, snapshot
from .errors import SessionError
from .projection import workload_label
from .projection import declared_workload, said, workload_label
from .store import warning_codes


Expand Down Expand Up @@ -77,7 +77,7 @@ def row_of(document: dict) -> dict:
# An observation is a number, or an object carrying one with its metadata (pyperf's).
values = [o["value"] if isinstance(o, dict) else o for o in measurement.get("observations", [])]
return {
"case": workload_label(document["coordinates"]["workload"]),
"case": workload_label(declared_workload(document)),
"status": measurement["status"] + (f" ({measurement['reason']})" if measurement.get("reason") else ""),
"n": len(values),
"median": statistics.median(values) if values else None,
Expand All @@ -87,6 +87,7 @@ def row_of(document: dict) -> dict:
"max": max(values) if values else None,
"loops": document.get("procedure", {}).get("inner_iterations"),
"warnings": warning_codes(document),
"message": None if measurement["status"] == "success" else said(document),
}


Expand All @@ -110,3 +111,18 @@ def render(rows, leading=()) -> str:
widths = [max(len(row[i]) for row in [header, *body]) for i in range(len(header))]
lines = [" ".join(cell.ljust(w) for cell, w in zip(row, widths)).rstrip() for row in [header, *body]]
return "\n".join([lines[0], " ".join("-" * w for w in widths), *lines[1:]])


def failures(rows, names=3) -> str:
"""The section under the table for the cases that were not measured: what each said, `case: message`,
cases with the same message as one line, `message (K cases: a, b, c, …)`. Empty when none has a message
(the status column is then all there is). A row's message is one line already (`projection.said`)."""
said_by = {}
for row in rows:
if row.get("message"):
said_by.setdefault(row["message"], []).append(row["case"])
if not said_by:
return ""
lines = [f" {message} ({len(cases)} cases: {', '.join(cases[:names])}{', …' if len(cases) > names else ''})"
if len(cases) > 1 else f" {cases[0]}: {message}" for message, cases in said_by.items()]
return "\n".join(["", f"not measured ({sum(len(c) for c in said_by.values())})", *lines])
Loading
Loading