Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: prototype tests

on:
push:
branches: [main]
paths:
- "prototype/**"
- "schemas/**"
- ".github/workflows/**"
pull_request:
branches: [main]
paths:
- "prototype/**"
- "schemas/**"
- ".github/workflows/**"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
test:
name: ${{ matrix.os }} / py${{ matrix.python-version }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python-version: ["3.11", "3.13"]
defaults:
run:
working-directory: prototype
steps:
- uses: actions/checkout@v4

- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: prototype/uv.lock

- name: Install Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Install package and test dependencies
run: uv sync --extra test --python ${{ matrix.python-version }}

- name: Run tests
run: uv run --extra test --python ${{ matrix.python-version }} python -m pytest tests -ra

- name: Run the pyperf-bench example
shell: bash
run: uv run --extra test --python ${{ matrix.python-version }} bash examples/pyperf-bench/run.sh

# Needs cmake and a C++ compiler (preinstalled on the Linux and macOS
# runners) and fetches Google Benchmark; the example is bash and CMake
# with Unix paths, so it is not run on Windows.
- name: Run the archery-local example
if: runner.os != 'Windows'
shell: bash
run: uv run --extra test --python ${{ matrix.python-version }} bash examples/archery-local/run.sh
83 changes: 68 additions & 15 deletions docs/design/prototype-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,53 @@ bx series [--workload … --quantity …] # list series
bx history <series> # one series' points (a query)
bx head [-n 10] [--json] # the latest results in the store, newest first
bx compare --run KEY --profile revisions|environments --baseline VALUE
[--label NAME] [--results DIR] [--k 3] [--json]
[--label NAME] [--results DIR] [--k 3] [--min-rounds 3] [--json]
# read mode: compare results already measured
bx target describe|prepare TARGET # a target's description (a provider's answer, or a build directory's sidecar)
bx compare BASELINE CONTENDER --profile revisions|environments
--suite NAME [--filter REGEX] [--quantity wall-time]
--rounds R [--repetitions N] [--min-time S]
[--project P] [--label NAME=BASE,CONTENDER]
[--run-key KEY] [--out DIR] [--k 3] [--json]
[--run-key KEY] [--out DIR] [--no-build] [--source-uri URI]
[--k 3] [--min-rounds 3] [--json]
# measure mode: run both sides, then compare
bx bench MODULE [FUNCTION …] [--processes N] [--values N] [--min-time S | --loops N] [--warmups N]
[-e NAME=VALUE] [--tree DIR] [--project P] [--run-key KEY] [--source-uri URI]
[--case-timeout S] [--out DIR] [--no-ingest]
# Python functions under pyperf → result files → store → table
bx config init [--dir DIR] # write .benchx/config.json stating pyperf's defaults
```

`bx bench` is a second front end to the same runner. It builds one work order
whose target is the `working_tree` of the git checkout containing MODULE (or
`--tree`), as found, with the dirty state and tree id recorded; the suite is the
module, a case is a function in it (`bench_*` and `benchmark_*` by default, or the
functions named), and `-e` fills `environment_variables`. Precision is resolved
before the order is written, flags over `.benchx/config.json` (found by walking up
from the current directory) over pyperf's defaults, so the order always states
what applied. The pyperf adapter takes `python` from the `PATH` the workers start
with and refuses the order if pyperf is not importable there. Delivery is as for
`bx run`. There is no comparison step: `bx bench` produces results, and comparing
them is not part of it.

`bx compare` has two modes, chosen by its arguments. Two positional targets
(`BUILD_DIR[:SOURCE_DIR]`) mean measure mode; `--run` means read mode; giving
(`BUILD_DIR[:SOURCE_DIR]` or `@CONFIG[=SOURCE_REF]`, a call to the project's
target provider) mean measure mode; `--run` means read mode; giving
both or neither is an error, as is a flag that belongs to the other mode
(`--baseline` and `--results` are read mode only, since measure mode derives
the baseline from its first target; `--suite`, `--rounds`, `--project`,
`--run-key`, `--out`, `--repetitions`, and `--min-time` are measure mode
only). `--label` names the label in read mode, and carries the two values in
`--run-key`, `--out`, `--repetitions`, `--min-time`, `--no-build`, and
`--source-uri` are measure mode only). `--label` names the label in read mode, and carries the two values in
measure mode. §5 describes measure mode.

A source URI given any way (`--source-uri`, a provider's description, the
checkout's `origin`) goes through one normalization before sides are
compared or recorded, and identity does the same: git's SCP-style
`user@host:/abs/path` becomes `ssh://user@host/abs/path`, a relative
`host:path` becomes `ssh://host/~/path` (relative to the login directory, so it
is not mistaken for the absolute `/path`), and a local path becomes a
`file://` URI. Windows drive paths and bracketed IPv6 hosts are recognized.

The CLI is a thin layer: each command is one call into the `benchx` package,
so the workbench, a CI script, or a notebook can do the same without a
subprocess. Measure mode is a call into the session function, which calls the
Expand All @@ -62,9 +90,10 @@ not post to the original store.
**Work order.** The prototype reads the work order of `runner-schema.md`
(`schemas/work-order/0.1.0`, `benchx/work-order/0.1.0`) and defines no format
of its own. One order is one side of one round: the runner executes one order
per invocation (`runner.md` §7), and the prototype accepts one target kind,
`build`, an existing build directory with an optional `source_dir`. The runner
never builds it (`runner.md` §1).
per invocation (`runner.md` §7), and the prototype accepts two target kinds:
`build`, an existing build directory with an optional `source_dir`, and
`working_tree`, a checkout measured as found, which only `bx bench` writes. The
runner never builds either (`runner.md` §1).

Interleaving needs two order fields. `round` is native to the schema and is
copied to `procedure.round`. `slot`, the realized position across all sides,
Expand Down Expand Up @@ -203,6 +232,11 @@ landing where its §3.4 puts it:
§5.5); OS, kernel, libc, governor, and load as observed context; an
allowlist of thread-control environment variables as observed context,
which is the allowlist of `benchmark-environments.md` §3.1.
The `env` key reads differently by adapter. Google Benchmark takes the
runner's own environment and omits `env` when no allowlisted variable is set.
pyperf records what the workers actually had, so `env` is `{}` when the
capture worked and none was set, and absent only when nothing was captured
(`env-capture-missing`). Both are kept as they are.
- **Source identity**, from `target.source_dir`, or from the
`CMAKE_HOME_DIRECTORY` the build directory records when the order names
none: revision, `subject_dirty`, and `subject_tree` (schema §4.1). When
Expand All @@ -219,6 +253,11 @@ the same way report the same configuration. Nobody declared these, so the detect
values are recorded as the intended ones in `coordinates.subject.configuration`
(§3.4). This is what makes the hardened and plain directories differ in
coordinates, which the `environments` profile requires.
The pyperf adapter's `python_cflags` is the exception to "never paths": it is
recorded verbatim in `comparison_context.runtime`, as `harness-adapter.md` §6.8
has it, so the same Python build installed under two prefixes reports different
flags (conda adds its `-isystem` and `-fdebug-prefix-map` paths) and lands in
different series.
- **Declared facts**, from `.benchx/setup.json` or `BENCHX_SETUP_FILE`
(`harness-adapter.md` §5.1), for what the cache cannot tell, such as how a
dependency was installed. The prototype has no placement rules, so they
Expand Down Expand Up @@ -380,9 +419,15 @@ compares them; it is the workbench's session loop (`system-decomposition.md`
first value of `--label NAME=BASE,CONTENDER` for `environments`.

It is exactly `bx run` 2R times followed by read-mode `bx compare`; it adds no
document, schema, or contract. It builds nothing: both targets must exist,
and whatever produced them plays the target-provider role
(`system-decomposition.md` §3.4). If it stops partway, the results already
document, schema, or contract. It builds nothing itself: both targets must exist
when it looks. A target given as `@CONFIG` is first requested from the
project's target provider (`system-decomposition.md` §3.4), which `prepare`s
it (and may build it) or, with `--no-build`, only `describe`s it; a target
given as a build directory is read from its `.benchx-target.json` sidecar if
it has one. The description's declared text (`how_built`, `activation`,
`shell`, `provider`) is copied into each order, recorded in each result's
`provenance.info`, and shown in the comparison; it is never executed or
checked. If it stops partway, the results already
delivered stay in the store, and a read-mode comparison of the run key reports
missing results or a failed alternation invariant, as UC-03 §10 requires.
Allowing a local-only `revisions` comparison of a dirty tree is open
Expand All @@ -394,7 +439,10 @@ Both profiles use one shape: two prepared build directories, one run key, R
rounds, and `bx compare` in measure mode, which alternates the sides and
assigns `round` and `slot`. The demo script plays the user, and in that role
the target provider: it prepares the sources and build directories with git
and CMake, since benchx builds nothing.
and CMake, since benchx builds nothing. `prototype/examples/archery-local`
shows the other route, where a provider script named in
`.benchx/provider.json` builds and describes the targets and the sides are
given as `@plain` and `@hardened`.

- **Revisions:** two git worktrees of one source at the base and head
commits, each built into its own directory with the same configuration.
Expand Down Expand Up @@ -425,12 +473,17 @@ history grow from run to run, and the demo ends with `bx head`.

## 7. Deliberate simplifications

- One harness and one target kind. The runner builds nothing, ever: the user
- Two harnesses (Google Benchmark, and pyperf through `bx bench`) and two target
kinds (`build`, `working_tree`). The runner builds nothing, ever: the user
prepares both sides (`benchmark-environments.md` §2). The prototype applies
`environment_variables` and defers the rest of the environment policy, and it
refuses an order that names one rather than ignoring it (§3).
- No target-provider hook: the demo prepares its builds by hand, and the
provider contract is open (`system-decomposition.md` §6 question 5).
- The target-provider hook is minimal: one provider command per project
(`.benchx/provider.json`), `describe` and `prepare` only, `build` targets
only, and a source ref other than `WORKSPACE` is the provider's business.
How cached builds are named and expired stays open
(`system-decomposition.md` §6 question 5). The demo prepares its builds by
hand.
- Environment identity is the snapshot's host name, CPU model, and core
count; continuity events, quarantine, and artifact storage are absent.
- One identity policy and one derived estimator, fixed in code.
Expand Down
6 changes: 3 additions & 3 deletions docs/design/prototype-scope.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ one run:

| Component | In prototype | Deferred | Out of scope |
|---|---|---|---|
| Runner | execute a `build` target per a work order; apply the order's `environment_variables` to the launched process; capture the zero-configuration snapshot, the build configuration, and source identity; account for planned cases; refuse an order it cannot apply exactly, including one that names a deferred field | `python_env` targets, context collectors beyond the built-in ones, environment policy (enforcing launch settings, verifying requested hardware, refusing an unsatisfiable run) | building revisions, tuning the machine (`benchmark-environments.md` §2.3) |
| Adapter | Google Benchmark, both halves: drive filter and repetitions, translate output | all other harnesses | — |
| Runner | execute a `build` or `working_tree` target per a work order; apply the order's `environment_variables` to the launched process; capture the zero-configuration snapshot, the build configuration, and source identity; account for planned cases; refuse an order it cannot apply exactly, including one that names a deferred field | `python_env` targets, context collectors beyond the built-in ones, environment policy (enforcing launch settings, verifying requested hardware, refusing an unsatisfiable run) | building revisions, tuning the machine (`benchmark-environments.md` §2.3) |
| Adapter | Google Benchmark, both halves: drive filter and repetitions, translate output; pyperf, both halves, for `bx bench` (Python functions in the working tree; no comparison step) | all other harnesses; pyperf's `tags`, `date`, per-run load and frequency, and `pyperf check` | — |
| Store | one Parquet file; ingest per schema §5.4 (validated and durable, auto-create, unit-conflict rejection, idempotency, attempt integrity); one default identity policy; queries: list series, series history, get result | continuity events, quarantine, artifact storage, identity-policy authoring | — |
| Comparator | run mode with the `revisions` and `environments` profiles; eligibility guard; comparison document | history mode; `single-run`, `variants`, and `cross-machine` profiles; batch requests; provisional noise rules | — |
| Workbench session loop | `bx compare` in measure mode: writes one work order per side and round, alternates them, delivers the results, then compares; takes each side as a build directory or as `@CONFIG`, a call to the project's target provider (`describe` or `prepare`); `bx target` asks for a description directly; copies the description's declared text (`how_built`, `activation`, `shell`, `provider`) into the order | provider-built revisions (a source ref other than `WORKSPACE` is the provider's business), `artifact` and `working_tree` targets, a declared `build` | workbench packaging, front ends such as `spin bench` |
| Workbench session loop | `bx compare` in measure mode: writes one work order per side and round, alternates them, delivers the results, then compares; takes each side as a build directory or as `@CONFIG`, a call to the project's target provider (`describe` or `prepare`); `bx target` asks for a description directly; copies the description's declared text (`how_built`, `activation`, `shell`, `provider`) into the order | provider-built revisions (a source ref other than `WORKSPACE` is the provider's business), an `artifact` target, `working_tree` targets in `bx compare`, a declared `build` | workbench packaging, front ends such as `spin bench` |
| Documents | work order (`schemas/work-order/0.1.0`), result (schema §5.2), comparison document | — | — |

**Out entirely:** scheduler, change detector, alerting, views beyond query
Expand Down
23 changes: 22 additions & 1 deletion prototype/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ bx compare BASELINE CONTENDER --profile revisions|environments
[--run-key KEY] [--out DIR] [--no-build] [--k 3] [--json]
# measure mode: run both sides, then compare
bx validate <file>… [--kind KIND] # every problem in each document, by schema and rules
bx bench MODULE [FUNCTION …] [--processes N] [--values N] [--min-time S | --loops N] [--warmups N]
[-e NAME=VALUE] [--tree DIR] [--project P] [--run-key KEY] [--source-uri URI]
[--case-timeout S] [--out DIR] [--no-ingest]
# Python functions under pyperf: order → results → store → table
bx config init [--dir DIR] # write .benchx/config.json stating pyperf's defaults
```

Each target is `BUILD_DIR[:SOURCE_DIR]` or `@CONFIG[=SOURCE_REF]`. Two targets
Expand All @@ -41,6 +46,18 @@ executed or checked: you activate the environment yourself, and the runner
records the allowlisted variables it inherited beside your declaration.
`examples/archery-local` is a complete walk-through.

`bx bench` is the other entry point: it benchmarks the functions of a Python file
with [pyperf](https://pyperf.readthedocs.io), as one `working_tree` order for the
checkout containing the module (uncommitted changes included, recorded as dirty).
Each `bench_*` or `benchmark_*` function, or the ones named, is a case; `-e` sets
environment variables for the workers and lands in the workload's parameters.
Precision resolves as flags, then `.benchx/config.json` (found by walking up from the
current directory; `bx config init` writes a template), then pyperf's defaults, and the
resolved values are written into the order. The workers run the `python` first on
`PATH`, which must have pyperf. It builds the order, runs it through the same runner and
delivers to the same store as `bx run`; it has no comparison step.
`examples/pyperf-bench` is a runnable walk-through.

## What it does

- **Work orders** are the `docs/design/runner-schema.md` schema
Expand Down Expand Up @@ -86,7 +103,7 @@ records the allowlisted variables it inherited beside your declaration.

```console
$ uv venv && uv pip install -e '.[test]' # editable: schemas are read from ../schemas
$ .venv/bin/python -m pytest tests # 222 tests, fake Google Benchmark binary, no compiler
$ .venv/bin/python -m pytest tests # 228 tests, fake Google Benchmark binary, no compiler
$ ./demo.sh # real builds; needs cmake, a C++ compiler, network once
```

Expand Down Expand Up @@ -134,6 +151,9 @@ benchx/
hashing.py canonical form, hashes, order reference
snapshot.py what the runner records (benchmark-environments.md §3)
adapters/gbench.py Google Benchmark, driving and translating halves
adapters/pyperf.py pyperf, both halves, for Python functions (bx bench)
bench.py bx bench: module + flags + config.json → one working_tree work order
config.py .benchx/config.json: the pyperf precision defaults and their template
runner.py one work order in, results out
session.py the session loop: alternate two targets over rounds, then compare
target.py target descriptions: the provider call, the sidecar, the declared text
Expand All @@ -144,6 +164,7 @@ benchx/
cli.py bx (compare has a read mode and a measure mode)
../schemas/ the result and work-order schemas, read in place ($BENCHX_SCHEMAS overrides)
examples/demo-suite a two-benchmark Google Benchmark suite for the demo
examples/pyperf-bench Python functions under `bx bench` (own README)
examples/archery-local an end-to-end, Arrow-local-shaped example with a target provider (own README)
tests/ test_prototype.py: the success criteria and the runner and store
rules, on a fake Google Benchmark binary; test_comparator.py: the
Expand Down
Loading
Loading