Skip to content

Prototype impl vs design docs - #3

Merged
ev-br merged 17 commits into
mainfrom
prototype_impl_vs_design_docs
Oct 20, 2026
Merged

ev-br merged 17 commits into
mainfrom
prototype_impl_vs_design_docs

Conversation

@ev-br

@ev-br ev-br commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Remaining open questions

All three are open questions about what the result should say. Each section gives how things stand today, the choices, and a suggested pick.

G: a dirty tree has no warning

Now: a dirty checkout is recorded as subject_dirty: "dirty", benchmark_dirty: "dirty" and a subject_tree id. The result has no quality block, so nothing says "dirty" in a form a person or a filter would pick up.

What the docs say: benchmark-result-schema.md §4.1 says the flag, the tree id "and a quality warning exist so that a detector or comparator can filter or annotate dirty results trivially". Its example uses "quality": {"warnings": ["dirty-benchmark-tree"]}. The schema only requires each warning to be a token, so the names are free. The runner today emits only env-mismatch and env-capture-missing.

Options

  1. Two warnings, dirty-subject-tree and dirty-benchmark-tree, one per flag. This matches the doc example. For bx bench both fire together, because the subject and the benchmark are the same tree.
  2. One warning, dirty-tree. This is simpler, but it doesn't match the example.
  3. None. The flag already says it, but this ignores the sentence in §4.1.

What it changes in practice: bx bench's table shows [warning] after the status, so a dirty run would start showing it. Every run of the pyperf example is on a dirty tree whenever any file in the repo is uncommitted, so the warning would appear often. Stored results and comparisons don't change, because the flag is what the comparator uses.

Suggested pick: option 1. It follows the doc example exactly, and the cost is just two strings.

L: the module isn't part of the result

Now: the workload is just {"name": "bench_join"}. The file it came from (.../bench_suite.py) is only in the work order. Two files in one project that both define bench_join would produce results with identical coordinates, so they'd share a series.

What the docs allow: the workload has name, parameters and dataset. harness-adapter.md §4.3 says the name is the harness's case name and the adapter never builds names out of other fields. batch_key holds "one suite file", but it is provenance and never part of identity.

Options

  1. Put the module in the name, such as bench_suite.bench_join. It's easy, and it separates the series. It breaks the §4.3 rule.
  2. Record it as a workload parameter, such as module: bench_suite.py. It stays structured and separates series, but a module isn't a parameter of the function the way a size is.
  3. Leave it and document it. Nothing breaks until two files in one project share a function name.
  4. Refuse the run when two selected cases have the same function name in different modules. In practice bx bench takes one module per run, so this does not arise within a run; the clash only happens across runs.

Suggested pick: option 3 for now, with a sentence in the docs. It only bites when two files share a function name, and any of the other options is a decision about the data model that you'd want to make deliberately.

M: environment variables as parameters, or as protocol

Now: a variable set with -e NAME=VALUE becomes coordinates.workload.parameters.NAME. Thread caps such as OMP_NUM_THREADS would be treated the same way.

What the docs say: the two docs disagree.

  • runner-schema.md (and benchmark-result-schema.md near line 165) says variables set for the measured process are workload parameters, reported under the names used. OMP_NUM_THREADS=1 and =10 are two ordinary variants of one benchmark.
  • benchmark-environments.md §3.4 (line 83) says thread caps and affinity go in comparison_context.protocol.threads and .affinity.

Why it matters: this changes what compares with what.

  • As a parameter, 1 thread against 10 threads is a comparison within a single suite, like a size of 10 against 1000.
  • As comparison context, they are different series. Line 213 of benchmark-environments.md says a differing cap is "usually an accident: fix the calling script", and it wants the results kept apart.

Options

  1. Follow runner-schema.md: parameters. Change benchmark-environments.md §3.4 to say caps set through the work order are parameters, and only an inherited cap is context.
  2. Follow benchmark-environments.md: comparison context. The runner would need a list of recognized variables and would put them in protocol.threads, which is more code and more choices.
  3. Leave both, and add the conflict to the prototype's open questions.

Suggested pick: option 1. It is what the prototype does now, and it supports comparing thread counts within one suite. But this decision isn't mine to make, since it changes a doc you or the other authors own.

Order of work

G is a small code change plus a test. L and M are decisions first, and then mostly doc edits (option 3 for L, option 1 for M).

rok and others added 11 commits October 13, 2026 14:24
GitHub does not dispatch pull_request runs for a workflow that exists only
in a pull request from a fork; the base repository must already carry a
workflow on its default branch. This placeholder bootstraps Actions so
PR benchx-dev#53, which brings the real workflow, can run.
CI: bootstrap GitHub Actions with a placeholder workflow
Co-authored-by: ev-br's Claude Code Agent <agent@localhost>
It fails all around: benchx-devgh-54 and other `os.` related failures.

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
ev-br and others added 5 commits October 13, 2026 19:15
- bx head: size each column to its longest cell
- prototype README, prototype-design and prototype-scope: add bx bench,
  bx config, the pyperf adapter and the working_tree target kind
- pyperf-bench example: PYTHON= now also selects the benchmark workers
  (its directory goes first on PATH); note that any uncommitted change
  in the repository makes the result dirty

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br force-pushed the prototype_impl_vs_design_docs branch from 2abf398 to 2e214c8 Compare October 13, 2026 17:22
Handle absolute and relative SCP paths (relative ones become ssh://host/~/path),
local paths (file://), Windows drive paths and bracketed IPv6 hosts. Apply it to
the provider URI, --source-uri and origin, and to identity._normalize_uri, so
equivalent spellings of one remote no longer trip the "different source URIs"
refusal.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br merged commit fb81530 into main Oct 20, 2026
4 checks passed
ev-br pushed a commit that referenced this pull request Nov 3, 2026
…kflow

prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as
examples/pyperf-suite.

- Remove it from this repository, and its step from tests.yml.
- e2e.yml gets a third job, pyperf-suite, that runs run.sh from
  benchx_examples and uploads its out/ directory. The header lists all three
  jobs.
- prototype/README.md points to benchx_examples where it described the example.

The job needs examples/pyperf-suite on the benchx_examples ref it clones
(main by default). benchx_examples PR #3 adds the same files unchanged, which
do not run from there; the port with adapted instructions is needed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br pushed a commit that referenced this pull request Nov 4, 2026
…kflow

prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as
examples/pyperf-suite.

- Remove it from this repository, and its step from tests.yml.
- e2e.yml gets a third job, pyperf-suite, that runs run.sh from
  benchx_examples and uploads its out/ directory. The header lists all three
  jobs.
- prototype/README.md points to benchx_examples where it described the example.

The job needs examples/pyperf-suite on the benchx_examples ref it clones
(main by default). benchx_examples PR #3 adds the same files unchanged, which
do not run from there; the port with adapted instructions is needed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br pushed a commit that referenced this pull request Nov 5, 2026
…kflow

prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as
examples/pyperf-suite.

- Remove it from this repository, and its step from tests.yml.
- e2e.yml gets a third job, pyperf-suite, that runs run.sh from
  benchx_examples and uploads its out/ directory. The header lists all three
  jobs.
- prototype/README.md points to benchx_examples where it described the example.

The job needs examples/pyperf-suite on the benchx_examples ref it clones
(main by default). benchx_examples PR #3 adds the same files unchanged, which
do not run from there; the port with adapted instructions is needed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants