Repository navigation
Prototype impl vs design docs - #3
Merged
Merged
Conversation
GitHub does not dispatch pull_request runs for a workflow that exists only in a pull request from a fork; the base repository must already carry a workflow on its default branch. This placeholder bootstraps Actions so PR benchx-dev#53, which brings the real workflow, can run.
CI: bootstrap GitHub Actions with a placeholder workflow
Co-authored-by: ev-br's Claude Code Agent <agent@localhost>
It fails all around: benchx-devgh-54 and other `os.` related failures. Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
ev-br
force-pushed
the
prototype_impl_vs_design_docs
branch
from
October 13, 2026 16:53
275700e to
2abf398
Compare
CI: add a CI run
- bx head: size each column to its longest cell - prototype README, prototype-design and prototype-scope: add bx bench, bx config, the pyperf adapter and the working_tree target kind - pyperf-bench example: PYTHON= now also selects the benchmark workers (its directory goes first on PATH); note that any uncommitted change in the repository makes the result dirty Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br
force-pushed
the
prototype_impl_vs_design_docs
branch
from
October 13, 2026 17:22
2abf398 to
2e214c8
Compare
Handle absolute and relative SCP paths (relative ones become ssh://host/~/path), local paths (file://), Windows drive paths and bracketed IPv6 hosts. Apply it to the provider URI, --source-uri and origin, and to identity._normalize_uri, so equivalent spellings of one remote no longer trip the "different source URIs" refusal. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br
pushed a commit
that referenced
this pull request
Nov 3, 2026
…kflow prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as examples/pyperf-suite. - Remove it from this repository, and its step from tests.yml. - e2e.yml gets a third job, pyperf-suite, that runs run.sh from benchx_examples and uploads its out/ directory. The header lists all three jobs. - prototype/README.md points to benchx_examples where it described the example. The job needs examples/pyperf-suite on the benchx_examples ref it clones (main by default). benchx_examples PR #3 adds the same files unchanged, which do not run from there; the port with adapted instructions is needed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br
pushed a commit
that referenced
this pull request
Nov 4, 2026
…kflow prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as examples/pyperf-suite. - Remove it from this repository, and its step from tests.yml. - e2e.yml gets a third job, pyperf-suite, that runs run.sh from benchx_examples and uploads its out/ directory. The header lists all three jobs. - prototype/README.md points to benchx_examples where it described the example. The job needs examples/pyperf-suite on the benchx_examples ref it clones (main by default). benchx_examples PR #3 adds the same files unchanged, which do not run from there; the port with adapted instructions is needed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br
pushed a commit
that referenced
this pull request
Nov 5, 2026
…kflow prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as examples/pyperf-suite. - Remove it from this repository, and its step from tests.yml. - e2e.yml gets a third job, pyperf-suite, that runs run.sh from benchx_examples and uploads its out/ directory. The header lists all three jobs. - prototype/README.md points to benchx_examples where it described the example. The job needs examples/pyperf-suite on the benchx_examples ref it clones (main by default). benchx_examples PR #3 adds the same files unchanged, which do not run from there; the port with adapted instructions is needed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remaining open questions
All three are open questions about what the result should say. Each section gives how things stand today, the choices, and a suggested pick.
G: a dirty tree has no warning
Now: a dirty checkout is recorded as
subject_dirty: "dirty",benchmark_dirty: "dirty"and asubject_treeid. The result has noqualityblock, so nothing says "dirty" in a form a person or a filter would pick up.What the docs say:
benchmark-result-schema.md§4.1 says the flag, the tree id "and a quality warning exist so that a detector or comparator can filter or annotate dirty results trivially". Its example uses"quality": {"warnings": ["dirty-benchmark-tree"]}. The schema only requires each warning to be a token, so the names are free. The runner today emits onlyenv-mismatchandenv-capture-missing.Options
dirty-subject-treeanddirty-benchmark-tree, one per flag. This matches the doc example. Forbx benchboth fire together, because the subject and the benchmark are the same tree.dirty-tree. This is simpler, but it doesn't match the example.What it changes in practice:
bx bench's table shows[warning]after the status, so a dirty run would start showing it. Every run of the pyperf example is on a dirty tree whenever any file in the repo is uncommitted, so the warning would appear often. Stored results and comparisons don't change, because the flag is what the comparator uses.Suggested pick: option 1. It follows the doc example exactly, and the cost is just two strings.
L: the module isn't part of the result
Now: the workload is just
{"name": "bench_join"}. The file it came from (.../bench_suite.py) is only in the work order. Two files in one project that both definebench_joinwould produce results with identical coordinates, so they'd share a series.What the docs allow: the workload has
name,parametersanddataset.harness-adapter.md§4.3 says the name is the harness's case name and the adapter never builds names out of other fields.batch_keyholds "one suite file", but it is provenance and never part of identity.Options
bench_suite.bench_join. It's easy, and it separates the series. It breaks the §4.3 rule.module: bench_suite.py. It stays structured and separates series, but a module isn't a parameter of the function the way a size is.bx benchtakes one module per run, so this does not arise within a run; the clash only happens across runs.Suggested pick: option 3 for now, with a sentence in the docs. It only bites when two files share a function name, and any of the other options is a decision about the data model that you'd want to make deliberately.
M: environment variables as parameters, or as protocol
Now: a variable set with
-e NAME=VALUEbecomescoordinates.workload.parameters.NAME. Thread caps such asOMP_NUM_THREADSwould be treated the same way.What the docs say: the two docs disagree.
runner-schema.md(andbenchmark-result-schema.mdnear line 165) says variables set for the measured process are workload parameters, reported under the names used.OMP_NUM_THREADS=1and=10are two ordinary variants of one benchmark.benchmark-environments.md§3.4 (line 83) says thread caps and affinity go incomparison_context.protocol.threadsand.affinity.Why it matters: this changes what compares with what.
benchmark-environments.mdsays a differing cap is "usually an accident: fix the calling script", and it wants the results kept apart.Options
runner-schema.md: parameters. Changebenchmark-environments.md§3.4 to say caps set through the work order are parameters, and only an inherited cap is context.benchmark-environments.md: comparison context. The runner would need a list of recognized variables and would put them inprotocol.threads, which is more code and more choices.Suggested pick: option 1. It is what the prototype does now, and it supports comparing thread counts within one suite. But this decision isn't mine to make, since it changes a doc you or the other authors own.
Order of work
G is a small code change plus a test. L and M are decisions first, and then mostly doc edits (option 3 for L, option 1 for M).