Repository navigation
Conversation
ev-br
force-pushed
the
context-collectors
branch
from
October 29, 2026 20:50
2f2590d to
8d93575
Compare
ev-br
force-pushed
the
context-collectors
branch
from
November 1, 2026 11:30
8d93575 to
f125103
Compare
ev-br
force-pushed
the
context-collectors
branch
from
November 1, 2026 14:10
f125103 to
984d68d
Compare
Context collectors
Compare two environments with separate `bx bench` calls joined by a run key. Place a run in a session with --label, --round and --slot, and describe each environment with the user's own collector, checked with verify/expect. Typed declared facts (--component, build) are deferred. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Calls that share a --run-key become the sides of one session, so two environments can be measured by separate calls and compared with `bx compare --run`. The caller's loop keeps the slot counter. Refused before an order is built: any of the three without --run-key (the default key is a timestamp per call), --label or --round without --slot (the ingest key only gets its slot suffix when a slot is set, so two sides would collide), a malformed or repeated label, a negative number. With a slot, the order is written to orders/order-<slot>.json, as in bx compare, so the calls do not overwrite each other's order. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The comparison now lists the facts that differ between the sides (unmatched_differences, which it already computed) and the warnings the sides' results carry (the schema's per-unit `caveats`), without changing a verdict. The refusals say what to change: the sides differ only in their label (describe the environment with a collector or a setup file), the sides do not alternate (same order every round), the attempt counts differ (same number of rounds), a call has no label or slot. tests/test_uc04.py runs UC-04 in process: separate `bx bench` calls with --run-key, --label, --round and --slot, the sides told apart by a context collector or by a setup file, then `bx compare --run`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
benchx leaves its own output (the store, the results directory, --out) out of the tree identity with exclude pathspecs. `git add -A` refuses an exclude pathspec that names a path the project's .gitignore already ignores, so the tree id came out unknown whenever the results directory was git-ignored, which is common. Such paths are now left out of the excludes: git leaves them out anyway. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two venvs hold different numpy versions, which no single process can, so run.sh drives separate `bx bench` calls that alternate round by round under one --run-key, with --label, --round and --slot, then compares them with `bx compare --run`. A context collector (describe_env.py) tells the sides apart: it prints the numpy its Python imports, which lands under environment.system.deps. The provider declares each side's numpy in how_built, and the collector's `verify: how_built` with `on_fail: warn` flags a side whose environment is not the declared one. The README carries the map from UC-04's spellings to the prototype's. The CI workflow runs the example on one matrix cell (it makes its own Python 3.12 venvs). prototype-design.md and the prototype README document the session flags and the collector route. The plan is updated to match the script as built. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run.sh assumes uv, fixes the numpy versions (1.26.4 and 2.0.2), the Python
(3.12) and the rounds (3) in upper-case variables at the top, and sources
each venv's activate directly: activating another venv replaces the one
before, so there is no deactivate and no subshell. Checked with uv venvs
under set -euo pipefail. provider.sh reads the versions from the
environment run.sh exports, so versions.env is gone.
Running the default pair for the first time showed that benchx cannot
import pyarrow 25 under numpy 1.26.4 ("pyarrow requires NumPy 2.0 or
newer"), and benchx imports pyarrow. The venvs install pyarrow<25 (up to
24 imports under numpy 1.26.4). The CI step loses its ROUNDS override.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sent towards
context-collectorsbranch, until it lands upstream in benchx-devgh-71