Skip to content

UC-04 - #6

Closed
ev-br wants to merge 10 commits into
context-collectorsfrom
uc-04
Closed

ev-br wants to merge 10 commits into
context-collectorsfrom
uc-04

Conversation

@ev-br

@ev-br ev-br commented Oct 27, 2026

Copy link
Copy Markdown
Owner

sent towards context-collectors branch, until it lands upstream in benchx-devgh-71

ev-br and others added 10 commits November 3, 2026 10:57
Compare two environments with separate `bx bench` calls joined by a run
key. Place a run in a session with --label, --round and --slot, and
describe each environment with the user's own collector, checked with
verify/expect. Typed declared facts (--component, build) are deferred.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Calls that share a --run-key become the sides of one session, so two
environments can be measured by separate calls and compared with
`bx compare --run`. The caller's loop keeps the slot counter.

Refused before an order is built: any of the three without --run-key
(the default key is a timestamp per call), --label or --round without
--slot (the ingest key only gets its slot suffix when a slot is set, so
two sides would collide), a malformed or repeated label, a negative
number. With a slot, the order is written to orders/order-<slot>.json,
as in bx compare, so the calls do not overwrite each other's order.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The comparison now lists the facts that differ between the sides
(unmatched_differences, which it already computed) and the warnings the
sides' results carry (the schema's per-unit `caveats`), without changing
a verdict. The refusals say what to change: the sides differ only in their
label (describe the environment with a collector or a setup file), the
sides do not alternate (same order every round), the attempt counts differ
(same number of rounds), a call has no label or slot.

tests/test_uc04.py runs UC-04 in process: separate `bx bench` calls with
--run-key, --label, --round and --slot, the sides told apart by a context
collector or by a setup file, then `bx compare --run`.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
benchx leaves its own output (the store, the results directory, --out) out
of the tree identity with exclude pathspecs. `git add -A` refuses an
exclude pathspec that names a path the project's .gitignore already
ignores, so the tree id came out unknown whenever the results directory
was git-ignored, which is common. Such paths are now left out of the
excludes: git leaves them out anyway.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two venvs hold different numpy versions, which no single process can, so
run.sh drives separate `bx bench` calls that alternate round by round
under one --run-key, with --label, --round and --slot, then compares them
with `bx compare --run`. A context collector (describe_env.py) tells the
sides apart: it prints the numpy its Python imports, which lands under
environment.system.deps. The provider declares each side's numpy in
how_built, and the collector's `verify: how_built` with `on_fail: warn`
flags a side whose environment is not the declared one.

The README carries the map from UC-04's spellings to the prototype's.
The CI workflow runs the example on one matrix cell (it makes its own
Python 3.12 venvs). prototype-design.md and the prototype README document
the session flags and the collector route. The plan is updated to match
the script as built.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run.sh assumes uv, fixes the numpy versions (1.26.4 and 2.0.2), the Python
(3.12) and the rounds (3) in upper-case variables at the top, and sources
each venv's activate directly: activating another venv replaces the one
before, so there is no deactivate and no subshell. Checked with uv venvs
under set -euo pipefail. provider.sh reads the versions from the
environment run.sh exports, so versions.env is gone.

Running the default pair for the first time showed that benchx cannot
import pyarrow 25 under numpy 1.26.4 ("pyarrow requires NumPy 2.0 or
newer"), and benchx imports pyarrow. The venvs install pyarrow<25 (up to
24 imports under numpy 1.26.4). The CI step loses its ROUNDS override.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant