Repository navigation
Closed
Labels
Description
Here's a repro:
mgorelli@marcoslaptop:~$ mkdir tmp-repro
mgorelli@marcoslaptop:~$ cd tmp-repro/
mgorelli@marcoslaptop:~/tmp-repro$ uv venv -p 3.13
Using CPython 3.13.13
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
mgorelli@marcoslaptop:~/tmp-repro$ . .venv/bin/activate
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro$ uv pip install -e ../benchx/prototype pyperf
Resolved 24 packages in 25ms
Built benchx-prototype @ file:///home/mgorelli/benchx/prototype Prepared 1 package in 520ms
Installed 24 packages in 64ms
+ arrow==1.4.0
+ attrs==26.1.0
+ benchx-prototype==0.1.0 (from file:///home/mgorelli/benchx/prototype)
+ fqdn==1.6.0
+ idna==3.20
+ isoduration==20.11.0
+ jsonpointer==3.8
+ jsonschema==4.26.0
+ jsonschema-specifications==2025.9.1
+ lark==1.3.1
+ psutil==7.2.2
+ pyarrow==26.0.1
+ pyperf==2.10.0
+ python-dateutil==4.0.0
+ referencing==0.37.0
+ rfc3339-validator==0.1.4
+ rfc3986-validator==0.1.1
+ rfc3987-syntax==1.1.0
+ rfc8785==0.1.4
+ rpds-py==2026.9.1
+ six==1.17.0
+ tzdata==2026.6
+ uri-template==1.3.0
+ webcolors==25.10.0
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro$ mkdir repro && cd repro && git init -q
printf 'def bench_a():\n sorted(range(1000))\n' > suite.py
printf 'store.parquet\nresults/\n' > .gitignore
git add . && git commit -qm base
git worktree add -q ../repro-base HEAD
printf 'def bench_a():\n sorted(range(5000))\n' > suite.py
git commit -qam slower
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro/repro$ bx compare ../repro-base . --adapter pyperf --profile revisions \
--suite suite.py --rounds 5 --no-project
/home/mgorelli/tmp-repro/repro/store.parquet
round 1/5 baseline: 2 result(s)
round 1/5 contender: 2 result(s)
round 2/5 baseline: 2 result(s)
round 2/5 contender: 2 result(s)
round 3/5 baseline: 2 result(s)
round 3/5 contender: 2 result(s)
round 4/5 baseline: 2 result(s)
round 4/5 contender: 2 result(s)
round 5/5 baseline: 2 result(s)
round 5/5 contender: 2 result(s)
run compare-20261105T123119Z profile revisions baseline d3ebe385e29d contender None [local-only]
failed invariants (1)
{'invariant': 'sides', 'detail': 'a result has no provenance.subject_tree'}Without the gitignore part, we get:
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro$ mkdir repro && cd repro && git init -q
printf 'def bench_a():\n sorted(range(1000))\n' > suite.py
git add . && git commit -qm base
git worktree add -q ../repro-base HEAD
printf 'def bench_a():\n sorted(range(5000))\n' > suite.py
git commit -qam slower
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro/repro$ bx compare ../repro-base . --adapter pyperf --profile revisions \
--suite suite.py --rounds 5 --no-project
/home/mgorelli/tmp-repro/repro/store.parquet
round 1/5 baseline: 2 result(s)
round 1/5 contender: 2 result(s)
round 2/5 baseline: 2 result(s)
round 2/5 contender: 2 result(s)
round 3/5 baseline: 2 result(s)
round 3/5 contender: 2 result(s)
round 4/5 baseline: 2 result(s)
round 4/5 contender: 2 result(s)
round 5/5 baseline: 2 result(s)
round 5/5 contender: 2 result(s)
run compare-20261105T123605Z profile revisions baseline cbf062bad0d0 contender d5d9b2e2bb16
REGRESSED (1)
bench_a wall-time effect +468.69% noise 6.92% rounds=5
Activity
MarcoGorelli commentedon Nov 5, 2026
nevermind, looks like this was already fixed by #81 , my own main was just behind 🤦