Skip to content

Comparison fails with "no provenance.subject_tree" when the store or results directory is gitignored #101

Closed

Description

@MarcoGorelli

Here's a repro:

mgorelli@marcoslaptop:~$ mkdir tmp-repro
mgorelli@marcoslaptop:~$ cd tmp-repro/
mgorelli@marcoslaptop:~/tmp-repro$ uv venv -p 3.13
Using CPython 3.13.13
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
mgorelli@marcoslaptop:~/tmp-repro$ . .venv/bin/activate
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro$ uv pip install -e ../benchx/prototype pyperf
Resolved 24 packages in 25ms
      Built benchx-prototype @ file:///home/mgorelli/benchx/prototype                                                                     Prepared 1 package in 520ms
Installed 24 packages in 64ms
 + arrow==1.4.0
 + attrs==26.1.0
 + benchx-prototype==0.1.0 (from file:///home/mgorelli/benchx/prototype)
 + fqdn==1.6.0
 + idna==3.20
 + isoduration==20.11.0
 + jsonpointer==3.8
 + jsonschema==4.26.0
 + jsonschema-specifications==2025.9.1
 + lark==1.3.1
 + psutil==7.2.2
 + pyarrow==26.0.1
 + pyperf==2.10.0
 + python-dateutil==4.0.0
 + referencing==0.37.0
 + rfc3339-validator==0.1.4
 + rfc3986-validator==0.1.1
 + rfc3987-syntax==1.1.0
 + rfc8785==0.1.4
 + rpds-py==2026.9.1
 + six==1.17.0
 + tzdata==2026.6
 + uri-template==1.3.0
 + webcolors==25.10.0
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro$ mkdir repro && cd repro && git init -q
printf 'def bench_a():\n    sorted(range(1000))\n' > suite.py
printf 'store.parquet\nresults/\n' > .gitignore
git add . && git commit -qm base
git worktree add -q ../repro-base HEAD
printf 'def bench_a():\n    sorted(range(5000))\n' > suite.py
git commit -qam slower
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro/repro$ bx compare ../repro-base . --adapter pyperf --profile revisions \
    --suite suite.py --rounds 5 --no-project
/home/mgorelli/tmp-repro/repro/store.parquet
round 1/5 baseline: 2 result(s)
round 1/5 contender: 2 result(s)
round 2/5 baseline: 2 result(s)
round 2/5 contender: 2 result(s)
round 3/5 baseline: 2 result(s)
round 3/5 contender: 2 result(s)
round 4/5 baseline: 2 result(s)
round 4/5 contender: 2 result(s)
round 5/5 baseline: 2 result(s)
round 5/5 contender: 2 result(s)
run compare-20261105T123119Z  profile revisions  baseline d3ebe385e29d  contender None  [local-only]

failed invariants (1)
  {'invariant': 'sides', 'detail': 'a result has no provenance.subject_tree'}

Without the gitignore part, we get:

(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro$ mkdir repro && cd repro && git init -q
printf 'def bench_a():\n    sorted(range(1000))\n' > suite.py
git add . && git commit -qm base
git worktree add -q ../repro-base HEAD
printf 'def bench_a():\n    sorted(range(5000))\n' > suite.py
git commit -qam slower
(tmp-repro) mgorelli@marcoslaptop:~/tmp-repro/repro$ bx compare ../repro-base . --adapter pyperf --profile revisions \
    --suite suite.py --rounds 5 --no-project
/home/mgorelli/tmp-repro/repro/store.parquet
round 1/5 baseline: 2 result(s)
round 1/5 contender: 2 result(s)
round 2/5 baseline: 2 result(s)
round 2/5 contender: 2 result(s)
round 3/5 baseline: 2 result(s)
round 3/5 contender: 2 result(s)
round 4/5 baseline: 2 result(s)
round 4/5 contender: 2 result(s)
round 5/5 baseline: 2 result(s)
round 5/5 contender: 2 result(s)
run compare-20261105T123605Z  profile revisions  baseline cbf062bad0d0  contender d5d9b2e2bb16

REGRESSED (1)
  bench_a                          wall-time  effect +468.69%  noise   6.92%  rounds=5

Activity

  1. MarcoGorelli commented on Nov 5, 2026

    @MarcoGorelli
    CollaboratorAuthor

    nevermind, looks like this was already fixed by #81 , my own main was just behind 🤦

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions