lightspeed: stop measure_impacted when the benchmarks import the project from outside the repository

[ArjunS07]

Purpose

A task image can have two copies of the project: the repository as an editable install, and a regular wheel pulled in as a dependency. asv starts each benchmark as a script, and that process can import the wheel instead of the repository. Then the timings measure the wrong code. In numpy#21464 every benchmark imported numpy 1.22.3 from site-packages, so no patch was ever timed. This PR adds a probe that finds where the benchmark processes import the project from. measure_impacted stops with ProjectShadowed before timing when a package comes from outside the repository.

How it works

flowchart LR
    A[measure_impacted] --> B[import names of the changed files]
    B --> C[probe benchmark, same spawner as the real benchmarks]
    C --> D{every package file inside the repository?}
    D -- yes --> E[run_benchmarks]
    D -- no --> F[raise ProjectShadowed with the paths]

Loading

asv/contrib/lightspeed/
├── project_imports.py   new: import_packages, probe_imports, outside_root
├── session.py           ProjectShadowed, check_project_imports, call in measure_impacted
└── __init__.py          exports ProjectShadowed, check_project_imports
test/test_lightspeed_project_imports.py   new

The probe is a one-function benchmark suite in a temporary folder. get_spawner(env, ..., launch_method) runs it through asv/benchmark.py, the same way as a real benchmark. It first imports the real benchmark suite package, because its __init__.py can change sys.path. Then it imports each package and writes __file__ to a JSON file.

Changes

1. New project_imports.py.
   import_packages(paths, repo_root) gives the top-level import names of changed files: numpy/core/x.c gives numpy, src/skimage/a.py gives skimage. A name is the first folder with an __init__.py. probe_imports(...) gives {package: file}. outside_root(...) keeps the packages whose real path is not inside the repository root. A meson or src/ layout passes when its files are inside the root.
   Effect: callers can see where the benchmark processes import the project from.
   Before: nothing checked this.
2. ProjectShadowed and check_project_imports in session.py.
   ProjectShadowed(ASVError) has .paths, for example {"numpy": ".../site-packages/numpy/__init__.py"}.
   Effect: one call checks the packages and raises with the paths.
   Before: no error type.
3. measure_impacted calls the check before run_benchmarks. The new argument packages=[...] gives the import names explicitly. When it is None, the names come from the changed files.
   Effect: a shadowed run stops before any timing, with the paths in the message.
   Before: the run timed the other copy and reported a speedup near 1.0.

Usage

from asv.contrib.lightspeed import ProjectShadowed, check_project_imports

try:
    check_project_imports(env, benchmark_dir, "spawn", ["numpy"], "/workspace/repo")
except ProjectShadowed as e:
    print(e.paths)  # {'numpy': '/opt/conda/envs/asv_3.9/lib/python3.9/site-packages/numpy/__init__.py'}

Verification

• uv venv -p 3.11 and uv pip install -e ".[test]", then pytest test/test_lightspeed_project_imports.py test/test_lightspeed_survey.py: 6 passed. The tests cover spawn and forkserver: the repository copy passes, and a copy earlier on the path is reported with its path.
• Task image numpy__numpy__21464, these files copied over the installed lsv, run through the task's lsv_measure.py: project shadowed: numpy -> /opt/conda/envs/asv_3.9/lib/python3.9/site-packages/numpy/__init__.py. After the wheel was removed, the same check passed and 4 selected benchmarks were timed.
• Control images pydata__bottleneck__304 and scikit-image__scikit-image__5399: the check passes and the timing is the same as before. Details are in the datasmith PR.

Notes

• initialize_diffcheck does not call the check. The task template removes the second copy before init.
• CI: test (ubuntu-latest, 3.9) has 6 failures. The last main run has the same 6 (test_environment_bench.py x5, test_publish.py::test_branch_name_is_also_filename). The 5 new tests pass in CI.
• The datasmith task template uses this check, and pins LSV_REF to this branch commit 3df656d.