Skip to content

refactor: separate Python sources from library runfiles - #1419

Open
jbedard wants to merge 1 commit into
mainfrom
py-library-runfiles-prefactor
Open

refactor: separate Python sources from library runfiles#1419
jbedard wants to merge 1 commit into
mainfrom
py-library-runfiles-prefactor

Conversation

@jbedard

@jbedard jbedard commented Aug 12, 2026

Copy link
Copy Markdown
Member

Avoid adding .py files to DefaultInfo.default_runfiles until absolutely necessary in terminal runnable rules such as py_venv_exec[_test] (which py_binary|test wrap).

This reduces confusion about providers vs runfiles.
This way the .py can more easily be swapped out for .pyc files in the future.

Changes are visible to end-users: no

Test plan

  • Covered by existing test cases

@aspect-workflows

aspect-workflows Bot commented Aug 12, 2026

Copy link
Copy Markdown

✨ Aspect Workflows Tasks

📅 Wed Aug 12 07:03:39 UTC 2026

✅ 42 successful tasks

  • ✅ buildifier · ⏱ 21.3s · 🐙 GitHub Actions · ☑️ Check
    💬 Format complete (clean)
  • ✅ gazelle · ⏱ 16.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Gazelle complete (clean)
  • ✅ test-e2e-bazel-8 [test] · ⏱ 5m 31s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (277/277 passed)
  • ✅ test-e2e-bazel-9 [test] · ⏱ 5m 13s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (271/271 passed)
  • ✅ test-e2e-interpreter-build-config-bazel-8 [test] · ⏱ 25.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-build-config-bazel-9 [test] · ⏱ 49s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-input-validation-bazel-8 [test] · ⏱ 17.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-input-validation-bazel-9 [test] · ⏱ 53.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-runtime-metadata-bazel-8 [test] · ⏱ 20.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-e2e-interpreter-runtime-metadata-bazel-9 [test] · ⏱ 47.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-e2e-interpreter-toolchain-settings-bazel-8 [test] · ⏱ 18.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-toolchain-settings-bazel-9 [test] · ⏱ 49.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-rules-proto-grpc-python-bazel-8 [test] · ⏱ 1m 47s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-rules-proto-grpc-python-bazel-9 [test] · ⏱ 1m 10s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-rules-python-interop-bazel-8 [test] · ⏱ 36.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (11/11 passed)
  • ✅ test-e2e-rules-python-interop-bazel-9 [test] · ⏱ 1m 3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (11/11 passed)
  • ✅ test-e2e-rules-python-provider-compat-bazel-8 [test] · ⏱ 24.9s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (3/3 passed)
  • ✅ test-e2e-rules-python-provider-compat-bazel-9 [test] · ⏱ 25.5s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (3/3 passed)
  • ✅ test-examples-debugger-bazel-8 [test] · ⏱ 26.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-debugger-bazel-9 [test] · ⏱ 38.4s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-dev_deps-bazel-8 [test] · ⏱ 28.3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-dev_deps-bazel-9 [test] · ⏱ 1m 3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-django-bazel-8 [test] · ⏱ 21.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-django-bazel-9 [test] · ⏱ 48.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-multi_version-bazel-8 [test] · ⏱ 30s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-examples-multi_version-bazel-9 [test] · ⏱ 1m 4s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-examples-protobuf-bazel-8 [test] · ⏱ 1m 28s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-protobuf-bazel-9 [test] · ⏱ 2m 3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-py_binary-bazel-8 [test] · ⏱ 25.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-py_binary-bazel-9 [test] · ⏱ 55.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_pex_binary-bazel-8 [test] · ⏱ 28.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-py_pex_binary-bazel-9 [test] · ⏱ 48.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_venv-bazel-8 [test] · ⏱ 33.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (3/3 passed)
  • ✅ test-examples-py_venv-bazel-9 [test] · ⏱ 47.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (3/3 passed)
  • ✅ test-examples-pytest-bazel-8 [test] · ⏱ 40.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (11/11 passed)
  • ✅ test-examples-pytest-bazel-9 [test] · ⏱ 1m 13s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (11/11 passed)
  • ✅ test-examples-uv_pip_compile-bazel-8 [test] · ⏱ 25.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-uv_pip_compile-bazel-9 [test] · ⏱ 1m 20s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-virtual_deps-bazel-8 [test] · ⏱ 28.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-virtual_deps-bazel-9 [test] · ⏱ 40.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-root-bazel-8 [test] · ⏱ 3m 3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (302/302 passed)
  • ✅ test-root-bazel-9 [test] · ⏱ 3m 37s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (301/301 passed)

⏱ Last updated Wed Aug 12 07:10:30 UTC 2026 · 📊 GitHub API quota 1,449/15,000 (10% used, resets in 22m)
🚀 Powered by Aspect CLI (v2026.28.2)  |  Aspect Build · X · LinkedIn · YouTube

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown

py_binary startup benchmark

Version Mean (ms) Median (ms) ± stddev vs BCR vs main Build (s)
BCR 1.11.7 (baseline) 151.590 135.677 ±32.709 30.13
HEAD main 47.321 46.791 ±1.581 -68.8% 8.89
This PR 49.522 48.781 ±4.220 -67.3% +4.7% 6.23

Measured with hyperfine --warmup 5 --runs 50 on Linux
Gate: PR vs HEAD main (threshold: 10%). BCR is shown only as a historical baseline.
Build time: cold bazel build //:bench with isolated output base, no disk cache.

sys.path quality

Version sys.path entries distinct site-packages roots duplicate realpaths
BCR 1.11.7 (baseline) 6 1 0
HEAD main 7 2 0
This PR 7 2 0

sys.path quality measured by bench_syspath inside the assembled venv. Duplicate realpaths indicate symlink redundancy; many distinct site-packages roots suggest an inefficient venv layout.

Bazel analysis benchmark

Version Mean (ms) Median (ms) ± stddev vs BCR vs main Targets Actions
BCR 2.0.0-alpha.5 (baseline) 10747.143 10722.585 ±191.853 301 13672
HEAD main 9482.358 9485.346 ±199.415 -11.8% 301 13740
This PR 9181.343 9202.318 ±134.293 -14.6% -3.2% 301 13740

Measured with hyperfine --warmup 1 --runs 10 on Linux
Gate: PR vs HEAD main (threshold: 10%). BCR is shown only as a historical baseline.
Command: cold bazel build --nobuild //workspace/... with isolated output base, no disk cache.

@jbedard
jbedard force-pushed the py-library-runfiles-prefactor branch 3 times, most recently from 9da2063 to a351428 Compare August 12, 2026 01:10
@jbedard
jbedard requested a review from xangcastle August 12, 2026 01:18
@jbedard
jbedard marked this pull request as ready for review August 12, 2026 01:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a3514283c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread py/private/py_venv/py_venv.bzl
@jbedard

jbedard commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

Robot summary, locations where .py files live:

  • py_library: sources remain in PyInfo and DefaultInfo.files, but not default_runfiles.
  • py_venv: sources remain in VirtualenvInfo.transitive_sources, but not default_runfiles.
  • py_venv_exec[_test]: the executable boundary adds them to runfiles.
  • py_image_layer: only sees the executable’s finalized runfiles.

Remaining .py files are intentional:

  • Files explicitly declared through data;
  • main on a source-mode executable, needed as the entrypoint;
  • DefaultInfo.files on py_library, which is the ordinary Bazel output contract;
  • coverage/instrumentation inputs.

@jbedard
jbedard force-pushed the py-library-runfiles-prefactor branch 5 times, most recently from acd7873 to 621cfe7 Compare August 12, 2026 02:07
@jbedard

jbedard commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 621cfe75aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread py/private/py_library.bzl
@jbedard
jbedard force-pushed the py-library-runfiles-prefactor branch from 621cfe7 to 0d74eb3 Compare August 12, 2026 06:53
@jbedard

jbedard commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d74eb35db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 92 to +94
extra_depsets = [
py_toolchain.files,
srcs_depset,
] + virtual_resolution.srcs + virtual_resolution.runfiles,
] + virtual_resolution.runfiles,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve data dependency sources in runnable venvs

When a standalone py_venv has data = [":plugin"] and that py_library depends on another Python library, ctx.files.data contributes only the plugin's direct source while the newly source-free library runfiles no longer contribute the helper source. The later include_sources = True merge restores only the venv's own srcs/deps closure, so bazel run :venv fails when the plugin imports its helper even though the data contract promises the transitive runtime closure; include Python sources reached through data when constructing the runnable venv.

Useful? React with 👍 / 👎.

Comment on lines +112 to +115
data_sources = [
get_py_info(target).transitive_sources
for target in ctx.attr.data
if has_py_info(target)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Traverse nested Python data edges when restoring sources

This recovery only reads PyInfo from each direct terminal data target, but PyInfo.transitive_sources follows deps, not that target's own data edges. For example, with py_binary(data = [":wrapper"]), wrapper.data = [":plugin"], and plugin.deps = [":helper"], the wrapper's runfiles retain plugin.py through ctx.files.data but lose helper.py because the plugin and helper now have source-free default runfiles; importing the plugin therefore fails. Preserve the source closure across nested data edges rather than restoring only direct data targets' PyInfo.

Useful? React with 👍 / 👎.

@jbedard
jbedard force-pushed the py-library-runfiles-prefactor branch from 0d74eb3 to 085394e Compare August 12, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant