[feat/precompile-tooling] docs: document the opt-in precompiled mode

[henryiii]

🤖 AI text below 🤖

Description

Stacked on #6137; final slice of the pre-compilation series. Adds a "Pre-compiling part of pybind11" section to docs/compiling.rst (CMake keyword/variable/function, config-guard behavior, caveats, and the non-CMake path including a Meson snippet), makes it the first suggestion under the FAQ's "How can I reduce the build time?", and documents pybind11_precompile / pybind11_SRC_DIR / the PRECOMPILE keyword in the CMake config docstring.

Benchmark (macOS arm64, AppleClang 21, Debug, Ninja, the ~50-TU pybind11_tests target; precompile OFF → ON):

Metric	header-only	precompiled	change
Clean build, CPU time	154.8 s	115.0 s	−26 %
Clean build, wall (10 jobs)	15.1 s	12.8 s	−15 %
Incremental (touch one test TU), wall	2.23 s	1.73 s	−22 %

The per-TU CPU saving is what scales for large projects and CI machines with fewer cores; the earlier slices also showed the precompiled test module shrinking ~1.6 MB in Debug.

Suggested changelog entry:

• Added an opt-in precompiled mode that compiles the non-template part of pybind11 once per project instead of once per translation unit: pybind11_add_module(... PRECOMPILE) / PYBIND11_PRECOMPILE in CMake, Pybind11Extension(precompile=True) in setuptools, and python -m pybind11 --srcdir plus a pkg-config srcdir variable for other build systems. Header-only remains the default.

---

📚 Documentation preview 📚: https://pybind11--6138.org.readthedocs.build/

[henryiii]

Codex review:

🤖 AI text below 🤖

Precompiled mode fails for consumers using pybind11's configurable namespace via target-specific definitions.

Review comment:

• [P2] Propagate custom namespaces to the precompiled library — /Users/henryfs/git/pybind/pybind11/tools/pybind11Common.cmake:497-497
  When a module sets PYBIND11_NAMESPACE through target-specific compile definitions, the module references symbols in that namespace while pybind11_precompiled is compiled in the default namespace, causing undefined references. Either propagate this supported configuration to the static target or explicitly reject/detect it alongside the other guarded configuration macros.

[henryiii]

Testing on boost-histogram:

Let's try pybind11's #6138 PR here. Enable the optional precompile mode, and see if it compiles faster and/or with less memory.

🤖 AI text below 🤖

Done — the precompile mode works here, with a modest but real win. Both builds link, import, and pass tests (168 passed against the precompiled build).

Setup (left in the working tree for you to inspect): I pointed the FetchContent GIT_TAG at the PR head (37431e2) and configured a second build with -DPYBIND11_PRECOMPILE=ON. One change was required: PYBIND11_DETAILED_ERROR_MESSAGES was PRIVATE on _core, but the precompiled library must agree on that macro (mismatch is a deliberate link error), so I moved it to add_compile_definitions() before pybind11_add_module. Builds were clean Release/Ninja builds on this machine (18 cores, AppleClang 21), instrumented per-TU with /usr/bin/time -l.

Compile time — every project TU gets faster by a roughly fixed ~0.6–0.9 s (the pybind11 non-template code it no longer compiles):

TU	header-only	precompiled	Δ
register_histograms	17.6 s	16.7 s	−5%
register_axis	8.5 s	7.7 s	−10%
register_accumulators	2.9 s	2.3 s	−22%
register_storage	2.6 s	1.9 s	−25%
register_transforms	2.1 s	1.5 s	−30%
register_algorithm	2.0 s	1.3 s	−35%
module	1.3 s	0.7 s	−44%

Project-TU CPU total dropped 37.0 s → 32.1 s (−13%). The one-time pybind11 library adds back ~5.8 s across 7 tiny TUs, so a clean build's total CPU is a wash (49.0 s → 49.4 s); wall time went 30.4 s → 28.9 s (−5%) because the tiny library TUs parallelize freely. The −13% is what you keep on every rebuild after the first, and per-TU savings show up directly in incremental edit–compile loops.

Memory — each TU's peak RSS drops ~40–70 MB (e.g. register_algorithm 356 → 292 MB, register_axis 1120 → 1076 MB). The build-wide peak barely moves (register_histograms 1914 → 1878 MB, −2%) since it's dominated by Boost.Histogram template instantiation, not pybind11.

Verdict: it does what the pybind11 PR claims — removes a fixed per-TU cost — but boost-histogram's TUs are so template-heavy that the relative gain is small (~13% CPU, ~2% peak memory). The mode would matter more on memory-constrained CI (the small TUs' footprints shrink noticeably) and for projects with many light binding TUs. The binary grew 33 KB, likely because the static library isn't covered by the module's LTO.

The CMakeLists.txt changes and both build trees (build-base, build-pre in the scratchpad) are still there; git checkout CMakeLists.txt reverts when you're done.

[henryiii]

And checking to see if we can enable LTO to get rid of that extra size:

🤖 AI text below 🤖

Yes — it works cleanly. One line in our CMakeLists after pybind11_add_module attaches pybind11's own LTO interface target to the static lib (it's created in our directory scope, so this is legal):

if(TARGET pybind11_precompiled)
  target_link_libraries(pybind11_precompiled PRIVATE pybind11::lto)
endif()

Results for the precompile+LTO variant:

• Binary size: 3,392,920 bytes — now 50 KB smaller than the non-LTO precompiled build (3,442,728) and even 16 KB smaller than the header-only baseline (3,409,144). The size regression is gone, and cross-TU LTO between the library bitcode and the module apparently trims a bit more than header-only mode achieved.
• Build time: unchanged within noise (28.7 s wall / 48.5 s user vs 28.9 / 49.4).
• Correctness: the archive of LLVM bitcode objects went through Apple's ar/ld without complaint, and the test suite passes against the build.

The caveat is portability rather than anything we hit here: with full -flto the static library members are bitcode, so the archive only works when the consumer links with the same LTO-aware toolchain — which is always true in this setup since the library is per-build-tree and never installed. That's arguably a point in favor of pybind11 doing this by default (matching the module's LTO mode in pybind11_add_module); GCC would want -ffat-lto-objects consideration and MSVC /GL has similar same-toolchain constraints, which is presumably why the PR left it off. Could be worth raising on the pybind11 PR.

The edit is in the working tree along with the earlier ones, and the third build tree is at build-prelto in the scratchpad.

[henryiii]

Let's try pybind11's #6138 PR here. Enable the optional precompile mode, and see if it compiles faster and/or with less memory, and if the binaries change size. We can link pybind11's compiled target with lto.

🤖 AI text below 🤖

Done. pybind11's precompiled mode (PR #6138 stack) is a clear win for PyTorch's torch_python on this machine — faster compiles, slightly less compiler memory, and a ~10% smaller binary.

What I set up

• Checked out the PR branch (pr-6138) in third_party/pybind11.
• PyTorch doesn't use pybind11's CMake (it builds its own pybind::pybind11 INTERFACE target from the headers), so I wired the mode manually: an uncommitted block in cmake/Dependencies.cmake behind a new USE_PYBIND11_PRECOMPILE option (auto-forwarded from the env by EnvVarForwarding.cmake). It builds third_party/pybind11/src/*.cpp into a static pybind11_precompiled lib with hidden visibility, PIC, and INTERPROCEDURAL_OPTIMIZATION ON (the LTO you asked for), and attaches it to pybind::pybind11 so torch_python/nnapi_backend get both the archive and the PYBIND11_PRECOMPILED define.

Measurements

A/B by flipping the flag, which rebuilds the identical set of 165 torch_python TUs each way. One methodology catch: ccache is now installed on this machine (the memory saying otherwise was stale), and my first header-only run was 100% cache hits — so I redid both runs cold with CCACHE_DISABLE=1. Cold results (18-core Apple Silicon, BUILD_TEST=0 USE_DISTRIBUTED=0):

Metric	header-only	precompiled	change
Wall time	84.5 s	72.7 s	−14%
CPU time (user+sys)	1014 s	810 s	−20%
ninja per-TU sum	1159 s	912 s	−21%
Peak RSS (largest process)	1.60 GB	1.54 GB	−3.7%
libtorch_python.dylib	27,033,256 B	24,345,672 B	−2.69 MB (−9.9%)

The heaviest pybind11 TUs (torch/csrc/init.cpp, script_init.cpp, guards.cpp, …) each improved 10–19%. Of the binary saving, 1.36 MB is __TEXT (deduplicated formerly-inline code) and the rest is symbol table/LINKEDIT; exported symbol count is unchanged. Costs are tiny: the 7-TU static lib builds once in a few seconds, and LTO raises the libtorch_python link from 0.1 s to 0.6 s.

Sanity check: import torch from the editable install works in precompiled mode and a small matmul computes correctly (also confirming July's editable-import caveat is fixed).

Current state: submodule on pr-6138, the Dependencies.cmake change uncommitted in the working tree, build left in precompiled mode. Numbers line up well with the PR's own ~26% CPU claim, so this would be a nice real-world data point for the pybind11 PR thread — happy to draft a comment for your review if you want one.