Skip to content

ext: compile the boundary shim optimized and pin the #1207 protocol to --release - #1522

Open
rotnov wants to merge 3 commits into
mainfrom
issue-1516-optimized-ext-shim
Open

rotnov wants to merge 3 commits into
mainfrom
issue-1516-optimized-ext-shim

Conversation

@rotnov

@rotnov rotnov commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Fixes #1516. Part of #1514, which stays open for #1517 and #1518.

Change

  • ext_build::ext_compile_args now leads with -O2 (EXT_SHIM_OPT_LEVEL) on the Linux, macOS and Windows arms, so the fixed C shim src/ext/pycc_ext_module.c is compiled optimized in every build profile. Before this change the driver got no -O flag, so the shim was built at -O0 even under --release. One GNU spelling serves every arm because every driver the build runs is GCC-compatible: the system cc on Linux and macOS, and the bundled clang targeting MSVC on Windows (linker_command). cl.exe is never invoked, so no /O2 arm is needed.
  • Profile decision. A build without --release keeps ordinary pycc build profile semantics: unoptimized IR and the debug pycc_rt. docs/RUNTIME.md has a new "Build profile of an ext artifact" paragraph, and the --ext entry in docs/CLI_SPEC.md now says the same. The hosted ext protocol's Versions bullet in docs/TESTING.md already pins the pycc arm to --release, so the protocol needed no amendment. The new re-measure paragraph says that ext timings are taken from --release and that the debug rows are context only. This decision is reversible and project-local, so no ADR.
  • Tests: the existing per-arm exact-argument tests now expect -O2. A new test, every_platform_compiles_the_shim_at_o2_and_exactly_once, checks that every arm leads with the flag and carries exactly one -O//O flag.

Re-measurement (lark 1.3.1, subject 419d76a7..., CPython 3.14.7 uv build)

The harness, machine and venvs are the same as for the #1515 re-measure. The artifact was built with --ext --foreign-relative-imports by a debug build of pycc. Five arms ran alternately, five processes each.

Arm parse(test_json) median (55 samples) 50-fold input callgrind Ir/parse
CPython 3.14.7 235 µs 11.0 ms 2.14M
main 431a3366, no --release 400 µs (1.71x) 19.4 ms (1.77x) 3.03M
this branch, no --release 337 µs (1.44x) 16.1 ms (1.47x) 2.84M
main 431a3366, --release 332 µs (1.41x) 15.8 ms (1.44x) 2.63M (1.23x)
this branch, --release 267 µs (1.14x) 13.7 ms (1.25x) 2.44M (1.14x)

Ir/parse is (Ir of 600 parses − Ir of 200 parses) / 400. CPython's per-process medians ranged from 214 to 275 µs, so the wall-clock ratios are noisier than the instruction counts. The parity driver matched CPython on the release artifact. The only difference was the order of set members in str(e), which also varies between CPython runs.

Verification

  • cargo fmt --check and cargo clippy -p pycc --all-targets -- -D warnings are clean.
  • cargo test --bin pycc -- ext --include-ignored: 352 passed.
  • 15 hosted ext integration suites passed with --include-ignored: 1036, 1048, 1049, 1050, 1054, 1066, 1067, 1112, 1143, 1145, 1199, 1397, 1427, 1453 and 1461.
  • scripts/check_diff_coverage.py --require-changed-lines 100: 25 of 25 changed lines covered. This ran locally against a cargo llvm-cov --bin pycc run of the toolchain tests, not the full workspace run.
  • Not run locally: the full workspace test suite and the Windows and macOS arms. Those arms are covered by unit tests on argument lists, and by CI.
  • Review: the pinned iEvo deep-reviewer (D-068) is unavailable in this environment, so no local pinned review was run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GLZC6kAwRGYgYdeXGRAjvM


Generated by Claude Code

`ext_compile_args` passed the driver no optimization flag, so the fixed
C shim (`src/ext/pycc_ext_module.c`) was compiled at -O0 even in a
`--release` build. It now leads with `-O2` on every platform arm; every
driver the build runs is GCC-compatible (system `cc`, or the bundled
`clang` targeting MSVC on Windows), so one spelling serves all three.

A build without `--release` keeps the ordinary profile semantics (debug
`pycc_rt`, unoptimized IR); docs/RUNTIME.md and docs/CLI_SPEC.md now say
so, and that ext timings come from `--release`, as the hosted ext
protocol's Versions bullet already requires. docs/TESTING.md records the
lark re-measurement: the release artifact goes from 2.63M to 2.44M
instructions per parse (CPython 2.14M), 1.41x to 1.14x wall clock.

Fixes #1516; part of #1514.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLZC6kAwRGYgYdeXGRAjvM
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T06:14:12.438899Z 93d5354 New commits
🔒 Security Review ✅ Completed 2026-10-10T05:16:20.868508Z b8bd224 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b8bd224164

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/RUNTIME.md
Comment on lines +519 to +526
**Build profile of an `ext` artifact ([#1516](https://github.com/rotnov/pycc/issues/1516)).**
An `--ext` build follows the ordinary `pycc build` profile rule
(`docs/CLI_SPEC.md`, `--release`): without `--release` (or a neighboring
`pycc.toml`'s `opt = "release"`) it emits unoptimized IR and links the
debug-profile `libpycc_rt.a`; with it, the O3-equivalent pipeline and the
release-profile runtime. The fixed C shim (`src/ext/pycc_ext_module.c`) is the
one exception: `ext_build::ext_compile_args` compiles it at `-O2` in every
profile, on every platform arm, since it is already-correct C on the path of

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the roadmap for the ext-profile change

This establishes new build --ext behavior—optimizing the C shim in every profile while leaving IR and pycc_rt profile-dependent—but the commit does not update docs/ROADMAP.md's commit-relative delivery status. Record the new artifact/profile behavior and evidence there so the roadmap continues to describe the containing tree as required.

AGENTS.md reference: AGENTS.md:L38-L38

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the next push: the CLI row of docs/ROADMAP.md now records that the shim is compiled at -O2 in every profile while IR and pycc_rt follow --release, with its evidence.


Generated by Claude Code

Comment thread docs/TESTING.md
Comment on lines +957 to +961
**Re-measured after #1516 (2026-10-10; Part 2 of #1514).** The #1516 branch
compiles the C shim at `-O2` in every build profile, where `main` passed the
driver no optimization flag ("Build profile of an `ext` artifact" in
`docs/RUNTIME.md`). A build without `--release` still links the debug
`pycc_rt` and emits unoptimized IR, as any `pycc build` does. Timings of an

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reconcile the contradictory #1516 status summary

Once this paragraph records #1516 as delivered, the immediately preceding “remaining parts” list still describes #1516 as future work that will optimize both the shim and runtime in the default artifact. That conflicts directly with these added lines, which say the non-release artifact deliberately retains the debug runtime; rewrite the earlier list as historical or mark #1516 delivered with its actual scope.

AGENTS.md reference: AGENTS.md:L36-L36

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the next push: the parts list is now introduced as the parts that remained after #1515, and the #1516 entry states the delivered scope (shim optimized in every profile; runtime and IR follow --release).


Generated by Claude Code

rotnov commented Oct 10, 2026

Copy link
Copy Markdown
Owner Author

native-build-test (macos-15-intel) is red on 93d53549 (and stayed red on its one re-run) only in nbody_release_binary_meets_required_speedup_over_cpython: 18.10x against the 20x gate. That is the known timing flake #641, not this PR: the diff only adds -O2 to the --ext shim's compile arguments, which an nbody native build never invokes, and main 67029281 is green on the same job. No fix exists for #641 yet. The next base-branch merge into this branch re-runs CI.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ext: build an optimized artifact (shim at -O2; no debug runtime in an --ext artifact)

2 participants