Skip to content

[AAASM-2189] 🐛 (ci): Install protoc in maturin wheel builds (Linux + macOS)#68

Merged
Chisanan232 merged 3 commits into
masterfrom
v0.0.1/AAASM-2109/fix/maturin_protoc
Jun 1, 2026
Merged

[AAASM-2189] 🐛 (ci): Install protoc in maturin wheel builds (Linux + macOS)#68
Chisanan232 merged 3 commits into
masterfrom
v0.0.1/AAASM-2109/fix/maturin_protoc

Conversation

@Chisanan232
Copy link
Copy Markdown
Contributor

@Chisanan232 Chisanan232 commented May 29, 2026

Description

The v0.0.1-alpha.2 dry-run's Release Python SDK workflow had all 4 wheel builds fail with:

error: failed to run custom build command for `aa-proto v0.0.1 ...`
Caused by: Could not find `protoc`. If `protoc` is installed, try
  setting the `PROTOC` environment variable to the path of the
  `protoc` binary. To install it on Debian, run
  `apt-get install protobuf-compiler`.

aa-proto (a git dep on agent-assembly) has a build.rs that uses prost-build, which requires the protoc compiler. Neither the manylinux container nor the GitHub macOS runners ship protoc by default.

Fix

Two install patterns depending on the build environment:

Job Approach
build-linux-x86_64 (manylinux: auto) Add before-script-linux: yum install -y protobuf-compiler || dnf install -y protobuf-compiler to the PyO3/maturin-action@v1 invocation. yum covers manylinux2014 (CentOS 7); dnf covers manylinux_2_28+ (Rocky/Alma).
build-linux-aarch64 (manylinux: auto) Same pattern.
build-macos-arm64 (macos-14 runner) Add run: brew install protobuf step BEFORE the maturin-action. macOS doesn't have docker; before-script-linux doesn't apply.
build-macos-x86_64 (macos-13 runner) Same pattern.
build-sdist No change — sdist doesn't compile, so protoc not needed.

Local verification

  • actionlint .github/workflows/release-python.yml clean (pre-existing macos-13 runner warning is unrelated — Apple retired that label from GitHub-hosted runners; same warning exists on master)
  • before-script-linux is a documented PyO3/maturin-action@v1 input (ref: https://github.com/PyO3/maturin-action#inputs)

Note on CI

Per maintainer direction, GitHub Actions billing limit is hit — CI won't run on this PR. The fix is verified by inspection + memory project_aa_cli_compat_protoc_requirement.md confirms this is the same root cause we've fixed in agent-assembly CI before. Re-running once billing resets is the verification.

Related Issues

Type of Change

  • 🐛 Bug fix

— Claude Code (Opus 4.7, 1M context)

The v0.0.1-alpha.2 dry-run surfaced this in the manylinux wheel
build for x86_64 and aarch64:

  error: failed to run custom build command for `aa-proto v0.0.1 ...`
  Caused by: Could not find `protoc`. If `protoc` is installed, try
  setting the `PROTOC` environment variable to the path of the
  `protoc` binary. To install it on Debian, run
  `apt-get install protobuf-compiler`.

`aa-proto` (a git dependency on agent-assembly) has a build.rs that
uses prost-build, which requires the protoc compiler. The manylinux2014
image (CentOS 7-based; manylinux_2_28+ uses Rocky/Alma) and the
GitHub macOS runners do not ship protoc by default.

Fix:

  * Linux (manylinux: auto):
    `before-script-linux: yum install -y protobuf-compiler || dnf install ...`
    Runs inside the manylinux container before maturin invokes cargo.
    yum covers manylinux2014; dnf covers manylinux_2_28+.

  * macOS (macos-14 / macos-13 runners):
    A regular `run: brew install protobuf` step before the
    maturin-action invocation. macOS doesn't have docker, so
    before-script-linux is not applicable.

Both x86_64 and aarch64 Linux variants, plus both macOS variants,
are patched uniformly.

Tracked: AAASM-2109
Local verification surfaced a bug in my prior fix: the CentOS 7-based
manylinux2014 image's `protobuf-compiler` package ships protoc 2.5.0
which only supports proto2 syntax. aa-proto's .proto files use proto3,
producing:

  test.proto:1:10: Unrecognized syntax identifier "proto3".
  This parser only recognizes "proto2".

Fix: download the official protoc binary release (v32.1) from GitHub
in the before-script-linux block instead of relying on the OS package.
Works for both x86_64 and aarch64 manylinux variants.

Verified locally inside `quay.io/pypa/manylinux2014_x86_64:latest`:
  * curl + unzip protoc-32.1-linux-x86_64.zip → libprotoc 32.1
  * proto3 sample compile → ✓ OK

The macOS step (`brew install protobuf`) was already correct —
homebrew tracks latest protoc.

Tracked: AAASM-2189
@Chisanan232
Copy link
Copy Markdown
Contributor Author

Chisanan232 commented Jun 1, 2026

Claude Code review — AAASM-2189

CI state

7 SUCCESS + 6 SKIPPED, 0 failuresmergeable=MERGEABLE, mergeStateStatus=CLEAN. The 6 SKIPPED checks are all by-design for PR context (same intentional-skip pattern as PR #67):

Skipped check Why
docs-build intent.yaml sets artifacts.docs: skip (PR #66 work)
Validate DockerHub Build / Validate GHCR Build No Dockerfile to validate
Security Scan (Supply Chain) Tag-push context only
Load Package Configuration Only the Parse Release Intent job needs to fire on PR

All 7 actually-applicable checks pass: parse-config, intent-parse, Dockerfile-check, build-git-tag-and-create-release, Python Package test, Validation Summary, Validation Summary (top-level).

Important note about the wheel-build verification

The Release Python SDK workflow (the one that contains the maturin wheel builds — the actual target of this fix) does NOT fire on PR events; it's tag-push triggered only. So CI here can't directly verify the protoc fix unblocks the wheel build.

That gap is covered by local end-to-end verification I ran on master before pushing:

Container: quay.io/pypa/manylinux2014_x86_64:latest (linux/amd64 via Rosetta)
Setup:     yum install unzip → curl GitHub releases protoc 32.1 → rustup → maturin
Command:   maturin build --release --interpreter 3.12 --manylinux 2014 ...

Result:    📦 Built wheel for CPython 3.12 to /io/dist/agent_assembly_core-0.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
           (621KB wheel, 1m00s Rust compile after install, exit 0)

The full chain runs cleanly: protoc 32.1 installs, aa-proto's build.rs (which uses prost-build for proto3) succeeds, the entire dep graph including pyo3-ffi compiles, and a valid manylinux wheel lands in dist/.

Scope vs. acceptance criteria

AC Verified Status
before-script-linux block downloads protoc 32.1 binary instead of the broken yum approach release-python.yml lines updated for both x86_64 and aarch64 Linux paths
macOS path adds brew install protobuf before maturin-action release-python.yml lines updated for both macos arm64 and x86_64 paths
The downloaded protoc actually handles proto3 (where the yum-installed protoc 2.5.0 doesn't) Local: libprotoc 32.1 + proto3 sample compile ✓ OK
aa-proto build.rs succeeds with new protoc Local maturin build advanced past the prost-build/aa-proto step
Full wheel build succeeds end-to-end inside manylinux2014 Local: 621KB manylinux wheel produced for CPython 3.12
actionlint clean (pre-existing macos-13 runner warning unrelated; same on master)

Commit history

# Commit Notes
1 ebf79b5 🐛 (ci): Install protoc in maturin wheel builds (Linux + macOS) — incomplete initial attempt using yum
2 b841666 🐛 (ci): Use protoc binary download instead of broken yum package — the actual fix, after local verification caught that yum's protobuf-compiler ships protoc 2.5.0 (proto2 only)

The two-commit history is the right trail of the work — the first commit's design (yum) was caught by local verification before merge, and replaced with the binary-download approach in the second commit.

Verdict

Ready for human approval and merge. Local end-to-end verification (protoc install + maturin build → manylinux wheel produced) covers the gap that the PR-CI can't reach. The actual release pipeline's wheel-build verification will happen on the next tag push.

— Claude Code (Opus 4.7, 1M context)

Supply-chain hardening per senior code review:

  * SHA256 verification: refuse to install protoc if the downloaded
    zip doesn't match the expected SHA256 (cross-verified against
    the GitHub release API's `digest` field on v32.1 assets).
  * Retry: --retry 3 --retry-delay 5 on curl so a single network
    blip doesn't fail the build.
  * Centralized env: PROTOC_VERSION + PROTOC_SHA256_X86_64 +
    PROTOC_SHA256_AARCH_64 all at workflow level. Bumping protoc
    is one edit (update version + 2 SHAs at the top).
  * GH expression interpolation (`${{ env.X }}`) instead of shell
    `$VAR` references. Substitutes at workflow-parse time before
    the script reaches the manylinux container, so we don't rely
    on maturin-action propagating env vars through `docker run`.

Without this gate, the previous commit downloaded an arbitrary
binary and ran it as root inside CI with no integrity check.

Verified locally end-to-end in quay.io/pypa/manylinux2014_x86_64:
  * good SHA → install succeeds, protoc --version returns libprotoc 32.1
  * tampered zip → rejected before install (exit 1, no execution)
  * bad SHA in script → exit 1, no install

SHAs:
  protoc-32.1-linux-x86_64.zip  : e9c129c176bb7df02546c4cd6185126ca53c89e7d2f09511e209319704b5dd7e
  protoc-32.1-linux-aarch_64.zip: 4a802ed23d70f7bad7eb19e5a3e724b3aa967250d572cadfd537c1ba939aee6a

(macOS path remains brew install protobuf — that's a separate trust
boundary handled by homebrew's own signing model. Further hardening
to also use a binary download on macOS is a follow-up if needed.)

Tracked: AAASM-2189
@Chisanan232
Copy link
Copy Markdown
Contributor Author

Claude Code follow-up — supply-chain hardening (commit e1b0424)

Per senior code review, hardening the protoc binary install against supply-chain risk.

What changed

 env:
   PYTHON_VERSION: '3.12'
+  PROTOC_VERSION: '32.1'
+  PROTOC_SHA256_X86_64: 'e9c129c176bb7df02546c4cd6185126ca53c89e7d2f09511e209319704b5dd7e'
+  PROTOC_SHA256_AARCH_64: '4a802ed23d70f7bad7eb19e5a3e724b3aa967250d572cadfd537c1ba939aee6a'

And in the before-script-linux for both Linux jobs:

- curl -sSLf "..." -o /tmp/protoc.zip
+ curl -sSLf --retry 3 --retry-delay 5 "..." -o /tmp/protoc.zip
+ echo "${EXPECTED_SHA}  /tmp/protoc.zip" | sha256sum --check --status \
+   || { echo "::error::SHA256 mismatch — refusing to install"; sha256sum /tmp/protoc.zip; exit 1; }
  unzip -o /tmp/protoc.zip -d /usr/local >/dev/null

Plus all the ${PROTOC_VERSION} shell-var references converted to ${{ env.PROTOC_VERSION }} GH expressions — they substitute at workflow-parse time, before the script reaches the manylinux container, so we don't rely on maturin-action propagating env vars through docker run.

SHA256s sourced

Both SHAs cross-verified against the GitHub release API's digest: field on the v32.1 release assets — not just from the artifact file itself. This is two independent integrity sources:

$ gh api /repos/protocolbuffers/protobuf/releases/tags/v32.1 --jq '.assets[] | select(.name|startswith("protoc-32.1-linux-")) | {name, digest}'
  protoc-32.1-linux-x86_64.zip   → sha256:e9c129c176bb7df02546c4cd6185126ca53c89e7d2f09511e209319704b5dd7e
  protoc-32.1-linux-aarch_64.zip → sha256:4a802ed23d70f7bad7eb19e5a3e724b3aa967250d572cadfd537c1ba939aee6a

$ sha256sum protoc-32.1-linux-x86_64.zip   # downloaded fresh
  e9c129c176bb7df02546c4cd6185126ca53c89e7d2f09511e209319704b5dd7e  protoc-32.1-linux-x86_64.zip
$ sha256sum protoc-32.1-linux-aarch_64.zip
  4a802ed23d70f7bad7eb19e5a3e724b3aa967250d572cadfd537c1ba939aee6a  protoc-32.1-linux-aarch_64.zip

Local end-to-end verification

Inside quay.io/pypa/manylinux2014_x86_64:latest (linux/amd64 via Rosetta), three paths tested:

Scenario Expected Actual
Good SHA + clean download Install proceeds; protoc --version returns libprotoc 32.1
Downloaded zip tampered (overwritten with "garbage" after download, before SHA check) Script exits 1 BEFORE unzip; no install
Wrong EXPECTED_SHA in script (e.g. all-zeros) sha256sum --check exits nonzero; install rejected

Threat model after hardening

Before: an attacker who compromised the GitHub release artifact (or MITM'd despite HTTPS) could install arbitrary code as root in CI.

After: the attacker also needs to substitute the SHA in this workflow file at the same time. That requires a write to the python-sdk repo's main branch, which is gated by PR review.

Net: shifts the trust boundary from "GitHub release storage" to "PR review + workflow file integrity".

Operational note

Bumping protoc in the future is now a single PR touching 3 lines in env: (version + 2 SHAs). Cross-verify the new SHAs against the GitHub release API's digest field at the time of bump, same way I did here.

— Claude Code (Opus 4.7, 1M context)

@Chisanan232 Chisanan232 merged commit ae7f8ff into master Jun 1, 2026
13 checks passed
@Chisanan232 Chisanan232 deleted the v0.0.1/AAASM-2109/fix/maturin_protoc branch June 1, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant