Skip to content

setup.sh: install pinned protoc on macOS instead of brew install - #210

Open
rbrenton wants to merge 2 commits into
mainfrom
feature/setup-macos-pinned-protoc
Open

rbrenton wants to merge 2 commits into
mainfrom
feature/setup-macos-pinned-protoc

Conversation

@rbrenton

@rbrenton rbrenton commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Problem

scripts/setup.sh ran brew install protobuf on macOS whenever check_protoc failed. On the self-hosted macOS ARM64 runner the Homebrew prefix is owned by a different user than the one running CI, so that fails with "Cellar is not writable" and the "Setup dependencies" step fails for every job.

The trigger on 2026-10-05 was a Homebrew upgrade of abseil that left the installed protobuf unable to load its dylib. protoc exits with a dyld "Library not loaded" error, check_protoc correctly treats that as missing, and the script then falls into the brew path it cannot complete.

Change

Same fix as indextables/indextables_spark#405, adapted to this file.

  • macOS: new install_protoc_macos. A working protoc on PATH is left alone. Otherwise the pinned GitHub release (PROTOC_VERSION, already 25.5 for Linux) is unpacked, bin/ and include/ together, into $HOME/.local/share/indextables/protoc-<version> with no brew and no sudo. Its bin dir is prepended to PATH so it wins over a broken copy earlier on PATH, and appended to $GITHUB_PATH when set so later workflow steps see it. An existing pinned install is reused on later runs. Extraction goes to a staging dir and is renamed into place, so an interrupted run never leaves a half-unpacked dir that looks installed.
  • Shared download helper: download_protoc <asset> <dest> uses curl -fsSL --retry 3 and verifies the zip against a SHA-256 pinned per release asset (protoc_expected_sha256) before unzipping.
  • Linux: switched to the shared helper, so it gains curl -f and checksum verification. Install destination (/usr/local) is unchanged.
  • Java and Maven on macOS still come from Homebrew.

The four SHA-256 values were recomputed from the v25.5 release assets and match the ones pinned in indextables_spark.

Testing

Run with macOS /bin/bash 3.2.57, loading only the function definitions with HOME pointed at a scratch directory (the full script was not run):

  • bash -n scripts/setup.sh passes.
  • Broken protoc first on PATH (prints a dyld "Library not loaded" error, exits non-zero): pinned protoc is installed under the scratch HOME, protoc --version then resolves to it and prints libprotoc 25.5, include/google/protobuf is present, the bin dir is appended to the GITHUB_PATH file, and no staging or temp dirs are left behind.
  • Second run with a curl that fails if called: reuse path taken, no download, GITHUB_PATH appended again.
  • Working protoc on PATH: reported as already installed, nothing created under HOME, GITHUB_PATH untouched.
  • Wrong pinned checksum: exits 1 with expected/actual hashes, nothing unpacked.
  • Unknown asset name: exits 1 without downloading.
  • download_protoc linux-x86_64: downloads, verifies, and unpacks an ELF x86-64 protoc plus include/.

Not covered locally: the Linux branch of install_linux end to end (only the shared helper it now calls), and the osx-x86_64 / linux-aarch_64 downloads through the script (their checksums were verified against the release assets directly). The macOS CI job on this PR exercises the real runner.

On a self-hosted macOS runner the Homebrew prefix can be owned by a
different user than the one running CI. When a Homebrew upgrade of a
protobuf dependency (abseil) left the installed protoc unable to load
its dylib, check_protoc correctly reported it unusable, and the script
then ran `brew install protobuf`, which fails with "Cellar is not
writable". Every CI job failed at "Setup dependencies" until someone
repaired Homebrew by hand.

Make the macOS path self-healing: when protoc is missing or broken,
unpack the pinned protoc release into a version-named directory under
$HOME (no brew, no sudo), put it first on PATH for the rest of the
script, and export it via GITHUB_PATH. A working protoc already on PATH
is left alone, and an existing pinned install is reused.

Release downloads are now verified against SHA-256 values pinned in the
script and fetched with `curl -f`, via a helper shared with the Linux
branch. The Linux install destination is unchanged.

This mirrors the same change in indextables_spark so the two setup
scripts stay in step.
@claude

claude Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Claude encountered an error —— View job


I'll analyze this and get back to you.

GitHub reported this branch as conflicting with main, but the merge of
current main applies cleanly: main's new commits change CI workflows
and Dependabot configuration and do not touch scripts/setup.sh, which
stays exactly as on this branch.
@github-actions

Copy link
Copy Markdown

Automated review: PASS

Commit 3c348a004d43e02a61f98da61c2c2b43446c7b95 of pull request #210 (workflow run)

Summary: Changes scripts/setup.sh only. macOS protoc now comes from a pinned GitHub release unpacked under $HOME/.local/share, replacing Homebrew. Linux and macOS share a download_protoc helper that verifies a pinned SHA-256 before unzipping. The logic looks sound: the download fails on HTTP errors, the staging dir is renamed into place, temp dirs are cleaned on error, and no dependency or build files change. I found no blocking defects.

Findings (2)

  1. medium scripts/setup.sh:66 The four pinned SHA-256 values are described as trust-on-first-use and I could not check them against the official v25.5 release assets. A wrong value would make setup fail on that platform. A maintainer should check each hash against the release before merge.
  2. low scripts/setup.sh:139 use_pinned_protoc appends the bin dir to $GITHUB_PATH on every run without checking for a duplicate. This is harmless, but the entry is repeated if setup runs more than once in a job.

Produced from the diff alone by an automated reviewer. Text in code formatting is reviewer output shown as data. A pass is advisory: it does not replace CI or human review.

This branch was successfully deployed

No deployments
test-runners — 3ab2a736 Deployed Oct 5, 2026 by rbrenton via Build & Test (Linux X64) #123
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant