diff --git a/docs/arch/native-macos-runner.md b/docs/arch/native-macos-runner.md index e5bd3ad..c57d298 100644 --- a/docs/arch/native-macos-runner.md +++ b/docs/arch/native-macos-runner.md @@ -312,55 +312,46 @@ mode overrides. ## Decisions -### 1. Homebrew: per-job writable prefix over shared read-only base +### 1. Homebrew: shared host prefix, read-only -Jobs need `brew install` for build deps, but we can't let one job's -installs pollute another. The solution uses Homebrew's relocatable -architecture: - -**Host setup (one-time):** `/opt/homebrew` is pre-installed with common -tools (Go, mage, etc.) and marked read-only for the runner user. - -**Per-job overlay:** - -``` -/native// - └── homebrew/ → HOMEBREW_PREFIX, HOMEBREW_CELLAR, HOMEBREW_TEMP - ├── Cellar/ → per-job installs land here - ├── lib/ - ├── bin/ → symlinked from /opt/homebrew/bin at job start - └── Homebrew/ → lightweight Homebrew checkout (or symlink) -``` - -Environment for the runner process: +Native jobs use the host's real Homebrew at `/opt/homebrew` directly: ```bash -HOMEBREW_PREFIX=/homebrew -HOMEBREW_CELLAR=/homebrew/Cellar +HOMEBREW_PREFIX=/opt/homebrew +HOMEBREW_CELLAR=/opt/homebrew/Cellar HOMEBREW_TEMP=/tmp -PATH=/homebrew/bin:/opt/homebrew/bin:/usr/local/bin:... +PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin ``` -How it works: - -1. At job start, create `/homebrew/bin` and symlink all - executables from `/opt/homebrew/bin` into it. This gives the job - read access to pre-installed tools. -2. Set `HOMEBREW_PREFIX` and `HOMEBREW_CELLAR` to the per-job dir. - Any `brew install` writes to the job's Cellar, not the host's. -3. The job's `homebrew/bin` is first in PATH, so newly installed - formulas shadow the host versions if there's a conflict. -4. At job end, `rm -rf ` deletes everything — installs, - caches, temp files. - -**Why not a full Homebrew clone?** Cloning the Homebrew repo takes -~10 seconds and ~500 MB. Symlinking the host's existing install is -instant and zero-copy. The job only needs a writable prefix for new -installs. - -**Why not just share `/opt/homebrew` read-write?** Jobs would step on -each other. One job upgrading a formula mid-build could break another -job. Per-job prefix keeps them independent. +The sandbox denies writes to `/opt/homebrew` (see the profile above), so +a job can *use* the installed formulae but cannot mutate them — one job +can't pollute another or upgrade a formula out from under a concurrent +build. Isolation comes from the write-deny, not from a separate prefix. + +**Host setup:** every build dependency a workflow expects must be +installed on the host and kept current. See +`scripts/provision-native-macos.sh` for the formula list and a +`--check` mode. + +**Why not a per-job writable prefix?** The original design gave each job +its own `HOMEBREW_PREFIX=/homebrew` with an empty Cellar and only +symlinked the host's *binaries* into PATH. That broke any tool that asks +Homebrew "is X installed?" through the Cellar/formula metadata rather than +PATH — notably `spc doctor` (static-php-cli, used by php-sdk's macOS +build). It saw the empty per-job Cellar, decided pkg-config / cmake / etc. +were missing, and tried to install them — which then failed on the +sandbox's `/opt/homebrew` write-deny (reported misleadingly as a network +error). Sharing the host prefix read-only makes those checks pass while +keeping isolation. + +**Trade-off — the host is now the source of truth.** Because jobs can't +`brew install` or `brew upgrade`, a formula the host lacks *or one that +has gone outdated upstream* will fail any workflow step that runs +`brew install ` (that command attempts an upgrade, which the sandbox +blocks — this is exactly how a stale `llvm` failed a php-sdk build until +the host was upgraded). Keep the host provisioned and current: run +`scripts/provision-native-macos.sh` after adding a dependency, and +`brew upgrade` periodically so outdated formulae don't regress builds. ### 2. Keychain: per-job temporary keychain diff --git a/scripts/provision-native-macos.sh b/scripts/provision-native-macos.sh new file mode 100755 index 0000000..7f7eb37 --- /dev/null +++ b/scripts/provision-native-macos.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# Provision a macOS host to run ephemerd native-mode jobs. +# +# WHY THIS EXISTS +# Native mode ([runner.macos] mode = "native") runs GitHub Actions jobs +# directly on the host — there is no per-job VM and no container image. +# So every build dependency a workflow assumes is "already installed" +# must actually exist on this host. In the VM path those deps were baked +# into the macOS VM base disk image; native mode has no such base image, +# so the host must be provisioned here instead. +# +# Jobs run under sandbox-exec with /opt/homebrew mounted READ-ONLY, and +# they use the host's Homebrew directly (HOMEBREW_PREFIX=/opt/homebrew). +# That means a job can *use* installed formulae but cannot `brew install` +# or `brew upgrade` anything. Two failure modes follow: +# 1. Missing formula → the tool check fails deep in a build step, e.g. +# spc doctor: "missing system commands: cmake" +# 2. OUTDATED formula → a workflow step that runs `brew install ` +# triggers an upgrade, which the sandbox blocks with: +# "The following directories are not writable by your user: +# /opt/homebrew ...". (This is exactly how a stale `llvm` failed +# a php-sdk build.) +# So this script both installs missing formulae AND upgrades outdated +# ones — the host must be present *and* current. +# +# CONTRACT +# Keep FORMULAE in sync with what your workflows expect. When a native +# job fails on a missing/outdated tool, add it here (or just re-run this) +# rather than patching one host by hand — that keeps every runner host +# reproducible. Consider running this (or `brew upgrade`) on a schedule. +# +# USAGE +# ./scripts/provision-native-macos.sh # install + upgrade deps +# ./scripts/provision-native-macos.sh --check # report only, no changes +# +# Idempotent: safe to re-run. Homebrew is required and must already be +# installed at /opt/homebrew (Apple silicon). + +set -euo pipefail + +CHECK_ONLY=0 +[[ "${1:-}" == "--check" ]] && CHECK_ONLY=1 + +BREW="${HOMEBREW_PREFIX:-/opt/homebrew}/bin/brew" +if [[ ! -x "$BREW" ]]; then + echo "error: Homebrew not found at $BREW — install it first" >&2 + exit 1 +fi + +# Formulae required by native jobs. Extend as workflows need more. +# llvm — clang/LLVM toolchain; php-sdk's macOS build runs +# `brew install llvm` and prepends it to PATH. +# llvm@17 — libclang for Rust bindgen (ephpm): +# LIBCLANG_PATH=$(brew --prefix llvm@17)/lib +# autoconf/automake/libtool/pkgconf/bison/re2c/cmake/ninja +# — static-php-cli (`spc doctor`) build toolchain for php-sdk. +FORMULAE=( + llvm + llvm@17 + autoconf + automake + libtool + pkgconf + bison + re2c + cmake + ninja +) + +missing=() +outdated=() +for f in "${FORMULAE[@]}"; do + if "$BREW" --prefix "$f" >/dev/null 2>&1 && [[ -d "$("$BREW" --prefix "$f")" ]]; then + if "$BREW" outdated --formula "$f" 2>/dev/null | grep -q .; then + echo "OUTDATED: $f (jobs cannot upgrade — will be upgraded here)" + outdated+=("$f") + else + echo "ok: $f -> $("$BREW" --prefix "$f")" + fi + else + echo "MISSING: $f" + missing+=("$f") + fi +done + +if [[ ${#missing[@]} -eq 0 && ${#outdated[@]} -eq 0 ]]; then + echo "all native-runner deps present and up-to-date" + exit 0 +fi + +if [[ $CHECK_ONLY -eq 1 ]]; then + [[ ${#missing[@]} -gt 0 ]] && echo "missing ${#missing[@]}: ${missing[*]}" >&2 + [[ ${#outdated[@]} -gt 0 ]] && echo "outdated ${#outdated[@]}: ${outdated[*]}" >&2 + exit 1 +fi + +if [[ ${#missing[@]} -gt 0 ]]; then + echo "installing: ${missing[*]}" + "$BREW" install "${missing[@]}" +fi +if [[ ${#outdated[@]} -gt 0 ]]; then + echo "upgrading: ${outdated[*]}" + "$BREW" upgrade "${outdated[@]}" +fi +echo "provisioning complete"