Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 34 additions & 43 deletions docs/arch/native-macos-runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,55 +312,46 @@ mode overrides.

## Decisions

### 1. Homebrew: per-job writable prefix over shared read-only base
### 1. Homebrew: shared host prefix, read-only

Jobs need `brew install` for build deps, but we can't let one job's
installs pollute another. The solution uses Homebrew's relocatable
architecture:

**Host setup (one-time):** `/opt/homebrew` is pre-installed with common
tools (Go, mage, etc.) and marked read-only for the runner user.

**Per-job overlay:**

```
<data_dir>/native/<job-id>/
└── homebrew/ → HOMEBREW_PREFIX, HOMEBREW_CELLAR, HOMEBREW_TEMP
├── Cellar/ → per-job installs land here
├── lib/
├── bin/ → symlinked from /opt/homebrew/bin at job start
└── Homebrew/ → lightweight Homebrew checkout (or symlink)
```

Environment for the runner process:
Native jobs use the host's real Homebrew at `/opt/homebrew` directly:

```bash
HOMEBREW_PREFIX=<job_dir>/homebrew
HOMEBREW_CELLAR=<job_dir>/homebrew/Cellar
HOMEBREW_PREFIX=/opt/homebrew
HOMEBREW_CELLAR=/opt/homebrew/Cellar
HOMEBREW_TEMP=<job_dir>/tmp
PATH=<job_dir>/homebrew/bin:/opt/homebrew/bin:/usr/local/bin:...
PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
```

How it works:

1. At job start, create `<job_dir>/homebrew/bin` and symlink all
executables from `/opt/homebrew/bin` into it. This gives the job
read access to pre-installed tools.
2. Set `HOMEBREW_PREFIX` and `HOMEBREW_CELLAR` to the per-job dir.
Any `brew install` writes to the job's Cellar, not the host's.
3. The job's `homebrew/bin` is first in PATH, so newly installed
formulas shadow the host versions if there's a conflict.
4. At job end, `rm -rf <job_dir>` deletes everything — installs,
caches, temp files.

**Why not a full Homebrew clone?** Cloning the Homebrew repo takes
~10 seconds and ~500 MB. Symlinking the host's existing install is
instant and zero-copy. The job only needs a writable prefix for new
installs.

**Why not just share `/opt/homebrew` read-write?** Jobs would step on
each other. One job upgrading a formula mid-build could break another
job. Per-job prefix keeps them independent.
The sandbox denies writes to `/opt/homebrew` (see the profile above), so
a job can *use* the installed formulae but cannot mutate them — one job
can't pollute another or upgrade a formula out from under a concurrent
build. Isolation comes from the write-deny, not from a separate prefix.

**Host setup:** every build dependency a workflow expects must be
installed on the host and kept current. See
`scripts/provision-native-macos.sh` for the formula list and a
`--check` mode.

**Why not a per-job writable prefix?** The original design gave each job
its own `HOMEBREW_PREFIX=<job_dir>/homebrew` with an empty Cellar and only
symlinked the host's *binaries* into PATH. That broke any tool that asks
Homebrew "is X installed?" through the Cellar/formula metadata rather than
PATH — notably `spc doctor` (static-php-cli, used by php-sdk's macOS
build). It saw the empty per-job Cellar, decided pkg-config / cmake / etc.
were missing, and tried to install them — which then failed on the
sandbox's `/opt/homebrew` write-deny (reported misleadingly as a network
error). Sharing the host prefix read-only makes those checks pass while
keeping isolation.

**Trade-off — the host is now the source of truth.** Because jobs can't
`brew install` or `brew upgrade`, a formula the host lacks *or one that
has gone outdated upstream* will fail any workflow step that runs
`brew install <it>` (that command attempts an upgrade, which the sandbox
blocks — this is exactly how a stale `llvm` failed a php-sdk build until
the host was upgraded). Keep the host provisioned and current: run
`scripts/provision-native-macos.sh` after adding a dependency, and
`brew upgrade` periodically so outdated formulae don't regress builds.

### 2. Keychain: per-job temporary keychain

Expand Down
105 changes: 105 additions & 0 deletions scripts/provision-native-macos.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# Provision a macOS host to run ephemerd native-mode jobs.
#
# WHY THIS EXISTS
# Native mode ([runner.macos] mode = "native") runs GitHub Actions jobs
# directly on the host — there is no per-job VM and no container image.
# So every build dependency a workflow assumes is "already installed"
# must actually exist on this host. In the VM path those deps were baked
# into the macOS VM base disk image; native mode has no such base image,
# so the host must be provisioned here instead.
#
# Jobs run under sandbox-exec with /opt/homebrew mounted READ-ONLY, and
# they use the host's Homebrew directly (HOMEBREW_PREFIX=/opt/homebrew).
# That means a job can *use* installed formulae but cannot `brew install`
# or `brew upgrade` anything. Two failure modes follow:
# 1. Missing formula → the tool check fails deep in a build step, e.g.
# spc doctor: "missing system commands: cmake"
# 2. OUTDATED formula → a workflow step that runs `brew install <it>`
# triggers an upgrade, which the sandbox blocks with:
# "The following directories are not writable by your user:
# /opt/homebrew ...". (This is exactly how a stale `llvm` failed
# a php-sdk build.)
# So this script both installs missing formulae AND upgrades outdated
# ones — the host must be present *and* current.
#
# CONTRACT
# Keep FORMULAE in sync with what your workflows expect. When a native
# job fails on a missing/outdated tool, add it here (or just re-run this)
# rather than patching one host by hand — that keeps every runner host
# reproducible. Consider running this (or `brew upgrade`) on a schedule.
#
# USAGE
# ./scripts/provision-native-macos.sh # install + upgrade deps
# ./scripts/provision-native-macos.sh --check # report only, no changes
#
# Idempotent: safe to re-run. Homebrew is required and must already be
# installed at /opt/homebrew (Apple silicon).

set -euo pipefail

CHECK_ONLY=0
[[ "${1:-}" == "--check" ]] && CHECK_ONLY=1

BREW="${HOMEBREW_PREFIX:-/opt/homebrew}/bin/brew"
if [[ ! -x "$BREW" ]]; then
echo "error: Homebrew not found at $BREW — install it first" >&2
exit 1
fi

# Formulae required by native jobs. Extend as workflows need more.
# llvm — clang/LLVM toolchain; php-sdk's macOS build runs
# `brew install llvm` and prepends it to PATH.
# llvm@17 — libclang for Rust bindgen (ephpm):
# LIBCLANG_PATH=$(brew --prefix llvm@17)/lib
# autoconf/automake/libtool/pkgconf/bison/re2c/cmake/ninja
# — static-php-cli (`spc doctor`) build toolchain for php-sdk.
FORMULAE=(
llvm
llvm@17
autoconf
automake
libtool
pkgconf
bison
re2c
cmake
ninja
)

missing=()
outdated=()
for f in "${FORMULAE[@]}"; do
if "$BREW" --prefix "$f" >/dev/null 2>&1 && [[ -d "$("$BREW" --prefix "$f")" ]]; then
if "$BREW" outdated --formula "$f" 2>/dev/null | grep -q .; then
echo "OUTDATED: $f (jobs cannot upgrade — will be upgraded here)"
outdated+=("$f")
else
echo "ok: $f -> $("$BREW" --prefix "$f")"
fi
else
echo "MISSING: $f"
missing+=("$f")
fi
done

if [[ ${#missing[@]} -eq 0 && ${#outdated[@]} -eq 0 ]]; then
echo "all native-runner deps present and up-to-date"
exit 0
fi

if [[ $CHECK_ONLY -eq 1 ]]; then
[[ ${#missing[@]} -gt 0 ]] && echo "missing ${#missing[@]}: ${missing[*]}" >&2
[[ ${#outdated[@]} -gt 0 ]] && echo "outdated ${#outdated[@]}: ${outdated[*]}" >&2
exit 1
fi

if [[ ${#missing[@]} -gt 0 ]]; then
echo "installing: ${missing[*]}"
"$BREW" install "${missing[@]}"
fi
if [[ ${#outdated[@]} -gt 0 ]]; then
echo "upgrading: ${outdated[*]}"
"$BREW" upgrade "${outdated[@]}"
fi
echo "provisioning complete"