fix(macos-vm): always run ephemerd fresh runner, not a stale baked copy#96
Merged
Merged
Conversation
…copy macOS VM jobs would register a runner, report "ready", but never get assigned their job — the VM booted, ran ~90s, exited 0, and the job sat queued forever, re-provisioning a 4GB VM every ~10 min. Native macOS jobs were unaffected, which was the tell. Root cause: the base image bakes a GHA runner into /Users/admin/actions- runner at provisioning time. ephemerd separately copies the *current* runner into /Library/ephemerd/runner over SSH, but the per-job setup script only copied it into the run dir when no run.sh already existed — so the baked runner always won. Once the pinned runner version was bumped (GitHub deprecated the old one; its broker returns 403 and refuses the JIT runner), every VM job silently ran the stale, deprecated runner. Fix: the setup script now always overwrites /Users/admin/actions-runner from the freshly-copied /Library/ephemerd/runner and kills any already- running runner (e.g. one auto-started from a baked LaunchDaemon) first, so the fresh runner owns the job's JIT config. The overwrite is gated on the fresh source existing so the baked fallback is never deleted when ephemerd provided nothing. Extracted the inline setup script to a testable const with a regression guard (TestMacOSRunnerSetupScript_*) asserting the always-refresh + kill behavior and that the old skip-if-exists guard never returns. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
macOS VM jobs would register a runner and report "ready", but GitHub never assigned them their job — the VM booted, ran ~90s, exited 0, and the job sat queued forever, re-provisioning a 4GB VM every ~10 min. Native macOS jobs were unaffected — the tell that pointed straight at the VM runner.
Root cause
The macOS base image bakes a GHA runner into
/Users/admin/actions-runnerat provisioning time. ephemerd separately copies the current runner into/Library/ephemerd/runnerover SSH, but the per-job setup script only copied it into the run dir when norun.shalready existed — so the baked runner always won. Once the pinned runner version was bumped (GitHub deprecated the old one; its broker returns 403 and refuses the JIT runner), every VM job silently ran the stale, deprecated runner → registered but never dispatched → looped.This is the same deprecation that PR #93 fixed for the host/native path — it was still hiding in the base image because we never rebuilt it.
Fix
The setup script now always overwrites
/Users/admin/actions-runnerfrom the freshly-copied/Library/ephemerd/runnerand kills any already-running runner first (e.g. one auto-started from a baked LaunchDaemon), so the fresh runner owns the job's JIT config. The overwrite is gated on the fresh source existing, so the baked fallback is never deleted when ephemerd provided nothing.Extracted the inline setup script to a testable const with a regression guard (
TestMacOSRunnerSetupScript_*) asserting the always-refresh + kill behavior and that the old skip-if-exists guard never returns.Verification
Diagnosed live: caught the looping php-sdk VM job (#168, current HEAD, not a zombie), confirmed the VM runner path via SSH into the guest, and traced it to the setup script's skip-if-exists guard. Deploying the fix to the runner host and confirming the job completes end-to-end (update to follow).
🤖 Generated with Claude Code