perf(ci): stop the mutation job from starving blocking checks - #99
Merged
Conversation
PR wall-clock was ~13-20 min against only ~3.5 min of actual compute. The bottleneck was never job speed — it was the single macmini runner service, which gives ledgr a concurrency of 1, so even `test` and `mutation` from one PR queued behind each other. Three fixes that cut how much of that scarce runner time we burn: - Drop `cache: pnpm` from both self-hosted jobs. The pnpm store already persists in $HOME between jobs on a self-hosted runner, so actions/cache was round-tripping a tarball over the network for nothing. It cost up to 79s in the post-job cache save — longer than the test step it was meant to speed up. - Gate the mutation job behind a cheap ubuntu-latest job. This repo is public, so hosted runners are free with unlimited concurrency. A PR touching no mutatable source previously spent ~35s of checkout/setup/install on the mini just to print "nothing to mutate"; now it never occupies a self-hosted slot. - Give scripts/mutate-diff.sh a --list-only mode and have the gate call it, so the gate and the run can never disagree about what counts as mutatable. Also adds .github/actionlint.yaml declaring the custom `macmini` label, so `actionlint .github/workflows/*.yml` runs clean locally instead of flagging every self-hosted `runs-on`.
Reverts the ubuntu-latest gate job. Everything stays on the mini — no GitHub-hosted runners in ci.yml. Same problem, local solution: the mutation job was starving the blocking checks. A 9-file diff kept Stryker busy for 30+ min while typecheck/lint/test of every other PR queued behind a job that is continue-on-error: true and gates nothing. 'needs: test' orders it after the blocking job releases the runner, so blocking checks always go first. Side effect, and it's a wanted one: a red test job now skips mutation instead of mutating a build that is already broken. Also drops the --list-only mode added to scripts/mutate-diff.sh for the gate, since nothing calls it any more.
KenTaniguchi-R
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR wall-clock has been 13–20 min against only ~3.5 min of actual compute. Two causes, both about the self-hosted runner pool being the scarce resource:
macmini-ledgrwas the only runner, so ledgr's runner concurrency was 1 — eventestandmutationfrom the same PR queued behind each other.mutationjob iscontinue-on-error: trueand gates nothing, but a wide diff keeps Stryker busy for a long time. Observed live: a 9-file diff onfix/liability-sign-conventionheld the runner for 30+ minutes while the blockingtypecheck · lint · testjobs of two other PRs sat queued behind it.Changes
1.
needs: teston the mutation job.Orders mutation after the blocking job releases the runner, so blocking checks always go first. Stays on the self-hosted runner — no GitHub-hosted runners are used.
Wanted side effect: a red
testjob now skips mutation rather than mutating an already-broken build.2. Drop
cache: pnpmfrom both jobs.On a self-hosted runner the pnpm store already persists in
$HOMEbetween jobs, soactions/cachewas round-tripping a tarball over the network for zero benefit. Measured cost: up to 79s in the post-job cache save — longer than the test step it was supposedly accelerating.3.
.github/actionlint.yamldeclaring the custommacminilabel, soactionlint .github/workflows/*.ymlruns clean locally instead of flagging every self-hostedruns-on.Also done (not a repo change)
Registered a second runner service on the mini (
macmini-ledgr-2, agent id 22), sotestjobs from different PRs now run in parallel instead of serially.Verification
actionlint .github/workflows/*.yml— clean across all three workflowsmutationjob kept explicit rather than inherited, so the protection is visible in the job it protectsubuntu-latestinci.yml;codeql.ymlanddocker-publish.ymlare untouched🤖 Generated with Claude Code