Skip to content

ci(cli): split the four slowest mutation scopes into parallel shards - #1007

Merged
blafourcade merged 3 commits into
nextfrom
chore/shard-mutation
Oct 12, 2026
Merged

blafourcade merged 3 commits into
nextfrom
chore/shard-mutation

Conversation

@blafourcade

@blafourcade blafourcade commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🎯 What & why

The CLI CI took 45 minutes, and the mutation jobs were all of it: framework 44.6 min, kernel 35.0, tools 29.6, telemetry 29.5, everything else 15 or less (run 38247560624). Each scope ran in one job with 4 Stryker processes. This splits the four slowest scopes into parallel legs, without changing what is measured.

🛠️ How it works

  • Split. mutation-scopes.json gives framework, tools and kernel 4 shards, telemetry 3, tools-codex and translate 2. A scope's files are cut by weight (line count): heaviest first, each to the lightest leg so far, ties by path. The same commit gives the same split on every machine. The heaviest leg stays within 1.25 of a fair share (tested for each sharded scope).
  • Legs. Each leg runs the whole stryker.conf.json with only its own mutate list, incremental file, sandbox and report paths (run-mutation.mjs --shard i/n). A leg applies no floor.
  • Score. cli / Mutation (<scope>) downloads every leg's report and manifest, merges them and applies the same scoreOf and floor as one run (--aggregate). It fails when a leg is missing, when a leg mutated other files than the split gives it at this commit, or when the legs hold no mutant. It runs if: always(), so a crashed leg fails the score instead of skipping it, and gate needs it.
  • Unchanged. The canary and every other scope stay one job. pnpm test:mutation:<scope> runs a scope whole, locally, as before.
  • Cache keys. Incremental keys now close the scope name with --: mutation-tools- used to match mutation-tools-codex-… too.

🧪 How to verify

  • cd cli && pnpm exec vitest run tests/architecture (164), node --test 'scripts/__tests__/**/*.test.js' (681).
  • mutation-shards.arch.test.ts: the split partitions each sharded scope, is balanced and deterministic, honours ! globs; a leg keeps the whole config; the score fails on a missing, drifted or empty leg.
  • Equivalence, measured on tools-codex split in two for the proof: one run and two legs plus the score give the same 444 detected, 21 undetected and the same survivors, score 95.5 both ways.

📏 Measured

CLI CI wall time Before After
Cold incremental cache 45.3 min (run 38247560624) 27.7 min (run 38258866316, attempt 1)
Warm incremental cache 27.5 min (run 38252812060) 13.8 min (run 38258866316, attempt 2)
Then kernel in 4, tools-codex and translate in 2: partly cold 18.3 min (run 38268276590, attempt 1; slowest leg 14.1, against 21.4 before)
Same, warm 15.2 min (run 38268276590, attempt 2)

The second split shortens the cold worst case; warm, it changes nothing measurable (13.8 then 15.2 min, within run-to-run spread): the slowest warm legs are now about 10 to 12 minutes whatever the split, the fixed cost of a leg (checkout, install, a dry run of the whole suite) plus the survivors Stryker reruns every time.

Scores over the shards, cold run: framework 93.0 (floor 93), kernel 96.8 (95), tools 95.2 (94), telemetry 95.4 (93). The cold run's slowest leg was kernel 3/3 (21.4 min): it holds 27 of kernel's 29 mutants that reach the 20 s timeout, which line weight does not see. The timeout is left as it is: lowering it would count a slow but passing test as a kill.

⚠️ Heads-up

  • This PR touches the harness, so its own CI runs every scope from a cold incremental cache under the new keys: its wall time is the honest comparison with the 45 minutes above. The first run after merge on each branch is cold for the same reason.
  • More parallel legs can queue behind the account's concurrent job limit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp

blafourcade and others added 2 commits October 12, 2026 15:51
framework, kernel, tools and telemetry took 30 to 45 minutes each in
one job, and set the wall time of every pull request. Each now runs as
3 or 4 legs in parallel: a deterministic split by file weight (heaviest
first, to the lightest leg, ties by path), one incremental file and
sandbox per leg, the full stryker.conf.json otherwise. A scoring job
merges the legs' reports and applies the same score and floor as one
run; it fails when a leg is missing, mutated other files than the split
gives it at this commit, or the legs hold no mutant, and it runs even
when a leg crashed. The canary and every other scope stay one job, and
running a scope locally is unchanged.

Measured on tools-codex, split in two for the proof: one run and two
legs give the same 444 detected, 21 undetected and survivors, score
95.5 both ways. The incremental cache keys now end their scope name with
--, so tools can never restore tools-codex's file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
After the first split, the warm run's slowest legs were tools-codex
(10.3 min), kernel (9.6) and translate. Their shares now run in
parallel too. Kernel's heaviest file alone outweighs a quarter of the
scope, so the balance check bounds a shard by a quarter over its fair
share or by its heaviest file, whichever is larger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
@blafourcade
blafourcade marked this pull request as ready for review October 12, 2026 15:46
@blafourcade
blafourcade requested a review from a team as a code owner October 12, 2026 15:46
run-mutation.mjs walked src/ on its own, beside the architecture tests'
sourceFiles(). Both now use the one in mutation-shards.mjs, so the CI
cuts exactly the file list the partition tests check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
@blafourcade
blafourcade merged commit ce80635 into next Oct 12, 2026
103 of 106 checks passed
blafourcade added a commit that referenced this pull request Oct 12, 2026
Brings the sharded mutation CI (#1007). Conflict: the telemetry scope
keeps this branch's globs and floor (96) and takes next's 3 shards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant