Skip to content

perf(mutation): stop booting a Postgres container for every mutant - #100

Merged
KenTaniguchi-R merged 1 commit into
mainfrom
perf/stryker-drop-db-bootstrap
Aug 29, 2026
Merged

perf(mutation): stop booting a Postgres container for every mutant#100
KenTaniguchi-R merged 1 commit into
mainfrom
perf/stryker-drop-db-bootstrap

Conversation

@KenTaniguchi-R

Copy link
Copy Markdown
Owner

Split out from #99 — that PR is pure CI config; this one changes mutation scores and deserves its own review.

Problem

Stryker was spending most of its runtime starting databases, not testing mutants. A 9-file diff kept the mutation job busy for 30+ minutes on the CI runner, with 18 Postgres containers alive simultaneously and the 16GB mini swapping (991k pageouts).

Cause

Three things compound:

  1. Stryker runs vitest once per mutant batch across concurrency workers — default is cpuCores - 1, so 9 on the Mac mini.
  2. vitest.config.ts declares globalSetup, which boots a testcontainers Postgres and replays the entire migration chain. That fires on every worker.
  3. vitest.related (default true) already limits which tests run — but globalSetup runs regardless of which tests are selected. So Postgres booted even to run money.test.ts.

Net effect: 8.92s of database bootstrap to run 29ms of tests.

Fix

A Stryker-only vitest config via the supported vitest.configFile option, with no globalSetup, excluding tests/ and the five colocated src tests that need a live database. Plus ignoreStatic: true, which skips mutants that only execute at file load — precisely the ones that force an environment reload.

vitest.config.ts is untouched, so pnpm test and CI's test job are unaffected.

Measured

Before After
One test file (29ms of tests) 6.66s 1.37s
Full unit suite (53 files, 390 tests) 6.47s, 0 containers
Stryker on money.ts — dry run 24s, 192 tests 3s, 111 tests
Stryker on money.ts — mutation phase 2m42s 1m52s
Peak Postgres containers 7 (this machine) / 18 (mini) 0

Measured on an M4 Pro (14 cores, 48GB). The gap is far wider on the mini — 10 cores and 16GB with swapping is where 7 containers became 18 and minutes became 30.

The score change, stated plainly

Mutation score on money.ts moves 77.32% → 75.26%. The composition is better than the number:

before: 71 killed, 4 timeout, 22 survived
after:  73 killed, 0 timeout, 24 survived

It kills more mutants outright. The old run's 4 timeouts — which Stryker counts as kills — were mutants timing out on container contention, not tests catching them. Part of the old score was infrastructure noise scored as success.

Mutants covered only by DB-backed tests now report NoCoverage. That's honest signal about unit-test strength, and it makes ci.yml's claim that Stryker "currently runs just the unit suite" true — it wasn't before.

Verification

  • pnpm typecheck — clean
  • pnpm lint — 0 errors (1 warning, pre-existing on main, verified by stashing)
  • pnpm test113 files / 751 tests pass in 32s
  • Full unit suite under the new config — 53 files / 390 tests pass, zero containers
  • End-to-end Stryker runs both ways on money.ts, numbers above

Note

coverageAnalysis: "perTest" is kept for documentation only — the vitest runner ignores it and always uses perTest, but ignoreStatic documents perTest as a prerequisite.

🤖 Generated with Claude Code

Stryker was spending most of its runtime starting databases, not testing
mutants. A 9-file diff kept it busy for 30+ minutes on the CI runner, with 18
Postgres containers alive at once and the box swapping.

Cause: Stryker runs vitest once per mutant batch across 'concurrency' workers
(default: CPU cores - 1, so 9 on the Mac mini). vitest.config.ts declares
globalSetup, which boots a testcontainers Postgres AND replays the whole
migration chain -- and that fires on every worker. vitest.related already
limits which tests run, but globalSetup runs regardless of which tests are
selected, so Postgres booted even to run money.test.ts.

Measured, one test file with 29ms of tests: 6.66s -> 1.37s. The full 53-file
DB-free unit suite now runs in 6.47s with zero containers. Stryker on
money.ts: dry run 24s/192 tests -> 3s/111 tests, mutation phase 2m42s ->
1m52s, peak containers 7 -> 0. The gap is far wider on the 16GB mini.

Fix: a Stryker-only vitest config via the supported vitest.configFile option,
with no globalSetup, excluding tests/ and the five colocated src tests that
need a live database. Adds ignoreStatic, which skips mutants that only run at
file load -- the ones that force an environment reload.

vitest.config.ts is untouched, so pnpm test and CI's test job are unaffected
(113 files / 751 tests still pass).

Mutation score on money.ts moves 77.32% -> 75.26%, and the composition is
better than the number suggests:

  before: 71 killed, 4 timeout, 22 survived
  after:  73 killed, 0 timeout, 24 survived

It kills more mutants outright. The old run's 4 timeouts -- which Stryker
counts as kills -- were mutants timing out on container contention, not tests
catching them. Part of the old score was infrastructure noise scored as
success.

Mutants covered only by DB-backed tests now report NoCoverage. That is honest
signal about unit-test strength, and it makes ci.yml's claim that Stryker
'runs just the unit suite' true -- it was not before.

Note: coverageAnalysis is kept for documentation only. The vitest runner
ignores it and always uses perTest, but ignoreStatic documents perTest as a
prerequisite.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant