mutation (diff) gate is unpassable for DB-heavy files: whole-file scoping + #100's no-coverage shift

[KenTaniguchi-R]

Symptom

PR #102 — a 2-line behavior fix in src/lib/simplefin/sync.ts plus 2 new tests — failed mutation (diff) at 28.07% against thresholds.break = 60.

The red was not caused by the diff. Stryker run locally on a clean checkout of main's sync.ts returns exactly the same numbers:

	killed	survived	no-coverage	errors	score
main baseline	32	15	67	131	28.07%
PR #102	32	15	67	131	28.07%

The PR's own mutants were killed. The score is entirely pre-existing debt in lines the PR never touched.

Mechanism — two things compounding

1. Whole-file scoping. scripts/mutate-diff.sh builds --mutate from file paths:

MUTATE=$(echo "$FILES" | paste -sd, -)
exec pnpm exec stryker run --mutate "$MUTATE"

So any edit to sync.ts pulls all ~700 lines into scope. This is the mechanism already documented in #34.

2. #100 changed what no-coverage means. #100 decoupled mutation testing from Postgres via vitest.stryker.config.ts, which was the right call for runtime (30min → <2min). But it means the DB half of sync.ts — syncConnection and everything under it — can no longer be exercised at all during mutation, so those mutants report no-coverage instead of being killed by the DB-backed tests that do cover them in pnpm test.

vitest.stryker.config.ts states this plainly and treats it as a feature:

Mutants covered only by those report as NoCoverage rather than Survived, which is honest signal about unit-test strength instead of a container tax.

That reasoning holds. What was missed is that thresholds.break = 60 was never re-tuned for the new, structurally lower scores.

Impact

Any PR touching a DB-heavy file under src/lib now fails this gate regardless of the change's quality. 67 of the 145 non-error mutants in sync.ts are unreachable by construction, so the file cannot clear 60% no matter how well the diff is tested. #102 was the first PR to hit this; every subsequent one touching sync.ts, plaid/sync.ts, or the query modules will too.

main has no branch protection, so this is currently advisory — but it trains everyone to merge through a red check, which defeats the gate.

Options

1. Scope to changed lines. Use Stryker's --since / diff integration so the gate reflects the diff's own quality rather than whole-file legacy debt. This is the "reconsider the gate shape" half of Raise mutation coverage on sync.ts / reports.ts (and other large touched files) to ≥60% #34 and fixes the root cause.
2. Exclude no-coverage from the score, or re-tune thresholds.break, acknowledging that no-coverage is now the expected state for DB-backed code rather than a signal of weak tests.
3. Split the config so DB-backed files are mutated against a real Postgres in a separate, slower job, and unit-only files stay in the fast path.

(1) is the most honest and helps every future PR; (2) is the cheapest.

Context

Found while merging #102 (fixes #91). That PR was merged with the gate red, with the baseline comparison recorded on the PR.

Related: #34 (raise coverage on sync.ts / reports.ts — the other half of this problem), #100 (introduced the no-coverage shift).

[KenTaniguchi-R]

Second data point: a PR that raises the score still fails the gate

PR #104 measured against an origin/main baseline over the same files:

file	main	PR #104	delta
date-utils.ts	41.67%	48.15%	+6.48pp, +15 killed, zero new survivors
accounts.ts	0.00%	0.00%	unchanged (0 killed, 0 survived — 44 → 57 no-coverage)

The PR adds 8 tests, kills every mutant its new code introduces, improves the score — and the gate still reports failure. That is about as clear as the case gets: the gate is not measuring the diff.

accounts.ts is the extreme illustration. Zero mutants covered, before and after. It has 11 integration tests, 5 of them new in that PR, but vitest.stryker.config.ts excludes tests/integration/ wholesale — so every query module scores a structural 0%, and any PR touching one is guaranteed to fail.

A third mechanism to fold in: stale-branch diffs

PR #104's first run mutated three files — including src/lib/simplefin/sync.ts, which the PR never touched. mutate-diff.sh diffs against a moving origin/main:

FILES=$(git diff --name-only --diff-filter=ACMR "$BASE" "$HEAD" -- src/lib src/actions src/queries ...)

The branch had been cut before #102 merged, so the old sync.ts read as a change and dragged that file's 28.07% into the gate. Any branch that falls behind main inherits unrelated files' legacy debt. Rebasing cleared it, but "rebase before every push or your gate is wrong" is not a workable contract.

So there are three compounding causes, not two:

1. whole-file scoping
2. no-coverage for DB-backed code since perf(mutation): stop booting a Postgres container for every mutant #100
3. stale-branch diffs pulling in untouched files

All three point the same way: scope the gate to changed lines, per option 1 above.

[KenTaniguchi-R]

Third failure mode: the gate doesn't just score low, it aborts

Hit on #120, which adds src/actions/category-rules.ts and src/queries/category-rules.ts — both DB-backed, both covered only by tests/integration/.

Because vitest.stryker.config.ts excludes tests/integration/ wholesale, Stryker's vitest.related found zero related test files across the entire mutated set, and the dry run failed hard:

WARN  VitestTestRunner  Vitest failed to find test files related to mutated files
INFO  DryRunExecutor    No tests were found
ERROR Stryker           No tests were executed. Stryker will exit prematurely.
ConfigError: No tests were executed.

No report, no score, exit 1 — a crash rather than a measurement.

This is qualitatively worse than the 0% case. When a PR touches some unit-tested file, DB-backed files score 0% and the run completes. When a PR touches only DB-backed files — which is the normal shape of a query or server-action change in this codebase — the gate cannot run at all. Nothing about the diff's quality is being measured; the tool simply fails.

Workaround used on #120

Extracting the pattern-validation logic into a pure, unit-tested module was worth doing on its own merits (the empty-pattern guard is the most consequential rule in that feature and was reachable only through a DB round trip). It also happens to give Stryker something to find:

rule-pattern.ts     100.00%   8 killed, 0 survived, 0 no-coverage
category-rules.ts     0.00%   38 no-coverage

The run now completes and reports 17.39% instead of crashing.

But that is a workaround, not a fix: it means every PR adding a query or action module must also add an unrelated pure module, or the gate crashes. That is not a workable contract.

Bearing on the options above

This strengthens option 1 (scope to changed lines) and adds weight to option 3 (a separate DB-backed mutation job). Whatever is chosen should ensure the gate always reports — a gate that crashes teaches people to ignore it faster than one that merely scores low.

[KenTaniguchi-R]

Concrete numbers from #118, which hit this exactly. The run scored 33.33% overall against a break threshold of 60:

File	Score	Killed	Survived	NoCoverage	Touched by that PR?
lib/account-name.ts	100.00	10	0	0	yes (new)
lib/uncategorized-share.ts	82.61	19	4	0	yes (new)
lib/theme.ts	0.00	0	0	14	yes (new)
lib/money.ts	76.47	78	24	0	no
lib/date-utils.ts	48.15	65	15	55	no
lib/plaid/sync.ts	7.34	8	7	94	no
queries/accounts.ts	0.00	0	0	57	no
queries/investments.ts	0.00	0	0	90	yes

Two distinct problems, worth separating:

1. Whole-file scoping pulls in untouched files. plaid/sync.ts, date-utils.ts, money.ts and queries/accounts.ts are 220 of the 310 NoCoverage mutants and were not modified by that PR at all — they appear because something else in the diff imports them or because the diff scoping resolves to whole files. A PR is being scored on code it did not write.

2. Files covered by integration tests report 0% by construction. queries/*.ts and plaid/sync.ts are excluded from vitest.stryker.config.ts (correctly — that's #100's container fix), so every mutant in them lands as NoCoverage rather than Survived. queries/investments.ts scoring 0.00 with 90 NoCoverage is not a test-quality signal at all.

The workflow already documents this and sets continue-on-error: true, so nothing is actually blocked — but the check reads as a red X on every PR, which trains people to ignore it.

Two things that would make the signal honest without waiting for the integration harness:

• Scope the mutation set to files actually changed in the PR, intersected with what vitest.stryker.config.ts can cover.
• Report a per-PR score over that intersection, so a PR whose own new files are at 100% shows green.

For reference, the three new files in #118 are now at 100% (40 killed, 0 survived) after fixing what the run legitimately caught — one unreachable guard and one vacuous test. The gate did find real problems in the code the PR owned; it just buried them under files the PR never touched.

[KenTaniguchi-R]

Fourth presentation: the same absence reported as survived, not no-coverage

On #121 (src/queries/investments.ts):

	killed	survived	no-cov	score
main	0	90	0	0.00%
PR #121	0	112	0	0.00%

Note the shape: 0 no-coverage, 112 survived. Compare accounts.ts on #104, which reported the identical underlying situation as 0 survived, 57 no-coverage.

Same root cause both times — every test covering the file is in tests/integration/, which the Stryker config excludes — but the two files report it in opposite columns.

This is the most misleading of the four presentations. "Survived" reads as tests ran and failed to catch the mutant, i.e. weak tests. "No coverage" reads as nothing exercises this, which is what is actually true. A reviewer looking at 112 survivors would reasonably conclude the author wrote bad tests, when in fact the tests are good and simply unreachable from this config.

So the gate's failure modes now number four:

1. whole-file scoping drags in legacy debt
2. DB-backed code reports 0% because tests/integration/ is excluded
3. stale-branch diffs pull in untouched files
4. that same exclusion sometimes surfaces as survived rather than no-coverage, actively misattributing the cause

(3) is fixed by rebasing. (1), (2) and (4) all point the same way: the gate cannot presently distinguish "badly tested" from "not reachable by this runner", which is the distinction the whole tool exists to make.

[KenTaniguchi-R]

Failure mode 2 recurred on #126 — worth noting how ordinary that PR is

src/actions/recurring.ts was the only file in scope. DB-backed, integration-tested, so zero related tests were found and Stryker aborted with ConfigError — no score, no report.

That is now twice (#120, #126). Both are unremarkable backend PRs: a server-action module with integration tests. This is the default shape of backend work in this repo, which means the gate cannot measure the majority of what gets written here.

On #120 I extracted a pure module and the run completed. I deliberately did not repeat that on #126: nothing in that action had the standalone value the pattern-validation had, so extracting one would have been contorting code to satisfy a tool. If the workaround is the only way to get a report, the contract is: every backend PR must also ship an unrelated pure module. That is not a contract anyone should accept, and it will quietly push people toward writing code shaped for the gate rather than for the domain.

Running tally of distinct presentations, all from the same root:

#	presentation	seen on
1	whole-file scoping drags in legacy debt	#102
2	aborts with ConfigError, no report at all	#120, #126
3	stale-branch diff pulls in untouched files	#104, #123
4	absence reported as survived, misattributing cause	#121

Worth contrasting with #123, where the gate worked exactly as intended: clean-name.ts is pure with colocated tests, so its 31 survivors were real signal — 9 of them on newly added code, and acting on them found a genuine gap (every test exercised the same string position). That is the tool earning its keep. The four modes above are the same tool being unable to tell "untested" from "unreachable".