test(ci): gate coverage at the measured baseline - #500
Merged
Conversation
No coverage instrumentation existed, so nothing said whether a new area arrived tested (#85). `@vitest/coverage-v8` and a `coverage` block in `vitest.config.ts` scoped to `src/**/*.{ts,vue}` — what ships. The extensions are spelled out because a bare `src/**` pulls in 36 design-token stylesheets and two token JSON files, which carry no statements and land in the report as three dozen files at 0%. CI runs `test:coverage` in place of the plain `vitest run` rather than as a second job: same suite, same assertions, plus the gate, for about 100s on top of 240s. `coverage/` is uploaded as an artifact. Locally it stays off unless asked for. Baseline, measured twice and byte-identical between runs — 69.34% statements, 67.76% branches, 69.04% functions, 68.81% lines. Thresholds are those rounded down to the whole percent, so the gate goes red on regression rather than demanding a level nobody has reached. What it catches is written down with the arithmetic, because the honest answer is narrower than "prevents erosion". At 69% statements there are 38 uncovered statements of headroom, so a large new untested area is red and a median 14-statement component is not. It does not catch a component losing its tests: `describe.skip` on the whole Button suite stops 212 tests and leaves statements, functions and lines unchanged to the digit, because `Button.vue` is mounted by dozens of other specs and keeps executing regardless. Coverage measures execution, not assertion. A per-file threshold would close that and cannot be turned on yet — 106 files sit at 0% today.
…t on failure Review found the threshold rule was wrong in a way the numbers hid. Rounding the measured baseline down to the whole percent gave statements 38 uncovered statements of headroom and functions exactly one, because 69.04 floors to 69. That is a tripwire on a single metric, not a baseline gate, and it made the headroom table in the docs wrong: a median component carries about six functions, so it would have failed on the metric the table said it passed. The rule is now baseline minus one point, floored, so the margin is comparable across all four — and wide enough to absorb the difference between a contributor's Node and the Node 24 that CI pins, which one function of headroom was not. Scope is now `src/runtime` and `src/theme` rather than all of `src`. The module half is exercised by `test/module`, which runs under its own config in its own process and is not instrumented, so about 175 statements were counted as untested while being tested, and no amount of module testing could have moved the number. `reportOnFailure` is on: the default drops `coverage/` when a test fails, which is exactly the run the CI artifact is worth having. `.gitignore` anchors the entry to the root so a future `docs/coverage/` cannot vanish silently. The lockfile was rebuilt to drop bumps unrelated to this change — one of them was `@vue/compiler-dom`, which generates the render functions being counted. Thresholds are zero in this commit on purpose: the baseline has to be re-measured on Node 24 in a clean tree, and putting a guessed number here would be the same mistake in a smaller font.
Baseline re-measured on Node 24, which is what CI pins, against the rescoped `src/runtime` + `src/theme`: 71.89% statements (5265/7323), 69.86% branches, 71.04% functions, 71.45% lines. Thresholds are that minus one point, floored — 70 / 68 / 70 / 70 — which buys 198 statements, 202 branches, 39 functions and 135 lines of new uncovered code before the gate goes red. A 190-statement file arriving with no tests is red on functions and lines. The point of margin turned out to be load-bearing for a reason the first version did not know about. Three full runs of the same clean tree produced two different denominators, 7323 and 7324 statements, and it is not the Node version: both values came out of Node 24, with Node 22 landing on the second. One fully-covered function in `Button.vue` appears or does not, worth about 0.01pp. The earlier claim that Node 22 and Node 24 differ by one unit was a coincidence of two runs and is corrected — the number is simply not reproducible to the digit, so a threshold set at the baseline would eventually go red on nothing. The docs section is rewritten around the measured figures rather than the old ones, and two things it did not say are now in it: that a partial run with coverage is red by construction, because the thresholds are checked against the whole of `src/` however little of it ran; and that 93 files sit at 0%, which is what keeps `perFile` off. The published contributor guide gains a pointer to it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issue
Resolves #85
Type of change
revert(Scope): ...)Description
@vitest/coverage-v8, acoverageblock invitest.config.ts, thresholds below the measured baseline, and CI running the instrumented suite instead of the plain one.The wiring was the easy part. Three separate things that looked settled turned out to be wrong when measured, and each is written down below rather than quietly fixed.
Scope: what ships, not all of
src/src/runtime/**/*.{ts,vue}andsrc/theme/**/*.ts.The module half of
src/—module.ts,unplugin.ts,vite.ts,plugins/,templates.ts— is deliberately out. It is exercised bytest/module, which runs under its own config in its own process and is not instrumented. Counting it here reported about 175 statements as untested while they were in fact tested, and no amount of module testing could ever have moved the number.Extensions are spelled out because a bare
**pulls in the 36 design-token stylesheets and two token JSON files: no statements, uncoverable, three dozen rows at 0% in the report.The threshold rule, and why the first one was wrong
The first version of this PR rounded the measured baseline down to the whole percent. That produced margins that differed by a factor of forty:
One new uncovered function turned CI red, because 69.04 floors to 69. That is not a baseline gate, it is a tripwire on a single metric — and it made the headroom table in the docs wrong, since a median component carries about six functions and would have failed on the metric the table said it passed.
The rule is now baseline minus one point, floored. Re-measured on Node 24, which is what CI pins, against the rescoped source:
Against the shape of this repository — 182 components, median 14 statements and 6 functions, p75 38 and 15, largest
InputMenu.vueat 190 and 88 — a 190-statement file arriving with no tests is red on functions and lines. A single median component is not.The baseline is not reproducible to the digit
The first version claimed two runs were byte-identical, and later that Node 22 and Node 24 differ by one unit. Both were coincidences of small samples.
Three full runs of the same clean tree gave two different denominators, 7323 and 7324 statements — and the split does not follow the Node version: both values came out of Node 24, with Node 22 landing on the second. It is one fully-covered function in
Button.vueappearing or not, worth about 0.01pp.Tiny, and entirely inside a one-point margin — but it is the reason the margin is a point rather than a rounding. A threshold set exactly at the baseline would eventually go red on nothing at all.
What the gate does not catch
A component losing the tests it had. Measured, not assumed —
describe.skipon the wholeButtonsuite stops 212 tests and leaves the numbers where they were:Button.vueis mounted by dozens of other specs — inside forms, menus, toolbars — so its code keeps executing whether or not anything asserts about it. Coverage measures execution, not assertion.A per-file threshold would close that gap and cannot be turned on today: 93 files sit at 0%, so
perFileis red on the first run. That is the coverage-expansion backlog (#86), not this PR.An earlier attempt to prove this by deleting
Button.spec.tsoutright went red — but onvitest-include.spec.ts's orphan-snapshot guard, before coverage was ever consulted. Worth knowing that guard exists; it is not this one.Partial runs are red by construction
pnpm run test:coverage --project vue, or with a path filter, still checks the thresholds against the whole ofsrc/however little of it ran. Two reviewers found this independently. It is vitest's behaviour, not a regression, and it is now called out in the config comment, the testing guide and the published contributor guide, becausepnpm run test Buttonsits a few lines away in the same document.Other review fixes
reportOnFailure: true. The default dropscoverage/when a test fails — precisely the run theif: always()artifact upload is worth having..gitignoreanchors to/coverage, so a futuredocs/coverage/cannot vanish silently.@vue/compiler-dom3.5.38→3.5.41, which generates the very render functions being counted;valibotwent the same way. Onlymagicastremains, a build-time dependency.actions/upload-artifactis pinned to330a01c490aca151604b8cf639adc76d48f6c5d4, resolved from the realv5.0.0tag rather than recalled.Cost
240s → 320s, about +33%, on every push. Peak RSS 3.0 GB against 16 GB available, so no repeat of #485. One run, not a second job. Local runs stay uninstrumented.
Checklist
Gate on Node 24:
lint0 ·typecheck0 ·test:coverage316/316 (7442) with thresholds met ·test:module3/3 ·test:workflows49/0.