fix(ci): measure coverage over all test targets, not just --lib - #7
Draft
h4x0r wants to merge 1 commit into
Draft
Conversation
`cargo llvm-cov --lib` builds only the lib target's own unit tests. Integration tests under tests/ are separate crates producing separate binaries, so `--lib` never builds them and nothing they cover is recorded. That makes the number mean something narrower than it reads. Any code reached through the public seam rather than from inside src/ shows as uncovered even when the suite exercises it fully. bluetooth-forensic hit exactly this: 54 lines reported uncovered on a library its own integration tests cover to 100%, a phantom debt that would have sent someone writing duplicate unit tests for code already tested -- or deleting the integration tests that made it green. Dropping `--lib` pulls binary targets into the report at 0% (built by the test profile, never executed), which would make a 100% gate unsatisfiable for reasons unrelated to test quality. Binary shells are excluded instead, matching the rule blob-decoder's gate script already carries: the CLI is a Humble Object over the library, and the library is what the gate is about. This repo passes today under `--lib`, which is the stronger claim -- the library is fully covered by unit tests alone -- so this changes nothing about its current result. Coverage can only grow when more test binaries run. What it prevents is the phantom debt appearing the first time someone covers new code from an integration test. 62 of the fleet's 76 repos with a coverage step already measure all test targets; this brings the remaining outliers in line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Why
--libbuilds only the lib target's own unit tests. Integration tests undertests/are separate crates producing separate binaries —--libnever buildsthem, so nothing they cover is ever recorded.
The number therefore means something narrower than it reads: code reached
through the public seam shows as uncovered even when the suite exercises it
fully.
bluetooth-forensichit exactly this — 54 lines reported uncovered ona library its integration tests cover to 100%. Acting on that number means
writing duplicate unit tests for already-tested code, or deleting the
integration tests that made it green.
Dropping
--libpulls binary targets in at 0% (built by the test profile, neverexecuted), which would make a 100% gate unsatisfiable for reasons unrelated to
test quality. Binary shells are excluded instead — the same rule
blob-decoder/scripts/coverage-gate.pyalready carries, for the same reason:the CLI is a Humble Object over the library, and the library is what the gate is
about.
Risk
None to the current result. This repo passes today under
--lib, which isthe stronger claim — the library is fully covered by unit tests alone.
Coverage can only grow when more test binaries run, and binaries are excluded.
What the change prevents is the phantom debt appearing the first time someone
covers new code from an integration test.
Fleet context
62 of the 76 repos with a coverage step already measure all test targets. This
brings the remaining outliers in line; ADR-0008 is being amended to match.