Close three fail-open gaps in the fixture-layout controls

[willhea]

Repairs the three places tests/test_fixture_layout.py fails open — where a real defect is planted and the suite stays green. One file changed, test-only, no production code.

Found by mutation-testing the module's 28 controls. The evidence is below rather than in a committed report: 37 mutations were applied to the checkout, run, and reverted, and every number here is a measurement, not a reading.

---

The three gaps

Each verified red-then-green against the mutation that previously passed silently.

Gap	Mutation	Before	After
G1 the committed-fixture oracle could go vacuous	committed_fixture_refs() returns set() + a real stale path planted in a scanned module	whole suite green	1 failed
G2 the exemption sets had no ceiling	exempt tests/test_diff_bill.py, then address a committed fixture through bills/ in it	30 passed	1 failed
G2 (same hole, other side)	exempt a new non-fetch tools/verify_manifest.py, same reference	30 passed	1 failed
G3 the gitignore control could not fail	append *.pdf to .gitignore	30 passed	1 failed, names all 53 PDFs
G3 (the original probe's format)	append tests/corpus/**/*.xml	28 passed	1 failed, names all 58 XML

G1 — the oracle nothing could see fail

committed_fixture_refs() decides what counts as an offence, and the gate consulting it cannot tell an empty answer from "no offences". Planting a genuine stale fixture path alongside a vacuous oracle left the entire non-browser suite green.

The new control asserts non-emptiness and cross-checks git ls-files tests/corpus — the index, against the working tree the oracle walks. That is a second oracle rather than the same one twice, which is precisely why the test that previously claimed this guarantee could not deliver it (below).

G2 — bounded by enumeration, not by prefix

A tools//tests/ prefix bound was written first and was itself fail-open: both directories hold ordinary product and test code. Membership of _DOWNLOAD_TIER_FILES switches off every bill rule for a file, so the prefix form let any test or tool address a committed fixture through bills/ forever.

Membership is now enumerated. There is no rule that separates a fetcher from its neighbours — "is acquisition code" is a judgement, not a path shape — so a list is the only honest bound. A genuinely new fetcher costs a line in the test as well as in the set; that friction is the point, because exempting a file is the cheapest way to defeat the whole module.

G3 — three defects, not one

1. Missing --no-index. Without it git check-ignore reports what git would do with a path, and it does nothing to a tracked file whatever .gitignore says. The control returned "not ignored" on a fixture tree that was fully ignored.
2. The probe crossed bills/, which this repo's own .gitignore says is commonly a symlink. Git refuses a pathspec "beyond a symbolic link" with 128, and the old code inferred work-tree-ness from that exit code — so it skipped, claiming "not a git work tree" while standing in one (measured with bills/ symlinked: 27 passed, 1 skipped). Any code that is neither 0 nor 1 is now a failure, not a verdict.
3. It probed one XML file out of 111 fixtures (58 XML, 53 PDF) — under 1% of the tree, and one of two formats. It now derives the set from git ls-files tests/corpus and asks git about all of it in a single check-ignore --no-index --stdin call, so a third format is covered the day one is committed, with no edit here.

---

One control deleted

test_download_only_versions_are_genuinely_uncommitted is removed, taking the module from 30 controls to 29. No meta-test was added to replace it.

Its detection set is test_no_source_reaches_into_bills_for_a_committed_fixture's minus the tmp_path skip — and that difference is a false positive: a synthetic tree naming a committed fixture is legal, per the module's own docstring. The guarantee it claimed (that the committed set has not quietly shrunk) is exactly what it could not deliver, because it consulted the same committed_fixture_refs() that had gone vacuous. G1's control holds it now, against git.

---

What the audit found and this PR deliberately does not do

A prior pass proposed deleting two controls as redundant. Attacking those findings before acting on them withdrew both — each is the sole detector of a failure nothing else in the suite sees.

test_conftest_refuses_a_foreign_engine was said to be dominated by its sibling, which asserts the same two things on a harder input. They actually differ in how their stand-in engine dies. Demoting the foreignness check into the except ModuleNotFoundError handler reddens it alone across the whole suite, because the sibling's partial engine raises ModuleNotFoundError (still caught) while this one's fuller engine dies on a plain ImportError (not caught). Against a complete engine copied out of another checkout:

	baseline	with the check demoted
child --collect-only	rc=4, RuntimeError: … DIFFERENT tree's source	rc=0, 191 tests collected

That is a silent green about another checkout's source.

test_the_known_bad_source_is_reported_end_to_end was said to add nothing over the discovery and rule tests. Giving find_stale_fixture_paths a directory roster reddens it alone: discovery still returns everything, and the rule still fires on every input the can-fire tests feed it. Every other input to that rule is keyed under a real top-level directory, so any roster written from today's tree spares them by construction.

Net: the cluster has no measured removal candidates. The count moves 28 → 29 only because of the G1/G2 additions and the G3-sibling deletion.

A limitation worth stating

Mutation testing measures detection, never worth. A sole detector of an irrelevant behaviour is still deletable, and nothing here establishes relevance. Roughly half these controls exist only to make the static text scan trustworthy; their value is inherited from that scan, so they cannot be judged one at a time.

Relatedly, the audit also asked whether the scan could shrink to the files the suite never executes. That question is unresolved and out of scope here. The coverage measurement behind it does not carry it: file-level execution does not establish that a file's fixture-path lines ran, and a skip ceiling sees only skips that actually occur. Answering it would need line- or branch-level coverage of the path-bearing code plus mutations proving a replacement boundary catches stale paths after the scan is removed. The scan's shape is untouched by this PR.

---

Verification

tests/test_fixture_layout.py	29 passed
neighbouring fixture/corpus modules	434 passed, 20 skipped
full non-browser suite	3824 passed, rc=0
ruff check / ruff format --check	clean, module and repository
git diff --stat vs develop	1 file changed

Every mutation applied, run, and reverted; the worktree carries no mutation or generated artifact. The mutation harness is deliberately uncommitted — its bodies contain bills/<id>/<file> literals that the very gates under audit would flag.

The audit ran on a machine with the bill corpus fetched (32 bills, 17 shared with tests/corpus/), which an earlier CI-shaped pass did not have. Two findings changed as a result.

Follow-up in #721, stacked on this branch: the runtime skip ceiling widened suite-wide, five live CWD-dependent defects, and a cwd-independence CI job. Also code-only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NwVE7osYo39KaUe7CEitFJ