Fuzz canonicalization and the policy loader, and record the finding it turned up

[arpanghoshal]

Why

Scorecard reads Fuzzing as 0/10 (weight 5 of 97.5, so ~+0.51). That is why it was picked up, not why it is worth having.

Canonicalization is the one place in this codebase where a defect is silent. Two distinct actions sharing a canonical form share an approval, and nothing in a receipt would look wrong. This repository has already had one bug of exactly that shape — str(key) folded {"1": "a", 1: "b"} into a single key, with the survivor decided by insertion order — and a review found it. These are the properties that would have found it without a review.

Two targets, each a promise already written down: canonicalization (SPEC-v0.1.md §2.3), and Policy.from_yaml, whose docstring says anything malformed raises PolicyError — the fail-closed rule in a sentence.

The invariants do not import Atheris

fuzz/properties.py is plain stdlib, so tests/test_fuzzing.py is a second driver for the same properties. They run on every commit whether or not anybody has a fuzzing toolchain, and a contributor can reproduce a finding without one. fuzz/fuzz_*.py hold no assertions of their own.

CI runs the seed corpus as a gate that cannot be skipped by a toolchain problem, then a bounded 2-minute campaign per target. A failure to install Atheris is a red build, not a skip.

A finding — recorded, not worked around

canonical_bytes raises UnicodeEncodeError, not InvalidArgument, for a string holding an unpaired UTF-16 surrogate.

Reachable: json.loads produces one from a \udXXX escape, so an MCP tool call can carry it. Action(...) accepts it; action_hash is where it fails.

Fail-closed holds — the gateway and Control both wrap the action path in except Exception — so this is a contract violation and a crash, not an authorization bypass. What it breaks is the closed error set in errors.py: a caller catching CTRLRunError does not catch this.

It is in KNOWN_FINDINGS with a reproducer in the corpus, rather than pruned out of the decoder, because a fuzzer whose corpus avoids its own findings reports zero forever. test_the_known_findings_still_reproduce asserts it still happens — the day it is fixed, that test goes red and the entry must be deleted.

The fix touches src/ so it is not in this PR, per the merge rule. It is narrow — reject unencodable strings in _no_floats beside the float and non-string-key checks — and changes no hash that previously succeeded, so §2.3's "old hashes still verify" rule is satisfied without a schema bump. Argued in fuzz/README.md.

What the positive controls found

Eleven controls, one per invariant. Writing them found three defects in the properties themselves:

• Only the first encode call was guarded. An encoder broken on its second call raised TypeError straight out of check_canonical. A property claiming "nothing but InvalidArgument escapes" has to hold for every call it makes.
• Nothing asserted that floats and non-string keys are refused. Checking only what comes back cannot distinguish "refused correctly" from "never refused at all".
• Every control was really testing one branch. The generated corpus is full of non-string keys, so must_refuse fired first and each control — whatever it was named for — passed on that. They use purpose-built documents now and assert which message they got.

18 mutations, 18 caught. 240,000 generated inputs across both properties produced exactly one finding — the one above.

One existing test changed

test_T124b_the_package_ships_nothing_from_research substring-matched the whole of MANIFEST.in including its comments, so a comment mentioning research/ beside the new prune fuzz failed a test about what setuptools ships. It parses directives now. Mutation-tested three ways — recursive-include research *.py, graft research, include research/soak/run.py — all caught, and the comment that broke it passes.

Say the word if you would rather that were a separate PR.

Checks

ruff format --check, ruff check, mypy --strict src clean; full suite 3936 passed, 45 skipped.

[arpanghoshal]

Follow-up commit: the campaign was instrumented by nothing

The fuzz job passed on the first commit — green tick, 9,788,746 executions on canonicalization and 5,074,499 on the policy loader. stat::new_units_added: 0 on both is what gave it away. A coverage-guided fuzzer that finds no new coverage unit in ten million runs is not coverage-guided.

import properties sat at module scope, so by the time atheris.instrument_imports() ran the module was already in sys.modules and the instrumented re-import was a no-op. Nothing was instrumented — not properties, and not ctrlrun.action underneath it, which is the code the campaign exists to cover.

It ran blind random input at full speed and reported success. That is worse than not running at all: the green tick was evidence for a claim nothing supported.

And the test asserted the string that said it was working

assert "instrument_imports" in source was true of the broken version — the call was right there, doing nothing. It is an AST walk now: no import properties at module scope, and an import properties inside a with whose context manager is instrument_imports. A structural property, because the difference between the two versions is structural and invisible to a substring.

Two mutations, two caught — reintroducing the top-level import, and removing the instrumentation block. Neither is detectable by the check it replaces.

Full suite after the fix: 3938 passed, 45 skipped.

I am flagging this rather than quietly amending because the shape of it is the thing worth reviewing: this is the repository's own false-green pattern, in the PR whose subject is catching false greens, written by the person adding the rule. Worth a second pair of eyes on whether the AST check is the right guard.

[arpanghoshal]

The fix, confirmed by CI rather than by the test

	canonical	policy
before — new_units_added	0	0
after — new_units_added	772	3143
after — coverage / features	cov: 134 ft: 647	cov: 1054 ft: 5034
after — corpus built	211 inputs	813 inputs
exec/s	80,898 → 4,091	41,938 → 2,743

Execution rate is down about 15×. That is the instrumentation overhead and it is the trade worth making: far fewer executions, each guided by coverage feedback and accumulating a corpus, rather than 15 million blind random inputs that explored nothing. Before the fix the log contained no cov: lines at all.

Both targets still report no new violations, and the seed gate still reproduces the one recorded finding.