Proposal: detection-quality evaluation pipeline and a task-level benchmark

[mazzasaverio]

Proposal: detection-quality evaluation pipeline (and a task-level benchmark nobody has)

Expands point 4 of #33 into a concrete plan. Filing it separately because it is bigger than a single comparative run, and it interacts with CI.

Where we stand

The benchmarks/ suite measures performance with real discipline (latency, import time, peak memory, wheel size, published with full machine context in docs/benchmarks.md). But there is no quality evaluation at all: no precision/recall/F1 per entity type, no comparison against other tools, no way to notice a silent recall regression. Correctness is covered by unit and corpus tests, which assert pass/fail but produce no metrics. In short: the repo can tell how fast it is to the millisecond, but not how good it is.

Side note on the existing numbers: the reference measurements in docs/benchmarks.md predate #29 (the quadratic overlap-resolution fix), so the published 64 KiB latency is very likely stale in our favor. I can send a small separate PR that re-measures and refreshes that page.

Level 1: measure ourselves on existing public benchmarks

A quality harness under benchmarks/quality/ that:

• computes precision / recall / F1 per entity type for the rules backend (and optionally the ML backend) on public synthetic datasets (ai4privacy pii-masking; the pii-anon benchmark datasets), respecting the repo rule that tests and benchmarks use synthetic data only;
• optionally runs the same datasets through Presidio and scrubadub for comparison, with competitors installed as extra dev dependencies in a separate environment (never as package dependencies), pinned versions, and their configurations documented in full. Fairness matters: a comparison perceived as rigged costs more credibility than it buys;
• reports results per configuration, since ML-backend accuracy depends on the model the user supplies; "package accuracy" is meaningless without naming the model;
• writes machine-readable results (JSON) plus a dated results page in docs/, following the same publishing discipline the perf benchmarks already have.

CI gate: same philosophy as the coverage floor. A minimum F1 per entity type on our own harness, enforced in CI, so detection quality cannot silently regress. Comparative runs against other tools stay out of CI (slow, env-heavy) and are refreshed manually per release.

Level 2: create the benchmark that does not exist

Every public PII benchmark measures NER accuracy only. Nobody measures the actual task this package is built for: the pseudonymize, use with an LLM, restore loop. A task-level benchmark could score, beyond detection F1:

• identity consistency: same normalized value gets the same alias, across blocks, batches, and scopes;
• round-trip fidelity: does restore reconstruct the original text exactly?
• leak rate under adversarial input: pre-existing placeholders, BiDi/RTL text, homoglyphs, deeply nested payloads (the existing adversarial corpus is the natural seed);
• structure preservation for nested JSON/JSONL payloads;
• throughput, reusing the existing perf suite.

Owning the reference benchmark for a task category also defines how the category is judged, and unlike code, a maintained benchmark is an asset a fork cannot copy meaningfully. The cost is real too: a public benchmark needs maintenance and fair-play arbitration if others submit results, so I would design it deliberately minimal (a spec + a runner + a dated results table, nothing more).

Suggested order

1. Small PR: re-measure perf numbers post-perf: linear-scaling overlap resolution and text reassembly #29 and refresh docs/benchmarks.md.
2. Level 1 harness with our own F1 floor in CI.
3. Comparative runs (Presidio, scrubadub) published in docs.
4. Level 2 task benchmark as a spec first, discussed here before any code.

Happy to implement in the usual one-PR-per-item style if the direction works for you.

[mazzasaverio]

Follow-up after re-reading ROADMAP.md and VISION.md properly: this issue was framed as a new idea, when part of it is actually an undelivered 0.2.0 roadmap commitment. The 0.2.0 section lists:

• Initial English, German, and Italian benchmarks
• Pinned model revisions, checksums, licences, memory, and latency measurements

0.2.0 shipped without the language benchmarks, and without published memory/latency numbers for the ML backend (the pinned checksums part landed with #27). So the accurate framing is: Level 1 closes an open 0.2.0 item, it does not add scope. That also settles the priority question, since finishing a committed milestone item outranks anything I proposed.

Two consequences for the plan above:

• The first quality datasets should be English, German, and Italian, matching the roadmap wording, rather than whatever public dataset is most convenient.
• The ML backend needs its own memory and latency measurements published in docs/benchmarks.md alongside the existing core numbers, since 0.2.0 promised them.

Level 2 (the task-level benchmark for the pseudonymize/use/restore loop) remains a genuinely new proposal and is subordinate to the above. Also worth noting that it maps directly onto the differentiation list in VISION.md (deterministic aliases, nested-payload processing, reports without raw detections): it would measure exactly the properties the vision claims as differentiators, which is currently asserted but not quantified.