Vision proposal: worldwide language and locale coverage

[mazzasaverio]

Vision proposal: worldwide language and locale coverage

Follow-up to #33, but bigger in scope, so it deserves its own discussion. The proposal: make "works for any language and any country" an explicit roadmap goal, reached in stages instead of ad hoc.

What the core already gives us

A useful part of the work is already language-agnostic by design: email, IBAN, payment card, IP, and URL credential detection are structural, normalization uses NFKC + casefold, and the adversarial corpus already covers RTL/BiDi text. There is even a German corpus (tests/corpus/de.jsonl) acting as a seed. What is missing is the country- and language-specific layer.

Staged plan

1. Locale identifier framework. Decide once how country packs plug in (this generalizes the core-vs-extension question asked in Proposal: Italian identifier detectors, ONNX long-text handling, and next contributions #33): naming, entity types (e.g. a generic NATIONAL_ID / TAX_ID rather than one enum member per country), detector registration, and how packs are discovered and documented.
2. Country identifier packs, prioritized by demand: national IDs and tax numbers with checksum validation wherever the spec allows it (codice fiscale, NIE/DNI, NINo, Steuer-ID, CPF, SSN patterns, etc.). Checksum-first keeps the zero-false-positive philosophy of the existing Luhn/mod-97 validators.
3. Phone realism per region. The current heuristic is deliberately conservative; a per-country prefix/length table (still stdlib-only, no external deps) could raise recall without hurting precision.
4. Multilingual ML guidance. The ONNX backend is model-agnostic, so multilingual coverage is mostly a documentation and label-mapping problem: document tested multilingual NER models, extend the label-suffix mapping beyond CoNLL/Ai4Privacy where needed, and state clearly which languages a given setup covers.
5. Per-language test corpora. Grow the tests/corpus/ set (synthetic only, as per the release gates) one language at a time, so every claimed language is actually tested.

Design questions

• Do you want locale packs in the core (stdlib-only detectors seem compatible with the zero-dependency rule) or as external pseudonymize-locale-* packages listed in the docs?
• One generic entity type per category (NATIONAL_ID, TAX_ID) or country-specific types? The first keeps the enum stable; the second gives finer policies.
• Which 3-5 countries first? I would start from where users actually are (EU first, given the regulatory pull).

I am happy to do the implementation work in the usual one-PR-per-item style once the framework decision is made.

[mazzasaverio]

Follow-up after re-reading ROADMAP.md: multilingual coverage is already anticipated there, so this issue should be read as an extension of an existing plan rather than a new direction. The 0.2.0 section commits to "initial English, German, and Italian benchmarks", so the language question is yours already; what is not covered anywhere is the country-identifier layer (national IDs, tax numbers) and the framework that would let it grow without bloating the core.

Re-scoping this issue accordingly:

• Already planned, not mine to propose: language coverage for the ML backend, starting from EN/DE/IT. I would simply help deliver it (see the quality-harness discussion in Proposal: detection-quality evaluation pipeline and a task-level benchmark #37, which turns out to close the same roadmap item).
• Actually new, and what this issue should be about: how country identifier packs plug in (naming, entity types, discovery, docs) and whether they live in the core or outside it.

On the entity-type question, VISION.md principle 9 ("earn compatibility after the architecture is proven") suggests deciding the shape now, while the enum can still change cheaply, rather than after 1.0.0. My suggestion remains generic categories (NATIONAL_ID, TAX_ID) over one enum member per country, since that keeps the public surface stable while the number of supported countries grows.

One check against principle 1 ("keep the trusted core small"): checksum-based identifier detectors are stdlib-only and structurally identical to the existing Luhn and mod-97 validators, so putting them in the core would not add dependencies. Whether it adds unwanted scope to the core is a different question, and that one is yours.