feat(nvisy-nlp): new NLP crate + nvisy-engine wiring

[martsokha]

Summary

New nvisy-nlp crate for NLP — NER, language detection, and tokenization — plus the nvisy-engine wiring that consumes it.

nvisy-nlp

Composable trait surface (NerBackend + LanguageDetector + Tokenizer) with a composite NlpEngine that orchestrates them in Presidio order (detect language → run NER with hint → tokenize → derive keywords). Second entrypoint analyze_in_language(text, lang) bypasses detection when the caller already knows the language.

Trait surface is transport-agnostic. v1 ships local-by-default implementations; LLM-mediated NER lives in nvisy-llm by deliberate crate split, not by trait restriction. Any third-party backend can implement NerBackend over any transport.

v1 implementations:

• OrtNerBackend — HuggingFace token-classification via ort (ONNX Runtime, load-dynamic). Real per-span softmax confidence. Truncates inputs at max_sequence_length. Validates language hints against supported_languages. Inference dispatched via tokio::task::spawn_blocking so it doesn't starve the async runtime.
• LinguaLanguageDetector — lingua-rs, real confidence scores, multilingual segmentation, two constructors (for_languages recommended, for_all_languages for unknown).
• UnicodeTokenizer — unicode-segmentation based, model-free.
• HfTokenizer — wraps tokenizers::Tokenizer for ORT NER alignment.
• NoopNerBackend — for tests.

Key surface decisions:

• NerBackend + LanguageDetector required at build; Tokenizer optional. Builder uses type-state so calling .build() without both required components is a compile error, not a runtime panic.
• Tokenizer::tokenize returns Result<Vec<Token>>.
• LanguageDetector::detect returns Result<Option<LanguageDetection>> — splits "no answer" (Ok(None)) from real failure (Err).
• LanguageDetector::detect_multiple returns Result<Vec<LanguageSpan>> for mixed-language documents; default impl falls back to single-language detect.
• LanguageDetection { language, confidence: Option<f64>, provenance: LanguageProvenance } — distinguishes detected from caller-asserted.
• Crate-wide Result<T> alias defaulting to Error; pub use nvisy_nlp::{Error, Result} at the crate root.
• Token { start, end, text, is_stop } — no is_punct (broken across both impls; reviewer caught it).
• User-provided model paths; no bundling, no auto-download. Users install libonnxruntime out-of-band.
• No lemmas (nvisy-nlp: lemmatization deferred: decide path when a consumer needs it #154 — no maintained Rust lemmatizer).

nvisy-engine wiring

• Renames EntityRecognition (LLM-driven) → LlmRecognition.
• Adds NerRecognition wrapping Arc<dyn nvisy_nlp::NerBackend>.
• Detection phase now runs three ops: LLM (skip if no provider), NER backend (skip if no backend), pattern (always).
• Engine::with_ner_backend(...) builder method for engine init.

Behaviour unchanged when no backend is configured.

Layout

crates/nvisy-nlp/src/
├── lib.rs
├── error.rs                          Error + Result alias
├── artifacts/
│   ├── mod.rs                        Artifacts
│   └── token.rs                      Token
├── engine/
│   ├── mod.rs                        re-exports + integration tests
│   ├── nlp_engine.rs                 NlpEngine
│   └── builder.rs                    NlpEngineBuilder (type-state)
├── language/
│   ├── mod.rs                        LanguageDetector + LanguageDetection + LanguageSpan + LanguageProvenance
│   └── lingua.rs                     LinguaLanguageDetector
├── ner/
│   ├── mod.rs                        NerBackend trait
│   ├── ort.rs                        OrtNerBackend + Inferencer (pub(crate))
│   └── noop.rs                       NoopNerBackend
└── tokenizer/
    ├── mod.rs                        Tokenizer trait + SUPPORTED_STOPWORD_LANGUAGES
    ├── hugging_face.rs               HfTokenizer
    └── unicode.rs                    UnicodeTokenizer

Deferred (tracked as follow-ups)

• GlotLID v3 as alternative language detector — better tail coverage, short-text accuracy, Apache-2.0.
• TiktokenTokenizer for LLM context-window counting.
• LinderaTokenizer for CJK content.
• Burn as alternative NER backend.
• Lemmatization (nvisy-nlp: lemmatization deferred: decide path when a consumer needs it #154).

Test plan

• cargo build --workspace clean
• cargo test --workspace — all ~440 tests pass, 28 in nvisy-nlp
• cargo clippy --workspace --all-targets clean
• cargo doc -p nvisy-nlp --no-deps clean (no broken intra-doc links)
• cargo fmt clean
• All commits independently buildable
• Manual smoke: load a real ONNX NER model via OrtNerBackend and run against documents
• Manual smoke: wire a backend into the engine and verify it runs alongside pattern recognition

🤖 Generated with Claude Code