Skip to content

Implement the MAP_REPLACE filter strategy with lookup table and generator fallback #51

Description

@jzonthemtn

Blocked by: philterd/phisql#30 (the MAP_REPLACE RFC). Do not start until that RFC is accepted (accepted label); the schema fields and generator contract are decided there and may change during review.

Summary

Implement the MAP_REPLACE filter strategy and the top-level generators block defined in philterd/phisql#30.

MAP_REPLACE replaces a detected value with a caller-supplied value from a lookup table. For values not in the table, it falls back to a named generator (a local LLM, Ollama first) that produces a structure-preserving replacement (for example Jon's Ice Cream Shop to Joe's Ice Cream Shop), and if the generator fails it falls back to a safe redaction.

Resolution order for a detected token:

  1. Map hit returns the mapped value.
  2. Miss with a generator configured invokes the generator.
  3. Generator failure / timeout / invalid output falls back to fallbackStrategy (default REDACT).

A detected token is never left in the clear, and a generator failure never blocks the pipeline.

Acceptance criteria

  • MAP_REPLACE strategy dispatched in the standard replacement path.
  • Map sources: inline mappings (object) and mappingFiles (list of local file paths), merged once at policy-load time into an in-memory map. Mirrors the existing custom-dictionary terms + files loading.
  • File format: one delimited key/value pair per row, per the delimiter decided in the RFC (TSV unless the RFC concludes otherwise).
  • Precedence: inline mappings override mappingFiles; documented, deterministic rule for duplicate keys across files. No undefined behavior on collision.
  • caseSensitive field (default false); key normalization at load and token normalization at lookup are identical.
  • Top-level generators block: name-keyed, reusable, with a type discriminator (ollama first). Strategy references a generator by name via generator.
  • Generator connection targets a local endpoint; no auth secrets read from inline policy literals (resolve any auth from server config/environment).
  • fallbackStrategy (default REDACT) and a per-generator timeout are honored. Unreachable / timed-out / invalid generator output falls back safely; the token is never emitted in the clear and the pipeline never blocks indefinitely.
  • Output-validation gate: reject and fall back if generator output equals input (after case normalization); re-scan the generated value to confirm no PII was reintroduced.
  • Consistency: mapped and generated replacements route through the same context-scoped caching used by RANDOM_REPLACE, so the same input yields the same output within a context and inference is not repeated for a repeated value.
  • Prompt token placeholder ({{token}}) supported; optionally expose the entity label to the prompt.
  • Validates against the MAP_REPLACE / generators schema shipped in phisql schema/1.3.0.
  • Unit tests: map hit, map miss to generator, generator failure to fallback, output-equals-input rejection, case sensitivity, precedence, file loading, caching/consistency.
  • User docs under docs/docs/ updated for the new strategy and the generators block (if this port ships those docs).

Cross-port parity

This must behave identically to the sibling implementations in the other two Phileas ports (map source/precedence/case rules and the generator fallback/validation/caching contract). Divergence here is the class of bug seen with ABBREVIATE (philterd/phileas#320, philterd/phileas-dotnet#58). The generator path is nondeterministic, so conformance tests the contract (fallback fires, input-equal output rejected, values re-scanned and cached), not generated text. Sibling issues: philterd/phileas, philterd/phileas-python, philterd/phileas-dotnet (linked below).

Implementation notes (Python)

  • Register a "MAP_REPLACE" action in the actions registry (the same registry that holds "ABBREVIATE": _abbreviate), and add the strategy fields to the policy model.
  • Load mappingFiles at policy-load time into the in-memory map; parse each row as a key/value pair.
  • Reuse the existing context-scoped replacement caching so a repeated token yields the same output.
  • Add the top-level generators config and an Ollama client for the generator call, with timeout, output-validation, and fallback.
  • Keep the delimiter, precedence, and case rules byte-for-byte identical to the Java and .NET ports.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions