You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Blocked by:philterd/phisql#30 (the MAP_REPLACE RFC). Do not start until that RFC is accepted (accepted label); the schema fields and generator contract are decided there and may change during review.
Summary
Implement the MAP_REPLACE filter strategy and the top-level generators block defined in philterd/phisql#30.
MAP_REPLACE replaces a detected value with a caller-supplied value from a lookup table. For values not in the table, it falls back to a named generator (a local LLM, Ollama first) that produces a structure-preserving replacement (for example Jon's Ice Cream Shop to Joe's Ice Cream Shop), and if the generator fails it falls back to a safe redaction.
Resolution order for a detected token:
Map hit returns the mapped value.
Miss with a generator configured invokes the generator.
Generator failure / timeout / invalid output falls back to fallbackStrategy (default REDACT).
A detected token is never left in the clear, and a generator failure never blocks the pipeline.
Acceptance criteria
MAP_REPLACE strategy dispatched in the standard replacement path.
Map sources: inline mappings (object) and mappingFiles (list of local file paths), merged once at policy-load time into an in-memory map. Mirrors the existing custom-dictionary terms + files loading.
File format: one delimited key/value pair per row, per the delimiter decided in the RFC (TSV unless the RFC concludes otherwise).
Precedence: inline mappings override mappingFiles; documented, deterministic rule for duplicate keys across files. No undefined behavior on collision.
caseSensitive field (default false); key normalization at load and token normalization at lookup are identical.
Top-level generators block: name-keyed, reusable, with a type discriminator (ollama first). Strategy references a generator by name via generator.
Generator connection targets a local endpoint; no auth secrets read from inline policy literals (resolve any auth from server config/environment).
fallbackStrategy (default REDACT) and a per-generator timeout are honored. Unreachable / timed-out / invalid generator output falls back safely; the token is never emitted in the clear and the pipeline never blocks indefinitely.
Output-validation gate: reject and fall back if generator output equals input (after case normalization); re-scan the generated value to confirm no PII was reintroduced.
Consistency: mapped and generated replacements route through the same context-scoped caching used by RANDOM_REPLACE, so the same input yields the same output within a context and inference is not repeated for a repeated value.
Prompt token placeholder ({{token}}) supported; optionally expose the entity label to the prompt.
Validates against the MAP_REPLACE / generators schema shipped in phisql schema/1.3.0.
Unit tests: map hit, map miss to generator, generator failure to fallback, output-equals-input rejection, case sensitivity, precedence, file loading, caching/consistency.
User docs under docs/docs/ updated for the new strategy and the generators block (if this port ships those docs).
Cross-port parity
This must behave identically to the sibling implementations in the other two Phileas ports (map source/precedence/case rules and the generator fallback/validation/caching contract). Divergence here is the class of bug seen with ABBREVIATE (philterd/phileas#320, philterd/phileas-dotnet#58). The generator path is nondeterministic, so conformance tests the contract (fallback fires, input-equal output rejected, values re-scanned and cached), not generated text. Sibling issues: philterd/phileas, philterd/phileas-python, philterd/phileas-dotnet (linked below).
Implementation notes (Python)
Register a "MAP_REPLACE" action in the actions registry (the same registry that holds "ABBREVIATE": _abbreviate), and add the strategy fields to the policy model.
Load mappingFiles at policy-load time into the in-memory map; parse each row as a key/value pair.
Reuse the existing context-scoped replacement caching so a repeated token yields the same output.
Add the top-level generators config and an Ollama client for the generator call, with timeout, output-validation, and fallback.
Keep the delimiter, precedence, and case rules byte-for-byte identical to the Java and .NET ports.
Blocked by: philterd/phisql#30 (the
MAP_REPLACERFC). Do not start until that RFC is accepted (acceptedlabel); the schema fields and generator contract are decided there and may change during review.Summary
Implement the
MAP_REPLACEfilter strategy and the top-levelgeneratorsblock defined in philterd/phisql#30.MAP_REPLACEreplaces a detected value with a caller-supplied value from a lookup table. For values not in the table, it falls back to a named generator (a local LLM, Ollama first) that produces a structure-preserving replacement (for exampleJon's Ice Cream ShoptoJoe's Ice Cream Shop), and if the generator fails it falls back to a safe redaction.Resolution order for a detected token:
generatorconfigured invokes the generator.fallbackStrategy(defaultREDACT).A detected token is never left in the clear, and a generator failure never blocks the pipeline.
Acceptance criteria
MAP_REPLACEstrategy dispatched in the standard replacement path.mappings(object) andmappingFiles(list of local file paths), merged once at policy-load time into an in-memory map. Mirrors the existing custom-dictionaryterms+filesloading.mappingsoverridemappingFiles; documented, deterministic rule for duplicate keys across files. No undefined behavior on collision.caseSensitivefield (defaultfalse); key normalization at load and token normalization at lookup are identical.generatorsblock: name-keyed, reusable, with atypediscriminator (ollamafirst). Strategy references a generator by name viagenerator.fallbackStrategy(defaultREDACT) and a per-generator timeout are honored. Unreachable / timed-out / invalid generator output falls back safely; the token is never emitted in the clear and the pipeline never blocks indefinitely.RANDOM_REPLACE, so the same input yields the same output within a context and inference is not repeated for a repeated value.{{token}}) supported; optionally expose the entity label to the prompt.MAP_REPLACE/generatorsschema shipped in phisqlschema/1.3.0.docs/docs/updated for the new strategy and thegeneratorsblock (if this port ships those docs).Cross-port parity
This must behave identically to the sibling implementations in the other two Phileas ports (map source/precedence/case rules and the generator fallback/validation/caching contract). Divergence here is the class of bug seen with
ABBREVIATE(philterd/phileas#320, philterd/phileas-dotnet#58). The generator path is nondeterministic, so conformance tests the contract (fallback fires, input-equal output rejected, values re-scanned and cached), not generated text. Sibling issues: philterd/phileas, philterd/phileas-python, philterd/phileas-dotnet (linked below).Implementation notes (Python)
"MAP_REPLACE"action in the actions registry (the same registry that holds"ABBREVIATE": _abbreviate), and add the strategy fields to the policy model.mappingFilesat policy-load time into the in-memory map; parse each row as a key/value pair.generatorsconfig and an Ollama client for the generator call, with timeout, output-validation, and fallback.