fix: stop PII redaction from matching look-alike keys#11
Open
ymeloch wants to merge 1 commit into
Open
Conversation
Substring matching on the key name meant fields like description, recipient, zip, and userAgent were redacted because they contain 'ip' or 'user'. Match on normalized whole key names and split tokens instead. Adds regression tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3
When SECURE_ACCESS_REDACT_PII=true, _redact_pii matched key names by substring against hints that included "ip" and "user". That redacts fields that merely contain those letters:
"ip" matches description, recipient, zip, relationship
"user" matches userAgent
So report/activity output gets mangled in non-obvious ways.
This PR replaces the substring check with matching on normalized whole key names plus split tokens (camelCase / snake_case aware). Real PII keys (identity, ip, internalIp, email, user, device, …) still redact; look-alikes like description and userAgent no longer do. "ip" and "email" are still matched as standalone tokens so compound keys like sourceIp are caught, while "user" is matched only as a whole key so userAgent is left alone.
Adds tests/test_pii_redaction.py covering both the true positives and the regression cases. Verified the new test fails against the old substring logic and passes with this change.