perf: ~15x faster anonymization via cheap substring gates - #21
Merged
Conversation
Run each regex family in TraceAnonymizer only when a cheap literal substring check says the text could match: emails need '@', home paths need 'home'/'Users', each API-key pattern needs its literal prefix, assignment/generic-secret scans need a sensitive word, JWTs need a lookaround-free pre-scan. Also cache the per-username owner/group patterns instead of recompiling them for every string. Output is byte-identical on a 3000-line benchmark (7.2s -> 0.48s); per-file replacement scoping is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The anonymization phase of
teich extractruns ~35 compiled regexes (several with expensive variable-length lookbehinds, plus assignment patterns that match everykey=value) over every string in every message — even plain prose that can't contain a secret. This PR gates each regex family behind a cheap literal substring check so the expensive patterns only run on text that could possibly match:@is present; home-path scrubbing only onhome/Userssk-ant-api03-,ghp_,AKIA, …) appears_is_sensitive_namefire is present (pass,secret,token,key,url, …)-----BEGIN; Bearer only onbearer; credential URLs only on://+@Also caches the per-username owner/group regexes, which were recompiled for every string.
Results
3000-line synthetic chat export seeded with real keys/emails/home paths: 7.22s → 0.48s (~15x), output byte-identical to the old code, replacement counts identical. Per-file anonymizer scoping (consistent within a trace, fresh across traces) is unchanged.
Testing
pytest tests/test_extract_anonymize_cli.py— 32 passed🤖 Generated with Claude Code