A comprehensive data redaction solution using spaCy NER (Named Entity Recognition) combined with regex pattern matching for detecting and redacting Personally Identifiable Information (PII) from text.
This project demonstrates an advanced approach to data redaction that goes beyond simple regex patterns. By leveraging spaCy's pre-trained NLP model (en_core_web_sm), the system can intelligently identify:
- Person Names - Named entities recognized by NLP model
- Organizations - Companies, institutions, organizations
- Locations - Geographic locations, cities, countries
- Structured PII - Email addresses, phone numbers, SSNs, credit cards, IP addresses, URLs
✨ Hybrid Detection Approach
- NLP-based entity recognition for context-aware PII detection
- Regex patterns for structured PII (email, phone, SSN, credit card, etc.)
- Automatic overlap detection and deduplication
🎯 Multiple Redaction Operators
replace: Replaces with<ENTITY_TYPE>placeholderredact: Masks with asteriskshash: Replaces with hashed values for consistency
📊 Rich Detection Results
- Entity type classification
- Original text preservation
- Position tracking (start/end indices)
- Detailed entity breakdown
- Python 3.7+
- pip package manager
# Clone the repository
git clone <repo-url>
cd TheOne
# Install spaCy
pip install spacy
# Download the English language model
python -m spacy download en_core_web_sm
# Verify installation
python -c "import spacy; nlp = spacy.load('en_core_web_sm'); print('✓ spaCy model loaded successfully')"TheOne/
├── README.md
├── src/
│ ├── redactor.py # Main TextRedactor class
│ └── test_redactor.py # Comprehensive test suite
└── requirements.txt # Project dependencies (optional)
from src.redactor import TextRedactor
# Initialize the redactor
redactor = TextRedactor()
# Redact PII from text
text = "John Smith works at Google. Contact: john.smith@google.com or (555) 123-4567"
result = redactor.redact_text(text, operator="replace")
print(result['redacted_text'])
# Output: <PERSON> works at <ORGANIZATION>. Contact: <EMAIL_ADDRESS> or (<CARDINAL>) <CARDINAL>-4567
print(f"Entities found: {result['total_entities']}")# Replace with entity type
result = redactor.redact_text(text, operator="replace")
# Output: <PERSON> works at <ORGANIZATION>...
# Mask with asterisks
result = redactor.redact_text(text, operator="redact")
# Output: ********** works at ******...
# Hash-based redaction
result = redactor.redact_text(text, operator="hash")
# Output: <HASH:5582620> works at <HASH:9211655>...# Full hybrid approach (NLP + Regex)
result_nlp = redactor.redact_text(text, operator="replace", use_nlp=True)
# Regex patterns only (no NLP)
result_regex = redactor.redact_text(text, operator="replace", use_nlp=False)The project includes a comprehensive test suite demonstrating all features:
python src/test_redactor.py- Basic Redaction - Tests all three operators (replace, redact, hash)
- NLP vs Regex - Compares detection accuracy between approaches
- Various PII Types - Tests emails, phones, SSNs, IPs, URLs, person names, organizations
- Complex Document - Real-world confidential document redaction
| Text | Regex Only | NLP + Regex | Accuracy Gain |
|---|---|---|---|
| "Dr. Sarah Johnson from Apple Inc. sent an email to michael.brown@apple.com about the project in San Francisco." | 1 entity (email only) | 4 entities (name, org, email, location) | +300% |
| PII Type | Detection Method | Example |
|---|---|---|
| Person Names | NLP (spaCy NER) | John Smith, Dr. Sarah Johnson |
| Organizations | NLP (spaCy NER) | Google, Apple Inc., Federal Reserve |
| Locations | NLP (spaCy NER) | San Francisco, New York, USA |
| Email Addresses | Regex Pattern | john.smith@google.com |
| Phone Numbers | Regex Pattern | (555) 123-4567, +1-555-987-6543 |
| Credit Cards | Regex Pattern | 4532-1234-5678-9010 |
| Social Security Numbers | Regex Pattern | 123-45-6789 |
| IP Addresses | Regex Pattern | 192.168.1.100 |
| URLs | Regex Pattern | https://example.com |
Initializes the redactor with spaCy model and regex patterns.
Parameters:
text(str): Input text to redactoperator(str): Redaction method -"replace","redact", or"hash"use_nlp(bool): Enable NLP-based entity detection (default: True)
Returns:
{
"redacted_text": str, # Redacted output text
"entities_found": list, # List of detected entities
"total_entities": int, # Count of redacted entities
"method": str, # Detection method used
"message": str # Status message (if no entities found)
}Entity Object:
{
"type": str, # Entity type (PERSON, ORGANIZATION, EMAIL_ADDRESS, etc.)
"start": int, # Start position in original text
"end": int, # End position in original text
"original": str # Original matched text
}- Healthcare - HIPAA compliance, patient record anonymization
- Finance - PCI-DSS compliance, customer data protection
- Legal - Sensitive document redaction
- Data Science - Creating anonymized datasets for analysis
- Content Publishing - Preparing data for public release
- Data Migration - Protecting PII during system transitions
- Context Awareness - Understands context to reduce false positives
- Better Name Detection - Catches person names in various contexts
- Organization Recognition - Identifies company names accurately
- Location Awareness - Detects geographic references
- Flexible - Can toggle between NLP and regex-only modes
- Comprehensive - Combines linguistic and pattern-based detection
- NLP model size (en_core_web_sm ~40MB)
- Language limitation (currently English only)
- Processing speed for very large documents
- Custom PII patterns support (to be added)
- Support for additional spaCy models (en_core_web_md, en_core_web_lg)
spacy>=3.0.0- Natural Language Processing library- Python 3.7+ - Required runtime
This project is part of a Towards Data Science article demonstrating the advantages of NLP-based over regex-only PII redaction approaches.