Skip to content

Commit ea19051

Browse files
SamarthegdeSamarthegde
committed
Refactor signing module to update installation instructions and improve error messages
Add vault module for evidence push to a hosted evidence vault with idempotent ingestion Enhance sample app models with ScanSample for PII auto-discovery testing Implement comprehensive tests for audit checkpoints and verification processes Update CLI tests to reflect changes in crypto installation instructions Introduce tests for schema-only PII auto-discovery and detection mechanisms Expand Django adapter tests to include PII scanning functionality Add rules linter tests for retention policies and compliance checks Implement tests for crypto-sealing functionality in audit logs Update signing tests to reflect changes in crypto installation instructions Add SQLAlchemy adapter tests for PII handling and audit logging Introduce value-level PII detection tests for breach classification Implement tests for the evidence-push client in the vault module
1 parent 8836697 commit ea19051

39 files changed

Lines changed: 3246 additions & 65 deletions

‎CHANGELOG.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Changelog
2+
3+
All notable changes to `dpdpstack-python-sdk`. The core stays dependency-free; optional
4+
features live behind extras (`[django]`, `[sqlalchemy]`, `[crypto]`).
5+
6+
## 0.6.0
7+
- **Sealing key rotation**: `seal`/`unseal` (and `AuditLog.record(seal_key=…)` /
8+
`open_sealed`) accept a *list* of keys (newest first) via MultiFernet — new payloads
9+
seal with the first key, unsealing tries all, so older-key payloads still open.
10+
11+
## 0.5.0
12+
- **Evidence client** (`EvidenceClient`): push the tamper-evident chain to a hosted vault.
13+
Zero-dependency (stdlib), evidence-only (never PII), idempotent, with a fire-and-forget
14+
`push_background` that never blocks or raises.
15+
16+
## 0.4.0
17+
- **Retention-safe checkpoints**: `AuditLog.checkpoint()` / `prune_through()` and
18+
`verify_report()` (pinpoints where a chain breaks) let a hash-chained log be pruned and
19+
still verify by anchoring to an immutable, self-chaining `Checkpoint`.
20+
- **Offline verification**: `dpdpstack verify-chain audit.jsonl [--checkpoints cp.jsonl]`.
21+
- **Crypto-shred** (`[crypto]`): `dpdpstack.sealing` — encrypt PII into a token the entry
22+
hash covers, so destroying the key satisfies right-to-erasure while the chain verifies.
23+
- **SQLAlchemy adapter** (`[sqlalchemy]`): `contrib.sqlalchemy` mirrors the Django adapter
24+
for FastAPI/Flask/any SQLAlchemy app; the `@pii` declaration is shared.
25+
26+
## 0.3.0
27+
- **DPDP linter + readiness score**: `dpdpstack.rules` — `lint_policy` flags retention
28+
compliance smells (each tied to a DPDP citation); `score_policies` grades a policy set
29+
(0–100, letter grade, tier). CLI: `dpdpstack lint [--score]`.
30+
- **Value-level PII detection**: `detect_values` (Aadhaar via Verhoeff, cards via Luhn,
31+
PAN/GST/IFSC/UPI/email/phone/IP) and `classify_breach_nature` for a breach report.
32+
- New presets: `companies_act()`, `third_schedule()`.
33+
34+
## 0.2.0
35+
- **PII auto-discovery** (`scan`): suggest `@pii(...)` declarations from Django model
36+
schema (`dpdp_scan`) or any field names — `dpdpstack scan`, `suggest_strategies`,
37+
`scan_mapping`. Schema-only, India-first catalog, advisory.
38+
39+
## 0.1.x
40+
- Core: legal-hold-aware `ErasureEngine`, `RetentionPolicy` + RBI/PMLA/CERT-In presets,
41+
anonymization strategies (`null`/`hashed`/`redact`/`constant`), hash-chained `AuditLog`
42+
(in-memory / JSONL / Django stores), `issue_certificate` (Certificate of Erasure),
43+
optional RS256 signing (`[crypto]`), Django adapter, and the `dpdpstack` CLI.

‎README.md‎

Lines changed: 199 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,10 @@ enterprise tools "cost more than a month's revenue."
2020

2121
## Install
2222
```bash
23-
pip install dpdpstack-python-sdk # core, no dependencies
24-
pip install "dpdpstack-python-sdk[django]" # + Django adapter
25-
pip install "dpdpstack-python-sdk[crypto]" # + RS256-signed certificates (PyJWT + cryptography)
23+
pip install dpdpstack-python-sdk # core, no dependencies
24+
pip install "dpdpstack-python-sdk[django]" # + Django adapter
25+
pip install "dpdpstack-python-sdk[sqlalchemy]" # + SQLAlchemy adapter (FastAPI/Flask/…)
26+
pip install "dpdpstack-python-sdk[crypto]" # + signed certs & crypto-shred (PyJWT + cryptography)
2627
```
2728

2829
## Quickstart (framework-agnostic)
@@ -72,6 +73,189 @@ erase_instance(user, policy=RetentionPolicy(purpose="profile", action=Action.ANO
7273
erase_instance(user, policy=rbi_kyc("kyc"), subject=user.external_ref)
7374
```
7475

76+
## FastAPI / Flask / any SQLAlchemy app (`[sqlalchemy]`)
77+
The same engine + DB-backed audit chain, against a SQLAlchemy `Session`. You map the
78+
audit entry once (you own the `Base`); the `@pii` declaration is shared with Django.
79+
```python
80+
from sqlalchemy.orm import DeclarativeBase
81+
from dpdpstack import RetentionPolicy, Action, null, redact, rbi_kyc
82+
from dpdpstack.contrib.sqlalchemy.models import DpdpAuditEntryMixin
83+
from dpdpstack.contrib.sqlalchemy.service import erase_instance, pii
84+
85+
class Base(DeclarativeBase): ...
86+
87+
class DpdpAuditEntry(Base, DpdpAuditEntryMixin): # the hash-chained audit store
88+
__tablename__ = "dpdp_audit_entries"
89+
90+
@pii(name=null, email=null, phone=redact(keep_last=4))
91+
class User(Base):
92+
__tablename__ = "users"
93+
...
94+
95+
# Anonymize PII, keep the (regulated) row; your session, your transaction.
96+
erase_instance(session, user, audit_model=DpdpAuditEntry, subject=user.external_ref,
97+
policy=RetentionPolicy(purpose="profile", action=Action.ANONYMIZE))
98+
99+
# KYC withdrawal -> deferred under RBI hold, nothing deleted, basis recorded
100+
erase_instance(session, user, audit_model=DpdpAuditEntry, subject=user.external_ref,
101+
policy=rbi_kyc("kyc"))
102+
session.commit()
103+
```
104+
105+
## Find your PII fields (`scan`)
106+
You declare PII once with `@pii(...)` - but which fields *are* PII? `scan` finds them
107+
for you. It reads **field names and types only** (never a single row), matches them
108+
against an India-first catalog (Aadhaar, PAN, GST, UPI, phone, email, special-category…),
109+
and suggests an anonymize strategy for each. Output is **advisory** - you review it, then
110+
paste. Zero-egress and zero-dependency.
111+
112+
**Django** - scan your models and get pasteable `@pii(...)` blocks:
113+
```bash
114+
python manage.py dpdp_scan --format python # or: text (default) | json
115+
# or, without a manage.py:
116+
dpdpstack scan --django --settings myproject.settings --app accounts --format python
117+
```
118+
```python
119+
# accounts.User
120+
@pii(
121+
name=null,
122+
email=null,
123+
phone=redact(keep_last=4),
124+
aadhaar_number=hashed(),
125+
)
126+
class User(models.Model):
127+
...
128+
```
129+
Re-running tags each field `new` (PII, not declared), `covered` (already declared), or
130+
`drift` (declared, but no longer looks like PII) - so it doubles as an ongoing audit.
131+
132+
**Anything else** - a sample dict, an API payload, a column list:
133+
```python
134+
from dpdpstack import anonymize_fields
135+
from dpdpstack.detect import scan_mapping, suggest_strategies
136+
137+
suggest_strategies(["email", "phone", "pan", "ledger_balance"])
138+
# {'email': <null>, 'phone': <redact>, 'pan': <hashed>} # 'ledger_balance' ignored
139+
140+
record = {"email": "a@b.com", "phone": "9876543210", "ledger_balance": 500}
141+
clean = anonymize_fields(record, suggest_strategies(record.keys()))
142+
```
143+
```bash
144+
dpdpstack scan --keys email,phone,pan --format python # comma-separated names
145+
dpdpstack scan --dict sample.json --format python # keys of a JSON object ('-' = stdin)
146+
```
147+
Bring your own catalog by passing a JSON file of the same shape to `load_catalog(path=...)`.
148+
149+
### Detect PII in values (and classify a breach)
150+
The scanner above reads field *names*; `detect_values` reads *values* / free text -
151+
useful to confirm a column really holds PII, or to fill a breach report's `nature`
152+
field. Aadhaar is checked with the **Verhoeff** checksum and cards with **Luhn**, so
153+
random 12-/16-digit numbers don't false-positive. Local, zero-dependency.
154+
```python
155+
from dpdpstack import detect_values, classify_breach_nature
156+
157+
detect_values("PAN ABCDE1234F, card 4111 1111 1111 1111")
158+
# [ValueMatch(type='PAN', ...), ValueMatch(type='Payment Card', ...)]
159+
160+
classify_breach_nature("leaked rows: asha@bank.in, Aadhaar 2341 2341 2346, plus medical records")
161+
# ['Email Address', 'Aadhaar Number', 'Health Data'] # for a Rule 7 breach report
162+
```
163+
164+
## Lint your retention policies (DPDP)
165+
`lint` statically checks a `RetentionPolicy` for compliance smells - a legal hold with
166+
no recorded basis, a hold that will hard-delete a regulated row, a basis cited without a
167+
hold period, retention far past what's justified - each tied to a DPDP citation. Offline
168+
and advisory.
169+
```python
170+
from dpdpstack import RetentionPolicy, Action, lint_policy
171+
172+
lint_policy(RetentionPolicy(purpose="kyc", legal_hold_days=1825, action=Action.DELETE))
173+
# [ERROR E001: ... no legal_basis recorded ... [DPDP Rules, 2025 - Rule 8],
174+
# WARNING W001: ... action=delete will hard-delete ... consider action=anonymize ...]
175+
```
176+
From the shell (exit code is non-zero if any **error** is found, so it drops into CI):
177+
```bash
178+
dpdpstack lint --presets # the built-in presets are clean
179+
dpdpstack lint --purpose kyc --legal-hold-days 1825 --action delete # E001 + W001
180+
```
181+
`dpdpstack.rules` also exposes `DPDP_RULES` and `STATUTORY_HOLDS` (RBI/PMLA/CERT-In/
182+
Companies Act) as a citable reference.
183+
184+
`score_policies(...)` rolls the findings into a graded **readiness report** - a deterministic
185+
0-100 score, letter grade, and tier across all your policies (great for a dashboard or an
186+
onboarding report):
187+
```python
188+
from dpdpstack import score_policies, rbi_kyc, pmla
189+
190+
score_policies([rbi_kyc(), pmla()]).summary
191+
# '100/100 (A+, exemplary) across 2 policies: 2 clean, 0 errors, 0 warnings.'
192+
```
193+
```bash
194+
dpdpstack lint --presets --score # ... Readiness: 100/100 (A+, exemplary) across 5 policies …
195+
```
196+
197+
## Retention-safe audit + offline verification
198+
The audit log is hash-chained, so any change breaks `verify()`. But a *retention* log
199+
must be prunable - and a pruned chain no longer starts at sequence 1, which would break
200+
verification. **Checkpoints** fix that: snapshot a run of entries into an immutable,
201+
self-chaining `Checkpoint`, then prune; verification anchors to the checkpoint instead
202+
of the genesis.
203+
```python
204+
log = AuditLog(JsonlAuditStore("audit.jsonl"))
205+
# ... record events ...
206+
cp = log.checkpoint(through_sequence=1000) # immutable snapshot (persist it)
207+
log.prune_through(1000) # drop the archived entries
208+
209+
log.verify_report([cp]) # VerifyResult(ok=True, checked=…, anchored_at=1000)
210+
log.verify_report() # ok=False, first_error_sequence pinpoints any tampering
211+
```
212+
An auditor can verify a chain straight from storage - **no backend, no API to trust**:
213+
```bash
214+
dpdpstack verify-chain audit.jsonl --checkpoints cp.jsonl
215+
# OK - verified 2400 entries (anchored at #1000).
216+
# (exits non-zero and names the broken entry if the chain was tampered with)
217+
```
218+
219+
## Crypto-shred PII in the audit log (optional, `[crypto]`)
220+
The chain normally holds no PII (`subject` is an opaque ref). When you must record PII
221+
*inside* an entry, `seal` it: the PII is encrypted into an opaque token that the entry
222+
hash covers. Verification runs on the ciphertext, so you can later **destroy the key**
223+
(right-to-erasure) - the payload becomes unreadable while the chain still verifies.
224+
```python
225+
from dpdpstack.sealing import generate_seal_key
226+
227+
key = generate_seal_key() # keep secret; deleting it shreds the data
228+
e = log.record("evidence", subject="user_42",
229+
private={"aadhaar": "2341 2341 2346"}, seal_key=key)
230+
AuditLog.open_sealed(e, key) # -> {"aadhaar": "…"} (with the key)
231+
log.verify() # True — even after the key is destroyed
232+
```
233+
**Key rotation** (zero-downtime): pass a *list* of keys, newest first. New entries seal with
234+
the first key; unsealing tries all, so older-key entries still open. The ciphertext is part
235+
of the entry hash, so chain entries are never re-encrypted — keep an old key around to read
236+
old entries, and retire it once they've been pruned or shredded.
237+
```python
238+
new = generate_seal_key()
239+
log.record("evidence", subject="user_43", private={…}, seal_key=[new, key]) # seals with `new`
240+
AuditLog.open_sealed(e, [new, key]) # still opens the old-key entry
241+
```
242+
243+
## Push evidence to the hosted vault (optional)
244+
Keep everything local, or push your tamper-evident chain to a vault (e.g. getdpdp.net)
245+
for an independent, server-timestamped, counter-signed copy. The push carries
246+
**evidence only** - opaque refs, event types, and hashes (plus any sealed ciphertext) -
247+
**never PII**, so it stays zero-egress. It's zero-dependency (stdlib), idempotent at the
248+
vault (re-pushing is a no-op), and the fire-and-forget variant never blocks or raises in
249+
your request path.
250+
```python
251+
from dpdpstack import EvidenceClient
252+
253+
vault = EvidenceClient("https://getdpdp.net/api/v1", api_key="dpdp_sk_…", source="api")
254+
255+
vault.push(log) # synchronous: -> {"stored": N, "chain_verified": True, …}
256+
vault.push_background(log) # fire-and-forget: returns immediately, errors swallowed
257+
```
258+
75259
## Signed certificates (optional, `[crypto]`)
76260
The hash-chained Certificate of Erasure is tamper-evident on its own; add an RS256
77261
signature so anyone can verify it with your public key (and you can't forge it):
@@ -99,20 +283,27 @@ dpdpstack verify cert.jwt --public-key ./keys/cert_public.pem
99283
(`python -m dpdpstack verify ...` works too.)
100284

101285
## Presets for the common conflicts
102-
`rbi_kyc()` (5-yr hold, anonymize) · `pmla()` · `cert_in_logs()` (180-day log hold). Or build your own `RetentionPolicy(retention_days=…, legal_hold_days=…, legal_basis="…", action=…)`.
286+
`rbi_kyc()` (5-yr hold, anonymize) · `pmla()` · `cert_in_logs()` (180-day log hold) · `companies_act()` (8-yr books of account) · `third_schedule()` (DPDP specified period). Or build your own `RetentionPolicy(retention_days=…, legal_hold_days=…, legal_basis="…", action=…)`.
103287

104288
## What's in the box
105289
| Module | What |
106290
|---|---|
107-
| `policies` | `RetentionPolicy` + RBI/PMLA/CERT-In presets |
291+
| `policies` | `RetentionPolicy` + RBI/PMLA/CERT-In/Companies-Act/Third-Schedule presets |
108292
| `anonymize` | `null` / `hashed` / `redact` / `constant` field strategies |
109-
| `audit` | hash-chained, tamper-evident log; pluggable store (in-memory, JSONL, Django) |
293+
| `audit` | hash-chained log + checkpoints/pruning + `verify_report`; store (in-memory, JSONL, Django, SQLAlchemy) |
110294
| `erasure` | `ErasureEngine` - legal-hold-aware resolve + your `executor` |
111295
| `certificate` | `issue_certificate()` → verifiable Certificate of Erasure |
296+
| `detect` | PII discovery - schema (`scan_mapping`) + values (`detect_values`, `classify_breach_nature`) |
297+
| `rules` | DPDP knowledge pack + `lint_policy()` / `dpdpstack lint` |
298+
| `vault` | `EvidenceClient` - push the chain to a hosted vault (evidence only, fire-and-forget) |
299+
| `sealing` *(extra)* | crypto-shred PII in the chain - `seal` / `unseal` / `AuditLog.open_sealed` |
112300
| `signing` *(extra)* | RS256-sign/verify a certificate - `pip install dpdpstack-python-sdk[crypto]` |
113-
| `contrib.django` | model-backed audit store + `erase_instance()` + `@pii(...)` |
301+
| `contrib.django` | model-backed audit store + `erase_instance()` + `@pii(...)` + `dpdp_scan` |
302+
| `contrib.sqlalchemy` *(extra)* | the same for any SQLAlchemy app (FastAPI/Flask/…) |
303+
304+
CLI: `dpdpstack scan` · `lint` · `verify-chain` · `verify` · `keygen` (`python -m dpdpstack …`).
114305

115306
## Status & scope
116-
Alpha (0.1). The core is dependency-free and framework-agnostic; the Django adapter is the first integration (FastAPI/Flask next, on demand). Hosted/managed version (dashboard, cross-system fan-out, certificate vault): [**getdpdp.net**](https://getdpdp.net/).
307+
Alpha (0.6). The core is dependency-free and framework-agnostic; Django and SQLAlchemy adapters ship today. Hosted/managed version (dashboard, cross-system fan-out, certificate vault): [**getdpdp.net**](https://getdpdp.net/).
117308

118309
dpdpstack is tooling, not legal advice; you remain the Data Fiduciary. MIT licensed.

‎examples/sign_and_verify.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
"""Sign a Certificate of Erasure and verify it (requires the 'crypto' extra).
22
3-
pip install "dpdpstack[crypto]"
3+
pip install "dpdpstack-python-sdk[crypto]"
44
python examples/sign_and_verify.py
55
66
Shows the full "show the evidence" loop: erase -> issue cert -> RS256-sign ->

‎pyproject.toml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "dpdpstack-python-sdk"
7-
version = "0.1.1"
7+
version = "0.6.0"
88
description = "DPDP-compliant data erasure for Indian apps: legal-hold-aware deletion, PII anonymization, and tamper-evident Certificates of Erasure. Zero-egress - runs inside your app."
99
readme = "README.md"
1010
requires-python = ">=3.9"
@@ -22,6 +22,7 @@ dependencies = []
2222

2323
[project.optional-dependencies]
2424
django = ["Django>=4.2"]
25+
sqlalchemy = ["SQLAlchemy>=2.0"]
2526
crypto = ["pyjwt[crypto]>=2.8"]
2627
dev = ["pytest"]
2728

@@ -36,3 +37,6 @@ Issues = "https://github.com/getdpdp/dpdpstack-python-sdk/issues"
3637

3738
[tool.setuptools.packages.find]
3839
where = ["src"]
40+
41+
[tool.setuptools.package-data]
42+
"dpdpstack.detect" = ["data/*.json"]

‎src/dpdpstack/__init__.py‎

Lines changed: 44 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,38 @@
55
mutation, so personal data never leaves your systems.
66
"""
77
from .anonymize import anonymize_fields, constant, hashed, null, redact
8-
from .audit import AuditEntry, AuditLog, InMemoryAuditStore, JsonlAuditStore, compute_hash
8+
from .audit import (
9+
AuditEntry,
10+
AuditLog,
11+
Checkpoint,
12+
InMemoryAuditStore,
13+
JsonlAuditStore,
14+
VerifyResult,
15+
compute_hash,
16+
)
917
from .certificate import ErasureCertificate, issue_certificate
18+
from .detect import (
19+
Detection,
20+
classify_breach_nature,
21+
detect_values,
22+
match_field,
23+
suggest_strategies,
24+
)
1025
from .erasure import ErasureEngine, ErasureResult, is_retention_due
11-
from .policies import Action, RetentionPolicy, Trigger, cert_in_logs, pmla, rbi_kyc
26+
from .policies import (
27+
Action,
28+
RetentionPolicy,
29+
Trigger,
30+
cert_in_logs,
31+
companies_act,
32+
pmla,
33+
rbi_kyc,
34+
third_schedule,
35+
)
36+
from .rules import LintFinding, ReadinessReport, lint_policies, lint_policy, score_policies
37+
from .vault import EvidenceClient
1238

13-
__version__ = "0.1.0"
39+
__version__ = "0.6.0"
1440

1541
__all__ = [
1642
"Action",
@@ -19,13 +45,17 @@
1945
"rbi_kyc",
2046
"pmla",
2147
"cert_in_logs",
48+
"companies_act",
49+
"third_schedule",
2250
"anonymize_fields",
2351
"null",
2452
"hashed",
2553
"redact",
2654
"constant",
2755
"AuditLog",
2856
"AuditEntry",
57+
"Checkpoint",
58+
"VerifyResult",
2959
"InMemoryAuditStore",
3060
"JsonlAuditStore",
3161
"compute_hash",
@@ -34,4 +64,15 @@
3464
"is_retention_due",
3565
"ErasureCertificate",
3666
"issue_certificate",
67+
"match_field",
68+
"suggest_strategies",
69+
"Detection",
70+
"detect_values",
71+
"classify_breach_nature",
72+
"lint_policy",
73+
"lint_policies",
74+
"LintFinding",
75+
"score_policies",
76+
"ReadinessReport",
77+
"EvidenceClient",
3778
]

0 commit comments

Comments
 (0)