Skip to content

Commit e2cc13a

Browse files
Merge pull request #86 from FreshCode-Org/feature/semantic-date-memory-jwd
feat(semantic): add DatePhraseExpert and retrieval-backed semantic memory
2 parents e9d52fa + 1e0c6f5 commit e2cc13a

16 files changed

Lines changed: 1172 additions & 51 deletions

‎README.md‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -435,20 +435,60 @@ cleaned, report = fd.clean(
435435
Modes: `"assist"` (suggest only), `"review"` (apply zero-risk deterministic repairs,
436436
suggest the rest), `"auto"` (apply high-confidence low-risk repairs that pass policy).
437437

438+
### Date phrase repair
439+
440+
`DatePhraseExpert` normalizes ISO/slash/dash dates, month-name dates, and
441+
`"today"`/`"yesterday"`/`"tomorrow"` in date-like columns — but relative phrases only
442+
resolve with an explicit `reference_date` (never the real wall-clock date), and
443+
ambiguous numeric dates (day/month both `<= 12`) are never auto-applied without an
444+
explicit `dayfirst` hint:
445+
446+
```python
447+
cleaned, report = fd.clean(
448+
df,
449+
semantic_mode="auto",
450+
semantic_context={
451+
"reference_date": "2026-07-01",
452+
"columns": {"signup_date": {"semantic_type": "date"}},
453+
},
454+
return_report=True,
455+
)
456+
```
457+
458+
### Semantic memory replay
459+
460+
Accepted semantic repairs can be learned into a `CleaningMemory` and replayed as
461+
policy-gated candidate proposals on similar future data:
462+
463+
```python
464+
cleaned, report = fd.clean(df, semantic_mode="auto", return_report=True)
465+
466+
memory = fd.learn_cleaning_memory(df, decisions=report, dataset_id="crm_signups")
467+
468+
next_cleaned, next_report = fd.clean(
469+
next_df, semantic_mode="auto", memory=memory, return_report=True,
470+
)
471+
```
472+
473+
Memory is local/server-free, never overrides policy (protected columns, confidence
474+
floor, and the ambiguity rules above still apply), and every replayed decision is
475+
audited with `memory_influenced=True` and `model_id="semantic:<issue_type>:memory"`.
476+
438477
**Guarantees and honesty about scope:**
439478

440479
- **Off by default** — existing behavior is byte-for-byte unchanged unless enabled.
441-
- **The first version is deterministic and fully offline** — no LLM, no network calls,
442-
no API keys. The interfaces leave clean extension points for future
443-
retrieval-backed memory and optional private LLM backends.
480+
- **Deterministic and fully offline** — no LLM, no network calls, no API keys, no vector
481+
database. Semantic memory replay is a local, no-dependency retrieval mechanism, not a
482+
neural/embedding search; the interfaces still leave a clean extension point for an
483+
optional future private-LLM candidate generator.
444484
- **An LLM never mutates the DataFrame.** Every change goes through profile → propose →
445485
score → policy → (maybe) apply.
446486
- **Ambiguous repairs are suggestions, not silent mutations** — they are recorded with
447487
`status="suggested"` and flagged for human review.
448488
- **ID, target, and `preserve_columns` are protected** — identifier-like columns are
449489
vetoed unless `semantic_context` explicitly marks them `mutable`.
450490
- **Every proposal is audited** in the report with rationale, risk, confidence, status,
451-
and `model_id` (e.g. `semantic:spelled_number:v1`).
491+
and `model_id` (e.g. `semantic:spelled_number:v1`, `semantic:date_phrase:memory`).
452492
- Semantic cleaning runs on the in-memory pandas path; on a native engine
453493
(Polars/DuckDB/Spark) it falls back to pandas and records the fallback in the report.
454494

‎docs/semantic-cleaning.md‎

Lines changed: 73 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ Examples it handles, deterministically and offline:
1414
| `currency_string` | `"$1,200.50"` | `1200.5` | money-like columns |
1515
| `unit_suffix` | `"10 kg"` | `10` | unit-consistent columns |
1616
| `category_synonym`| `"M"`, `"Male "` | `"male"` | low-cardinality categoricals|
17+
| `date_phrase` | `"2026-07-01"`, `"today"` | `Timestamp` | date-like columns |
1718

1819
## How it works
1920

@@ -27,9 +28,10 @@ Every change flows through a fixed, auditable pipeline:
2728
6. otherwise **record a suggestion** for review;
2829
7. **preserve every decision** in the `CleanReport`.
2930

30-
> An LLM never mutates the DataFrame. The first version is fully deterministic and offline
31-
> (no network, no API keys). The interfaces leave clean extension points for future
32-
> retrieval-backed memory and optional private LLM candidate generation.
31+
> An LLM never mutates the DataFrame. Everything is fully deterministic and offline (no
32+
> network, no API keys). Retrieval-backed [semantic memory replay](#semantic-memory-replay)
33+
> is implemented as a local, server-free extension; the interfaces still leave a clean
34+
> extension point for an optional future private-LLM candidate generator.
3335
3436
## Enabling it
3537

@@ -64,6 +66,72 @@ cleaned, report = fd.clean(
6466
)
6567
```
6668

69+
### Date phrase repair
70+
71+
`DatePhraseExpert` normalizes obvious date phrases and strings in date-like columns
72+
(matched by role, `semantic_type`, a date-like column name, or a strong share of
73+
parseable values): ISO/slash-ordered dates (`2026-07-01`, `2026/07/01`), month-name dates
74+
(`July 1 2026`, `1 July 2026`), and numeric dates where day/month order is unambiguous.
75+
76+
`"today"`/`"yesterday"`/`"tomorrow"` are **only** resolved when you supply an explicit
77+
`reference_date` — the real wall-clock date is never used silently:
78+
79+
```python
80+
cleaned, report = fd.clean(
81+
df,
82+
semantic_mode="auto",
83+
semantic_context={
84+
"reference_date": "2026-07-01",
85+
"columns": {"signup_date": {"semantic_type": "date"}},
86+
},
87+
return_report=True,
88+
)
89+
```
90+
91+
Numeric dates where **both** day and month are `<= 12` (`"01/02/2026"`) are ambiguous and
92+
are never auto-applied — they are recorded as high-risk, `human_review=True` suggestions
93+
unless you resolve the convention explicitly with `dayfirst`:
94+
95+
```python
96+
semantic_context={"columns": {"event_date": {"semantic_type": "date", "dayfirst": True}}}
97+
```
98+
99+
Applied date repairs convert the column to pandas `datetime64` dtype only when *every*
100+
value in the column resolved successfully — a column with any leftover unconverted
101+
string stays as-is, so the conversion never silently drops information.
102+
103+
### Semantic memory replay
104+
105+
Accepted semantic repairs can be learned into a `CleaningMemory`
106+
(`fd.learn_cleaning_memory`) and replayed as candidate proposals on similar future data:
107+
108+
```python
109+
cleaned, report = fd.clean(df, semantic_mode="auto", return_report=True)
110+
111+
memory = fd.learn_cleaning_memory(df, decisions=report, dataset_id="crm_signups")
112+
113+
next_cleaned, next_report = fd.clean(
114+
next_df,
115+
semantic_mode="auto",
116+
memory=memory,
117+
return_report=True,
118+
)
119+
```
120+
121+
- Memory is local and server-free — it stores learned repairs in
122+
`memory.value_patterns["semantic_repairs"]`, no different from the rest of
123+
`CleaningMemory`'s JSON/SQLite storage.
124+
- Memory is **evidence, not authority**: every retrieved repair still passes through the
125+
same policy gate as a deterministic proposal, so it can never override
126+
`target_column`/`id_columns`/`preserve_columns` protection, the confidence floor, or the
127+
ambiguous-date rules above.
128+
- Replayed repairs are fully audited: `memory_influenced=True` and
129+
`model_id="semantic:<issue_type>:memory"` mark which decisions came from memory, and
130+
`Action.metadata` carries the raw/proposed value and evidence either way.
131+
- Retrieval matches on the exact normalized value first, falling back to a lightweight,
132+
no-dependency similarity check (`difflib`) for minor value drift; low-similarity or
133+
conflicting repairs are never auto-applied.
134+
67135
## Configuration
68136

69137
| `CleanConfig` field | Default | Meaning |
@@ -74,7 +142,7 @@ cleaned, report = fd.clean(
74142
| `semantic_max_distinct_values` | `500` | skip very high-cardinality columns |
75143
| `semantic_sample_size` | `10_000` | rows sampled when profiling distinct values |
76144
| `semantic_backends` | `("deterministic",)` | candidate backends (extension point) |
77-
| `semantic_context` | `None` | per-column hints (semantic_type/unit/allowed_values/mutable) |
145+
| `semantic_context` | `None` | per-column hints (semantic_type/unit/allowed_values/mutable/dayfirst) plus a top-level `reference_date` for relative date phrases |
78146
| `semantic_privacy_policy` | `"local_only"` | privacy posture for future external inference |
79147
| `semantic_budget` | `None` | budget hints (extension point) |
80148

@@ -96,6 +164,7 @@ Each decision appears in the report with `step="semantic"`:
96164
for a in report:
97165
if a.step == "semantic":
98166
print(a.status, a.column, a.confidence, a.risk, a.model_id, a.description)
167+
print(a.memory_influenced, a.metadata) # metadata carries raw/proposed value + evidence
99168
```
100169

101170
Preview proposal counts before running, via `suggest_plan`:

‎src/freshdata/api.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,7 @@ def clean(
248248

249249
want_report = return_report or memory is not None
250250
cleaner = Cleaner(config=config, **options)
251-
result = cleaner.clean(df, report=want_report)
251+
result = cleaner.clean(df, report=want_report, memory=memory)
252252
if want_report:
253253
cleaned, rep = result
254254
if memory is not None and mem_match is not None:

‎src/freshdata/cleaner.py‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ def _validate_input(df: object, config: CleanConfig) -> pd.DataFrame:
4444
return frame
4545

4646

47-
def run_pipeline(df: pd.DataFrame, config: CleanConfig) -> tuple[pd.DataFrame, CleanReport]:
47+
def run_pipeline(
48+
df: pd.DataFrame, config: CleanConfig, *, memory: object | None = None
49+
) -> tuple[pd.DataFrame, CleanReport]:
4850
"""Run every enabled step, in a fixed and documented order.
4951
5052
With ``preserve_original=True`` (the default) the input frame is never
@@ -57,6 +59,10 @@ def run_pipeline(df: pd.DataFrame, config: CleanConfig) -> tuple[pd.DataFrame, C
5759
duplicates), ``strategy="auto"`` runs the decision engine for missing
5860
values and outliers; explicit ``impute=`` / ``outliers=`` settings always
5961
override the corresponding engine stage.
62+
63+
``memory`` (a :class:`~freshdata.CleaningMemory`, or ``None``) is passed
64+
through to the semantic stage so it can retrieve and replay compatible
65+
learned semantic repairs; every other step ignores it.
6066
"""
6167
df = _validate_input(df, config)
6268
report = CleanReport(
@@ -87,7 +93,7 @@ def run_pipeline(df: pd.DataFrame, config: CleanConfig) -> tuple[pd.DataFrame, C
8793
# Lazily imported to keep ``import freshdata`` light.
8894
from .semantic.apply import run_semantic # noqa: PLC0415
8995

90-
out = run_semantic(out, config, report)
96+
out = run_semantic(out, config, report, memory=memory)
9197
if config.engine_mode is not None:
9298
cache = build_engine_cache(out, config)
9399
out = auto_missing(out, config, report, contexts=cache.contexts,
@@ -132,15 +138,17 @@ def __init__(self, config: CleanConfig | None = None, **options: object) -> None
132138
self.report_: CleanReport | None = None
133139

134140
def clean(
135-
self, df: pd.DataFrame, *, report: bool = False
141+
self, df: pd.DataFrame, *, report: bool = False, memory: object | None = None
136142
) -> pd.DataFrame | tuple[pd.DataFrame, CleanReport]:
137143
"""Clean *df* and return the result (the input is left unchanged
138144
unless ``preserve_original=False`` was configured).
139145
140146
With ``report=True``, returns ``(cleaned_df, CleanReport)`` instead.
141-
The latest report is always available as :attr:`report_`.
147+
The latest report is always available as :attr:`report_`. ``memory``
148+
(a :class:`~freshdata.CleaningMemory`) lets the semantic stage replay
149+
compatible learned repairs; see :func:`freshdata.clean`'s ``memory=``.
142150
"""
143-
cleaned, rep = run_pipeline(df, self.config)
151+
cleaned, rep = run_pipeline(df, self.config, memory=memory)
144152
self.report_ = rep
145153
if self.config.verbose:
146154
print(rep.brief())

‎src/freshdata/memory.py‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,7 @@ def learn_cleaning_memory(
281281
) -> CleaningMemory:
282282
"""Learn a :class:`CleaningMemory` from reviewed decisions on *df*."""
283283
from . import __version__
284+
from .semantic.memory import learn_semantic_repairs
284285

285286
accepted, rejected = _normalize_decisions(decisions)
286287
if roles is None:
@@ -291,14 +292,22 @@ def learn_cleaning_memory(
291292
for r in infer_roles(df).to_dict("records")}
292293
except Exception: # pragma: no cover - role inference is best-effort
293294
roles = {}
295+
296+
vp = dict(value_patterns or {})
297+
semantic_repairs = learn_semantic_repairs(decisions)
298+
if semantic_repairs:
299+
for repair in semantic_repairs:
300+
repair["dataset_id"] = dataset_id
301+
vp["semantic_repairs"] = list(vp.get("semantic_repairs", [])) + semantic_repairs
302+
294303
return CleaningMemory(
295304
dataset_id=dataset_id,
296305
signature=_signature(df),
297306
roles=roles,
298307
accepted=accepted,
299308
rejected=rejected,
300309
thresholds=dict(thresholds or {}),
301-
value_patterns=dict(value_patterns or {}),
310+
value_patterns=vp,
302311
exceptions=list(exceptions or []),
303312
created_at=datetime.now(timezone.utc).isoformat(),
304313
freshdata_version=__version__,
@@ -373,7 +382,11 @@ def annotate_report(report: CleanReport, memory: CleaningMemory, match: MemoryMa
373382
new_actions = []
374383
for a in report.actions:
375384
key = (str(a.column) if a.column is not None else None, str(a.step))
376-
if key in accepted:
385+
# Semantic actions replay via the dedicated, policy-gated semantic-memory
386+
# pathway (see freshdata.semantic.apply); marking them here too would let
387+
# this coarse column+step match silently override a suggested/skipped,
388+
# high-risk semantic decision with "approved".
389+
if a.step != "semantic" and key in accepted:
377390
new_actions.append(replace(a, status="approved", memory_influenced=True))
378391
n_marked += 1
379392
else:

‎src/freshdata/report.py‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,10 @@ class Action:
6464
memory_influenced: bool = False
6565
#: ``True`` when the action is flagged for human review.
6666
human_review: bool = False
67+
#: Free-form, JSON-friendly audit detail (e.g. the semantic layer's
68+
#: raw/proposed value, evidence, and memory-replay key). Empty for steps
69+
#: that don't need it, so ``to_dict()`` omits it for those.
70+
metadata: dict[str, Any] = field(default_factory=dict)
6771

6872
def __str__(self) -> str:
6973
target = f"{self.column!r}: " if self.column is not None else ""
@@ -171,6 +175,7 @@ def add(
171175
reversible: bool | None = None,
172176
memory_influenced: bool = False,
173177
human_review: bool = False,
178+
metadata: dict[str, Any] | None = None,
174179
) -> None:
175180
"""Record one action (internal; called by the pipeline)."""
176181
self.actions.append(
@@ -187,6 +192,7 @@ def add(
187192
reversible=reversible,
188193
memory_influenced=memory_influenced,
189194
human_review=human_review,
195+
metadata=dict(metadata) if metadata else {},
190196
)
191197
)
192198

@@ -216,6 +222,26 @@ def cells_changed(self) -> int:
216222
"""Total affected cells/rows summed across all actions."""
217223
return sum(a.count for a in self.actions)
218224

225+
@staticmethod
226+
def _action_dict(a: Action) -> dict[str, Any]:
227+
entry: dict[str, Any] = {
228+
"step": a.step,
229+
"column": a.column,
230+
"description": a.description,
231+
"count": a.count,
232+
"rationale": a.rationale,
233+
"risk": a.risk,
234+
"confidence": a.confidence,
235+
"model_id": a.model_id,
236+
"status": a.status,
237+
"reversible": a.reversible,
238+
"memory_influenced": a.memory_influenced,
239+
"human_review": a.human_review,
240+
}
241+
if a.metadata:
242+
entry["metadata"] = dict(a.metadata)
243+
return entry
244+
219245
def to_dict(self) -> dict[str, Any]:
220246
"""Return a JSON-friendly dictionary representation of the report.
221247
@@ -248,23 +274,7 @@ def to_dict(self) -> dict[str, Any]:
248274
"columns_preserved": list(self.columns_preserved),
249275
"warnings": list(self.warnings),
250276
"recommendations": list(self.recommendations),
251-
"actions": [
252-
{
253-
"step": a.step,
254-
"column": a.column,
255-
"description": a.description,
256-
"count": a.count,
257-
"rationale": a.rationale,
258-
"risk": a.risk,
259-
"confidence": a.confidence,
260-
"model_id": a.model_id,
261-
"status": a.status,
262-
"reversible": a.reversible,
263-
"memory_influenced": a.memory_influenced,
264-
"human_review": a.human_review,
265-
}
266-
for a in self.actions
267-
],
277+
"actions": [self._action_dict(a) for a in self.actions],
268278
}
269279
if self.domain is not None:
270280
payload["domain"] = self.domain

0 commit comments

Comments
 (0)