@@ -14,6 +14,7 @@ Examples it handles, deterministically and offline:
1414| ` currency_string ` | ` "$1,200.50" ` | ` 1200.5 ` | money-like columns |
1515| ` unit_suffix ` | ` "10 kg" ` | ` 10 ` | unit-consistent columns |
1616| ` category_synonym ` | ` "M" ` , ` "Male " ` | ` "male" ` | low-cardinality categoricals|
17+ | ` date_phrase ` | ` "2026-07-01" ` , ` "today" ` | ` Timestamp ` | date-like columns |
1718
1819## How it works
1920
@@ -27,9 +28,10 @@ Every change flows through a fixed, auditable pipeline:
27286 . otherwise ** record a suggestion** for review;
28297 . ** preserve every decision** in the ` CleanReport ` .
2930
30- > An LLM never mutates the DataFrame. The first version is fully deterministic and offline
31- > (no network, no API keys). The interfaces leave clean extension points for future
32- > retrieval-backed memory and optional private LLM candidate generation.
31+ > An LLM never mutates the DataFrame. Everything is fully deterministic and offline (no
32+ > network, no API keys). Retrieval-backed [ semantic memory replay] ( #semantic-memory-replay )
33+ > is implemented as a local, server-free extension; the interfaces still leave a clean
34+ > extension point for an optional future private-LLM candidate generator.
3335
3436## Enabling it
3537
@@ -64,6 +66,72 @@ cleaned, report = fd.clean(
6466)
6567```
6668
69+ ### Date phrase repair
70+
71+ ` DatePhraseExpert ` normalizes obvious date phrases and strings in date-like columns
72+ (matched by role, ` semantic_type ` , a date-like column name, or a strong share of
73+ parseable values): ISO/slash-ordered dates (` 2026-07-01 ` , ` 2026/07/01 ` ), month-name dates
74+ (` July 1 2026 ` , ` 1 July 2026 ` ), and numeric dates where day/month order is unambiguous.
75+
76+ ` "today" ` /` "yesterday" ` /` "tomorrow" ` are ** only** resolved when you supply an explicit
77+ ` reference_date ` — the real wall-clock date is never used silently:
78+
79+ ``` python
80+ cleaned, report = fd.clean(
81+ df,
82+ semantic_mode = " auto" ,
83+ semantic_context = {
84+ " reference_date" : " 2026-07-01" ,
85+ " columns" : {" signup_date" : {" semantic_type" : " date" }},
86+ },
87+ return_report = True ,
88+ )
89+ ```
90+
91+ Numeric dates where ** both** day and month are ` <= 12 ` (` "01/02/2026" ` ) are ambiguous and
92+ are never auto-applied — they are recorded as high-risk, ` human_review=True ` suggestions
93+ unless you resolve the convention explicitly with ` dayfirst ` :
94+
95+ ``` python
96+ semantic_context= {" columns" : {" event_date" : {" semantic_type" : " date" , " dayfirst" : True }}}
97+ ```
98+
99+ Applied date repairs convert the column to pandas ` datetime64 ` dtype only when * every*
100+ value in the column resolved successfully — a column with any leftover unconverted
101+ string stays as-is, so the conversion never silently drops information.
102+
103+ ### Semantic memory replay
104+
105+ Accepted semantic repairs can be learned into a ` CleaningMemory `
106+ (` fd.learn_cleaning_memory ` ) and replayed as candidate proposals on similar future data:
107+
108+ ``` python
109+ cleaned, report = fd.clean(df, semantic_mode = " auto" , return_report = True )
110+
111+ memory = fd.learn_cleaning_memory(df, decisions = report, dataset_id = " crm_signups" )
112+
113+ next_cleaned, next_report = fd.clean(
114+ next_df,
115+ semantic_mode = " auto" ,
116+ memory = memory,
117+ return_report = True ,
118+ )
119+ ```
120+
121+ - Memory is local and server-free — it stores learned repairs in
122+ ` memory.value_patterns["semantic_repairs"] ` , no different from the rest of
123+ ` CleaningMemory ` 's JSON/SQLite storage.
124+ - Memory is ** evidence, not authority** : every retrieved repair still passes through the
125+ same policy gate as a deterministic proposal, so it can never override
126+ ` target_column ` /` id_columns ` /` preserve_columns ` protection, the confidence floor, or the
127+ ambiguous-date rules above.
128+ - Replayed repairs are fully audited: ` memory_influenced=True ` and
129+ ` model_id="semantic:<issue_type>:memory" ` mark which decisions came from memory, and
130+ ` Action.metadata ` carries the raw/proposed value and evidence either way.
131+ - Retrieval matches on the exact normalized value first, falling back to a lightweight,
132+ no-dependency similarity check (` difflib ` ) for minor value drift; low-similarity or
133+ conflicting repairs are never auto-applied.
134+
67135## Configuration
68136
69137| ` CleanConfig ` field | Default | Meaning |
@@ -74,7 +142,7 @@ cleaned, report = fd.clean(
74142| ` semantic_max_distinct_values ` | ` 500 ` | skip very high-cardinality columns |
75143| ` semantic_sample_size ` | ` 10_000 ` | rows sampled when profiling distinct values |
76144| ` semantic_backends ` | ` ("deterministic",) ` | candidate backends (extension point) |
77- | ` semantic_context ` | ` None ` | per-column hints (semantic_type/unit/allowed_values/mutable) |
145+ | ` semantic_context ` | ` None ` | per-column hints (semantic_type/unit/allowed_values/mutable/dayfirst) plus a top-level ` reference_date ` for relative date phrases |
78146| ` semantic_privacy_policy ` | ` "local_only" ` | privacy posture for future external inference |
79147| ` semantic_budget ` | ` None ` | budget hints (extension point) |
80148
@@ -96,6 +164,7 @@ Each decision appears in the report with `step="semantic"`:
96164for a in report:
97165 if a.step == " semantic" :
98166 print (a.status, a.column, a.confidence, a.risk, a.model_id, a.description)
167+ print (a.memory_influenced, a.metadata) # metadata carries raw/proposed value + evidence
99168```
100169
101170Preview proposal counts before running, via ` suggest_plan ` :
0 commit comments