Skip to content

Commit eaeb924

Browse files
Merge pull request #69 from FreshCode-Org/feature/cdc-gate-jwd
feat: CDC event-time quality gate (fd.cdc_profile, CDCReport, CDCDefect)
2 parents 59533c4 + 27e5750 commit eaeb924

5 files changed

Lines changed: 649 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,14 @@ adheres to [Semantic Versioning](https://semver.org/).
77
## [Unreleased]
88

99
### Added
10+
- New **CDC / event-time quality gate** `fd.cdc_profile(df, event_time=..., key=...)`
11+
(module `freshdata.cdc`, also exporting `CDCReport` / `CDCDefect`): classifies
12+
change-data-capture defects that are *not* nulls — stale, late (past-watermark),
13+
out-of-order, duplicate-key, invalid-operation, missing-event-time, and
14+
replay-risk batches — with per-key ordering, an explicit-watermark mode, and
15+
freshness/ordering/CDC trust penalties (each `0..1`). Read-only; never imputes.
16+
`CDCReport` supports `.summary()` / `.to_dict()` / `.to_json()` / `.to_frame()` /
17+
`.passed` / `.trust_penalties` / `.freshness_seconds`.
1018
- New **provenance-aware cleaning** for document/OCR-extracted tables (module
1119
`freshdata.provenance`): `fd.clean(df, source_provenance=..., return_report=True)`
1220
and `clean_enterprise(..., source_provenance=...)` preserve per-column

‎examples/cdc_event_time_gate.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
"""CDC / event-time quality gate with ``fd.cdc_profile``.
2+
3+
Change-data-capture streams fail in ways that are *not* missing values: records
4+
arrive stale, late, out of order, with duplicate keys, bad operation codes, or as
5+
a replayed batch. ``cdc_profile`` classifies those freshness/ordering defects
6+
separately from nulls and reports trust penalties you can gate on. Run:
7+
8+
python examples/cdc_event_time_gate.py
9+
"""
10+
11+
import pandas as pd
12+
13+
import freshdata as fd
14+
15+
16+
def main() -> None:
17+
# An order-change feed captured from a CDC connector, in arrival order.
18+
changes = pd.DataFrame(
19+
{
20+
"order_id": [101, 101, 102, 103, 103, 101, 104],
21+
"event_ts": [
22+
"2024-03-01 09:00:00",
23+
"2024-03-01 09:05:00",
24+
"2024-03-01 09:06:00",
25+
"2024-03-01 09:05:30", # 102/103 interleave is fine (different keys)
26+
None, # missing event time
27+
"2024-03-01 08:30:00", # 101 arrives 35m late, past the watermark
28+
"2024-03-01 09:05:30",
29+
],
30+
"op": ["c", "u", "c", "c", "u", "u", "Q"], # 'Q' is not a valid op code
31+
}
32+
)
33+
34+
report = fd.cdc_profile(
35+
changes,
36+
event_time="event_ts",
37+
key="order_id",
38+
lateness="10m",
39+
operation_col="op",
40+
now=pd.Timestamp("2024-03-01 09:10:00"),
41+
stale_after="30m",
42+
)
43+
44+
print(report.summary())
45+
print("\ntrust penalties (0=clean, 1=worst):", report.trust_penalties)
46+
print(f"freshness: {report.freshness_seconds:.0f}s behind reference")
47+
print(f"\ngate passed: {report.passed}\n")
48+
print("defect table:")
49+
print(report.to_frame()[["kind", "level", "n_rows", "rationale"]].to_string(index=False))
50+
51+
52+
if __name__ == "__main__":
53+
main()

‎src/freshdata/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
profile,
3333
suggest_plan,
3434
)
35+
from .cdc import CDCDefect, CDCReport, cdc_profile
3536
from .cleaner import Cleaner
3637

3738
# Compliance report generators (additive — Phase 1 roadmap). Light import:
@@ -55,6 +56,8 @@
5556

5657
__all__ = [
5758
"Action",
59+
"CDCDefect",
60+
"CDCReport",
5861
"CleanConfig",
5962
"CleanPlan",
6063
"CleanReport",
@@ -72,6 +75,7 @@
7275
"StreamingState",
7376
"TimeSeriesCleanConfig",
7477
"__version__",
78+
"cdc_profile",
7579
"clean",
7680
"clean_csv",
7781
"clean_domain_file",

0 commit comments

Comments
 (0)