Repository navigation
fix(contracts): tie-aware drift maths, constant baselines, Arrow dtypes, full-precision stats - #362
Merged
Merged
Conversation
…lies, exact saved stats KS and PSI (#234): the baseline CDF is rebuilt from stored quantiles. When several quantiles shared a value, KS kept the lowest probability, so a frame with tied values (zero-inflated counts, small integer ratings) failed against its own baseline. A stored quantile p at value x only bounds the CDF, F(x-) <= p <= F(x), so KS now measures the distance to the nearest baseline CDF within those bounds. The possible jump at x is capped by the largest point mass the baseline's distinct-value count allows, so a spike on a quantile of a continuous baseline (e.g. median imputation) is still drift. PSI splits current values that equal a bin edge the same way. For continuous data both statistics are unchanged. Constant baseline columns (#235): all quantiles collapsed to one value and KS returned None, so 5.0 -> 1000.0 passed. The same bounds now give a KS of max(P(X < c), P(X > c)); PSI stays undefined for a single bin. ArrowDtype (#241): _normalize_dtype strips a trailing "[pyarrow]" and maps double/halffloat/decimal* to float, large_string/large_utf8 and dictionary to string. Numeric profiling casts to float, so Arrow decimals (Decimal objects on pandas 2) can be profiled. Saved precision (#275): ColumnBaseline.to_dict rounded min/max/mean/std and quantiles to 6 decimal places, collapsing small-magnitude columns. These stats are now stored unrounded (NaN/inf still become null), so a loaded baseline gives the same verdict as the in-memory one. Closes #234 Closes #235 Closes #241 Closes #275
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes four problems in baseline drift maths and dtype handling in
enterprise/contracts.py.[0]*99+[1]scored KS 0.99). A stored quantilepat valuexonly bounds the CDF (F(x-) <= p <= F(x)). KS now measures the distance to the nearest baseline CDF inside those bounds. The jump allowed atxis capped by the largest share of rows one value can hold, taken from the baseline's distinct-value count, so a spike on a quantile of a continuous baseline (e.g. median imputation) is still drift. PSI splits current values equal to a bin edge the same way. KS and PSI are unchanged for continuous data (max difference vs main below 1e-13 over 500 random pairs). 3000 random discrete, zero-inflated and constant self-compares now give no drift findings, both in memory and after save/load.None, so 5.0 → 1000.0 passed. KS is nowmax(P(X<c), P(X>c)): that case fails with KS 1.0, and 20% of rows moving to 4.0 warns. PSI stays undefined for a single bin.pd.ArrowDtypewasn't recognised._normalize_dtypestrips[pyarrow]and mapsdouble/halffloat/decimal*to float andlarge_string/large_utf8/dictionary<…>to string. Numeric profiling casts to float, so Arrow decimals (returned asDecimalobjects on pandas 2) no longer crashstd().ColumnBaseline.to_dictnow stores min/max/mean/std/quantiles at full precision (NaN/inf still becomenull). Rounding to significant digits isn't enough: the tie-aware checks compare stored quantiles exactly with data values, and even 12 digits turned0.30000000000000004into0.3.Behaviour notes:
drift.ksfindings.Tests
tests/test_baseline_compare.py:nulltests/test_enterprise_contracts.py: a parametrised_normalize_dtypefamily table; Arrow double/decimal/large_string columns satisfy contracts, get numeric or categorical baselines, and show numeric drift (importorskip("pyarrow")).Verification
ruff check .: all checks passedmypy src/freshdata: no issues in 202 source filespytest -m "not online and not large": 4272 passed, 6 skippedCloses #234
Closes #235
Closes #241
Closes #275