Skip to content

fix(contracts): tie-aware drift maths, constant baselines, Arrow dtypes, full-precision stats - #362

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/baseline-drift-math
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/baseline-drift-math

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

Fixes four problems in baseline drift maths and dtype handling in enterprise/contracts.py.

  • compare_to_baseline fails a frame against its own baseline when numeric values tie #234 – a frame failed against its own baseline when values tie. The baseline CDF comes from stored quantiles, and when several shared a value KS used the lowest probability ([0]*99+[1] scored KS 0.99). A stored quantile p at value x only bounds the CDF (F(x-) <= p <= F(x)). KS now measures the distance to the nearest baseline CDF inside those bounds. The jump allowed at x is capped by the largest share of rows one value can hold, taken from the baseline's distinct-value count, so a spike on a quantile of a continuous baseline (e.g. median imputation) is still drift. PSI splits current values equal to a bin edge the same way. KS and PSI are unchanged for continuous data (max difference vs main below 1e-13 over 500 random pairs). 3000 random discrete, zero-inflated and constant self-compares now give no drift findings, both in memory and after save/load.
  • Numeric drift checks are skipped for a constant baseline column: 5.0 -> 1000.0 passes #235 – a constant baseline column skipped drift checks. All quantiles collapsed to one value and KS returned None, so 5.0 → 1000.0 passed. KS is now max(P(X<c), P(X>c)): that case fails with KS 1.0, and 20% of rows moving to 4.0 warns. PSI stays undefined for a single bin.
  • Contracts and baselines don't recognise pd.ArrowDtype: double[pyarrow] fails dtype='float' #241 – pd.ArrowDtype wasn't recognised. _normalize_dtype strips [pyarrow] and maps double/halffloat/decimal* to float and large_string/large_utf8/dictionary<…> to string. Numeric profiling casts to float, so Arrow decimals (returned as Decimal objects on pandas 2) no longer crash std().
  • save_baseline rounds stats to 6 decimals, causing false drift for small-valued columns #275 – saved stats were rounded to 6 decimal places. ColumnBaseline.to_dict now stores min/max/mean/std/quantiles at full precision (NaN/inf still become null). Rounding to significant digits isn't enough: the tie-aware checks compare stored quantiles exactly with data values, and even 12 digits turned 0.30000000000000004 into 0.3.

Behaviour notes:

  • Constant baseline columns now produce drift.ks findings.
  • Saved baseline JSON has full-precision floats. Existing files load unchanged.
  • Some point-mass shifts score lower than before, because main's higher scores came from the bug behind the false failures. For example, ratings moving from uniform to 10/80/10 now warn instead of failing. Real shifts still fail (ratings all becoming 3, refunds rising to 20%, 60% median imputation).

Tests

  • tests/test_baseline_compare.py:
    • the compare_to_baseline fails a frame against its own baseline when numeric values tie #234 repro
    • self-comparison with tied values, both in memory and after save/load: zero-inflated counts, ratings, constant, a value spanning several quantiles, zero-inflated continuous, thirds, and inexact tenths
    • real drift still flagged: tied baselines, and a spike on a continuous baseline's quantile
    • constant baseline: fails on a shift, warns on a partial shift, passes against itself
    • small-magnitude save/load keeps exact stats and the same verdict
    • non-finite stats serialise as null
  • tests/test_enterprise_contracts.py: a parametrised _normalize_dtype family table; Arrow double/decimal/large_string columns satisfy contracts, get numeric or categorical baselines, and show numeric drift (importorskip("pyarrow")).

Verification

  • ruff check .: all checks passed
  • mypy src/freshdata: no issues in 202 source files
  • py3.12: pytest -m "not online and not large": 4272 passed, 6 skipped
  • py3.9 / pandas 1.5: 4268 passed, 10 skipped
  • All four issue repros fail on main and pass on this branch on both lanes.

Closes #234
Closes #235
Closes #241
Closes #275

…lies, exact saved stats

KS and PSI (#234): the baseline CDF is rebuilt from stored quantiles. When
several quantiles shared a value, KS kept the lowest probability, so a frame
with tied values (zero-inflated counts, small integer ratings) failed against
its own baseline. A stored quantile p at value x only bounds the CDF,
F(x-) <= p <= F(x), so KS now measures the distance to the nearest baseline
CDF within those bounds. The possible jump at x is capped by the largest point
mass the baseline's distinct-value count allows, so a spike on a quantile of a
continuous baseline (e.g. median imputation) is still drift. PSI splits
current values that equal a bin edge the same way. For continuous data both
statistics are unchanged.

Constant baseline columns (#235): all quantiles collapsed to one value and KS
returned None, so 5.0 -> 1000.0 passed. The same bounds now give a KS of
max(P(X < c), P(X > c)); PSI stays undefined for a single bin.

ArrowDtype (#241): _normalize_dtype strips a trailing "[pyarrow]" and maps
double/halffloat/decimal* to float, large_string/large_utf8 and dictionary
to string. Numeric profiling casts to float, so Arrow decimals (Decimal
objects on pandas 2) can be profiled.

Saved precision (#275): ColumnBaseline.to_dict rounded min/max/mean/std and
quantiles to 6 decimal places, collapsing small-magnitude columns. These stats
are now stored unrounded (NaN/inf still become null), so a loaded baseline
gives the same verdict as the in-memory one.

Closes #234
Closes #235
Closes #241
Closes #275
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 66799ae9-3578-47b4-aa96-ad6a087924de


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit 4ff13d8 into main Sep 15, 2026
21 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant