Skip to content

fix: inclusive mostly thresholds and a deterministic repair-plan hash - #368

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/mostly-boundary-plan-hash
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/mostly-boundary-plan-hash

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

  • Inclusive mostly thresholds (mostly= tolerance fails at the exact boundary (mostly=0.9 with 1 of 10 rows violating) #307). ColumnContract/ColumnRule value checks and CrossColumnRule compared the violation ratio with 1.0 - mostly. Since 1.0 - 0.9 == 0.09999999999999998, exactly 1 of 10 violating rows under mostly=0.9 failed instead of warning. Both sites now use a shared _mostly_tolerated helper that compares the pass count inclusively against mostly * n_total with a tiny relative slack (1e-12, far below one row for any realistic frame). mostly=1.0 still never tolerates a violation.
  • Deterministic RepairPlan.decisions_hash (RepairPlan.decisions_hash varies with PYTHONHASHSEED (set params) and numpy version #312). _json_safe emitted set members in iteration order, which varies with PYTHONHASHSEED, and serialized numpy scalars via repr(), which differs between numpy 1.x and 2.x. Sets and frozensets are now sorted by canonical JSON, numpy scalars convert via .item(), datetime64/timedelta64 map to their pandas forms (NaT becomes null), and ndarrays become lists. Hashes of plans without sets or numpy values are unchanged.

Behaviour changes

  • A rule with exactly the allowed share of violating rows now warns and passes instead of failing.
  • decisions_hash / to_json change for plans whose params contain sets (members now sorted) or numpy scalars (now JSON numbers/bools); plans with ndarray params no longer raise.

Tests

New tests/test_plan_hash_and_mostly.py:

  • mostly boundary grid (0.9 with 1/10, 0.8 with 2/10, 0.95 with 1/20, 0.7 with 3/10 pass; 0.9 with 2/10 and 0.95 with 2/20 fail) run against the helper, enforce_contract, suite ColumnRule, and CrossColumnRule, plus the issue reproduction and a billion-row exactness check.
  • Subprocess runs of a set/frozenset-carrying plan under four PYTHONHASHSEED values, asserting one hash and sorted serialized members.
  • numpy vs Python scalar parity for int64, int32, float32, float64, bool_, str_ (same hash, same JSON, native JSON types), plus _json_safe unit checks.

Verification

Closes #307
Closes #312

mostly boundaries (#307): ColumnContract/ColumnRule value checks and
CrossColumnRule graded a violation ratio against `1.0 - mostly`. Because
`1.0 - 0.9` is 0.09999999999999998, exactly 1 of 10 violating rows under
mostly=0.9 failed instead of warning. Both sites now share
`_mostly_tolerated`, which compares the pass count inclusively against
`mostly * n_total` with a 1e-12 relative slack (far below one row for any
realistic frame). mostly=1.0 still never tolerates a violation, and empty
or violation-free inputs keep their existing early returns.

repair-plan hash (#312): `_json_safe` turned sets into lists in iteration
order, which depends on PYTHONHASHSEED for strings, and fell back to
`repr()` for numpy scalars, which differs between numpy 1.x ("42") and
2.x ("np.int64(42)"). Set and frozenset members are now sorted by their
canonical JSON, numpy scalars become Python scalars via `.item()`,
datetime64/timedelta64 map to the same forms as their pandas
counterparts, and ndarrays become lists. Hashes of plans without sets or
numpy values are unchanged.

Closes #307
Closes #312
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 4b08366e-a5dd-480a-b4f0-7890249a3979


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit 648b7f6 into main Sep 15, 2026
21 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant