Repository navigation
fix(learning): JSON-safe learned patterns, non-mutating merge, honest drift counts in apply_plan - #364
Merged
Conversation
…onest plan audit #256: fd.learn stored learned clean values in CleaningMemory.value_patterns as the numpy/pandas scalars pulled from the aligned frame (int64, bool_, Timestamp, Decimal). save_profile's canonical json.dumps rejected them, and CleaningMemory.to_json (default=str) silently turned 1 into "1". Replay patterns are now built by a shared _replay_patterns helper that encodes each clean value like ValueMap entries (encode_value) and unwraps it to plain bool/int/float/str. The holdout precision gate in fit_eval rebuilt the same patterns the same buggy way and now uses the helper too. #257: _merge_memory returned the calling profile's CleaningMemory and, for the default strategy, wrote the other profile's patterns into it, so p1.merge(p2) changed p1's replay behaviour and saved bytes. Every branch now returns a deep copy, so the merge never mutates or aliases either parent. #258: with allow_drift=True, _collect_mappings only checked that an action's column existed, and _record_action logged the plan-time n_affected, so actions whose raw values were gone were reported as applied. The actual match count is now computed with the same dict lookup _apply_column uses for replacement. Actions with no matching cells are skipped and recorded as "value ... not present (frame drift)". Applied actions record the observed count. The follow-up "did not apply" record for such actions now says "frame drift" instead of repeating the gate reason. Closes #256 Closes #257 Closes #258
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fd.learnstored clean values inCleaningMemory.value_patternsas numpy/pandas scalars (int64,bool_,Timestamp,Decimal).save_profileraisedTypeErrorandCleaningMemory.to_jsonsilently stringified them. A shared_replay_patternshelper now encodes each value likeValueMapentries do and stores plainbool/int/float/str. The holdout precision gate infit_eval.py, which rebuilt the same patterns, uses it too.LearningProfile.mergemutated the calling profile's memory._merge_memoryreturned a parent'sCleaningMemoryobject and, for the default strategy, wrote the other profile's patterns into it. Every branch now returns a deep copy, so neither parent is mutated or shared.apply_plan(allow_drift=True)recorded actions as applied when their raw values were gone._collect_mappingsnow counts matching cells with the same dict lookup_apply_columnuses to replace them. Actions with no match are skipped and recorded asvalue … not present in column … (frame drift).n_affected.frame driftinstead of the gate reason.CleanReport(...)constructor inexecute_planis untouched.Behaviour changes
1, not"1"); Timestamps and Decimals are strings, as inValueMapentries.p1.merge(p2)no longer changes either parent.allow_drift=True, actions whose raw value is gone are recorded as skipped with count 0; applied actions record the observed count (normally equal ton_affectedwithout drift).Tests
New module
tests/test_learning_and_plan_audit.py(26 tests):_merge_memorybranch returns a copy.n_affected(1), and the undo log holds only the applied action.n_affected._apply_columnand tolerates unhashable cells.All new behaviour tests fail on main, except the no-drift guard, which is meant to pass there.
Verification
ruff check .— all checks passedmypy src/freshdata— no issues in 202 source files-m "not online and not large") — 4265 passed, 6 skipped, 12 deselectedCloses #256
Closes #257
Closes #258