Skip to content

fix(learning): JSON-safe learned patterns, non-mutating merge, honest drift counts in apply_plan - #364

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/learning-and-plan-audit
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/learning-and-plan-audit

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

  • fd.learn profiles with integer clean values cannot be saved (int64 not JSON serializable) #256 — learned profiles with numpy clean values could not be saved. fd.learn stored clean values in CleaningMemory.value_patterns as numpy/pandas scalars (int64, bool_, Timestamp, Decimal). save_profile raised TypeError and CleaningMemory.to_json silently stringified them. A shared _replay_patterns helper now encodes each value like ValueMap entries do and stores plain bool/int/float/str. The holdout precision gate in fit_eval.py, which rebuilt the same patterns, uses it too.
  • LearningProfile.merge mutates the calling profile's embedded CleaningMemory #257 — LearningProfile.merge mutated the calling profile's memory. _merge_memory returned a parent's CleaningMemory object and, for the default strategy, wrote the other profile's patterns into it. Every branch now returns a deep copy, so neither parent is mutated or shared.
  • apply_plan(allow_drift=True) records actions as applied when their raw values are gone #258 — apply_plan(allow_drift=True) recorded actions as applied when their raw values were gone.
    • _collect_mappings now counts matching cells with the same dict lookup _apply_column uses to replace them. Actions with no match are skipped and recorded as value … not present in column … (frame drift).
    • Applied actions record the observed cell count instead of the plan-time n_affected.
    • The follow-up "did not apply" record says frame drift instead of the gate reason.
    • The CleanReport(...) constructor in execute_plan is untouched.

Behaviour changes

  • Learned clean values in memory JSON are plain JSON types: integers stay numbers (1, not "1"); Timestamps and Decimals are strings, as in ValueMap entries.
  • p1.merge(p2) no longer changes either parent.
  • With allow_drift=True, actions whose raw value is gone are recorded as skipped with count 0; applied actions record the observed count (normally equal to n_affected without drift).

Tests

New module tests/test_learning_and_plan_audit.py (26 tests):

All new behaviour tests fail on main, except the no-drift guard, which is meant to pass there.

Verification

  • ruff check . — all checks passed
  • mypy src/freshdata — no issues in 202 source files
  • pytest py3.12 (-m "not online and not large") — 4265 passed, 6 skipped, 12 deselected
  • pytest py3.9 / pandas 1.5 — 4261 passed, 10 skipped, 12 deselected

Closes #256
Closes #257
Closes #258

…onest plan audit

#256: fd.learn stored learned clean values in CleaningMemory.value_patterns
as the numpy/pandas scalars pulled from the aligned frame (int64, bool_,
Timestamp, Decimal). save_profile's canonical json.dumps rejected them, and
CleaningMemory.to_json (default=str) silently turned 1 into "1". Replay
patterns are now built by a shared _replay_patterns helper that encodes each
clean value like ValueMap entries (encode_value) and unwraps it to plain
bool/int/float/str. The holdout precision gate in fit_eval rebuilt the same
patterns the same buggy way and now uses the helper too.

#257: _merge_memory returned the calling profile's CleaningMemory and, for
the default strategy, wrote the other profile's patterns into it, so
p1.merge(p2) changed p1's replay behaviour and saved bytes. Every branch now
returns a deep copy, so the merge never mutates or aliases either parent.

#258: with allow_drift=True, _collect_mappings only checked that an action's
column existed, and _record_action logged the plan-time n_affected, so
actions whose raw values were gone were reported as applied. The actual
match count is now computed with the same dict lookup _apply_column uses
for replacement. Actions with no matching cells are skipped and recorded as
"value ... not present (frame drift)". Applied actions record the observed
count. The follow-up "did not apply" record for such actions now says
"frame drift" instead of repeating the gate reason.

Closes #256
Closes #257
Closes #258
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 23c2a4d0-e360-4976-b8be-13ce4c54b078


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit 2d8c401 into main Sep 15, 2026
21 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant