Skip to content

fix: use dynamic min_nonzero/10 for get_binned_data fill value - #1923

Open
yamashita721 wants to merge 1 commit into
evidentlyai:mainfrom
yamashita721:fix-binned-data-fill
Open

fix: use dynamic min_nonzero/10 for get_binned_data fill value#1923
yamashita721 wants to merge 1 commit into
evidentlyai:mainfrom
yamashita721:fix-binned-data-fill

Conversation

@yamashita721

Copy link
Copy Markdown

This PR updates the hardcoded 0.0001 fill value in get_binned_data to be dynamically calculated as one-tenth of the minimum non-zero percent. This prevents KL/PSI score distortions when actual data percents are smaller than 0.0001.

Addresses #334.

Changes made:

  • Added safe calculation for ref_fill and curr_fill.
  • Handled empty array edge cases to prevent ValueError.
  • Updated both pandas and spark legacy utils.py implementations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant