Repository navigation
fix: median imputation crash on nullable integer columns - #187
Merged
Merged
Conversation
pandas computes a masked-integer median by filling masked slots with iNaT, which numpy 2.5+ refuses to cast into Int8/Int16/Int32/UInt arrays (OverflowError: Python integer ... out of bounds for int8). Every median imputation path crashed on such columns with missing values, including the explicit config of the weekly large performance workflow. Add _util.safe_median, which takes the median of the Float64 view for IntegerArray columns and defers to Series.median() for every other dtype, and use it in steps/missing, engine/missing, simple.fill_missing, MissForest seeding/fallback and seasonal time-series imputation. The numeric_median_fill benchmark baseline now works in float64.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
kevincostner17
added a commit
that referenced
this pull request
Sep 14, 2026
test-matrix (3.9) has failed on main since #187: with pandas 1.5, the streaming state converted numeric columns via pd.to_numeric(s).to_numpy(dtype="float64"), which raises "cannot convert to 'float64'-dtype NumPy array with missing values" for nullable (masked) integer columns with NA. fd.clean_timeseries and StreamingCleaner hit it on any Int*/UInt* column with gaps. Pass na_value=np.nan there and in the time-series interpolation helper, which has the same conversion. Also pin the alert step #188 added to fetch-fixtures.yml, which #190's SHA sweep predated.
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Median imputation crashes on nullable integer columns that contain missing values when numpy 2.5+ is installed:
pandas 2.3.x computes a masked-integer median via
nanops.nanmedian, which fills masked slots withiNaT; numpy 2.5 no longer lets that value be cast intoint8/int16/int32/unsigned arrays.Int64and float columns are unaffected, and numpy ≤2.4 is unaffected.This is what has failed the weekly Large performance investigation workflow every week: all 18 failed cases are the
explicitconfig (impute="median"), crashing on the benchmark'sInt8targetcolumn.Changes
freshdata._util.safe_median: forpd.arrays.IntegerArrayit takes the median of theFloat64view (the same value, since pandas already computes it in float64); every other dtype still callss.median(), so Int64, float, Float32 and Arrow results are unchanged.steps/missing.py(impute=)engine/missing.py(low / medium / high bands)simple.py(fill_missing)imputation/missforest.py(initial fill + fallback)streaming/_timeseries.py(seasonal global median)benchmarks/performance/baselines.py: thenumeric_median_fillbaseline now works in float64. It crashed on the same columns, and it also failed on a fractional median going intoInt64.Tests
New
tests/test_nullable_int_median.py, 35 cases acrossInt8/Int16/Int32/UInt8/UInt64, withInt64/Float32/float64as no-change checks. Covers:impute="median"/"auto",fill_missing, the default engine and seasonal imputationsafe_mediantest_has_outlierssingle-definition check (_has_outliers is duplicated in engine/missing.py and engine/model_select.py #33)Results:
mainwith numpy 2.5.3 the crash tests fail with theOverflowErrorabove; on this branch all pass.test_missing,test_engine_missing,test_simple,test_streaming_timeseries,test_missforest_imputation,test_model_select): pass.ruff check .andmypy src/freshdata: clean.make truthbench-pr:overall: PASS.python -m benchmarks.performance run --rows 100000 --widths narrow --configs explicit,default --repetitions 1: both casescompleted(theexplicitcase crashed before this change).