Skip to content

fix: median imputation crash on nullable integer columns - #187

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/nullable-int-median
Sep 14, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/nullable-int-median

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

Median imputation crashes on nullable integer columns that contain missing values when numpy 2.5+ is installed:

OverflowError: Python integer -9223372036854775808 out of bounds for int8

pandas 2.3.x computes a masked-integer median via nanops.nanmedian, which fills masked slots with iNaT; numpy 2.5 no longer lets that value be cast into int8/int16/int32/unsigned arrays. Int64 and float columns are unaffected, and numpy ≤2.4 is unaffected.

This is what has failed the weekly Large performance investigation workflow every week: all 18 failed cases are the explicit config (impute="median"), crashing on the benchmark's Int8 target column.

Changes

  • Add freshdata._util.safe_median: for pd.arrays.IntegerArray it takes the median of the Float64 view (the same value, since pandas already computes it in float64); every other dtype still calls s.median(), so Int64, float, Float32 and Arrow results are unchanged.
  • Use it at every median site that sees user columns:
    • steps/missing.py (impute=)
    • engine/missing.py (low / medium / high bands)
    • simple.py (fill_missing)
    • imputation/missforest.py (initial fill + fallback)
    • streaming/_timeseries.py (seasonal global median)
  • benchmarks/performance/baselines.py: the numeric_median_fill baseline now works in float64. It crashed on the same columns, and it also failed on a fractional median going into Int64.
  • CHANGELOG entry under Unreleased → Fixed.

Tests

New tests/test_nullable_int_median.py, 35 cases across Int8/Int16/Int32/UInt8/UInt64, with Int64/Float32/float64 as no-change checks. Covers:

Results:

  • On main with numpy 2.5.3 the crash tests fail with the OverflowError above; on this branch all pass.
  • Python 3.11 / numpy 2.4.6: all pass.
  • Related suites (test_missing, test_engine_missing, test_simple, test_streaming_timeseries, test_missforest_imputation, test_model_select): pass.
  • ruff check . and mypy src/freshdata: clean.
  • make truthbench-pr: overall: PASS.
  • python -m benchmarks.performance run --rows 100000 --widths narrow --configs explicit,default --repetitions 1: both cases completed (the explicit case crashed before this change).

pandas computes a masked-integer median by filling masked slots with iNaT,
which numpy 2.5+ refuses to cast into Int8/Int16/Int32/UInt arrays
(OverflowError: Python integer ... out of bounds for int8). Every median
imputation path crashed on such columns with missing values, including the
explicit config of the weekly large performance workflow.

Add _util.safe_median, which takes the median of the Float64 view for
IntegerArray columns and defers to Series.median() for every other dtype,
and use it in steps/missing, engine/missing, simple.fill_missing,
MissForest seeding/fallback and seasonal time-series imputation. The
numeric_median_fill benchmark baseline now works in float64.
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 45e5aa82-e0a9-467e-a88c-b37070dbc1fb


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kevincostner17
kevincostner17 merged commit d3b0169 into main Sep 14, 2026
16 of 17 checks passed
@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

kevincostner17 added a commit that referenced this pull request Sep 14, 2026
test-matrix (3.9) has failed on main since #187: with pandas 1.5, the
streaming state converted numeric columns via
pd.to_numeric(s).to_numpy(dtype="float64"), which raises "cannot convert to
'float64'-dtype NumPy array with missing values" for nullable (masked)
integer columns with NA. fd.clean_timeseries and StreamingCleaner hit it on
any Int*/UInt* column with gaps.

Pass na_value=np.nan there and in the time-series interpolation helper,
which has the same conversion. Also pin the alert step #188 added to
fetch-fixtures.yml, which #190's SHA sweep predated.
@kevincostner17
kevincostner17 deleted the fix/nullable-int-median branch September 14, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant