Skip to content

Commit 96f29a1

Browse files
fix(freshcore): fall back for non-scalar dtypes and wide ints, restore integer dtypes, map non-string labels
The FreshCore native arrays only carry float, bool and string values, and the adapter rebuilt results from plain lists keyed by str(label). That silently changed dtypes and column labels: - Datetime (naive and tz-aware), timedelta, categorical, period and interval columns were sent as str(v) and came back as object columns of strings. _unsupported_reason() now falls back for them, naming the columns and dtype kinds, so the run goes through the pandas reference path (and fallback_policy="error" raises FallbackError). - Integer columns holding a value beyond +/-2**53 were rounded by the float64 round-trip. They now fall back too. - int64, nullable Int64 and other integer widths came back as float64. _frame_from_native() now casts each integer input column back to its input dtype when every returned value is integral and in range (missing values only for nullable dtypes). Otherwise the column stays float64 and the change is recorded in report.backend_differences (step "dtypes"). - Distinct labels whose str() collides, before or after column-name normalization (1 and "1", or 1 and " 1 "), collapsed into one native column with no record. They now fall back. Non-colliding non-string labels (for example 0 and 1 from a headerless CSV) are mapped back to the original labels instead of coming back stringified. Docs: new rows in docs/fallback-matrix.md and matching notes in docs/backends.md and docs/freshcore.md. Tests: tests/test_execution/test_freshcore_dtype_fidelity.py covers each fallback (identical pandas output, recorded reason, FallbackError under fallback_policy="error"), integer dtype restoration, recorded dtype differences, and non-string label round-trips with a fake native module. Closes #262 Refs #232 (part 2)
1 parent 7d6e461 commit 96f29a1

5 files changed

Lines changed: 479 additions & 15 deletions

File tree

‎docs/backends.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,11 @@ Notes:
163163
since native frames carry no index.
164164
- FreshCore also falls back, based on the data, when `outliers` is set and a
165165
float column holds `±inf`, or when `impute="mode"`/`"auto"` would fill a
166-
nullable `boolean` column. With `drop_duplicates=False` and
166+
nullable `boolean` column. It falls back for datetime, timedelta,
167+
categorical, period and interval columns, for integer columns holding values
168+
beyond ±2\*\*53, and for column labels that collide once stringified (such as
169+
`1` and `"1"`). Other integer columns are cast back to their input dtype, and
170+
non-string labels come back unchanged. With `drop_duplicates=False` and
167171
`duplicate_ratio_action="error"`, it falls back unless the native module
168172
reports a duplicate-row count (see the [fallback matrix](fallback-matrix.md)).
169173
- FreshCore v1 is a cleaning-first native engine, not an out-of-core engine. It

‎docs/fallback-matrix.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,9 @@ FreshCore also runs its own config and data checks in
3737
| semantic cleaning | native-distinct | native-distinct | pandas | pandas | polars/duckdb run it over a natively extracted distinct table; non-default semantic backends force pandas |
3838
| pandas input with a mixed-type object column (e.g. numbers and strings) | pandas | pandas | — | pandas | native ingestion would reject the column (polars) or cast every value to text (duckdb) |
3939
| pandas input with duplicate column labels | pandas | pandas | — | pandas | native frames need unique column names; the pandas pipeline deduplicates them (`"x", "x"` → `"x", "x_2"`) |
40+
| pandas input whose column labels collide once stringified (e.g. `1` and `"1"`) | native | native | — | pandas | FreshCore names columns by `str(label)`, so one column would overwrite the other; distinct non-string labels (e.g. `0`, `1`) stay native and come back unchanged |
41+
| pandas input with datetime / timedelta / categorical / period / interval columns | native | native | — | pandas | FreshCore v1 carries only float, bool and string columns, so these dtypes would come back as strings |
42+
| pandas input with an integer column holding a value beyond ±2\*\*53 | native | native | — | pandas | FreshCore v1 carries numbers as float64, which cannot represent such integers exactly; other integer columns are cast back to their input dtype (or reported in `backend_differences` when the result is no longer integral) |
4043
| contracts / validation / memory / profile replay | pandas | pandas | pandas | pandas | in-memory reference features (see [limitations](limitations.md)) |
4144

4245
“pandas” means the **whole pipeline** runs on the pandas reference (fallbacks

‎docs/freshcore.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,16 @@ columns with missing values. Detection-only dedup under
5959
`duplicate_ratio_action="error"` falls back unless the native module reports
6060
`duplicates_detected`.
6161

62+
The native arrays carry only float, bool and string values, so FreshCore also
63+
falls back for datetime, timedelta, categorical, period and interval columns,
64+
for integer columns holding values beyond ±2\*\*53 (float64 cannot represent
65+
them exactly), and for column labels that collide once stringified (such as
66+
`1` and `"1"`). Integer columns (`int64`, nullable `Int64`, and other widths)
67+
are cast back to their input dtype when every returned value is integral and
68+
in range; otherwise they come back as `float64` and the change is recorded in
69+
`report.backend_differences`. Non-string column labels such as `0` and `1`
70+
come back unchanged.
71+
6272
## Benchmarking
6373

6474
Build the native module before benchmarking:

‎src/freshdata/execution/backends/_freshcore.py‎

Lines changed: 168 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,21 @@
1313
from __future__ import annotations
1414

1515
import importlib
16+
import math
1617
import time
18+
from collections import Counter
1719
from typing import TYPE_CHECKING, Any
1820

1921
import numpy as np
2022
import pandas as pd
21-
from pandas.api.types import is_bool_dtype, is_float_dtype, is_numeric_dtype, is_object_dtype
23+
from pandas.api.types import (
24+
is_bool_dtype,
25+
is_datetime64_any_dtype,
26+
is_float_dtype,
27+
is_integer_dtype,
28+
is_numeric_dtype,
29+
is_object_dtype,
30+
)
2231

2332
from ..._util import memory_bytes
2433
from ...config import _DEFAULT_FACTOR
@@ -31,9 +40,14 @@
3140
from ._pandas import materialize_to_pandas
3241

3342
if TYPE_CHECKING: # pragma: no cover - typing only
43+
from collections.abc import Sequence
44+
3445
from ...config import CleanConfig
3546
from .._config import EngineConfig
3647

48+
#: Largest integer magnitude float64 represents exactly (native numbers are f64).
49+
_MAX_EXACT_FLOAT_INT = 2**53
50+
3751

3852
class FreshCoreEngine(ExecutionEngine):
3953
"""Optional native backend backed by the ``freshdata_freshcore`` extension."""
@@ -89,8 +103,10 @@ def execute(
89103
"which this FreshCore native module does not report",
90104
)
91105

92-
cleaned = self._frame_from_native(native)
106+
cleaned, dtype_changes = self._frame_from_native(native, frame, config)
93107
report = self._report_from_native(frame, cleaned, native, started, config)
108+
for column, detail in dtype_changes:
109+
report.record_backend_difference("freshcore", "dtypes", detail, column=column)
94110
return cleaned, report
95111

96112
@staticmethod
@@ -150,8 +166,29 @@ def _unsupported_reason(self, frame: pd.DataFrame, config: CleanConfig) -> str |
150166
return "protected/id/target column semantics require the pandas reference path"
151167
if frame.columns.duplicated().any():
152168
return "duplicate input column labels require the pandas reference path"
169+
colliding = self._colliding_labels(frame, config)
170+
if colliding:
171+
return (
172+
f"column labels {self._shown(colliding)} collide once stringified: "
173+
"FreshCore v1 names columns by str(label), so distinct labels such as "
174+
"1 and '1' require the pandas reference path"
175+
)
153176
if not isinstance(frame.index, pd.RangeIndex):
154177
return "non-default pandas index semantics require the pandas reference path"
178+
non_scalar, kinds = self._non_scalar_dtype_columns(frame)
179+
if non_scalar:
180+
return (
181+
f"{'/'.join(kinds)} column(s) {self._shown(non_scalar)}: FreshCore v1 "
182+
"carries only float, bool and string columns, so datetime, timedelta, "
183+
"categorical, period and interval dtypes require the pandas reference path"
184+
)
185+
wide = self._wide_integer_columns(frame)
186+
if wide:
187+
return (
188+
f"integer column(s) {self._shown(wide)} hold values beyond ±2**53: "
189+
"FreshCore v1 carries numbers as float64, which cannot represent them "
190+
"exactly, so they require the pandas reference path"
191+
)
155192
if self._has_unsupported_object_values(frame):
156193
return "mixed object columns with non-string values require the pandas reference path"
157194
if config.outliers is not None:
@@ -173,11 +210,75 @@ def _unsupported_reason(self, frame: pd.DataFrame, config: CleanConfig) -> str |
173210
return None
174211

175212
@staticmethod
176-
def _shown(columns: list[str], limit: int = 5) -> str:
213+
def _shown(columns: Sequence[object], limit: int = 5) -> str:
177214
shown = ", ".join(repr(c) for c in columns[:limit])
178215
extra = len(columns) - limit
179216
return f"{shown} (+{extra} more)" if extra > 0 else shown
180217

218+
@staticmethod
219+
def _output_labels(frame: pd.DataFrame, config: CleanConfig) -> list[object]:
220+
"""Column labels after the (optional) column-name normalization step."""
221+
if config.column_names:
222+
return normalized_column_labels(frame.columns)
223+
return list(frame.columns)
224+
225+
def _colliding_labels(self, frame: pd.DataFrame, config: CleanConfig) -> list[object]:
226+
"""Distinct labels whose ``str()`` clashes before or after renaming.
227+
228+
The native module keys columns by string name, so ``1`` and ``"1"``
229+
(or ``1`` and ``" 1 "``, which normalizes to ``"1"``) would collapse.
230+
"""
231+
clashing: set[int] = set()
232+
for labels in (list(frame.columns), self._output_labels(frame, config)):
233+
names = [str(label) for label in labels]
234+
counts = Counter(names)
235+
clashing.update(i for i, name in enumerate(names) if counts[name] > 1)
236+
return [frame.columns[i] for i in sorted(clashing)]
237+
238+
@staticmethod
239+
def _non_scalar_dtype_columns(frame: pd.DataFrame) -> tuple[list[object], list[str]]:
240+
"""Columns whose dtype the native float/bool/string arrays cannot carry."""
241+
found: list[object] = []
242+
kinds: list[str] = []
243+
for i, col in enumerate(frame.columns):
244+
dtype = frame.iloc[:, i].dtype
245+
if isinstance(dtype, pd.CategoricalDtype):
246+
kind = "categorical"
247+
elif isinstance(dtype, pd.PeriodDtype):
248+
kind = "period"
249+
elif isinstance(dtype, pd.IntervalDtype):
250+
kind = "interval"
251+
elif is_datetime64_any_dtype(dtype):
252+
kind = "datetime"
253+
elif isinstance(dtype, np.dtype) and dtype.kind == "m":
254+
kind = "timedelta"
255+
else:
256+
continue
257+
found.append(col)
258+
if kind not in kinds:
259+
kinds.append(kind)
260+
return found, kinds
261+
262+
@staticmethod
263+
def _wide_integer_columns(frame: pd.DataFrame) -> list[object]:
264+
"""Integer columns holding a value float64 cannot represent exactly."""
265+
found: list[object] = []
266+
for i, col in enumerate(frame.columns):
267+
s = frame.iloc[:, i]
268+
if is_bool_dtype(s) or not is_integer_dtype(s):
269+
continue
270+
info = np.iinfo(getattr(s.dtype, "numpy_dtype", s.dtype))
271+
if int(info.min) >= -_MAX_EXACT_FLOAT_INT and int(info.max) <= _MAX_EXACT_FLOAT_INT:
272+
continue # int8..int32 and their unsigned widths always fit
273+
non_null = s.dropna()
274+
if non_null.empty:
275+
continue
276+
if int(non_null.max()) > _MAX_EXACT_FLOAT_INT or (
277+
int(non_null.min()) < -_MAX_EXACT_FLOAT_INT
278+
):
279+
found.append(col)
280+
return found
281+
181282
@staticmethod
182283
def _non_finite_float_columns(frame: pd.DataFrame) -> list[str]:
183284
"""Columns holding ±inf. Only float dtypes can store infinities."""
@@ -215,10 +316,7 @@ def _has_unsupported_object_values(frame: pd.DataFrame) -> bool:
215316
return False
216317

217318
def _payload(self, frame: pd.DataFrame, config: CleanConfig) -> dict[str, Any]:
218-
renamed = (
219-
normalized_column_labels(frame.columns)
220-
if config.column_names else list(frame.columns)
221-
)
319+
renamed = self._output_labels(frame, config)
222320
rename_map = [
223321
(str(old), str(new))
224322
for old, new in zip(frame.columns, renamed)
@@ -270,18 +368,74 @@ def _column_payload(name: str, series: pd.Series) -> dict[str, Any]:
270368
]
271369
return {"name": name, "dtype": "string", "values": string_values}
272370

273-
@staticmethod
274-
def _frame_from_native(native: dict[str, Any]) -> pd.DataFrame:
275-
data: dict[str, Any] = {}
276-
for column in native["columns"]:
371+
def _frame_from_native(
372+
self,
373+
native: dict[str, Any],
374+
original: pd.DataFrame | None = None,
375+
config: CleanConfig | None = None,
376+
) -> tuple[pd.DataFrame, list[tuple[str, str]]]:
377+
"""Rebuild the cleaned frame and list the dtype changes it could not undo.
378+
379+
Native column names are ``str(label)``; with *original* and *config*
380+
they are mapped back to the original (possibly renamed) labels, and
381+
integer columns are cast back to their input dtype when possible.
382+
"""
383+
sources: dict[str, tuple[object, pd.Series]] = {}
384+
if original is not None and config is not None:
385+
for i, label in enumerate(self._output_labels(original, config)):
386+
sources[str(label)] = (label, original.iloc[:, i])
387+
labels: list[object] = []
388+
data: dict[int, Any] = {}
389+
changes: list[tuple[str, str]] = []
390+
for position, column in enumerate(native["columns"]):
277391
name = column["name"]
278392
values = column["values"]
279393
dtype = column.get("dtype")
394+
label, source = sources.get(name, (name, None))
395+
labels.append(label)
280396
if dtype == "bool":
281-
data[name] = pd.Series(values, dtype="boolean")
397+
data[position] = pd.Series(values, dtype="boolean")
398+
elif dtype == "float" and source is not None and is_integer_dtype(source):
399+
restored, detail = self._restore_integer(values, source.dtype)
400+
data[position] = restored
401+
if detail is not None:
402+
changes.append((str(label), detail))
282403
else:
283-
data[name] = values
284-
return pd.DataFrame(data)
404+
data[position] = values
405+
cleaned = pd.DataFrame(data)
406+
cleaned.columns = pd.Index(labels)
407+
return cleaned, changes
408+
409+
@staticmethod
410+
def _restore_integer(values: list[Any], dtype: Any) -> tuple[Any, str | None]:
411+
"""Cast native float values back to the input integer *dtype* when exact."""
412+
nullable = not isinstance(dtype, np.dtype)
413+
info = np.iinfo(getattr(dtype, "numpy_dtype", dtype))
414+
ints: list[int | None] = []
415+
problem: str | None = None
416+
for v in values:
417+
if v is None or (isinstance(v, float) and math.isnan(v)):
418+
if not nullable:
419+
problem = "missing values, which a non-nullable integer dtype cannot hold"
420+
break
421+
ints.append(None)
422+
elif not (math.isfinite(v) and float(v).is_integer()):
423+
problem = "non-integral values"
424+
break
425+
elif not int(info.min) <= int(v) <= int(info.max):
426+
problem = f"values outside the {dtype} range"
427+
break
428+
else:
429+
ints.append(int(v))
430+
if problem is None:
431+
if nullable:
432+
return pd.array(ints, dtype=dtype), None
433+
return np.array(ints, dtype=dtype), None
434+
detail = (
435+
f"FreshCore v1 returned {problem} for this {dtype} input column, "
436+
"so it comes back as float64"
437+
)
438+
return pd.Series(values, dtype="float64"), detail
285439

286440
def _report_from_native(
287441
self,

0 commit comments

Comments
 (0)