You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 96f29a1
Browse filesBrowse the repository at this point in the historyBrowse files
fix(freshcore): fall back for non-scalar dtypes and wide ints, restore integer dtypes, map non-string labels
The FreshCore native arrays only carry float, bool and string values, and the
adapter rebuilt results from plain lists keyed by str(label). That silently
changed dtypes and column labels:
- Datetime (naive and tz-aware), timedelta, categorical, period and interval
columns were sent as str(v) and came back as object columns of strings.
_unsupported_reason() now falls back for them, naming the columns and dtype
kinds, so the run goes through the pandas reference path (and
fallback_policy="error" raises FallbackError).
- Integer columns holding a value beyond +/-2**53 were rounded by the float64
round-trip. They now fall back too.
- int64, nullable Int64 and other integer widths came back as float64.
_frame_from_native() now casts each integer input column back to its input
dtype when every returned value is integral and in range (missing values
only for nullable dtypes). Otherwise the column stays float64 and the change
is recorded in report.backend_differences (step "dtypes").
- Distinct labels whose str() collides, before or after column-name
normalization (1 and "1", or 1 and " 1 "), collapsed into one native column
with no record. They now fall back. Non-colliding non-string labels (for
example 0 and 1 from a headerless CSV) are mapped back to the original
labels instead of coming back stringified.
Docs: new rows in docs/fallback-matrix.md and matching notes in
docs/backends.md and docs/freshcore.md.
Tests: tests/test_execution/test_freshcore_dtype_fidelity.py covers each
fallback (identical pandas output, recorded reason, FallbackError under
fallback_policy="error"), integer dtype restoration, recorded dtype
differences, and non-string label round-trips with a fake native module.
Closes#262
Refs #232 (part 2)
Copy file name to clipboardExpand all lines: docs/fallback-matrix.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,6 +37,9 @@ FreshCore also runs its own config and data checks in
37
37
| semantic cleaning | native-distinct | native-distinct | pandas | pandas | polars/duckdb run it over a natively extracted distinct table; non-default semantic backends force pandas |
38
38
| pandas input with a mixed-type object column (e.g. numbers and strings) | pandas | pandas | — | pandas | native ingestion would reject the column (polars) or cast every value to text (duckdb) |
39
39
| pandas input with duplicate column labels | pandas | pandas | — | pandas | native frames need unique column names; the pandas pipeline deduplicates them (`"x", "x"` → `"x", "x_2"`) |
40
+
| pandas input whose column labels collide once stringified (e.g. `1` and `"1"`) | native | native | — | pandas | FreshCore names columns by `str(label)`, so one column would overwrite the other; distinct non-string labels (e.g. `0`, `1`) stay native and come back unchanged |
41
+
| pandas input with datetime / timedelta / categorical / period / interval columns | native | native | — | pandas | FreshCore v1 carries only float, bool and string columns, so these dtypes would come back as strings |
42
+
| pandas input with an integer column holding a value beyond ±2\*\*53 | native | native | — | pandas | FreshCore v1 carries numbers as float64, which cannot represent such integers exactly; other integer columns are cast back to their input dtype (or reported in `backend_differences` when the result is no longer integral) |
0 commit comments