Skip to content

Commit bcd8b5d

Browse files
fix(engine): materialize Spark DataFrames in the pandas fallback (#425)
fd.clean(spark_df) raised TypeError: cannot materialize source of type DataFrame. The balanced-strategy pandas fallback probed to_pandas (polars) and df (duckdb) but not a Spark DataFrame's toPandas(), so any Spark input crashed before cleaning. Add a toPandas() probe. Found by the production-readiness campaign (FDC-L1a-001).
1 parent bbbd93b commit bcd8b5d

3 files changed

Lines changed: 27 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,11 @@ adheres to [Semantic Versioning](https://semver.org/).
77
## [Unreleased]
88

99
### Fixed
10+
- `fd.clean` on a Spark DataFrame no longer raises
11+
`TypeError: cannot materialize source of type DataFrame`. Under the default
12+
`strategy="balanced"` the pandas fallback now materializes a Spark source
13+
through its `toPandas()` method, alongside the existing polars and DuckDB
14+
paths.
1015
- `fd.evaluate_quality_debt` no longer scores a dimension a clean 0.0 when it
1116
was never measured. `type_instability` when profiling fails, `pii_risk` when
1217
the PII scan is unavailable or fails, and `schema_drift` and `category_churn`

‎src/freshdata/execution/backends/_pandas.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,11 @@ def materialize_to_pandas(source: Any) -> pd.DataFrame:
5151
to_df = getattr(source, "df", None)
5252
if callable(to_df):
5353
return to_df()
54+
# Spark DataFrame (collects to the driver, matching the balanced-strategy
55+
# pandas fallback for a Spark source).
56+
to_pandas_spark = getattr(source, "toPandas", None)
57+
if callable(to_pandas_spark):
58+
return to_pandas_spark()
5459
raise TypeError(f"cannot materialize source of type {type(source).__name__}")
5560

5661

‎tests/test_execution/test_dispatch_and_config.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,20 @@ def test_engine_config_spark_options():
6161
spark_shuffle_partitions=4)
6262
assert cfg.spark_shuffle_partitions == 4
6363
assert cfg.output_format == "spark"
64+
65+
66+
def test_materialize_to_pandas_accepts_spark_style_frame():
67+
"""A Spark DataFrame exposes ``toPandas`` (not ``to_pandas``/``df``); the
68+
balanced-strategy pandas fallback must materialize it, not raise TypeError."""
69+
import pandas as pd
70+
71+
from freshdata.execution.backends._pandas import materialize_to_pandas
72+
73+
expected = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
74+
75+
class _FakeSparkFrame:
76+
def toPandas(self) -> pd.DataFrame: # noqa: N802 - matches pyspark's API
77+
return expected.copy()
78+
79+
out = materialize_to_pandas(_FakeSparkFrame())
80+
pd.testing.assert_frame_equal(out, expected)

0 commit comments

Comments
 (0)