Repository navigation
Commit bcd8b5d
authored
fix(engine): materialize Spark DataFrames in the pandas fallback (#425)
fd.clean(spark_df) raised TypeError: cannot materialize source of type
DataFrame. The balanced-strategy pandas fallback probed to_pandas (polars) and
df (duckdb) but not a Spark DataFrame's toPandas(), so any Spark input crashed
before cleaning. Add a toPandas() probe. Found by the production-readiness
campaign (FDC-L1a-001).1 parent bbbd93b commit bcd8b5d
3 files changed
Lines changed: 27 additions & 0 deletions
File tree
- src/freshdata/execution/backends
- tests/test_execution
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
10 | 15 | | |
11 | 16 | | |
12 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
54 | 59 | | |
55 | 60 | | |
56 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
0 commit comments