Repository navigation
Commit 9f6e89d
refactor(spark): derive schema, cast decision, and cast target from one authority
Enumerating "maps to a Spark type" and "readable by ArrowColumnVector"
by hand let them drift: a type could map to a Spark DataType yet have no
vectorized accessor, crashing at task time (FixedSizeList, and latent for
LargeList / FixedSizeBinary / Date64). Replace the three hand-kept switches
with a single authority.
- sparkConsumable(ArrowType): the read gate, mirroring Spark 4.0's
ArrowColumnVector.initAccessor accessor set (verified against the class),
deliberately the narrower gate.
- sparkTarget(Field): the nearest consumable field, recursing into children
(identity when already consumable). One place normalizes unsigned->signed,
Float16->Float32, non-us timestamp->us, Date64->Date32, Time->int,
FixedSizeBinary->Binary, FixedSizeList/LargeList->List.
toSparkType, needsCast, and castTargetString now all derive from
sparkTarget, so the gates cannot disagree and adding a type is one case.
toSparkSchema asserts at plan time that every target is consumable, so an
unsupported type (e.g. Interval) fails in planning naming the column,
instead of an opaque executor UNSUPPORTED_ARROWTYPE.
Closes three latent bugs alongside FixedSizeList: LargeList, FixedSizeBinary,
and Date64 previously mapped to a Spark type without a cast.
Tests: SchemaConverterTest gains cases for LargeList/FixedSizeBinary/Date64,
a "sparkTarget is always consumable" invariant, and a fail-fast case; the
example `types` table gains labels (LargeList<Utf8>), digest
(FixedSizeBinary), and day (Date64) columns, asserted end-to-end in
AdbcSourceTest and the pyspark harness. The E2E session enables the Java 8
date API so DateType decodes without needing sun.util.calendar opened.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>1 parent 390691b commit 9f6e89d
5 files changed
Lines changed: 366 additions & 150 deletions
File tree
- examples/adbc-datafusion-driver
- src
- spark/src
- main/java/org/apache/datafusion/spark
- test/java/org/apache/datafusion/spark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
122 | 132 | | |
123 | 133 | | |
124 | 134 | | |
| |||
163 | 173 | | |
164 | 174 | | |
165 | 175 | | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
166 | 190 | | |
167 | 191 | | |
168 | 192 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
52 | | - | |
53 | | - | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
152 | 158 | | |
153 | 159 | | |
154 | 160 | | |
| |||
166 | 172 | | |
167 | 173 | | |
168 | 174 | | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
169 | 178 | | |
170 | 179 | | |
171 | 180 | | |
| |||
178 | 187 | | |
179 | 188 | | |
180 | 189 | | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
181 | 193 | | |
182 | 194 | | |
183 | 195 | | |
| |||
190 | 202 | | |
191 | 203 | | |
192 | 204 | | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
193 | 208 | | |
194 | 209 | | |
195 | 210 | | |
| |||
217 | 232 | | |
218 | 233 | | |
219 | 234 | | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
220 | 238 | | |
221 | 239 | | |
222 | 240 | | |
| |||
265 | 283 | | |
266 | 284 | | |
267 | 285 | | |
268 | | - | |
| 286 | + | |
269 | 287 | | |
270 | 288 | | |
271 | 289 | | |
| |||
279 | 297 | | |
280 | 298 | | |
281 | 299 | | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
282 | 309 | | |
283 | 310 | | |
284 | 311 | | |
| |||
307 | 334 | | |
308 | 335 | | |
309 | 336 | | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
310 | 340 | | |
311 | 341 | | |
0 commit comments