Repository navigation
Conversation
Rewrite arrow_cast(x, 'T') as x::SQLTYPE in the sqllogictest files when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name: Int8/16/32/64, Float32/64, Boolean, Binary, Date32, Utf8View (STRING), Decimal128(p, s) with p <= 38 and List(Int32) (ARRAY<INT>). Negative literals and expressions are parenthesized because :: binds tighter than unary minus. arrow_cast is kept for Utf8 (STRING maps to Utf8View), unsigned integers, timestamps with a precision or time zone, and Arrow types without a SQL name. Spark has no arrow_cast, so the PySpark runner can now run these queries instead of skipping them.
…h date_from_unix_date Spark cannot cast an integer to DATE, so integer day counts now go through date_from_unix_date, and DATE results go back to day counts with unix_date.
andygrove
force-pushed
the
test-arrow-cast-to-double-colon
branch
from
November 18, 2026 22:11
f0c496c to
cc2249f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A. Follow-up to #54 (the PySpark runner skips any query that calls
arrow_cast, because Spark has no such function).Rationale for this change
Many
.sltqueries usearrow_castonly to pick a type that has a plain SQL name. Writing those as::casts makes the tests easier to read, and Spark can run them, so the PySpark runner checks these queries instead of skipping them.What changes are included in this PR?
arrow_cast(x, 'T')becomesx::SQLTYPEin 58 files (482 lines) when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name:Int8/Int16/Int32/Int64TINYINT/SMALLINT/INT/BIGINTFloat32/Float64FLOAT/DOUBLEBoolean,Binary,Date32BOOLEAN,BINARY,DATEUtf8ViewSTRINGDecimal128(p, s), p ≤ 38DECIMAL(p, s)List(Int32)ARRAY<INT>::binds tighter than unary minus, so negative literals and expressions are wrapped in parentheses, for example(-128)::TINYINT. Without them,-128::TINYINTcasts128first and overflows.Spark cannot cast an integer to
DATE, so anarrow_cast(n, 'Date32')of an integer day count becomesdate_from_unix_date(n)rather thann::DATE. A test that reads aDATEresult back as a day count usesunix_date(...), because Spark'sCASTof aDATEtoINTreturns NULL.arrow_castis kept where no SQL type gives the same Arrow type:Utf8, becauseSTRINGandVARCHARmap toUtf8Viewby defaultFloat16,Large*,*View,Date64,Time32,Decimal32/Decimal64/Decimal256,FixedSize*,Dictionary,StructandMapTime64Utf8Viewcasts of a column that is alreadyUtf8View(inbround.sltandpositive.slt), wherecolumn1::STRINGgets the same name as plaincolumn1and DataFusion rejects the duplicate projectionAre these changes tested?
Only test files changed.
cargo test -p datafusion-spark-sqllogictestpasses, apart frommath/expm1.slt, which also fails on main on macOS and is fixed by #359. The PySpark runner now checks the converted queries on Spark 4.2, and none of them adds a new difference from Spark.Are there any user-facing changes?
No.