Skip to content

test: replace arrow_cast with :: casts where a SQL type is equivalent - #76

Open
andygrove wants to merge 3 commits into
apache:mainfrom
andygrove:test-arrow-cast-to-double-colon
Open

andygrove wants to merge 3 commits into
apache:mainfrom
andygrove:test-arrow-cast-to-double-colon

Conversation

@andygrove

@andygrove andygrove commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A. Follow-up to #54 (the PySpark runner skips any query that calls arrow_cast, because Spark has no such function).

Rationale for this change

Many .slt queries use arrow_cast only to pick a type that has a plain SQL name. Writing those as :: casts makes the tests easier to read, and Spark can run them, so the PySpark runner checks these queries instead of skipping them.

What changes are included in this PR?

arrow_cast(x, 'T') becomes x::SQLTYPE in 58 files (482 lines) when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name:

Arrow type SQL type
Int8 / Int16 / Int32 / Int64 TINYINT / SMALLINT / INT / BIGINT
Float32 / Float64 FLOAT / DOUBLE
Boolean, Binary, Date32 BOOLEAN, BINARY, DATE
Utf8View STRING
Decimal128(p, s), p ≤ 38 DECIMAL(p, s)
List(Int32) ARRAY<INT>

:: binds tighter than unary minus, so negative literals and expressions are wrapped in parentheses, for example (-128)::TINYINT. Without them, -128::TINYINT casts 128 first and overflows.

Spark cannot cast an integer to DATE, so an arrow_cast(n, 'Date32') of an integer day count becomes date_from_unix_date(n) rather than n::DATE. A test that reads a DATE result back as a day count uses unix_date(...), because Spark's CAST of a DATE to INT returns NULL.

arrow_cast is kept where no SQL type gives the same Arrow type:

  • Utf8, because STRING and VARCHAR map to Utf8View by default
  • unsigned integers, Float16, Large*, *View, Date64, Time32, Decimal32/Decimal64/Decimal256, FixedSize*, Dictionary, Struct and Map
  • timestamps with a precision or a time zone, and Time64
  • Utf8View casts of a column that is already Utf8View (in bround.slt and positive.slt), where column1::STRING gets the same name as plain column1 and DataFusion rejects the duplicate projection

Are these changes tested?

Only test files changed. cargo test -p datafusion-spark-sqllogictest passes, apart from math/expm1.slt, which also fails on main on macOS and is fixed by #359. The PySpark runner now checks the converted queries on Spark 4.2, and none of them adds a new difference from Spark.

Are there any user-facing changes?

No.

Rewrite arrow_cast(x, 'T') as x::SQLTYPE in the sqllogictest files when
DataFusion maps the SQL type to exactly the same Arrow type and Spark
accepts the type name: Int8/16/32/64, Float32/64, Boolean, Binary,
Date32, Utf8View (STRING), Decimal128(p, s) with p <= 38 and
List(Int32) (ARRAY<INT>). Negative literals and expressions are
parenthesized because :: binds tighter than unary minus.

arrow_cast is kept for Utf8 (STRING maps to Utf8View), unsigned
integers, timestamps with a precision or time zone, and Arrow types
without a SQL name. Spark has no arrow_cast, so the PySpark runner can
now run these queries instead of skipping them.
…h date_from_unix_date

Spark cannot cast an integer to DATE, so integer day counts now go through
date_from_unix_date, and DATE results go back to day counts with unix_date.
@andygrove
andygrove force-pushed the test-arrow-cast-to-double-colon branch from f0c496c to cc2249f Compare November 18, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant