You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hash/xxhash64: reject map and variant arguments at planning time like Spark #69
Follow-up to #68 (adds hash). Not started; to be done after #68 merges.
Problem
Spark's HashExpression.checkInputDataTypes (sql/catalyst/.../expressions/hash.scala, 4.2.0) rejects some inputs for hash and xxhash64:
Any argument whose type contains a map, including maps nested in arrays, structs or other maps, fails analysis with HASH_MAP_TYPE. The exception is the internal flag spark.sql.legacy.allowHashOnMapType=true.
Any argument whose type is a variant fails with HASH_VARIANT_TYPE.
Zero arguments fails with a wrong-number-of-arguments error.
Here both functions use Signature::variadic_any and the shared macros in hash/utils.rs hash maps, so hash(map(...)) and xxhash64(map(...)) succeed instead of failing. Unsupported types only fail at execution (utils.rs, "Unsupported data type in hasher").
Proposed work
Add a recursive "contains a map" type check in hash/utils.rs, used from return_field_from_args of both functions, raising a Spark-style HASH_MAP_TYPE error.
Decide how to handle variant extension types.
Decide whether to support spark.sql.legacy.allowHashOnMapType. This needs a config mechanism, which the crate doesn't appear to have. Comet keeps map hashing because it supports the legacy flag.
Update after review of #68: the zero-argument bullet is out of date. #68 no longer adds its own zero-argument check. DataFusion's variadic_any signature now rejects hash() and xxhash64() at planning time, and #68 adds tests for both. What remains for this issue is only the error text: DataFusion's message instead of Spark's WRONG_NUM_ARGS.WITHOUT_SUGGESTION.
Also in scope here: types that Spark can produce but the hasher doesn't handle (for example Time64, Duration and Interval) fail at execution with an internal "Unsupported data type in hasher" error. They should be rejected at planning time, or hashed the way Spark does.
Activity
andygrove commentedon Nov 15, 2026
Update after review of #68: the zero-argument bullet is out of date. #68 no longer adds its own zero-argument check. DataFusion's
variadic_anysignature now rejectshash()andxxhash64()at planning time, and #68 adds tests for both. What remains for this issue is only the error text: DataFusion's message instead of Spark'sWRONG_NUM_ARGS.WITHOUT_SUGGESTION.Also in scope here: types that Spark can produce but the hasher doesn't handle (for example
Time64,DurationandInterval) fail at execution with an internal "Unsupported data type in hasher" error. They should be rejected at planning time, or hashed the way Spark does.feat(spark): add Spark-compatible hash (Murmur3) function (#68)