Skip to content

hash/xxhash64: reject map and variant arguments at planning time like Spark #69

Open

Description

@andygrove

Follow-up to #68 (adds hash). Not started; to be done after #68 merges.

Problem

Spark's HashExpression.checkInputDataTypes (sql/catalyst/.../expressions/hash.scala, 4.2.0) rejects some inputs for hash and xxhash64:

  • Any argument whose type contains a map, including maps nested in arrays, structs or other maps, fails analysis with HASH_MAP_TYPE. The exception is the internal flag spark.sql.legacy.allowHashOnMapType=true.
  • Any argument whose type is a variant fails with HASH_VARIANT_TYPE.
  • Zero arguments fails with a wrong-number-of-arguments error.

Here both functions use Signature::variadic_any and the shared macros in hash/utils.rs hash maps, so hash(map(...)) and xxhash64(map(...)) succeed instead of failing. Unsupported types only fail at execution (utils.rs, "Unsupported data type in hasher").

Proposed work

  • Add a recursive "contains a map" type check in hash/utils.rs, used from return_field_from_args of both functions, raising a Spark-style HASH_MAP_TYPE error.
  • Decide how to handle variant extension types.
  • Decide whether to support spark.sql.legacy.allowHashOnMapType. This needs a config mechanism, which the crate doesn't appear to have. Comet keeps map hashing because it supports the legacy flag.
  • Use a Spark-style error for the zero-argument case in hash (added in feat(spark): add Spark-compatible hash (Murmur3) function #68 as a plain planning error) and add the same check to xxhash64, which currently accepts zero arguments.
  • Add slt cases: a top-level map, a map nested in an array and in a struct, and zero arguments.

Activity

  1. andygrove commented on Nov 15, 2026

    @andygrove
    MemberAuthor

    Update after review of #68: the zero-argument bullet is out of date. #68 no longer adds its own zero-argument check. DataFusion's variadic_any signature now rejects hash() and xxhash64() at planning time, and #68 adds tests for both. What remains for this issue is only the error text: DataFusion's message instead of Spark's WRONG_NUM_ARGS.WITHOUT_SUGGESTION.

    Also in scope here: types that Spark can produce but the hasher doesn't handle (for example Time64, Duration and Interval) fail at execution with an internal "Unsupported data type in hasher" error. They should be rejected at planning time, or hashed the way Spark does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions