Skip to content

fix: [branch-1.2] hash NullType and ANSI interval values natively (#7084) - #7647

Merged
andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-7084-branch-1.2
Nov 17, 2026
Merged

andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-7084-branch-1.2

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #7084 to branch-1.2. Closes #7063 on this branch.

Rationale for this change

The native Murmur3 and xxHash64 kernels have no arm for Null, Interval(YearMonth) or Duration(Microsecond), so hash(a, null), xxhash64(a, null) and native round-robin shuffle over those types fail with Unsupported data type in hasher. Hashing a null struct that has a NullType field also panics (NullArray data should not contain a null buffer). branch-1.2 has the same kernels. See #7084 and #7063 for the full analysis.

What changes are included in this PR?

Cherry-pick of f1778b9 (git cherry-pick -x) with no conflicts and no adaptations. It adds the missing arms to create_hashes_internal!, makes child_with_parent_nulls return a Null child unchanged, and includes the source PR's unit tests and SQL file tests. The diff against the source commit is identical apart from line numbers and blob hashes.

Other release branches: branch-1.2 is the newest release branch, so there is no newer one to port to.

How are these changes tested?

On this branch, with JDK 17 and the default Spark profile only: the 88 hash_funcs tests in datafusion-comet-spark-expr and the struct_nulls tests in datafusion-comet-common pass, cargo check --all-targets on both crates and cargo fmt --check are clean, and ./mvnw test ran CometSqlFileTestSuite for hash_null_type.sql, round_robin_null_and_intervals.sql, get_array_struct_fields.sql and get_struct_field.sql against a native library built from this branch (4 tests, 0 failures). With the native sources (native/common, native/spark-expr) restored to branch-1.2's version, all four SQL file tests fail (Unsupported data type in hasher: Null and the NullArray null-buffer panic), so the branch has the bug and the tests catch it.

ump45nose and others added 2 commits November 16, 2026 10:32
* fix: hash NullType and ANSI interval values natively

* test: cover NullType and ANSI interval hashing with SQL file tests

Adds SQL file tests for the native paths this PR fixes: hash and
xxhash64 with a NULL argument and with a NullType struct field, including
under a null struct; reading a NullType field through a null struct with
GetStructField and GetArrayStructFields; and native round-robin shuffle,
with positional placement off, over a NULL column and both ANSI interval
types.

* test: hash lists of NullType and intervals; fold the NullType arm in struct_nulls

Adds murmur3 and xxhash64 unit tests that hash a list of NullType,
which must leave the hash unchanged, and lists of both ANSI interval
types, which must hash like the same lists of ints and longs.

Folds the NullType check in struct_nulls into the null-parent arm, and
drops the unneeded localTableScan config from the round-robin SQL file
test, noting why positional placement is pinned off.

---------

Co-authored-by: ump45nose <ump45nose@users.noreply.github.com>
(cherry picked from commit f1778b9)

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@andygrove
andygrove merged commit 1a6f043 into apache:branch-1.2 Nov 17, 2026
138 checks passed
@andygrove
andygrove deleted the backport-7084-branch-1.2 branch November 17, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants