Skip to content

perf: hash fixed-width columns with nulls 64 rows at a time - #7691

Open
mbutrovich wants to merge 5 commits into
apache:mainfrom
mbutrovich:perf-hash-nulls-chunks
Open

mbutrovich wants to merge 5 commits into
apache:mainfrom
mbutrovich:perf-hash-nulls-chunks

Conversation

@mbutrovich

@mbutrovich mbutrovich commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Once a fixed-width column has a null, Comet's murmur3 and xxhash64 test the null bit row by row (hash_array_primitive!), so the loop has a branch per row and doesn't vectorize. Shuffle hash partitioning computes partition ids with murmur3 (multi_partition.rs), so it takes this loop on every nullable key. In TPC-DS, the foreign keys in store_sales are 4.5% null, so nearly every batch has a null.

What changes are included in this PR?

Integer and float columns with nulls are hashed 64 rows at a time, adapting apache/datafusion#27559. A chunk with at least 16 valid rows hashes every row with the vectorized loop from #7687 and then restores the previous hashes of its null rows from a copy of the chunk. A sparser chunk hashes only its valid rows. Small decimals, intervals, and booleans keep the per-row loop, because i64::try_from must not run on a null slot.

Benchmark results

TODO

apache/datafusion#27559 chose its threshold of 48 valid rows from a benchmark-bot sweep across null densities on Neoverse-V2 and Xeon. This PR starts lower, at 16, because Comet's murmur3 loop vectorizes and DataFusion's foldhash loop doesn't, so hashing a whole chunk costs less here. xxhash64 stays scalar on ARM, as foldhash does, so it may want a higher value. The hash_nulls cases from 25% to 85% nulls put chunks between about 48 and 10 valid rows, around the threshold.

The value needs the same sweep on x86-64-v3 and Neoverse-N1 before it's final.

Could a reviewer run the same comparison on x86 with -Ctarget-cpu=x86-64-v3 and on Neoverse-N1 with -Ctarget-cpu=neoverse-n1, against #7687? On both sides:

cd native
RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo bench -p datafusion-comet-spark-expr --bench hash_nulls

The no_nulls case doesn't reach the changed code, so it shows the noise floor. To sweep the threshold, set MIN_VALID_FOR_DENSE in native/spark-expr/src/hash_funcs/utils.rs to 8, 16, 32, and 48 in turn and rerun hash_nulls for each.

How are these changes tested?

hash_fixed_width_with_nulls_matches_per_row in hash_funcs/utils.rs checks the chunk loop against hashing row by row, for lengths from 1 to 200, offsets 0 and 5, and validity patterns at and one below the threshold, with a null run longer than a chunk, and all null. hash_fixed_width_with_nulls_length_mismatch_is_error checks the length check. hash_fixed_width_nulls.sql checks murmur3 and xxhash64 over int, bigint, and double columns with nulls against Spark, including -0.0 and NaN, in aligned chunks at and one below the threshold and in a partial last chunk.

AI usage disclosure

I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.

@mbutrovich
mbutrovich marked this pull request as ready for review November 19, 2026 18:42
@mbutrovich

Copy link
Copy Markdown
Contributor Author

The 99% nulls case is 33x faster because the loop this PR replaces (hash_array_primitive!) pays for every row, null or not. In a thin-LTO aarch64 build of #7687, a null row runs 18 instructions in create_murmur3_hashes. Four of them reload the null buffer's fields from the array on every row, and two more bounds-check the row index against the null buffer's length. Then it loads a validity byte, shifts, and tests the bit.

The hash_nulls input at 99% has 80 valid rows out of 8192, and 70 of its 128 chunks are all null. Splitting the measured times into a cost per row and a cost per valid row gives about 0.51 ns per row plus 4.6 ns per valid row before this PR. That's 4.16 us of null checks and 0.37 us of hashing at 99%. This PR pays about 0.44 ns per 64-row chunk plus 1.0 ns per valid row, or 0.06 us and 0.08 us. Those are two data points fitted to two parameters, so the disassembly is what supports the split.

apache/datafusion#27559 lost at 99% nulls with the same chunk loop (Neoverse-V2, Xeon) because its baseline was different. DataFusion's loop iterates valid_indices() (hash_utils.rs), which walks the bitmap a word at a time and jumps to each valid row. Its cost already grows with words and valid rows, not with all rows, so the chunk loop's overhead per chunk decides the comparison there.

So most of the gain at high null rates here comes from no longer checking every row. A valid_indices() loop would capture much of it too, because its cost grows the same way, but I haven't measured that. The 64-row chunks add the gains at low null rates, where a dense chunk runs the vectorized loop from #7687.

@github-actions github-actions Bot added enhancement New feature or request performance area:expressions Expression evaluation labels Nov 19, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The chunk loop is correct as far as I can trace it. Full chunks use all 64 bits, the remainder bits cover only len % 64 rows, at most 64 - MIN_VALID_FOR_DENSE rows need restoring, and the length check from #7687 keeps the three slices aligned. The first inline point is measured and changes the dense-chunk cost, so it would be good to settle it before the threshold sweep on x86-64-v3 and Neoverse-N1.

On the xxhash64 threshold: at seed 42, SQL xxhash64 over integer columns goes to datafusion-spark's SparkXxhash64. This loop sees xxhash64 for float and struct arguments and for approx_count_distinct (hash_values_into in hll_plus_plus.rs), so those are the workloads an xxhash64-specific value would be tuned for.

continue;
}
// The check above leaves at most `64 - MIN_VALID_FOR_DENSE` null rows.
let mut kept = [H::default(); 64 - MIN_VALID_FOR_DENSE as usize];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dense branch walks the null bits twice (save, then restore) and sizes kept from MIN_VALID_FOR_DENSE. Copying the whole chunk first needs one walk and no H: Default bound:

let previous = *hashes;
for (h, &v) in hashes.iter_mut().zip(values) {
    *h = hash(v, *h);
}
for i in set_bits(!valid) {
    hashes[i] = previous[i];
}

The copy is 256 or 512 bytes and vectorizes. I timed the chunk loop on its own (int32 murmur3, 8192 rows, random nulls, M-series Mac, -C target-cpu=native). This is a local micro-benchmark, not the hash_nulls bench. This PR against the copy: 1.85 vs 1.87 us at 10% nulls, 2.18 vs 1.91 us at 25%, 4.29 vs 3.08 us at 50%, 5.10 vs 3.60 us at 60%, and 3.49 vs 2.58 us at 75%. A cheaper dense chunk moves the break-even toward fewer valid rows, so the threshold sweep would need to run after this change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, switched to the copy. For murmur3's 256-byte chunk the copy stays inline: 8 vmovups pairs on x86-64-v3 and 8 ldp/stp q pairs on Neoverse-N1. The restore then runs 7 or 8 instructions per null row, where the save and restore loops took about 18. For xxhash64's 512-byte chunk the copy becomes a memcpy call on both targets, so the x86 and Neoverse runs should include an xxhash64 case before the threshold is final.

/// back after. Hashing a null slot is harmless because `hash` can't fail. A sparser chunk, and the
/// last `len % 64` rows, hash only their valid rows.
#[inline]
pub(crate) fn hash_fixed_width_with_nulls<T: Copy, H: Copy + Default>(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This safe pub(crate) function relies on values, nulls and hashes having the same length. If hashes is longer, its extra rows are silently left alone, which is the failure #7687 added the length check for. The check in create_hashes_internal! guarantees it today. A debug_assert_eq! on the three lengths here would keep it true for a future caller.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A debug_assert_eq! on the three lengths here would keep it true for a future caller.

Thanks. A debug_assert_eq! doesn't run in release builds, so a future caller with mismatched lengths would still skip rows or panic on hash_tail[i] there. The function now returns an internal error when the lengths differ, like the check in create_hashes_internal!, and both macro callers use ?. hash_fixed_width_with_nulls_length_mismatch_is_error covers it.

}

#[test]
fn create_hashes_with_nulls_matches_per_row() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This compares Comet with itself. The files under sql-tests/expressions/hash/ all have fewer than 64 rows, so no SQL test reaches a full chunk today. A SQL file with a CTAS over range(200) that nulls rows in these patterns and selects hash(i), hash(l), hash(d), xxhash64(d) and xxhash64(i, d) would check both loops against Spark. A float argument keeps xxhash64 on Comet's kernel at the default seed (args_compatible_with_spark_xxhash64), and the -0.0 and NaN rows fit in the same table. hash_fixed_width_with_nulls_matches_per_row still covers the sliced offsets, so this test could be replaced by that file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added hash_fixed_width_nulls.sql. It writes one Parquet file of 1300 rows, so each block of 256 rows is four aligned chunks with its own validity pattern, and the last 20 rows are a partial chunk. Two blocks have 16 and 15 valid rows per chunk, at and one below MIN_VALID_FOR_DENSE. The double column has -0.0 and NaN rows. I dropped create_hashes_with_nulls_matches_per_row.

}

#[cfg(test)]
mod null_chunk_tests {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utils.rs already has a #[cfg(test)] mod tests (line 1356), so these tests can go there instead of a second test module. MIN_VALID_FOR_DENSE can then drop pub(super), since only this file uses it and a child module can read private items.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, moved them into mod tests, and MIN_VALID_FOR_DENSE is private now.

Comment thread native/spark-expr/benches/hash_nulls.rs Outdated
}

/// Every tenth row is null, a pattern the branch predictor learns.
fn stride_nulls() -> NullBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_null(i, 0.1) in benches/common/mod.rs builds this same every-tenth-row pattern, and most benches in this crate pull that module in. common::i32_array(NUM_ROWS, 0.1, ...) can replace stride_nulls here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, switched to common::i32_array.

@mbutrovich

mbutrovich commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback, @comphead!

Benchmarking your suggested changes locally.

@mbutrovich
mbutrovich requested a review from comphead November 19, 2026 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants