Repository navigation
perf: hash fixed-width columns with nulls 64 rows at a time - #7691
mbutrovich wants to merge 5 commits into
Conversation
|
The 99% nulls case is 33x faster because the loop this PR replaces ( The apache/datafusion#27559 lost at 99% nulls with the same chunk loop (Neoverse-V2, Xeon) because its baseline was different. DataFusion's loop iterates So most of the gain at high null rates here comes from no longer checking every row. A |
comphead
left a comment
There was a problem hiding this comment.
The chunk loop is correct as far as I can trace it. Full chunks use all 64 bits, the remainder bits cover only len % 64 rows, at most 64 - MIN_VALID_FOR_DENSE rows need restoring, and the length check from #7687 keeps the three slices aligned. The first inline point is measured and changes the dense-chunk cost, so it would be good to settle it before the threshold sweep on x86-64-v3 and Neoverse-N1.
On the xxhash64 threshold: at seed 42, SQL xxhash64 over integer columns goes to datafusion-spark's SparkXxhash64. This loop sees xxhash64 for float and struct arguments and for approx_count_distinct (hash_values_into in hll_plus_plus.rs), so those are the workloads an xxhash64-specific value would be tuned for.
| continue; | ||
| } | ||
| // The check above leaves at most `64 - MIN_VALID_FOR_DENSE` null rows. | ||
| let mut kept = [H::default(); 64 - MIN_VALID_FOR_DENSE as usize]; |
There was a problem hiding this comment.
The dense branch walks the null bits twice (save, then restore) and sizes kept from MIN_VALID_FOR_DENSE. Copying the whole chunk first needs one walk and no H: Default bound:
let previous = *hashes;
for (h, &v) in hashes.iter_mut().zip(values) {
*h = hash(v, *h);
}
for i in set_bits(!valid) {
hashes[i] = previous[i];
}The copy is 256 or 512 bytes and vectorizes. I timed the chunk loop on its own (int32 murmur3, 8192 rows, random nulls, M-series Mac, -C target-cpu=native). This is a local micro-benchmark, not the hash_nulls bench. This PR against the copy: 1.85 vs 1.87 us at 10% nulls, 2.18 vs 1.91 us at 25%, 4.29 vs 3.08 us at 50%, 5.10 vs 3.60 us at 60%, and 3.49 vs 2.58 us at 75%. A cheaper dense chunk moves the break-even toward fewer valid rows, so the threshold sweep would need to run after this change.
There was a problem hiding this comment.
Thanks, switched to the copy. For murmur3's 256-byte chunk the copy stays inline: 8 vmovups pairs on x86-64-v3 and 8 ldp/stp q pairs on Neoverse-N1. The restore then runs 7 or 8 instructions per null row, where the save and restore loops took about 18. For xxhash64's 512-byte chunk the copy becomes a memcpy call on both targets, so the x86 and Neoverse runs should include an xxhash64 case before the threshold is final.
| /// back after. Hashing a null slot is harmless because `hash` can't fail. A sparser chunk, and the | ||
| /// last `len % 64` rows, hash only their valid rows. | ||
| #[inline] | ||
| pub(crate) fn hash_fixed_width_with_nulls<T: Copy, H: Copy + Default>( |
There was a problem hiding this comment.
This safe pub(crate) function relies on values, nulls and hashes having the same length. If hashes is longer, its extra rows are silently left alone, which is the failure #7687 added the length check for. The check in create_hashes_internal! guarantees it today. A debug_assert_eq! on the three lengths here would keep it true for a future caller.
There was a problem hiding this comment.
A
debug_assert_eq!on the three lengths here would keep it true for a future caller.
Thanks. A debug_assert_eq! doesn't run in release builds, so a future caller with mismatched lengths would still skip rows or panic on hash_tail[i] there. The function now returns an internal error when the lengths differ, like the check in create_hashes_internal!, and both macro callers use ?. hash_fixed_width_with_nulls_length_mismatch_is_error covers it.
| } | ||
|
|
||
| #[test] | ||
| fn create_hashes_with_nulls_matches_per_row() { |
There was a problem hiding this comment.
This compares Comet with itself. The files under sql-tests/expressions/hash/ all have fewer than 64 rows, so no SQL test reaches a full chunk today. A SQL file with a CTAS over range(200) that nulls rows in these patterns and selects hash(i), hash(l), hash(d), xxhash64(d) and xxhash64(i, d) would check both loops against Spark. A float argument keeps xxhash64 on Comet's kernel at the default seed (args_compatible_with_spark_xxhash64), and the -0.0 and NaN rows fit in the same table. hash_fixed_width_with_nulls_matches_per_row still covers the sliced offsets, so this test could be replaced by that file.
There was a problem hiding this comment.
Thanks, added hash_fixed_width_nulls.sql. It writes one Parquet file of 1300 rows, so each block of 256 rows is four aligned chunks with its own validity pattern, and the last 20 rows are a partial chunk. Two blocks have 16 and 15 valid rows per chunk, at and one below MIN_VALID_FOR_DENSE. The double column has -0.0 and NaN rows. I dropped create_hashes_with_nulls_matches_per_row.
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod null_chunk_tests { |
There was a problem hiding this comment.
utils.rs already has a #[cfg(test)] mod tests (line 1356), so these tests can go there instead of a second test module. MIN_VALID_FOR_DENSE can then drop pub(super), since only this file uses it and a child module can read private items.
There was a problem hiding this comment.
Thanks, moved them into mod tests, and MIN_VALID_FOR_DENSE is private now.
| } | ||
|
|
||
| /// Every tenth row is null, a pattern the branch predictor learns. | ||
| fn stride_nulls() -> NullBuffer { |
There was a problem hiding this comment.
is_null(i, 0.1) in benches/common/mod.rs builds this same every-tenth-row pattern, and most benches in this crate pull that module in. common::i32_array(NUM_ROWS, 0.1, ...) can replace stride_nulls here.
There was a problem hiding this comment.
Thanks, switched to common::i32_array.
…match, check against Spark
|
Thanks for the feedback, @comphead! Benchmarking your suggested changes locally. |
Which issue does this PR close?
Rationale for this change
Once a fixed-width column has a null, Comet's murmur3 and xxhash64 test the null bit row by row (
hash_array_primitive!), so the loop has a branch per row and doesn't vectorize. Shuffle hash partitioning computes partition ids with murmur3 (multi_partition.rs), so it takes this loop on every nullable key. In TPC-DS, the foreign keys instore_salesare 4.5% null, so nearly every batch has a null.What changes are included in this PR?
Integer and float columns with nulls are hashed 64 rows at a time, adapting apache/datafusion#27559. A chunk with at least 16 valid rows hashes every row with the vectorized loop from #7687 and then restores the previous hashes of its null rows from a copy of the chunk. A sparser chunk hashes only its valid rows. Small decimals, intervals, and booleans keep the per-row loop, because
i64::try_frommust not run on a null slot.Benchmark results
TODO
apache/datafusion#27559 chose its threshold of 48 valid rows from a benchmark-bot sweep across null densities on Neoverse-V2 and Xeon. This PR starts lower, at 16, because Comet's murmur3 loop vectorizes and DataFusion's foldhash loop doesn't, so hashing a whole chunk costs less here. xxhash64 stays scalar on ARM, as foldhash does, so it may want a higher value. The
hash_nullscases from 25% to 85% nulls put chunks between about 48 and 10 valid rows, around the threshold.The value needs the same sweep on x86-64-v3 and Neoverse-N1 before it's final.
Could a reviewer run the same comparison on x86 with
-Ctarget-cpu=x86-64-v3and on Neoverse-N1 with-Ctarget-cpu=neoverse-n1, against #7687? On both sides:The
no_nullscase doesn't reach the changed code, so it shows the noise floor. To sweep the threshold, setMIN_VALID_FOR_DENSEinnative/spark-expr/src/hash_funcs/utils.rsto 8, 16, 32, and 48 in turn and rerunhash_nullsfor each.How are these changes tested?
hash_fixed_width_with_nulls_matches_per_rowinhash_funcs/utils.rschecks the chunk loop against hashing row by row, for lengths from 1 to 200, offsets 0 and 5, and validity patterns at and one below the threshold, with a null run longer than a chunk, and all null.hash_fixed_width_with_nulls_length_mismatch_is_errorchecks the length check.hash_fixed_width_nulls.sqlchecks murmur3 and xxhash64 over int, bigint, and double columns with nulls against Spark, including -0.0 and NaN, in aligned chunks at and one below the threshold and in a partial last chunk.AI usage disclosure
I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.