Skip to content

perf: slice contiguous rows in native shuffle instead of gathering them one at a time - #7636

Open
comphead wants to merge 4 commits into
apache:mainfrom
comphead:hash-shuffle-contiguous-runs
Open

comphead wants to merge 4 commits into
apache:mainfrom
comphead:hash-shuffle-contiguous-runs

Conversation

@comphead

@comphead comphead commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7474.

Rationale for this change

Hash, range, and hash round robin partitioning record one (batch, row) pair per row and build each output chunk with interleave_record_batch. Arrow copies primitive and string columns there one row at a time (interleave_primitive, interleave_bytes) and merges contiguous rows only in interleave_fallback, which Boolean columns use. So a chunk whose rows are already one span of a buffered batch is still gathered row by row.

That is the shape an Iceberg write gets when Spark rebalances on a partition column that holds one value per run, such as the ds of an hourly or daily ETL. Every row hashes to one partition, and each buffered batch is a single run. In the production job described in #7474, partition interleaving was 54% of the native shuffle writer time in the map stage.

DataFusion's RepartitionExec gained the whole-batch case of this in 56.0.0. Its partition_grouped_take passes a batch through unchanged when all of its rows go to one partition (apache/datafusion#24598). This PR also covers runs that span buffered batches and output chunks.

What changes are included in this PR?

  • RowIterator (partitioned_batch_iterator.rs) splits each output chunk into runs of consecutive rows of one buffered batch with chunk_by. When the runs average at least 64 rows, it slices them out of the buffered batches and concatenates the slices, and a chunk that is a single run reaches the writer as a zero-copy slice. Otherwise it gathers as before.
  • Deciding is cheap on both sides of the threshold. One probe per 64 rows gives a lower bound on the run count that rules out a chunk whose runs are all shorter than 64 rows, and an exact count that stops one run past the budget decides the rest. Without the bound, a chunk with runs just under the threshold paid for reading most of itself before being gathered anyway.
  • The partition index is unchanged, so placement, memory accounting, and spilling are untouched. This answers the open question in Native hash shuffle gathers rows one at a time even when a whole batch goes to one partition #7474 with a choice made per output chunk at flush, rather than a mixed index of rows and runs.
  • A schema with a view type keeps the gather, because a sliced view array serializes every data buffer it shares. This is the rule partitioning_for_schema already applies to positional round robin, so contains_view_type becomes pub(crate).
  • RunIterator and RowIterator share the single-slice passthrough through a new concat_slices.
  • native_shuffle.md describes how a hash partition's rows are written, Range Partitioning points at it, and the RowGroups section no longer says positional placement is the only strategy that hands the IPC writer slices.
  • A shuffle_hash_contiguous bench group in benches/shuffle_writer.rs, with constant, clustered (runs of 48 and 96 rows) and per-row hash keys.

Range partitioning over clustered input takes the same path.

The threshold comes from a sweep with arrow 59.3.2 on Apple Silicon, over chunks of 8192 rows drawn from 16 batches. Slicing and concatenating beats interleave_record_batch from an average run of about 32 rows on 4-column and 16-column flat schemas, and breaks even at about 64 on a nested schema (8 columns of a three-leaf struct inside two single-field structs, plus a list column). At the shuffle level, the 40-column nested fixture already runs 0.63x to 0.72x of the merge base at 64-row runs. I did not measure x86.

How are these changes tested?

New unit tests in partitioned_batch_iterator.rs:

  • contiguous_rows_are_sliced_not_gathered: over sliced sources with an Int64, a nullable Utf8 and a nullable Boolean column, chunks that are a whole batch, two runs, and part of a batch come out equal to interleave_record_batch. The whole-batch chunk shares its source buffers, which shows the slice path ran.
  • view_columns_are_gathered_even_when_contiguous: a contiguous Utf8View chunk long enough to slice is still gathered.
  • long_run_bound_matches_exact_count: the lower bound never changes the exact count's decision, on random mixes of short, long and out-of-order runs.

offsets_as_lengths_round_trip_every_mapped_type now also round-trips its batch of every mapped type (struct, list, large list, map, fixed size list, decimal, boolean and others, with nulls, empty values and inner arrays that start past offset 0) sliced from row 5, through ShuffleBlockWriter and both readers under every codec. That is the shape the slice path hands the IPC writer. The existing boolean_interleave_matches_arrow_for_sliced_nullable_states also sends its contiguous selections through the slice path.

Benchmarks ran on an 18-core Apple Silicon machine that was shared with other work, so the before and after binaries ran alternately and the numbers below are means over the rounds.

shuffle_hash_contiguous, place and gather of 65,536 rows into 200 partitions, PR time over the merge base. Rows at or under 48 are the mean of four rounds interleaved per case, the others are two rounds. Flat is the 4-column fixture, nested is the bench's 40-column struct fixture with a key column added.

hash key flat nested structs
per row (scattered) 0.99
runs of 32 rows 1.01
runs of 48 rows 1.00 1.00
runs of 64 rows 0.76 / 0.82 0.63 / 0.72
runs of 96 rows 0.72 / 0.79 0.65 / 0.66
runs of 120 rows 0.69 / 0.77 0.60 / 0.61
runs of 136 rows 0.71 / 0.71 0.57 / 0.59
one key per task 0.48 / 0.51

The same group in absolute numbers, three rounds, measured before the lower bound was added. The bound changes no decision and costs about 20 ns of probes per scattered chunk.

Hash key Rows per batch Before After
constant 8192 693 µs 333 µs
constant 1000 702 µs 408 µs
per row 8192 981 µs 997 µs
per row 1000 1.369 ms 1.329 ms

shuffle_bench over a generated 4M-row Parquet file with 16 columns (8 strings, 7 longs, 1 int) and a constant ds, 1600 partitions, zstd(1), two rounds:

Hash key Write time interleave time Encode time Output
constant ds, before 1.301 s 0.151 s 0.94 s 335 MiB
constant ds, after 1.161 s 0.016 s 0.94 s 335 MiB
per-row seq, before 2.317 s 0.913 s 1.18 s 451 MiB
per-row seq, after 2.304 s 0.886 s 1.17 s 451 MiB

Encode time does not change, so the IPC writer does not pay more for slices. zstd encoding takes most of the write time on this data, so the end-to-end gain here is smaller than the production share in #7474 suggests.

Input and commands

The input was generated with DuckDB:

COPY (
  SELECT
    '2026-11-03' AS ds,
    md5(i::VARCHAR) AS record_id,
    sha256(i::VARCHAR) AS payload_hash,
    CASE WHEN i % 20 = 0 THEN NULL ELSE 'device-' || (i % 5000)::VARCHAR END AS device,
    ['US', 'DE', 'FR', 'JP', 'IN', 'BR', 'GB', 'CN'][(i % 8) + 1] AS country,
    (i % 256)::VARCHAR || '.' || ((i // 256) % 256)::VARCHAR || '.'
      || ((i // 65536) % 256)::VARCHAR || '.' || (i % 251)::VARCHAR AS ip,
    'app/' || (i % 37)::VARCHAR || '.' || (i % 11)::VARCHAR AS app_version,
    CASE WHEN i % 7 = 0 THEN NULL ELSE 'city_' || (i % 997)::VARCHAR END AS city,
    i::BIGINT AS seq,
    (1730000000000000 + i * 1000)::BIGINT AS event_time_us,
    (i * 2654435761 % 1000000007)::BIGINT AS user_id,
    (i % 1000)::BIGINT AS bucket,
    CASE WHEN i % 13 = 0 THEN NULL ELSE (i * 31)::BIGINT END AS bytes_in,
    (i * 17)::BIGINT AS bytes_out,
    (i % 86400)::BIGINT AS second_of_day,
    (i % 100)::INTEGER AS shard
  FROM range(4000000) t(i)
) TO 'constant_ds.parquet' (FORMAT parquet, ROW_GROUP_SIZE 1048576);
cd native
cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- shuffle_hash_contiguous
cargo build --release -p datafusion-comet-shuffle --features shuffle-bench --bin shuffle_bench
# --hash-columns 0 is the constant ds, 8 is the per-row seq
target/release/shuffle_bench --input constant_ds.parquet --partitions 1600 --partitioning hash \
  --hash-columns 0 --codec zstd --zstd-level 1 --warmup 1 --iterations 3

@comphead comphead added the run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue label Nov 15, 2026
@github-actions github-actions Bot added enhancement New feature or request performance area:shuffle Shuffle (JVM and native) labels Nov 15, 2026
…em one at a time

Hash, range, and hash round robin partitioning record one (batch, row) pair
per row and build each output chunk with `interleave_record_batch`, which
copies primitive and string columns a row at a time even when the rows sit
together. A constant hash key, which an Iceberg write gets when Spark
rebalances on a partition column holding one value per run, sends every row
of a batch to one partition, so every chunk is one long run.

`RowIterator` now splits each chunk into runs of consecutive rows of one
buffered batch. When they average at least 128 rows it slices them out and
concatenates them, and a chunk that is a single run reaches the writer as a
slice. Scattered chunks and schemas with view types are gathered as before.
The native shuffle guide no longer says positional placement is the only
strategy that hands the IPC writer slices.
@comphead
comphead force-pushed the hash-shuffle-contiguous-runs branch from 8a203b4 to 07765c2 Compare November 15, 2026 19:51
@comphead comphead removed the run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue label Nov 15, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead, this is a nice targeted win, and the per-chunk decision at flush is a clean answer to the open question in #7474.

I ran the benches on x86 (Ryzen 9 7950X3D) against the merge base, alternating the two binaries over two rounds. Your shuffle_hash_contiguous numbers reproduce closely: 0.46x and 0.59x for the constant key, and flat for the per-row key. I also added a case with the hash key clustered in runs of N rows, over your flat fixture, the 40-column nested struct fixture, and a schema with list<int64> (with null and empty lists), list<utf8>, map<utf8, int64> and struct<int64, list<utf8>>. From 136-row runs up, it's 0.45x to 0.73x on flat, down to 0.24x on nested and 0.27x on complex types. A task that is all one key runs at 0.02x on nested. That's great.

average run (rows) flat nested structs complex (list/map)
32 1.00 / 1.01 0.99 / 1.01 1.00 / 1.00
96 1.02 / 1.02 1.00 / 1.00 1.00 / 1.00
120 0.91 / 1.02 0.87 / 0.99 0.83 / 1.00
136 0.72 / 0.73 0.64 / 0.56 0.51 / 0.56
192 0.66 / 0.65 0.60 / 0.45 0.44 / 0.44
512 0.58 / 0.51 0.45 / 0.24 0.35 / 0.27
one key per task 0.46 / 0.46 0.02 / 0.02 0.16 / 0.16

(PR time over base for place+gather of 65,536 rows, at 16 / 200 partitions.)

There's one band where it's slower. On the flat schema with runs of about 96 rows, place+gather goes from 908 µs to 930 µs. That's +2.4%, consistent across both rounds and at both 16 and 200 partitions. I think has_long_runs is scanning nearly the whole chunk there before falling back to the gather. Your sweep put flat break-even at about 32 rows and nested at about 64, so between 64 and 127 we pay for the scan and then take the slower path anyway. Would lowering MIN_AVG_RUN_ROWS to 64 work, since that's what you benchmarked with originally? It would also be good to add a clustered-key case near the threshold to shuffle_hash_contiguous, so the boundary stays covered.

Two other things:

  1. Until now only RowGroups, which is off by default, sent slices to the IPC writer. With this PR the default hash and range paths do too. From reading arrow-ipc 59.3.2, I believe it truncates every type Comet shuffles except the view types, which this PR keeps on the gather. But contiguous_rows_are_sliced_not_gathered covers only Int64, Utf8 and Boolean, and compares in-memory batches. Could you add a case that sends struct, list, map and decimal columns through the slice path from sources with a non-zero offset, then round-trips them through ShuffleBlockWriter and the reader? An end-to-end check with checkSparkAnswer in CometNativeShuffleSuite, using repartition on a constant column with nested data columns, would also work.

  2. In native_shuffle.md the new behavior is only described inside the RowGroups section. Since it mainly affects hash and range partitioning, could you add a short note to the Hash Partitioning section, and point Range Partitioning at it?

The exact run count reads about L/T of a chunk whose runs of L rows fall
under the threshold T before the chunk is gathered anyway, which made
place+gather about 2.4% slower for clustered runs just under the
threshold. A probe every T rows now gives a lower bound on the run count
that rules those chunks out first without changing any decision, and the
threshold drops back to 64 rows, where slicing breaks even on nested
schemas.

Every mapped type now also round-trips through the block writer as a
slice that starts past row 0, and the shuffle guide describes how hash
and range partitions are written.
@comphead

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove, the clustered cases found a real band, and your reading of it was right.

Runs just under the threshold. The exact count stops one run past the budget, so with runs of L rows under the threshold T it reads about L/T of the chunk and then gathers anyway. Timing the decision alone on 8192-row chunks, it costs 1.7 µs at 96-row runs with T=128, against about 55 µs for the 4-column gather it guards. Lowering T to 64 on its own moves the same cost down to runs of 32 to 63 rows. With the binaries alternated per case over four rounds, T=64 alone was 2.3% and 2.4% slower than the merge base at runs of 32 and 48 rows, and slower in every round. That is your band one octave down. On this machine the T=128 build stayed within noise at 96 rows, so I took the x86 number as the one to fix.

So has_long_runs now starts with a lower bound. A stretch of T steps whose ends are not T rows apart in one batch has to contain a run boundary, and the stretches are disjoint, so the number of broken stretches bounds the run count from below. When every run is shorter than T, every stretch is broken and the chunk is ruled out after one probe per T rows. Below the threshold the decision now costs 0.08 to 0.10 µs whatever the run length, and scattered chunks pay about 20 ns more for the probes. The bound never changes a decision, which long_run_bound_matches_exact_count checks against the exact count on random mixes of short, long and out-of-order runs.

With that cost gone, the threshold only decides where slicing starts, so I lowered MIN_AVG_RUN_ROWS to 64 as you suggested. PR time over the merge base for place+gather of 65,536 rows into 200 partitions, on Apple Silicon:

average run (rows) flat nested structs
scattered 0.99
32 1.01
48 1.00 1.00
64 0.76 / 0.82 0.63 / 0.72
96 0.72 / 0.79 0.65 / 0.66
120 0.69 / 0.77 0.60 / 0.61
136 0.71 / 0.71 0.57 / 0.59
one key per task 0.48 / 0.51

Rows at or under 48 are the mean of four rounds interleaved per case, the others are two rounds. Flat is the bench's 4-column create_batch_from fixture and nested is its 40-column struct fixture with a key column added.

Nested types through the writer. The slice path's only new output is a RecordBatch::slice of a buffered batch, since a chunk of several runs is concatenated into compact arrays, as the gather's output is. So the risk you describe is the IPC writer on sliced arrays, and an existing test already had the fixture for it. offsets_as_lengths_round_trip_every_mapped_type round-trips one column of every type Comet shuffles, including struct, list, large list, map, fixed size list, decimal and boolean, with nulls, empty values and inner arrays that start past offset 0. It goes through ShuffleBlockWriter and both readers, under every codec, with and without offsets as lengths. It now also round-trips that batch sliced from row 5, and passes. At 64 the existing boolean_interleave_matches_arrow_for_sliced_nullable_states also sends its contiguous selections through the slice path again.

Docs. The Hash Partitioning section now describes how a partition's rows are written, and Range Partitioning and the RowGroups paragraph point at it.

shuffle_hash_contiguous also has clustered keys over runs of 48 and 96 rows now, just under and over the threshold.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed ee46702ac999605573453378a883ca65c34ced01 with five independent review passes. No new actionable findings.

The slice path preserves partition membership, row order and chunk boundaries. Source buffers stay owned by the producer until local or RSS encoding finishes, and recursive view-type exclusion avoids serializing unrelated shared data buffers. The lower-bound precheck addresses the previously discussed threshold-band cost, and the added nonzero-offset round trips cover the IPC boundary. Compared with Apache 1.2.0, the intended change is materialization of long row-indexed runs; partition assignment and the wire format are unchanged.

Validation: no local tests or benchmarks. The exact-head fork CI run passed its Rust suite, including all three new iterator tests and the expanded mapped-type IPC round trip, plus Spark 4.1 and Iceberg 1.11 suites. The overall run failed because Maven dependency downloads stopped the Java lint and Celeborn compatibility jobs before their checks ran. The upstream green wrapper is not a separate full test pass. No independent performance measurement is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:shuffle Shuffle (JVM and native) enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native hash shuffle gathers rows one at a time even when a whole batch goes to one partition

3 participants