Repository navigation
Conversation
…em one at a time Hash, range, and hash round robin partitioning record one (batch, row) pair per row and build each output chunk with `interleave_record_batch`, which copies primitive and string columns a row at a time even when the rows sit together. A constant hash key, which an Iceberg write gets when Spark rebalances on a partition column holding one value per run, sends every row of a batch to one partition, so every chunk is one long run. `RowIterator` now splits each chunk into runs of consecutive rows of one buffered batch. When they average at least 128 rows it slices them out and concatenates them, and a chunk that is a single run reaches the writer as a slice. Scattered chunks and schemas with view types are gathered as before. The native shuffle guide no longer says positional placement is the only strategy that hands the IPC writer slices.
8a203b4 to
07765c2
Compare
andygrove
left a comment
There was a problem hiding this comment.
Thanks @comphead, this is a nice targeted win, and the per-chunk decision at flush is a clean answer to the open question in #7474.
I ran the benches on x86 (Ryzen 9 7950X3D) against the merge base, alternating the two binaries over two rounds. Your shuffle_hash_contiguous numbers reproduce closely: 0.46x and 0.59x for the constant key, and flat for the per-row key. I also added a case with the hash key clustered in runs of N rows, over your flat fixture, the 40-column nested struct fixture, and a schema with list<int64> (with null and empty lists), list<utf8>, map<utf8, int64> and struct<int64, list<utf8>>. From 136-row runs up, it's 0.45x to 0.73x on flat, down to 0.24x on nested and 0.27x on complex types. A task that is all one key runs at 0.02x on nested. That's great.
| average run (rows) | flat | nested structs | complex (list/map) |
|---|---|---|---|
| 32 | 1.00 / 1.01 | 0.99 / 1.01 | 1.00 / 1.00 |
| 96 | 1.02 / 1.02 | 1.00 / 1.00 | 1.00 / 1.00 |
| 120 | 0.91 / 1.02 | 0.87 / 0.99 | 0.83 / 1.00 |
| 136 | 0.72 / 0.73 | 0.64 / 0.56 | 0.51 / 0.56 |
| 192 | 0.66 / 0.65 | 0.60 / 0.45 | 0.44 / 0.44 |
| 512 | 0.58 / 0.51 | 0.45 / 0.24 | 0.35 / 0.27 |
| one key per task | 0.46 / 0.46 | 0.02 / 0.02 | 0.16 / 0.16 |
(PR time over base for place+gather of 65,536 rows, at 16 / 200 partitions.)
There's one band where it's slower. On the flat schema with runs of about 96 rows, place+gather goes from 908 µs to 930 µs. That's +2.4%, consistent across both rounds and at both 16 and 200 partitions. I think has_long_runs is scanning nearly the whole chunk there before falling back to the gather. Your sweep put flat break-even at about 32 rows and nested at about 64, so between 64 and 127 we pay for the scan and then take the slower path anyway. Would lowering MIN_AVG_RUN_ROWS to 64 work, since that's what you benchmarked with originally? It would also be good to add a clustered-key case near the threshold to shuffle_hash_contiguous, so the boundary stays covered.
Two other things:
-
Until now only
RowGroups, which is off by default, sent slices to the IPC writer. With this PR the default hash and range paths do too. From reading arrow-ipc 59.3.2, I believe it truncates every type Comet shuffles except the view types, which this PR keeps on the gather. Butcontiguous_rows_are_sliced_not_gatheredcovers onlyInt64,Utf8andBoolean, and compares in-memory batches. Could you add a case that sends struct, list, map and decimal columns through the slice path from sources with a non-zero offset, then round-trips them throughShuffleBlockWriterand the reader? An end-to-end check withcheckSparkAnswerinCometNativeShuffleSuite, usingrepartitionon a constant column with nested data columns, would also work. -
In
native_shuffle.mdthe new behavior is only described inside theRowGroupssection. Since it mainly affects hash and range partitioning, could you add a short note to the Hash Partitioning section, and point Range Partitioning at it?
The exact run count reads about L/T of a chunk whose runs of L rows fall under the threshold T before the chunk is gathered anyway, which made place+gather about 2.4% slower for clustered runs just under the threshold. A probe every T rows now gives a lower bound on the run count that rules those chunks out first without changing any decision, and the threshold drops back to 64 rows, where slicing breaks even on nested schemas. Every mapped type now also round-trips through the block writer as a slice that starts past row 0, and the shuffle guide describes how hash and range partitions are written.
|
Thanks @andygrove, the clustered cases found a real band, and your reading of it was right. Runs just under the threshold. The exact count stops one run past the budget, so with runs of L rows under the threshold T it reads about L/T of the chunk and then gathers anyway. Timing the decision alone on 8192-row chunks, it costs 1.7 µs at 96-row runs with T=128, against about 55 µs for the 4-column gather it guards. Lowering T to 64 on its own moves the same cost down to runs of 32 to 63 rows. With the binaries alternated per case over four rounds, T=64 alone was 2.3% and 2.4% slower than the merge base at runs of 32 and 48 rows, and slower in every round. That is your band one octave down. On this machine the T=128 build stayed within noise at 96 rows, so I took the x86 number as the one to fix. So With that cost gone, the threshold only decides where slicing starts, so I lowered
Rows at or under 48 are the mean of four rounds interleaved per case, the others are two rounds. Flat is the bench's 4-column Nested types through the writer. The slice path's only new output is a Docs. The Hash Partitioning section now describes how a partition's rows are written, and Range Partitioning and the
|
sunchao
left a comment
There was a problem hiding this comment.
Reviewed ee46702ac999605573453378a883ca65c34ced01 with five independent review passes. No new actionable findings.
The slice path preserves partition membership, row order and chunk boundaries. Source buffers stay owned by the producer until local or RSS encoding finishes, and recursive view-type exclusion avoids serializing unrelated shared data buffers. The lower-bound precheck addresses the previously discussed threshold-band cost, and the added nonzero-offset round trips cover the IPC boundary. Compared with Apache 1.2.0, the intended change is materialization of long row-indexed runs; partition assignment and the wire format are unchanged.
Validation: no local tests or benchmarks. The exact-head fork CI run passed its Rust suite, including all three new iterator tests and the expanded mapped-type IPC round trip, plus Spark 4.1 and Iceberg 1.11 suites. The overall run failed because Maven dependency downloads stopped the Java lint and Celeborn compatibility jobs before their checks ran. The upstream green wrapper is not a separate full test pass. No independent performance measurement is claimed.
Which issue does this PR close?
Closes #7474.
Rationale for this change
Hash, range, and hash round robin partitioning record one
(batch, row)pair per row and build each output chunk withinterleave_record_batch. Arrow copies primitive and string columns there one row at a time (interleave_primitive,interleave_bytes) and merges contiguous rows only ininterleave_fallback, which Boolean columns use. So a chunk whose rows are already one span of a buffered batch is still gathered row by row.That is the shape an Iceberg write gets when Spark rebalances on a partition column that holds one value per run, such as the
dsof an hourly or daily ETL. Every row hashes to one partition, and each buffered batch is a single run. In the production job described in #7474, partition interleaving was 54% of the native shuffle writer time in the map stage.DataFusion's
RepartitionExecgained the whole-batch case of this in 56.0.0. Itspartition_grouped_takepasses a batch through unchanged when all of its rows go to one partition (apache/datafusion#24598). This PR also covers runs that span buffered batches and output chunks.What changes are included in this PR?
RowIterator(partitioned_batch_iterator.rs) splits each output chunk into runs of consecutive rows of one buffered batch withchunk_by. When the runs average at least 64 rows, it slices them out of the buffered batches and concatenates the slices, and a chunk that is a single run reaches the writer as a zero-copy slice. Otherwise it gathers as before.partitioning_for_schemaalready applies to positional round robin, socontains_view_typebecomespub(crate).RunIteratorandRowIteratorshare the single-slice passthrough through a newconcat_slices.native_shuffle.mddescribes how a hash partition's rows are written, Range Partitioning points at it, and theRowGroupssection no longer says positional placement is the only strategy that hands the IPC writer slices.shuffle_hash_contiguousbench group inbenches/shuffle_writer.rs, with constant, clustered (runs of 48 and 96 rows) and per-row hash keys.Range partitioning over clustered input takes the same path.
The threshold comes from a sweep with arrow 59.3.2 on Apple Silicon, over chunks of 8192 rows drawn from 16 batches. Slicing and concatenating beats
interleave_record_batchfrom an average run of about 32 rows on 4-column and 16-column flat schemas, and breaks even at about 64 on a nested schema (8 columns of a three-leaf struct inside two single-field structs, plus a list column). At the shuffle level, the 40-column nested fixture already runs 0.63x to 0.72x of the merge base at 64-row runs. I did not measure x86.How are these changes tested?
New unit tests in
partitioned_batch_iterator.rs:contiguous_rows_are_sliced_not_gathered: over sliced sources with anInt64, a nullableUtf8and a nullableBooleancolumn, chunks that are a whole batch, two runs, and part of a batch come out equal tointerleave_record_batch. The whole-batch chunk shares its source buffers, which shows the slice path ran.view_columns_are_gathered_even_when_contiguous: a contiguousUtf8Viewchunk long enough to slice is still gathered.long_run_bound_matches_exact_count: the lower bound never changes the exact count's decision, on random mixes of short, long and out-of-order runs.offsets_as_lengths_round_trip_every_mapped_typenow also round-trips its batch of every mapped type (struct, list, large list, map, fixed size list, decimal, boolean and others, with nulls, empty values and inner arrays that start past offset 0) sliced from row 5, throughShuffleBlockWriterand both readers under every codec. That is the shape the slice path hands the IPC writer. The existingboolean_interleave_matches_arrow_for_sliced_nullable_statesalso sends its contiguous selections through the slice path.Benchmarks ran on an 18-core Apple Silicon machine that was shared with other work, so the before and after binaries ran alternately and the numbers below are means over the rounds.
shuffle_hash_contiguous, place and gather of 65,536 rows into 200 partitions, PR time over the merge base. Rows at or under 48 are the mean of four rounds interleaved per case, the others are two rounds. Flat is the 4-column fixture, nested is the bench's 40-column struct fixture with a key column added.The same group in absolute numbers, three rounds, measured before the lower bound was added. The bound changes no decision and costs about 20 ns of probes per scattered chunk.
shuffle_benchover a generated 4M-row Parquet file with 16 columns (8 strings, 7 longs, 1 int) and a constantds, 1600 partitions, zstd(1), two rounds:interleave timeds, beforeds, afterseq, beforeseq, afterEncode time does not change, so the IPC writer does not pay more for slices. zstd encoding takes most of the write time on this data, so the end-to-end gain here is smaller than the production share in #7474 suggests.
Input and commands
The input was generated with DuckDB: