Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions docs/source/contributor-guide/native_shuffle.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,13 @@ Native shuffle implements Spark-compatible hash partitioning:
- Computes hash of partition key columns
- Applies modulo by partition count: `partition_id = hash % num_partitions`

The partitioner writes each partition's rows out of the buffered batches in chunks of up to
`batch_size` rows. When a chunk's rows sit in contiguous runs that average at least 64 rows
(`MIN_AVG_RUN_ROWS`), as when every row of a batch hashes to one partition, it slices the runs out
and concatenates them, and it hands a chunk that is a single run to the IPC writer as a slice of
the buffered batch. It gathers any other chunk a row at a time with `interleave_record_batch`, as it
does every chunk when the schema has a view type.

### Range Partitioning

For range partitioning:
Expand All @@ -346,6 +353,10 @@ For range partitioning:
3. Native code converts sort key columns to comparable row format.
4. Binary search (`partition_point`) determines which partition each row belongs to.

The partitioner then writes each partition's rows as described in
[Hash Partitioning](#hash-partitioning), so input that is already clustered on the sort key is
sliced rather than gathered.

### Single Partition

The simplest case: all rows go to partition 0. Uses `SinglePartitionShufflePartitioner`, which
Expand Down Expand Up @@ -446,11 +457,12 @@ one `(batch, row)` pair per row, so the index list charged against the spill res
smaller, and `RunIterator` builds each output chunk by slicing and concatenating runs. A run that
covers an entire `batch_size` buffered batch is passed through without copying.

One schema-level restriction is applied in `create_repartitioner`: positional placement is the only
strategy that hands a sliced array to the IPC writer, and while the writer truncates a slice's
buffers for every other type, for `Utf8View` and `BinaryView` it serializes every shared data
buffer in full. A schema containing a view type anywhere therefore falls back to `HashAll`, with
the same `maxHashColumns`.
One schema-level restriction is applied in `create_repartitioner`. The IPC writer truncates a
slice's buffers for every type except `Utf8View` and `BinaryView`, where it serializes every shared
data buffer in full. Positional placement always hands the writer slices, so a schema containing a
view type anywhere falls back to `HashAll`, with the same `maxHashColumns`. The other strategies
also slice chunks that sit in long contiguous runs (see [Hash Partitioning](#hash-partitioning)),
and on a schema with a view type they gather those rows instead.

#### Retry safety under `RowGroups`

Expand Down
55 changes: 53 additions & 2 deletions native/shuffle/benches/shuffle_writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ use arrow::row::{RowConverter, SortField};
use criterion::{criterion_group, criterion_main, BatchSize, Bencher, Criterion};
use datafusion::datasource::memory::MemorySourceConfig;
use datafusion::datasource::source::DataSourceExec;
use datafusion::physical_expr::expressions::{col, Column};
use datafusion::logical_expr::Operator;
use datafusion::physical_expr::expressions::{binary, col, lit, Column};
use datafusion::physical_expr::{LexOrdering, PhysicalSortExpr};
use datafusion::physical_plan::metrics::Time;
use datafusion::{
Expand Down Expand Up @@ -467,6 +468,56 @@ fn partitioning_benchmark(c: &mut Criterion) {
group.finish();
}

/// Hash placement when every row of a task goes to one output partition, as when a V2 write
/// rebalances on a partition column holding one value per run, such as an hourly or daily `ds`.
/// The literal key places rows as such a column would, so each partition's rows are contiguous
/// in the buffered batches. The clustered keys repeat over runs of 48 and 96 rows, just under and
/// over the shortest average run that is sliced, and a per-row key over the same rows is the
/// scattered control. `rows_per_batch=8192` makes every output chunk one whole buffered batch,
/// `rows_per_batch=1000` makes a chunk span several. The writer discards its batches, as in
/// `partitioning_benchmark`.
fn contiguous_hash_benchmark(c: &mut Criterion) {
const BATCH_SIZE: usize = 8192;
const NUM_PARTITIONS: usize = 200;

let mut group = c.benchmark_group("shuffle_hash_contiguous");
for (rows_per_batch, num_batches) in [(BATCH_SIZE, 8), (1000, 66)] {
let batches: Vec<_> = (0..num_batches)
.map(|b| create_batch_from(b * rows_per_batch, rows_per_batch, true))
.collect();
let schema = batches[0].schema();
let row = col("c0", &schema).unwrap();
let mut keys = vec![
("constant".to_string(), lit(20261103i32)),
("per_row".to_string(), Arc::clone(&row)),
];
if rows_per_batch == BATCH_SIZE {
for run in [48, 96] {
let key = binary(Arc::clone(&row), Operator::Divide, lit(run), &schema).unwrap();
keys.push((format!("clustered_{run}"), key));
}
}
for (key, expr) in keys {
let partitioning = CometPartitioning::Hash(vec![expr], NUM_PARTITIONS);
group.bench_function(
format!("place+gather (key={key}, rows_per_batch={rows_per_batch})"),
|b| {
b.iter_batched(
|| BenchRepartitioner::try_new(partitioning.clone(), BATCH_SIZE).unwrap(),
|mut repartitioner| {
repartitioner.place(&batches).unwrap();
repartitioner.gather().unwrap();
repartitioner
},
BatchSize::LargeInput,
);
},
);
}
}
group.finish();
}

/// Benchmarks the per-block IPC encoding cost (schema + record batch) in isolation, using the
/// `None` codec so that compression does not obscure the schema-encoding cost. Covers a wide flat
/// schema and a deeply nested schema, where the schema flatbuffer is largest.
Expand Down Expand Up @@ -684,6 +735,6 @@ fn config() -> Criterion {
criterion_group! {
name = benches;
config = config();
targets = criterion_benchmark, boolean_state_benchmark, partitioning_benchmark, schema_encoding_benchmark, ipc_context_reuse_benchmark
targets = criterion_benchmark, boolean_state_benchmark, partitioning_benchmark, contiguous_hash_benchmark, schema_encoding_benchmark, ipc_context_reuse_benchmark
}
criterion_main!(benches);
Loading