Skip to content

perf: combine small input batches before a native sort that merges them - #7696

Open
parthchandra wants to merge 1 commit into
apache:mainfrom
parthchandra:sort-coalesce-shuffle-read
Open

parthchandra wants to merge 1 commit into
apache:mainfrom
parthchandra:sort-coalesce-shuffle-read

Conversation

@parthchandra

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5686.

Rationale for this change

A shuffle written by many map tasks into many partitions gives each reduce task one tiny Arrow batch per shuffle block. On a production job, about 330,000 map tasks write to 16,000 partitions, so each reduce task reads about 330,000 blocks of about 62 rows. When the native sort over that input spills, it is extremely slow. In one task, SortExec took 16,778 s of elapsed_compute for 19.7 M rows and 5 spills, about 97% of the task. The shuffle read took 33 s.

The cause is DataFusion 55.2's ExternalSorter:

  • When it spills, sort_and_spill_in_mem_batches calls in_mem_sort_stream(false, false), which does not combine buffered batches into runs. Every buffered batch becomes its own merge stream, so a spill of about 4 M rows merges about 66,000 streams.
  • For every output batch, BatchBuilder calls arrow interleave over every live input batch, for each column and each struct child. The cost per output batch grows with the number of streams.
  • Without spilling, DataFusion combines runs only for single-key sorts. Multi-key sorts merge one stream per batch either way.

#5686 found the same problem and measured a large native gain. It was set aside after an end-to-end run regressed by 11%. In that run each partition held about 7,800 rows, under the 1 MiB sort_in_place_threshold_bytes, so the sort never merged: it concatenated in place, and the extra CoalesceBatchesExec was pure overhead. The issue also pointed out that the coalescer's buffers were not charged to a memory pool. This PR follows the analysis in #5686 and addresses both points.

What changes are included in this PR?

  • SortInputCoalesceExec (native/operators/src/sort_input_coalesce.rs), a new operator in front of a native SortExec:
    • Gate: batches pass through unchanged until a task's sort input reaches the sort's in-place threshold, counted the way the sort counts it. Small sorts, the case that regressed in perf: coalesce small shuffle-read batches before native sort #5686, are unaffected.
    • Combining: after the gate, batches with fewer rows than half of spark.comet.batchSize and fewer than 512 are combined in arrival order. A combined batch holds at most spark.comet.batchSize rows and half the in-place threshold in bytes, 512 KiB by default, kept between 64 KiB and 4 MiB. The sort then reserves no more for a combined batch than its own in-place threshold. Input batches are never split.
    • Memory: it reserves exactly what it holds, the batches plus the combined copy, in a consumer that cannot spill. It releases the reservation before the combined batch reaches the sort, so it holds nothing while the sort reserves. If the pool refuses, it hands the group and the batch to the sort and passes the next 64 small batches through before asking again. Errors other than ResourcesExhausted are returned.
    • Never combined: batches with a different schema start a new group. Batches with small dictionary keys or small run-end types, at any nesting depth, pass through, since concatenating them can overflow the key type. A small slice of a large batch counts as large and passes through.
    • Metrics: coalesced_input_batches and coalesced_output_batches are reported under the sort. The operator's time is part of the sort's time.
  • Planner: the Sort arm adds the coalescer only when Sort.coalesce_input (a new proto field) is set, the sort has no fetch (TopK never merges), and the input does not already satisfy the ordering (the sort passes such input through). The semi-join key reduction in join.rs looks through the coalescer and replaces it.
  • Config: spark.comet.exec.sort.coalesceInput.enabled, on by default. CometSortExec sets the proto field from it and declares the two metrics.
  • Docs:
    • The migration guide gets a 1.3.0 note.
    • metrics.md gets a CometSort section.
    • memory_management.md describes the reservation and the trade-off below.
  • Benchmark: native/operators/benches/sort_small_batches.rs.

Trade-off to review. A combined batch is larger than the blocks it replaces. A sort whose memory share is very close to the least it can run with can now fail with "Not enough memory to continue external sort" where the tiny blocks fit. The same is already true of any input that delivers full-size batches. Halving the cap and holding nothing while the sort reserves made this much narrower: in tight-pool sweeps, the coalesced sort spilled slightly less than the plain one and never failed where it fit. It is documented, with the config as the way out. The complete fix belongs in DataFusion, as a follow-up below.

Follow-ups, not in this PR:

  • In DataFusion, combine buffered batches into runs on the spill path, within the memory budget. Also have ExternalSorter split a batch it cannot reserve after spilling, instead of failing. Either would make this operator unnecessary or safer.
  • Hold the input before the gate without copying it, so the start of each task's input is combined too.

How are these changes tested?

  • Rust unit tests in sort_input_coalesce.rs (28):
    • Gate and grouping: pass-through below the gate without a copy; the gate's byte counting; the row target, byte cap and 512-row limit; large batches flushing the group first; empty, zero-row and zero-column batches.
    • Memory: the reservation covers exactly the held batches and is released; nothing is held while the sort receives a batch; a refused reservation flushes the held group first; backoff and resumption; combining when the pool fits the group but not a full one.
    • Errors and early drop: an input error reaches the sort and releases memory; dropping the stream early releases the reservation.
    • Batches never combined: schema changes; small dictionary keys at the top level, in a struct and in a list; slices of large batches.
    • Metrics: a rebuilt node reports through the original's metrics.
    • Types: a nested map, list and struct mix, decimal(38,10) at its limits, years 1 and 9999, a time zone, CJK and other non-ASCII strings, and dictionaries.
    • Spilling sorts: single-key and multi-key spilling SortExecs give identical output with and without the coalescer.
  • CometFairMemoryPool test (fair_pool.rs): a sort whose fair share is halved by a second spillable consumer spills and finishes with combined input wherever it does without. It failed with the earlier 1 MiB cap.
  • Planner tests (4): only a full sort over unordered input is wrapped, with the session's batch size and threshold; a sort with only a skip is wrapped; the semi-join key reduction replaces the coalescer.
  • sql-tests/operators/sort_coalesce_input.sql: compared with Spark. Each sort task reads blocks of about 12 rows.
    • The data covers nested structs, maps and arrays of structs, decimal bounds, dates, timestamps and timestamp_ntz from year 1 to 9999, and non-ASCII strings.
    • Sort keys include a single string key, multiple keys with both null orderings, and a struct field.
    • The config matrix covers the in-place threshold (0 and the default), a normal and a tiny memory pool so the sort spills, and two session time zones.
  • CometTaskMetricsSuite: a sort over 80 map tasks' small blocks spills and matches Spark. With the config on it combines batches; with it off it does not. The test checks metrics, which a SQL file cannot.
  • Micro-benchmark sort_small_batches: 200,000 wide rows (fixed-length string key, 8 longs, a struct, no nulls), one or two sort keys, in memory and with a memory limit that forces spills, with keys scattered so every batch spans the whole key range (poorly predicted) and with keys already in order (the most predictable merge). Times in ms, raw -> combined, on a laptop with about +/-10% run-to-run noise. Scattered keys:
Rows per input batch In memory, 1 key In memory, 2 keys Spilling, 1 key Spilling, 2 keys
16 86.3 -> 51.9 330.2 -> 51.2 256.2 -> 79.3 258.2 -> 70.7
64 38.8 -> 31.9 97.5 -> 32.0 102.0 -> 58.6 97.7 -> 49.8
256 32.1 -> 29.7 45.1 -> 29.0 64.2 -> 57.4 62.6 -> 50.3
384 31.2 -> 28.3 38.7 -> 28.3 58.0 -> 57.0 55.3 -> 51.2
1024 (passed through) 27.0 -> 27.2 31.0 -> 31.1 40.1 -> 42.7 48.3 -> 45.7

Keys already in order:

Rows per input batch In memory, 1 key In memory, 2 keys Spilling, 1 key Spilling, 2 keys
16 66.2 -> 38.3 310.0 -> 47.1 221.0 -> 68.3 259.0 -> 73.2
64 29.2 -> 22.1 90.7 -> 26.9 76.6 -> 47.4 99.1 -> 56.2
256 19.7 -> 18.1 35.2 -> 23.4 44.7 -> 42.5 50.1 -> 48.2
1024 (passed through) 16.9 -> 17.1 21.2 -> 20.9 34.5 -> 34.3 40.0 -> 47.2

When 512-row batches were combined, a spilling single-key sort got slower (51.4 -> 54.0 ms), so the limit is fewer than 512 rows. Multi-key sorts gain the most, even in memory, where DataFusion does not combine runs itself.

Cost: batches the coalescer passes through are unchanged, but the extra operator is not free. With keys in order, spilling and two keys, 1024-row input is about 7 ms (~18%) slower with the coalescer in place, with the same spill count. An identity ProjectionExec in the same place costs the same (38.6 -> 44.3 ms), so this is the cost of one more operator between an in-memory source and the sort, not of the coalescer's work. In a Comet plan the sort's input is already a shuffle reader or scan. Sorts whose input is declared in order are not wrapped. In the TPC-H runs below, total time did not get worse; q7 is the one query where it might show.

  • End-to-end: TPC-H SF100 on one machine (Spark 4.0 standalone, 2 executors x 8 cores), with spark.comet.exec.sort.coalesceInput.enabled off and on, spark.sql.shuffle.partitions=2000 and AQE partition coalescing off, at 16g and at 2g off-heap. Three iterations each; times are the average.
Off-heap Off (s) On (s) Change Failed tasks, on
16g 1148.3 1144.9 -0.3% 0
2g 1257.7 1234.8 -1.8% 0
  • The coalescer combined batches in q3, q4, q9, q10 and q21, including about 8.4 M small batches in q9 and 5.2 M in q10. Those queries changed by 1% or less: their sorts did not spill and are not where the time goes. TPC-H does not reach the spill-heavy sort over tiny blocks that perf: coalesce small shuffle-read batches before native sort #5686 describes, so this run shows that the change costs nothing on a standard workload, not the speed-up.
  • No query failed with the coalescer on, and spill counts did not change. The only failures were in the run with it off: an executor was lost during one q5 iteration (256.8 s against about 87 s for the others), which is left out of the 16g "off" total above (1204.9 s with it).
  • q11 (about 3 s or 25 s per iteration whatever the setting) and q12 (82 to 112 s) vary too much between iterations to compare. q7 was about 10% slower with the coalescer on at 16g, though nothing was combined there. That could be the cost of the extra operator described above, or noise; three iterations do not separate the two.

A shuffle from many map tasks gives each reduce task one tiny batch per
block. DataFusion's sort merges one stream per buffered batch when it
spills, so such sorts become very slow.

Add SortInputCoalesceExec in front of a full native sort over unordered
input. It passes batches through until the input reaches the sort's
in-place threshold, then combines small batches into batches of at most
spark.comet.batchSize rows and half that threshold in bytes. It reserves
exactly what it holds and holds nothing while the sort takes a batch.

Controlled by spark.comet.exec.sort.coalesceInput.enabled (default
true). Closes apache#5686.
@parthchandra
parthchandra requested review from andygrove, comphead, mbutrovich and sunchao and removed request for andygrove November 19, 2026 21:39
@github-actions github-actions Bot added enhancement New feature or request performance area:memory Memory pools, reservations, OOM handling labels Nov 19, 2026
@parthchandra parthchandra added run-spark-4.0-tests run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue labels Nov 19, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the operator against DataFusion 55.2's ExternalSorter. It reserves before it holds a batch and frees before the sort reserves, and the planner wraps only full sorts over unordered input. One fair_pool.rs test no longer tests what it says once merged with #7630, and a few tests repeat others. For the DataFusion follow-up, the two-key in-memory numbers are worth including: in_mem_sort_stream in 55.2 and 56.0 keeps multi-column sorts at one run per batch, on the grounds that merging many small runs is faster, and sort_small_batches shows the opposite for tiny batches.

Comment on lines +413 to +414
// Another consumer that can spill, such as a second sort in the task, halves the share.
let _other_sort = spillable("ExternalSorter[1]", &pool);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #7630 landed on main, CometFairMemoryPool splits the share only among consumers that can spill and are active. A sort that has not started takes no share from another sort (is_sort and unstarted_sorts in try_grow). This ExternalSorter[1] registers but never reserves. So after a merge with main, I expect the sort under test to get the whole 3 MiB share, and the test stops covering the halved share it describes. It should still pass, because the plain sort spills either way, so CI will not flag it. hold(&other_sort, 2) from memory_pools/testing.rs makes it active, as the tests added in #7630 do. The final pool.reserved() check then has to allow for those 2 bytes. Untested.

Comment on lines +680 to +722
#[test]
fn a_refused_reservation_flushes_the_held_group_first() {
let input = ten_row_batches(4);
let one = get_record_batch_memory_size(&input[0]);
// Room for two batches and their copy. The third is refused, so the first two go out
// combined and the third passes through. The fourth starts a new group.
let output = run_in_pool(input.clone(), 1000, 2 * 2 * one);
assert_eq!(row_counts(&output), [20, 10, 10]);
same_rows(&output, &input);
}

#[tokio::test]
async fn holds_no_memory_while_the_sort_takes_a_batch() {
let input = ten_row_batches(10);
let schema = input[0].schema();
let pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(1 << 30));
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::clone(&pool))
.build_arc()
.unwrap();
let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);
let source = MemorySourceConfig::try_new_exec(&[input], schema, None).unwrap();
// A 25-row target makes every third batch overflow the group.
let exec = SortInputCoalesceExec::new(source, 25, 0);
let mut stream = exec.execute(0, ctx.task_ctx()).unwrap();
let mut rows = vec![];
while let Some(batch) = stream.try_next().await.unwrap() {
assert_eq!(pool.reserved(), 0, "held memory while handing on a batch");
rows.push(batch.num_rows());
}
assert_eq!(rows, [20, 20, 20, 20, 20]);
}

#[test]
fn after_a_refusal_the_next_batches_pass_through() {
let input = ten_row_batches(6);
let one = get_record_batch_memory_size(&input[0]);
// Room for two batches and their copy, so the third is refused. The ones after it pass
// through without asking the pool again.
let output = run_in_pool(input.clone(), 1000, 2 * 2 * one);
assert_eq!(row_counts(&output), [20, 10, 10, 10, 10]);
same_rows(&output, &input);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two tests use the same pool size and target as combining_resumes_after_the_backoff below. Their expected outputs, [20, 10, 10] and [20, 10, 10, 10, 10], are prefixes of its output, so that test alone covers all three. Also, the comment in the first one is off: the fourth batch passes through under the backoff and does not start a new group. The three dictionary tests at lines 767 to 816 differ only in how the column wraps the Int8 dictionary. One test that loops over the three columns would cover them.

Comment on lines +979 to +1044
/// A sort in a fair pool so tight that it spills about every 50 KiB. It failed with combined
/// input when the coalescer still held memory while the sort reserved a combined batch.
async fn sort_in_tight_fair_pool(
input: Vec<RecordBatch>,
coalesce: bool,
) -> Result<MetricsSet> {
use datafusion::execution::memory_pool::FairSpillPool;
use datafusion::physical_expr::expressions::col;
use datafusion::physical_expr::{LexOrdering, PhysicalSortExpr};
use datafusion::physical_plan::sorts::sort::SortExec;

let schema = input[0].schema();
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::new(FairSpillPool::new(2560 * 1024)))
.build_arc()
.unwrap();
let config = SessionConfig::new().with_sort_spill_reservation_bytes(64 * 1024);
let gate = config.options().execution.sort_in_place_threshold_bytes;
let ctx = SessionContext::new_with_config_rt(config, runtime);
let source = MemorySourceConfig::try_new_exec(&[input], Arc::clone(&schema), None).unwrap();
let coalescer = Arc::new(SortInputCoalesceExec::new(
Arc::clone(&source) as _,
8192,
gate,
));
let sort_input: Arc<dyn ExecutionPlan> = if coalesce {
Arc::clone(&coalescer) as _
} else {
source
};
let ordering = LexOrdering::new(vec![PhysicalSortExpr::new(
col("id", &schema).unwrap(),
Default::default(),
)])
.unwrap();
let sort = Arc::new(SortExec::new(ordering, sort_input));
collect(sort.execute(0, ctx.task_ctx())?).await?;
Ok(coalescer.metrics().unwrap())
}

#[tokio::test]
async fn combined_input_does_not_fail_a_sort_in_a_tight_fair_pool() {
let input: Vec<_> = (0..4000)
.map(|b| {
let ids: Vec<i64> = (0..50)
.map(|i| ((b * 50 + i) * 7919 % 200_000) as i64)
.collect();
let text = ids.iter().map(|i| format!("value-{i:010}-padding-padding"));
RecordBatch::try_from_iter([
("id", Arc::new(Int64Array::from(ids.clone())) as ArrayRef),
(
"s",
Arc::new(StringArray::from_iter_values(text)) as ArrayRef,
),
])
.unwrap()
})
.collect();
sort_in_tight_fair_pool(input.clone(), false)
.await
.expect("the plain sort fits");
let metrics = sort_in_tight_fair_pool(input, true)
.await
.expect("the sort over combined batches fits too");
assert!(metric(&metrics, "coalesced_input_batches") > 0);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs the same scenario as a_sort_fits_its_share_with_combined_input_where_it_fits_without in fair_pool.rs: a sort over small batches in a tight fair pool, with and without the coalescer. The difference is that it uses DataFusion's FairSpillPool, which Comet does not use outside tests. The failure it describes, the coalescer holding memory while the sort reserves, is the invariant that holds_no_memory_while_the_sort_takes_a_batch checks directly. I think this test and sort_in_tight_fair_pool can be removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:memory Memory pools, reservations, OOM handling enhancement New feature or request performance run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue run-spark-4.0-tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: coalesce small shuffle-read batches before native sort

2 participants