Repository navigation
perf: combine small input batches before a native sort that merges them - #7696
parthchandra wants to merge 1 commit into
Conversation
A shuffle from many map tasks gives each reduce task one tiny batch per block. DataFusion's sort merges one stream per buffered batch when it spills, so such sorts become very slow. Add SortInputCoalesceExec in front of a full native sort over unordered input. It passes batches through until the input reaches the sort's in-place threshold, then combines small batches into batches of at most spark.comet.batchSize rows and half that threshold in bytes. It reserves exactly what it holds and holds nothing while the sort takes a batch. Controlled by spark.comet.exec.sort.coalesceInput.enabled (default true). Closes apache#5686.
comphead
left a comment
There was a problem hiding this comment.
I checked the operator against DataFusion 55.2's ExternalSorter. It reserves before it holds a batch and frees before the sort reserves, and the planner wraps only full sorts over unordered input. One fair_pool.rs test no longer tests what it says once merged with #7630, and a few tests repeat others. For the DataFusion follow-up, the two-key in-memory numbers are worth including: in_mem_sort_stream in 55.2 and 56.0 keeps multi-column sorts at one run per batch, on the grounds that merging many small runs is faster, and sort_small_batches shows the opposite for tiny batches.
| // Another consumer that can spill, such as a second sort in the task, halves the share. | ||
| let _other_sort = spillable("ExternalSorter[1]", &pool); |
There was a problem hiding this comment.
Since #7630 landed on main, CometFairMemoryPool splits the share only among consumers that can spill and are active. A sort that has not started takes no share from another sort (is_sort and unstarted_sorts in try_grow). This ExternalSorter[1] registers but never reserves. So after a merge with main, I expect the sort under test to get the whole 3 MiB share, and the test stops covering the halved share it describes. It should still pass, because the plain sort spills either way, so CI will not flag it. hold(&other_sort, 2) from memory_pools/testing.rs makes it active, as the tests added in #7630 do. The final pool.reserved() check then has to allow for those 2 bytes. Untested.
| #[test] | ||
| fn a_refused_reservation_flushes_the_held_group_first() { | ||
| let input = ten_row_batches(4); | ||
| let one = get_record_batch_memory_size(&input[0]); | ||
| // Room for two batches and their copy. The third is refused, so the first two go out | ||
| // combined and the third passes through. The fourth starts a new group. | ||
| let output = run_in_pool(input.clone(), 1000, 2 * 2 * one); | ||
| assert_eq!(row_counts(&output), [20, 10, 10]); | ||
| same_rows(&output, &input); | ||
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn holds_no_memory_while_the_sort_takes_a_batch() { | ||
| let input = ten_row_batches(10); | ||
| let schema = input[0].schema(); | ||
| let pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(1 << 30)); | ||
| let runtime = RuntimeEnvBuilder::new() | ||
| .with_memory_pool(Arc::clone(&pool)) | ||
| .build_arc() | ||
| .unwrap(); | ||
| let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime); | ||
| let source = MemorySourceConfig::try_new_exec(&[input], schema, None).unwrap(); | ||
| // A 25-row target makes every third batch overflow the group. | ||
| let exec = SortInputCoalesceExec::new(source, 25, 0); | ||
| let mut stream = exec.execute(0, ctx.task_ctx()).unwrap(); | ||
| let mut rows = vec![]; | ||
| while let Some(batch) = stream.try_next().await.unwrap() { | ||
| assert_eq!(pool.reserved(), 0, "held memory while handing on a batch"); | ||
| rows.push(batch.num_rows()); | ||
| } | ||
| assert_eq!(rows, [20, 20, 20, 20, 20]); | ||
| } | ||
|
|
||
| #[test] | ||
| fn after_a_refusal_the_next_batches_pass_through() { | ||
| let input = ten_row_batches(6); | ||
| let one = get_record_batch_memory_size(&input[0]); | ||
| // Room for two batches and their copy, so the third is refused. The ones after it pass | ||
| // through without asking the pool again. | ||
| let output = run_in_pool(input.clone(), 1000, 2 * 2 * one); | ||
| assert_eq!(row_counts(&output), [20, 10, 10, 10, 10]); | ||
| same_rows(&output, &input); | ||
| } |
There was a problem hiding this comment.
These two tests use the same pool size and target as combining_resumes_after_the_backoff below. Their expected outputs, [20, 10, 10] and [20, 10, 10, 10, 10], are prefixes of its output, so that test alone covers all three. Also, the comment in the first one is off: the fourth batch passes through under the backoff and does not start a new group. The three dictionary tests at lines 767 to 816 differ only in how the column wraps the Int8 dictionary. One test that loops over the three columns would cover them.
| /// A sort in a fair pool so tight that it spills about every 50 KiB. It failed with combined | ||
| /// input when the coalescer still held memory while the sort reserved a combined batch. | ||
| async fn sort_in_tight_fair_pool( | ||
| input: Vec<RecordBatch>, | ||
| coalesce: bool, | ||
| ) -> Result<MetricsSet> { | ||
| use datafusion::execution::memory_pool::FairSpillPool; | ||
| use datafusion::physical_expr::expressions::col; | ||
| use datafusion::physical_expr::{LexOrdering, PhysicalSortExpr}; | ||
| use datafusion::physical_plan::sorts::sort::SortExec; | ||
|
|
||
| let schema = input[0].schema(); | ||
| let runtime = RuntimeEnvBuilder::new() | ||
| .with_memory_pool(Arc::new(FairSpillPool::new(2560 * 1024))) | ||
| .build_arc() | ||
| .unwrap(); | ||
| let config = SessionConfig::new().with_sort_spill_reservation_bytes(64 * 1024); | ||
| let gate = config.options().execution.sort_in_place_threshold_bytes; | ||
| let ctx = SessionContext::new_with_config_rt(config, runtime); | ||
| let source = MemorySourceConfig::try_new_exec(&[input], Arc::clone(&schema), None).unwrap(); | ||
| let coalescer = Arc::new(SortInputCoalesceExec::new( | ||
| Arc::clone(&source) as _, | ||
| 8192, | ||
| gate, | ||
| )); | ||
| let sort_input: Arc<dyn ExecutionPlan> = if coalesce { | ||
| Arc::clone(&coalescer) as _ | ||
| } else { | ||
| source | ||
| }; | ||
| let ordering = LexOrdering::new(vec![PhysicalSortExpr::new( | ||
| col("id", &schema).unwrap(), | ||
| Default::default(), | ||
| )]) | ||
| .unwrap(); | ||
| let sort = Arc::new(SortExec::new(ordering, sort_input)); | ||
| collect(sort.execute(0, ctx.task_ctx())?).await?; | ||
| Ok(coalescer.metrics().unwrap()) | ||
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn combined_input_does_not_fail_a_sort_in_a_tight_fair_pool() { | ||
| let input: Vec<_> = (0..4000) | ||
| .map(|b| { | ||
| let ids: Vec<i64> = (0..50) | ||
| .map(|i| ((b * 50 + i) * 7919 % 200_000) as i64) | ||
| .collect(); | ||
| let text = ids.iter().map(|i| format!("value-{i:010}-padding-padding")); | ||
| RecordBatch::try_from_iter([ | ||
| ("id", Arc::new(Int64Array::from(ids.clone())) as ArrayRef), | ||
| ( | ||
| "s", | ||
| Arc::new(StringArray::from_iter_values(text)) as ArrayRef, | ||
| ), | ||
| ]) | ||
| .unwrap() | ||
| }) | ||
| .collect(); | ||
| sort_in_tight_fair_pool(input.clone(), false) | ||
| .await | ||
| .expect("the plain sort fits"); | ||
| let metrics = sort_in_tight_fair_pool(input, true) | ||
| .await | ||
| .expect("the sort over combined batches fits too"); | ||
| assert!(metric(&metrics, "coalesced_input_batches") > 0); | ||
| } |
There was a problem hiding this comment.
This runs the same scenario as a_sort_fits_its_share_with_combined_input_where_it_fits_without in fair_pool.rs: a sort over small batches in a tight fair pool, with and without the coalescer. The difference is that it uses DataFusion's FairSpillPool, which Comet does not use outside tests. The failure it describes, the coalescer holding memory while the sort reserves, is the invariant that holds_no_memory_while_the_sort_takes_a_batch checks directly. I think this test and sort_in_tight_fair_pool can be removed.
Which issue does this PR close?
Closes #5686.
Rationale for this change
A shuffle written by many map tasks into many partitions gives each reduce task one tiny Arrow batch per shuffle block. On a production job, about 330,000 map tasks write to 16,000 partitions, so each reduce task reads about 330,000 blocks of about 62 rows. When the native sort over that input spills, it is extremely slow. In one task,
SortExectook 16,778 s ofelapsed_computefor 19.7 M rows and 5 spills, about 97% of the task. The shuffle read took 33 s.The cause is DataFusion 55.2's
ExternalSorter:sort_and_spill_in_mem_batchescallsin_mem_sort_stream(false, false), which does not combine buffered batches into runs. Every buffered batch becomes its own merge stream, so a spill of about 4 M rows merges about 66,000 streams.BatchBuildercalls arrowinterleaveover every live input batch, for each column and each struct child. The cost per output batch grows with the number of streams.#5686 found the same problem and measured a large native gain. It was set aside after an end-to-end run regressed by 11%. In that run each partition held about 7,800 rows, under the 1 MiB
sort_in_place_threshold_bytes, so the sort never merged: it concatenated in place, and the extraCoalesceBatchesExecwas pure overhead. The issue also pointed out that the coalescer's buffers were not charged to a memory pool. This PR follows the analysis in #5686 and addresses both points.What changes are included in this PR?
SortInputCoalesceExec(native/operators/src/sort_input_coalesce.rs), a new operator in front of a nativeSortExec:spark.comet.batchSizeand fewer than 512 are combined in arrival order. A combined batch holds at mostspark.comet.batchSizerows and half the in-place threshold in bytes, 512 KiB by default, kept between 64 KiB and 4 MiB. The sort then reserves no more for a combined batch than its own in-place threshold. Input batches are never split.ResourcesExhaustedare returned.coalesced_input_batchesandcoalesced_output_batchesare reported under the sort. The operator's time is part of the sort's time.Sortarm adds the coalescer only whenSort.coalesce_input(a new proto field) is set, the sort has no fetch (TopK never merges), and the input does not already satisfy the ordering (the sort passes such input through). The semi-join key reduction injoin.rslooks through the coalescer and replaces it.spark.comet.exec.sort.coalesceInput.enabled, on by default.CometSortExecsets the proto field from it and declares the two metrics.metrics.mdgets aCometSortsection.memory_management.mddescribes the reservation and the trade-off below.native/operators/benches/sort_small_batches.rs.Trade-off to review. A combined batch is larger than the blocks it replaces. A sort whose memory share is very close to the least it can run with can now fail with "Not enough memory to continue external sort" where the tiny blocks fit. The same is already true of any input that delivers full-size batches. Halving the cap and holding nothing while the sort reserves made this much narrower: in tight-pool sweeps, the coalesced sort spilled slightly less than the plain one and never failed where it fit. It is documented, with the config as the way out. The complete fix belongs in DataFusion, as a follow-up below.
Follow-ups, not in this PR:
ExternalSortersplit a batch it cannot reserve after spilling, instead of failing. Either would make this operator unnecessary or safer.How are these changes tested?
sort_input_coalesce.rs(28):SortExecs give identical output with and without the coalescer.CometFairMemoryPooltest (fair_pool.rs): a sort whose fair share is halved by a second spillable consumer spills and finishes with combined input wherever it does without. It failed with the earlier 1 MiB cap.skipis wrapped; the semi-join key reduction replaces the coalescer.sql-tests/operators/sort_coalesce_input.sql: compared with Spark. Each sort task reads blocks of about 12 rows.CometTaskMetricsSuite: a sort over 80 map tasks' small blocks spills and matches Spark. With the config on it combines batches; with it off it does not. The test checks metrics, which a SQL file cannot.sort_small_batches: 200,000 wide rows (fixed-length string key, 8 longs, a struct, no nulls), one or two sort keys, in memory and with a memory limit that forces spills, with keys scattered so every batch spans the whole key range (poorly predicted) and with keys already in order (the most predictable merge). Times in ms, raw -> combined, on a laptop with about +/-10% run-to-run noise. Scattered keys:Keys already in order:
When 512-row batches were combined, a spilling single-key sort got slower (51.4 -> 54.0 ms), so the limit is fewer than 512 rows. Multi-key sorts gain the most, even in memory, where DataFusion does not combine runs itself.
Cost: batches the coalescer passes through are unchanged, but the extra operator is not free. With keys in order, spilling and two keys, 1024-row input is about 7 ms (~18%) slower with the coalescer in place, with the same spill count. An identity
ProjectionExecin the same place costs the same (38.6 -> 44.3 ms), so this is the cost of one more operator between an in-memory source and the sort, not of the coalescer's work. In a Comet plan the sort's input is already a shuffle reader or scan. Sorts whose input is declared in order are not wrapped. In the TPC-H runs below, total time did not get worse; q7 is the one query where it might show.spark.comet.exec.sort.coalesceInput.enabledoff and on,spark.sql.shuffle.partitions=2000and AQE partition coalescing off, at 16g and at 2g off-heap. Three iterations each; times are the average.