Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions docs/source/contributor-guide/memory_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,7 +413,7 @@ partially sorted, but it is likewise checked only against the pool and Spark's g
`PartialMerge` over input sorted on every grouping key stays on the hash aggregate, since the final
aggregate has no spill path there and would only halve the limit of a sort below it.

Three details matter for tuning:
Four details matter for tuning:

- The pool is task-shared (see below), so the consumers that divide the task's share include those
of every native plan in the task, not just the plan making the request. Registering a consumer
Expand All @@ -433,7 +433,21 @@ Three details matter for tuning:
the task's whole share, because the limit is then zero. Spark lets such a consumer hold more than
`off_heap_size / running_tasks` while other tasks have not yet asked for memory. A larger
`spark.memory.offHeap.size`, a smaller `spark.comet.batchSize` or `spark.comet.exec.memoryPool=greedy_unified`
avoids it, and the migration guide says so too.
avoids it, and the migration guide says so too. For a sort over a shuffle with many small
blocks, `spark.comet.exec.sort.coalesceInput.enabled=false` also avoids it, as described next.
- `CometSort` can put a consumer that cannot spill in front of the sort, which combines small input
batches, such as shuffle blocks from many map tasks, before the sort buffers them. It reserves
twice the bytes of the batches it holds, enough for them and the combined copy, and releases that
before the sort reserves the combined batch, so it holds nothing while the sort takes a batch. A
combined batch is larger than the blocks it replaces, so a sort whose share is very close to the
least it can run with may still fail with the error above where the blocks fit, as it can with a
full-size batch from any other input. A combined batch holds at most half the sort's in-place
threshold, 512 KiB by default, so the sort reserves at most about that threshold for it. That cap
is kept between 64 KiB and 4 MiB, so with a threshold below 128 KiB or above 8 MiB it is not half.
While it holds a group, its reservation, like any that cannot spill, lowers the share of the
task's consumers that can spill, such as another plan's sort in the same task. When the pool
refuses it a reservation, it passes the next 64 small batches through, then asks again.
`spark.comet.exec.sort.coalesceInput.enabled=false` turns it off.

### Task-shared pools and their lifetime

Expand Down Expand Up @@ -562,8 +576,10 @@ unbounded, so nobody charged for them at allocation time. Whether they are charg
on who holds them. DataFusion's `ExternalSorter` reserves `get_reserved_bytes_for_record_batch` for
every batch it retains, and the hash join build side reserves `get_record_batch_memory_size` for
each incoming batch; both read buffer sizes off the `ArrayData` and apply equally to imported
buffers. So an imported batch that a sort or join has buffered _is_ reserved in Comet's pool, and
through a unified pool is charged to Spark. The same batch held by an operator that does not reserve
buffers. The coalescer that can sit in front of a native sort also reserves the batches it holds,
at twice their size to cover the combined copy. So an imported batch that a sort, a join or that
coalescer has buffered _is_ reserved in Comet's pool, and through a unified pool is charged to
Spark. The same batch held by an operator that does not reserve
(a projection or filter, or `ScanExec` itself between polls) is charged to nothing.

**Native → JVM (`CometExecIterator`).** DataFusion produces the batch in Rust, and whether it is
Expand Down
14 changes: 14 additions & 0 deletions docs/source/user-guide/latest/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,20 @@ Native aggregates with grouping keys report these additional metrics:
Spill bytes from native sorts, aggregates, and sort-merge joins are also added to Spark's task-level
`diskBytesSpilled` metric in every stage, not only in shuffle stages.

### CometSort

Native sorts report these additional metrics:

| Metric | Description |
| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `number of spills` | Number of times the sort spilled to disk. |
| `total spilled bytes` | Bytes written to sort spill files. |
| `number of small input batches combined into larger ones` | Small input batches, such as shuffle blocks from many map tasks, combined before the sort. |
| `number of batches produced by combining small input batches` | Batches the sort received from that combining. Zero means the sort got its input batches unchanged. |

The combining is controlled by `spark.comet.exec.sort.coalesceInput.enabled`. Its time is included in
the sort's `total time`.

### Hash Joins

With `spark.comet.exec.join.dynamicFilter.enabled=true`, native broadcast and shuffled hash joins
Expand Down
11 changes: 11 additions & 0 deletions docs/source/user-guide/latest/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,17 @@ Kubernetes can kill an executor that runs out of it. The plugin warns at startup
but check it before you upgrade; see
[Configuring Comet Memory](tuning/memory.md#configuring-comet-memory).

### Sorts Combine Small Input Batches

`CometSort` now combines small input batches before sorting them, once a task's sort input is too
large to sort in place. A shuffle written by many map tasks gives each reduce task many tiny
batches, and DataFusion's sort merges one stream per batch, at least when it spills or sorts on more
than one key, which made such sorts very slow. The results do not change. A combined batch is
larger than the blocks it replaces, so a sort with a very small memory share can fail with "Not
enough memory to continue external sort" where it used to run. If that happens, set
`spark.comet.exec.sort.coalesceInput.enabled=false`, or use the remedies in
[Memory Pool Limits](#memory-pool-limits).

### Deprecated and Removed Settings

`spark.comet.exec.columnarToRow.native.enabled` is deprecated and will be removed in a future major
Expand Down
91 changes: 91 additions & 0 deletions native/core/src/execution/memory_pools/fair_pool.rs
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,97 @@ mod tests {
(0..n).map(|_| tasks.start()).collect()
}

/// Sorts 3000 batches of 60 wide rows in `pool`, optionally combining them first. Returns
/// how many rows came out, how many times the sort spilled, and how many input batches were
/// combined.
async fn sort_small_batches(
pool: &Arc<dyn MemoryPool>,
coalesce: bool,
) -> datafusion::common::Result<(usize, usize, usize)> {
use arrow::array::{ArrayRef, Int64Array, RecordBatch, StringArray};
use datafusion::datasource::memory::MemorySourceConfig;
use datafusion::execution::runtime_env::RuntimeEnvBuilder;
use datafusion::physical_expr::expressions::col;
use datafusion::physical_expr::{LexOrdering, PhysicalSortExpr};
use datafusion::physical_plan::sorts::sort::SortExec;
use datafusion::physical_plan::{collect, ExecutionPlan};
use datafusion::prelude::{SessionConfig, SessionContext};
use datafusion_comet_operators::SortInputCoalesceExec;

let input: Vec<RecordBatch> = (0..3000)
.map(|b| {
let ids: Vec<i64> = (0..60).map(|i| (b * 60 + i) * 7919 % 180_000).collect();
let text = ids.iter().map(|i| format!("{i:0>210}"));
RecordBatch::try_from_iter([
("id", Arc::new(Int64Array::from(ids.clone())) as ArrayRef),
(
"s",
Arc::new(StringArray::from_iter_values(text)) as ArrayRef,
),
])
.unwrap()
})
.collect();
let schema = input[0].schema();
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::clone(pool))
.build_arc()?;
let config = SessionConfig::new().with_sort_spill_reservation_bytes(64 * 1024);
let gate = config.options().execution.sort_in_place_threshold_bytes;
let ctx = SessionContext::new_with_config_rt(config, runtime);
let source: Arc<dyn ExecutionPlan> =
MemorySourceConfig::try_new_exec(&[input], Arc::clone(&schema), None)?;
let coalescer = Arc::new(SortInputCoalesceExec::new(Arc::clone(&source), 8192, gate));
let sort_input: Arc<dyn ExecutionPlan> = if coalesce {
Arc::clone(&coalescer) as _
} else {
source
};
let ordering = LexOrdering::new(vec![PhysicalSortExpr::new(
col("id", &schema)?,
Default::default(),
)])
.unwrap();
let sort = Arc::new(SortExec::new(ordering, sort_input));
let output = collect(Arc::clone(&sort) as _, ctx.task_ctx()).await?;
let spills = sort
.metrics()
.and_then(|metrics| metrics.spill_count())
.unwrap_or(0);
let combined = coalescer
.metrics()
.and_then(|metrics| metrics.sum_by_name("coalesced_input_batches"))
.map_or(0, |value| value.as_usize());
Ok((
output.iter().map(|batch| batch.num_rows()).sum(),
spills,
combined,
))
}

/// The coalescer in front of a sort cannot spill, so this pool limits it only by the pool
/// size. A combined batch must still be small enough for the sort's fair share, or the sort
/// fails after a spill where the uncombined batches fit.
#[tokio::test(flavor = "multi_thread")]
async fn a_sort_fits_its_share_with_combined_input_where_it_fits_without() {
let fake = FakeSpark::with(usize::MAX);
let pool = pool(&fake, 3 * 1024 * 1024);
// Another consumer that can spill, such as a second sort in the task, halves the share.
let _other_sort = spillable("ExternalSorter[1]", &pool);
Comment on lines +413 to +414

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #7630 landed on main, CometFairMemoryPool splits the share only among consumers that can spill and are active. A sort that has not started takes no share from another sort (is_sort and unstarted_sorts in try_grow). This ExternalSorter[1] registers but never reserves. So after a merge with main, I expect the sort under test to get the whole 3 MiB share, and the test stops covering the halved share it describes. It should still pass, because the plain sort spills either way, so CI will not flag it. hold(&other_sort, 2) from memory_pools/testing.rs makes it active, as the tests added in #7630 do. The final pool.reserved() check then has to allow for those 2 bytes. Untested.


let (rows, spills, _) = sort_small_batches(&pool, false).await.unwrap();
assert_eq!(rows, 180_000);
assert!(
spills > 0,
"the plain sort must spill for this test to mean anything"
);
let (rows, spills, combined) = sort_small_batches(&pool, true).await.unwrap();
assert_eq!(rows, 180_000);
assert!(spills > 0, "the sort over combined input must spill too");
assert!(combined > 0, "no input batches were combined");
assert_eq!(pool.reserved(), 0);
}

#[test]
fn consumers_that_can_spill_share_their_tasks_share_of_the_pool() {
// Eight running tasks share a 1040-byte pool, and Spark grants this one its 130 bytes.
Expand Down
38 changes: 31 additions & 7 deletions native/core/src/execution/planner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,9 @@ use datafusion::{
},
prelude::SessionContext,
};
use datafusion_comet_operators::{range_exec, range_sort_expr, ExpandExec, SampleExec};
use datafusion_comet_operators::{
range_exec, range_sort_expr, ExpandExec, SampleExec, SortInputCoalesceExec,
};
use datafusion_comet_spark_expr::{
create_comet_physical_fun, create_comet_physical_fun_with_eval_mode, BinaryOutputStyle,
BloomFilterMightContain, CheckedBinaryExpr, CsvWriteOptions, EvalMode, SparkArraysZipFunc,
Expand Down Expand Up @@ -1860,12 +1862,32 @@ impl PhysicalPlanner {
.collect();

let fetch = sort.fetch.map(|num| num as usize);
let ordering = LexOrdering::new(exprs?).unwrap();

// Only a full sort over unordered input merges: TopK never does, and a sort passes
// ordered input through.
let (sort_input, coalescer): (Arc<dyn ExecutionPlan>, Vec<_>) =
if sort.coalesce_input
&& fetch.is_none()
&& !child
.native_plan
.properties()
.equivalence_properties()
.ordering_satisfy(ordering.clone())?
{
let config = self.session_ctx.copied_config();
let coalescer: Arc<dyn ExecutionPlan> =
Arc::new(SortInputCoalesceExec::new(
Arc::clone(&child.native_plan),
config.batch_size(),
config.options().execution.sort_in_place_threshold_bytes,
));
(Arc::clone(&coalescer), vec![coalescer])
} else {
(Arc::clone(&child.native_plan), vec![])
};

let sort_plan = SortExec::new(
LexOrdering::new(exprs?).unwrap(),
Arc::clone(&child.native_plan),
)
.with_fetch(fetch);
let sort_plan = SortExec::new(ordering, sort_input).with_fetch(fetch);
let mut sort_exec: Arc<dyn ExecutionPlan> = if sort.dynamic_filter_enabled {
match TopKReaderFilterExec::try_new(
&sort_plan,
Expand All @@ -1885,10 +1907,11 @@ impl PhysicalPlanner {
Ok((
scans,
shuffle_scans,
Arc::new(SparkPlan::new(
Arc::new(SparkPlan::new_with_additional(
spark_plan.plan_id,
sort_exec,
vec![Arc::clone(&child)],
coalescer,
)),
))
}
Expand Down Expand Up @@ -3183,6 +3206,7 @@ mod tests {
mod empty_native_scan;
mod multi_store_scan;
mod semi_join_reduction;
mod sort_input_coalesce;

use futures::{poll, StreamExt};
use std::{
Expand Down
11 changes: 8 additions & 3 deletions native/core/src/execution/planner/join.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ use datafusion::{
},
};
use datafusion_comet_operators::{
CometFilterExec, CometProjectionExec, MembershipKeyReductionExec,
CometFilterExec, CometProjectionExec, MembershipKeyReductionExec, SortInputCoalesceExec,
};
use datafusion_comet_proto::{
spark_expression::Expr,
Expand Down Expand Up @@ -220,7 +220,8 @@ fn reduce_before_sort(
let Some(sort) = params.right.native_plan.downcast_ref::<SortExec>() else {
return Ok(None);
};
let input = sort.input();
// The reducer packs its own output, so it replaces any input coalescer.
let input = SortInputCoalesceExec::unwrap_input(sort.input());
let ([(_, key)], [options]) = (params.join_on.as_slice(), sort_options) else {
return Ok(None);
};
Expand Down Expand Up @@ -268,7 +269,11 @@ fn reduce_before_sort(
vec![Arc::clone(&reducer)],
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
)?;
Arc::make_mut(&mut params.right).native_plan = sort;
let right = Arc::make_mut(&mut params.right);
right.native_plan = sort;
right
.additional_native_plans
.retain(|plan| !plan.is::<SortInputCoalesceExec>());
// The Spark sort collects from its replacement. The join owns the reducer's named metrics.
Ok(Some(reducer))
}
Expand Down
Loading
Loading