Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/source/contributor-guide/memory_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,19 @@ JNI, which goes through Spark's ordinary `TaskMemoryManager`. That means:
granted. While overcommit is outstanding, `try_grow` asks Spark for it on top of the request and
is refused unless Spark can cover both, so operators spill until the debt is repaid. Both pools'
`Display` output and their `try_grow` errors report the current overcommit.
- When Spark throws while acquiring memory, for instance because a spill of another consumer
failed, every later `try_grow` of the task fails with that exception, without asking Spark and
before the fair pool checks its own limits. A later `grow` records the full amount as
overcommit, also without asking Spark. The exception is not `ResourcesExhausted`, so an operator
that checks for a refusal, such as the shuffle repartitioner, fails the task rather than
spilling, and one that takes any error for a refusal and asks again, such as DataFusion's sort,
fails when it asks again. The JNI boundary rethrows the exception as itself, unless an operator
has wrapped it in `Context` or `Shared`, which turns it into a `CometNativeException`. Nothing
checks for the failure when a plan's output ends, so the exception is still only logged if no
operator returns it: if no `try_grow` follows a failed `grow` in the task, or if every `try_grow`
that fails comes from DataFusion's sort-merge join, whose `materializing_stream.rs` and
`bitwise_stream.rs` take any error for a refusal, spill the batch and carry on
([issue #7615](https://github.com/apache/datafusion-comet/issues/7615)).

`CometFairMemoryPool` exists because Spark cannot ask a native operator to spill. Spark limits each
task to an even share of its off-heap pool among the tasks that hold memory from it. Once one
Expand Down
8 changes: 5 additions & 3 deletions docs/source/contributor-guide/native_shuffle.md
Original file line number Diff line number Diff line change
Expand Up @@ -491,9 +491,11 @@ to handle sliced batches or an indeterminate stage's rollback.
Native shuffle uses DataFusion's memory management with spilling support:

- **Memory pool**: Tracks memory usage across the shuffle operation.
- **Spill triggers**: Partitions spill to disk when the memory pool denies an allocation, or
when the buffered bytes reach `spark.comet.shuffle.native.maxBufferBytes`. That config defaults to
0, which disables the fixed limit and leaves memory pressure as the only trigger.
- **Spill triggers**: Partitions spill to disk when the memory pool refuses an allocation with
`ResourcesExhausted`, or when the buffered bytes reach
`spark.comet.shuffle.native.maxBufferBytes`. That config defaults to 0, which disables the fixed
limit and leaves memory pressure as the only trigger. Any other error from the pool, such as an
exception Spark threw while spilling another consumer, fails the task.
- **One spill file per task**: Every output partition spills into the same file, not one file per
partition. `PartitionedSpill` records the byte ranges each partition's blocks occupy, in write
order, and the final output copies a partition's ranges back in that order.
Expand Down
12 changes: 12 additions & 0 deletions native/common/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1067,6 +1067,18 @@ pub fn unwrap_arrow_external_error(error: ArrowError) -> DataFusionError {
}
}

/// Whether `e`, from growing or resizing a memory reservation, is the pool refusing the memory.
///
/// Every pool reports a refusal as `ResourcesExhausted`, which [`DataFusionError::find_root`]
/// finds under `Context`, `Shared` and `External`. An operator that can give memory back, by
/// spilling or by closing files early, answers a refusal that way. A pool backed by Spark can also
/// fail with an exception Spark threw, such as a failed spill of another consumer or the
/// `InterruptedException` of a task killed while it waited for memory. That comes back as another
/// kind of error and has to fail the task: giving memory back and carrying on would hide it.
pub fn is_refusal(e: &DataFusionError) -> bool {
matches!(e.find_root(), DataFusionError::ResourcesExhausted(_))
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down
2 changes: 1 addition & 1 deletion native/common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ mod utils;

pub use calendar::{civil_from_days, days_from_civil};
pub use error::{
decimal_overflow_error, error_chain, unwrap_arrow_external_error, SparkError,
decimal_overflow_error, error_chain, is_refusal, unwrap_arrow_external_error, SparkError,
SparkErrorWithContext, SparkResult,
};
pub use ffi_offsets::zero_offsets;
Expand Down
129 changes: 119 additions & 10 deletions native/core/src/execution/delta_dv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
use std::mem::size_of;
use std::sync::Arc;

use datafusion::common::DataFusionError;
use datafusion::datasource::listing::PartitionedFile;
use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
Expand All @@ -45,8 +46,10 @@ use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
use roaring::{RoaringBitmap, RoaringTreemap};

use crate::errors::CometError;
use crate::execution::operators::ExecutionError;
use crate::execution::operators::ExecutionError::GeneralError;
use datafusion_comet_common::is_refusal;
use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;

const NATIVE_MAGIC: i32 = 1681511376;
Expand Down Expand Up @@ -710,10 +713,12 @@ async fn attach_access_plan(
admission_bound_bytes(dv.cardinality, row_counts.len(), page_bound_selectors)?;
let reservation = partition_reservation.new_empty();
reservation.try_grow(admission_bytes).map_err(|e| {
GeneralError(format!(
"Deletion vector access plan for {file_path} needs up to {admission_bytes} \
bytes to construct, exceeding the execution memory pool: {e}"
))
reservation_error(e, |e| {
format!(
"Deletion vector access plan for {file_path} needs up to {admission_bytes} \
bytes to construct, exceeding the execution memory pool: {e}"
)
})
})?;

let plan = build_access_plan(&metadata, &deleted, file.range.as_ref())
Expand Down Expand Up @@ -763,11 +768,13 @@ async fn attach_access_plan(
.min(total_rows);
let retained_bytes_bound = reader_peak_bytes(reader_selector_bound, row_counts.len())?;
reservation.try_resize(retained_bytes_bound).map_err(|e| {
GeneralError(format!(
"Deletion vector access plan for {file_path} retains {selector_count} row \
selectors, needing up to {retained_bytes_bound} bytes at the reader's \
normalization peak, exceeding the execution memory pool: {e}"
))
reservation_error(e, |e| {
format!(
"Deletion vector access plan for {file_path} retains {selector_count} row \
selectors, needing up to {retained_bytes_bound} bytes at the reader's \
normalization peak, exceeding the execution memory pool: {e}"
)
})
})?;

// Keyed by concrete type: the parquet opener looks up
Expand All @@ -784,13 +791,41 @@ async fn attach_access_plan(
.with_extension(DvAccessPlanReservation(reservation)))
}

/// The error for a reservation that the memory pool failed with `e`: `refused(e)` if the pool
/// refused the memory, and otherwise `e` itself (see [`is_refusal`]). An exception Spark threw
/// stays that exception, which the JNI boundary rethrows as itself.
fn reservation_error(
e: DataFusionError,
refused: impl FnOnce(&DataFusionError) -> String,
) -> ExecutionError {
if is_refusal(&e) {
return GeneralError(refused(&e));
}
if let DataFusionError::External(inner) = e.find_root() {
if let Some(CometError::JavaException {
class,
msg,
throwable,
}) = inner.downcast_ref::<CometError>()
{
return ExecutionError::JavaException {
class: class.clone(),
msg: msg.clone(),
throwable: Arc::clone(throwable),
};
}
}
e.into()
}

#[cfg(test)]
mod tests {
use super::*;
use datafusion::arrow::datatypes::Schema;
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::execution::memory_pool::{GreedyMemoryPool, MemoryPool};
use datafusion::execution::memory_pool::{GreedyMemoryPool, MemoryPool, UnboundedMemoryPool};
use datafusion::execution::runtime_env::RuntimeEnvBuilder;
use jni::objects::{Global, JThrowable};
use parquet::arrow::ArrowWriter;
use parquet::file::metadata::ParquetMetaDataReader;
use parquet::file::properties::WriterProperties;
Expand Down Expand Up @@ -1685,6 +1720,80 @@ mod tests {
);
}

/// A pool with room to spare that fails every `try_grow` with an exception Spark threw, as a
/// pool backed by Spark does once a spill of another consumer has failed.
#[derive(Debug)]
struct ThrowingPool {
inner: UnboundedMemoryPool,
throwable: Arc<Global<JThrowable<'static>>>,
}

impl std::fmt::Display for ThrowingPool {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "ThrowingPool({})", self.inner)
}
}

impl MemoryPool for ThrowingPool {
fn name(&self) -> &str {
"ThrowingPool"
}

fn grow(&self, reservation: &MemoryReservation, additional: usize) {
self.inner.grow(reservation, additional)
}

fn shrink(&self, reservation: &MemoryReservation, shrink: usize) {
self.inner.shrink(reservation, shrink)
}

fn try_grow(&self, _: &MemoryReservation, _: usize) -> datafusion::common::Result<()> {
Err(CometError::JavaException {
class: "org.apache.spark.memory.SparkOutOfMemoryError".to_string(),
msg: "[SPILL_OUT_OF_MEMORY] Error while calling spill()".to_string(),
throwable: Arc::clone(&self.throwable),
}
.into())
}

fn reserved(&self) -> usize {
self.inner.reserved()
}
}

/// An exception Spark threw while the access plan reserves its memory fails the scan with
/// that exception, not with a message that the plan exceeds the pool.
#[tokio::test]
async fn attach_access_plans_fails_with_the_exception_spark_threw() {
let tmp = tempfile::tempdir().unwrap();
// A null reference stands in for the throwable, since no JVM runs here.
let throwable: Arc<Global<JThrowable<'static>>> = Arc::new(Global::null());
let pool: Arc<dyn MemoryPool> = Arc::new(ThrowingPool {
inner: UnboundedMemoryPool::default(),
throwable: Arc::clone(&throwable),
});
let runtime_env = Arc::new(
RuntimeEnvBuilder::new()
.with_memory_pool(Arc::clone(&pool))
.build()
.unwrap(),
);
let scan_file = alternating_dv_scan_file(&runtime_env, tmp.path(), 1024);

let err = attach_access_plans(Arc::clone(&runtime_env), vec![scan_file])
.await
.unwrap_err();
assert!(
matches!(
&err,
ExecutionError::JavaException { throwable: thrown, .. }
if Arc::ptr_eq(thrown, &throwable)
),
"{err:?}"
);
assert_eq!(pool.reserved(), 0);
}

/// A pool with room for the plan succeeds, reserves exactly the reader-lifecycle peak
/// bound (`reader_peak_bytes`, never a hardcoded constant) once construction's transient
/// peak has passed, attaches the reservation alongside the access plan, and releases it
Expand Down
3 changes: 3 additions & 0 deletions native/core/src/execution/memory_pools/fair_pool.rs
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,9 @@ impl MemoryPool for CometFairMemoryPool {
) -> Result<(), DataFusionError> {
if additional > 0 {
let mut state = self.state.lock();
// Before the pool's own limits, whose refusal would make the operator spill and carry
// on, and under the lock, which a `grow` holds while Spark fails it.
self.spark.check_failure()?;
let can_spill = reservation.consumer().can_spill();
let used = state.used;
if limited_to_its_share(reservation.consumer()) {
Expand Down
45 changes: 45 additions & 0 deletions native/core/src/execution/memory_pools/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ pub(crate) fn create_memory_pool_with_fake_spark(
mod tests {
use super::testing::spillable;
use super::*;
use datafusion::common::DataFusionError;
use datafusion::execution::memory_pool::MemoryConsumer;

#[test]
Expand All @@ -186,6 +187,50 @@ mod tests {
}
}

/// Spark refusing memory is a refusal, which makes an operator spill. Spark throwing in a
/// `grow` is not: every later `try_grow` of the task fails with the exception Spark threw, at
/// the top of the error, through the wrappers of each pool type. That holds also where the
/// pool's own limits would have refused the request, and for a final aggregate's replay,
/// which records a refused request with `grow` but must not record a failed one.
#[test]
fn a_failed_grow_fails_every_later_try_grow_of_the_task() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test overlaps with a_failed_spark_call_during_the_replay_is_not_recorded in spill_replay.rs. Both run the two pool types, fail merge.grow, check that the replay's try_grow fails without ResourcesExhausted, and check that reserved() does not change. This one adds the fair pool's limits and the repeated calls. Could the older test be removed, or extended with the new assertions, so the case is tested in one place? In the same way, a_failed_acquire_fails_every_later_try_acquire in spark_memory.rs repeats the overcommit and release assertions of failed_acquire_is_all_overcommit.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: removed the older replay test and the repeated overcommit and release checks.

for (name, task_attempt_id, pool_type) in [
("greedy_unified", -3009, MemoryPoolType::GreedyUnified),
("fair_unified", -3010, MemoryPoolType::FairUnified),
] {
let fake = spark_memory::fake::FakeSpark::with(1000);
let config = MemoryPoolConfig::new(pool_type, 1000);
let pool = create_pool(&config, task_attempt_id, &RunningTasks::default(), || {
fake.memory()
});
let sort = spillable("ExternalSorter[0]", &pool);
let refusal = sort.try_grow(2000).unwrap_err();
assert!(
matches!(refusal.find_root(), DataFusionError::ResourcesExhausted(_)),
"{name}: {refusal:?}"
);

// A final aggregate reading its spill files back holds the task's whole share, which
// Spark failed to grant.
let merge = spillable("FinalHashAggregateStream[0]", &pool);
let replay = merge.new_empty();
fake.throw_next_acquire();
merge.grow(1000);
assert_eq!(pool.reserved(), 1000, "{name}");
// The sort asks again, as DataFusion's does after spilling on any error.
for reservation in [&sort, &replay, &sort] {
let error = reservation.try_grow(1).unwrap_err();
assert!(fake.is_thrown(0, &error), "{name}: {error:?}");
}
assert_eq!(pool.reserved(), 1000, "{name}");

drop(merge);
drop(replay);
assert_eq!(pool.reserved(), 0, "{name}");
assert_eq!(fake.held(), 0, "{name}");
}
}

/// The tasks of an executor share one count, which `create_memory_pool` takes from
/// [`RunningTasks::of_executor`]. A pool created with a count of its own would see only its
/// own task and give it the whole pool. No other test counts tasks in the executor's count.
Expand Down
Loading