Skip to content

fix: fail the task when Spark throws during a native memory request - #7607

Open
JonSnow1807 wants to merge 10 commits into
apache:mainfrom
JonSnow1807:fix/7244-memory-errors
Open

JonSnow1807 wants to merge 10 commits into
apache:mainfrom
JonSnow1807:fix/7244-memory-errors

Conversation

@JonSnow1807

@JonSnow1807 JonSnow1807 commented Nov 11, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Part of #7244.

Rationale for this change

Two native memory paths treat an exception from Spark during a memory request as a refusal:

  • The multi-partition shuffle repartitioner spills on any try_grow error, so an exception (for example a failed spill of another consumer, or a killed task) makes it spill and carry on.
  • An infallible grow on the Spark-backed pools logs the failed call and books the size as overcommit, so the exception only reaches the log.

What changes are included in this PR?

  • multi_partition.rs: spill only when the error is a refusal (ResourcesExhausted, found with find_root, the same is_refusal as the Iceberg writer, now in datafusion-comet-common). Any other error fails the shuffle.
  • SparkMemory keeps the first request to Spark that fails, in grow or try_grow, and every later try_grow of the task then fails without asking Spark. Each of them fails with the exception Spark threw, so the task fails with it when the operator passes the error on unchanged.
  • The fair pool checks this under its lock, before its own limits, so its own refusal can’t hide the failure.
  • Once a failure is recorded, a later grow books its whole size as overcommit without asking Spark.
  • Overcommit accounting is unchanged, and grow stays infallible.
  • delta_dv.rs: only a refusal gets the size message. An exception from Spark keeps its class.
  • Notes in memory_management.md and native_shuffle.md.

The failure is kept rather than handed out once because DataFusion’s sort and multi-level merge take any try_grow error as a refusal and ask again, which would drop a one-time error.

One gap is left: a failed grow with no later try_grow in the task is still only logged. Closing it would mean checking for the failure when a plan’s stream ends in jni_api.rs, which changes the JNI plan lifecycle, so I left it out. I’ll do that in a follow-up. DataFusion’s sort-merge join also treats any try_grow error as a refusal, so it is covered by #7615.

How are these changes tested?

New unit tests, each failing on main and passing with this change:

  • pool_errors_other_than_refusal_fail_the_shuffle
  • a_failed_acquire_fails_every_later_try_acquire
  • a_failed_try_acquire_fails_every_later_one
  • a_failed_grow_fails_every_later_try_grow_of_the_task (greedy and fair unified pools)
  • attach_access_plans_fails_with_the_exception_spark_threw

a_refused_reservation_spills checks that a refusal, also when wrapped in Context or Shared, still spills. cargo test -p datafusion-comet-shuffle and cargo test -p datafusion-comet --lib pass, and clippy is clean.

The multi-partition shuffle repartitioner spilled on any error from
`try_grow`, so an exception Spark threw while spilling another consumer
made it spill and carry on. It now spills only on a `ResourcesExhausted`
refusal, found with `find_root` as the Iceberg writer does, and returns
any other error.

An infallible `grow` on the Spark-backed pools logged a failed call to
Spark and carried the whole size as overcommit, so the exception only
reached the log. `SparkMemory` now keeps the first request to Spark that
fails, in `grow` or `try_grow`, and every later `try_grow` of the task
fails without asking Spark: the first one after a failed `grow` with the
exception itself, the others with its message. The fair pool checks
this, under its lock, before its own limits, whose refusal would make an
operator spill and carry on. An operator that takes any error for a
refusal and asks again fails when it asks again.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design holds up: check_exception clears the pending Java exception, so the error has to travel back through the pool, and DataFusion's sort retries on any try_grow error, which is why the failure has to stay recorded. The state lives in the per-task SparkMemory, so other tasks are not affected. My questions are about how much of #7244 this closes, which exception the task ends with, and a duplicated helper.

Comment on lines +337 to +341
checks its own limits. A `grow` still records the full amount as overcommit, and when it was the
first to fail, the first `try_grow` after it is handed the exception itself. The others fail with
its message. That error is not `ResourcesExhausted`, so an operator that checks for a refusal,
such as the shuffle repartitioner, fails the task rather than spilling, and one that takes any
error for a refusal and asks again fails when it asks again.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description leaves one case open: a failed grow with no later try_grow in the task is still only logged, which is the second half of the expected behavior in #7244. Could this paragraph say so, or could the gap go under Open problems? As written it reads as if every exception now reaches the task. Also, Closes #7244 will close the issue on merge. If the end-of-stream check goes to a follow-up, Part of #7244 would keep it open.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The doc now says a failed grow with no later try_grow is still only logged, and the PR is now Part of #7244. The end-of-stream check will be a follow-up.


/// Fails once a request to Spark has failed: the first time after a failed [`Self::acquire`]
/// with its error, and otherwise with the message of the first request that failed.
pub(super) fn check_failure(&self) -> CometResult<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original exception goes only to the first try_grow after a failed grow. If that caller is ExternalSorter with batches in memory, it takes the error for a refusal, spills and asks again (reserve_memory_for_batch_and_maybe_spill in DataFusion 55.2). The task then fails with the DataFusionError::Execution message as a generic CometNativeException, where a JVM consumer would get Spark's SparkOutOfMemoryError from TaskMemoryManager.spillConsumer. A failed try_acquire never stores the throwable, so its retries end the same way. Could the throwable be kept and rethrown on every later failure? Global is not Clone in jni 0.22, so that needs an env, and I have not tried it. If that is too much for this PR, could the description's "the task fails with the exception Spark threw" be limited to operators that pass the first error on?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the throwable is now kept (in an Arc), so every later request fails with the same exception. A new Spark test shows the task now ends with SparkOutOfMemoryError instead of CometNativeException.

/// by Spark can also fail with an exception Spark threw, such as a failed spill of another
/// consumer or the interrupt of a killed task. That comes back as another kind of error and has
/// to fail the task. Spilling would hide it, and the task would carry on as if Spark had refused.
fn is_refusal(e: &DataFusionError) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has the same body as is_refusal in native/core/src/execution/operators/iceberg_write.rs. datafusion-comet and datafusion-comet-shuffle both depend on datafusion-comet-common. One pub fn is_refusal there would give both callers the same definition of a refusal, and its doc comment could explain the Spark exception case once.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: one pub fn is_refusal in datafusion-comet-common, used by both.

/// each pool type, also where the pool's own limits or the spill replay would have refused it
/// or recorded it past a refusal.
#[test]
fn a_failed_grow_fails_every_later_try_grow_of_the_task() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test overlaps with a_failed_spark_call_during_the_replay_is_not_recorded in spill_replay.rs. Both run the two pool types, fail merge.grow, check that the replay's try_grow fails without ResourcesExhausted, and check that reserved() does not change. This one adds the fair pool's limits and the repeated calls. Could the older test be removed, or extended with the new assertions, so the case is tested in one place? In the same way, a_failed_acquire_fails_every_later_try_acquire in spark_memory.rs repeats the overcommit and release assertions of failed_acquire_is_all_overcommit.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: removed the older replay test and the repeated overcommit and release checks.

The shuffle repartitioner and the native Iceberg writer each had a
private copy of the same predicate, which tells a pool's refusal
(`ResourcesExhausted`, under any wrapper) from an exception Spark threw.
Move it into `datafusion-comet-common`, which both crates depend on, and
explain the Spark exception case once in its doc comment.
Only the first `try_grow` after a failed `grow` was handed the exception
Spark threw, and a failed `try_grow` never kept it. Every later request
failed with its message as `DataFusionError::Execution`. DataFusion's
sort takes any error for a refusal, spills and asks again, so a sort
holding batches ended the task with a `CometNativeException` instead of
Spark's `SparkOutOfMemoryError`.

`CometError::JavaException` now holds its throwable in an `Arc`, since
`Global` is not `Clone`. `SparkMemory` keeps the first failure and fails
every later request with the same exception, which the JNI boundary
rethrows as itself, with no JNI call per request. A call that failed
without a Java exception still fails later requests with its message.

The pool tests now check that each later error carries the exception
thrown first. `a_failed_grow_fails_every_later_try_grow_of_the_task`
covers what `a_failed_spark_call_during_the_replay_is_not_recorded`
did, which is removed. A new Spark test runs a native sort in a task
whose other consumer fails its first spill, and checks that the task
fails with `SparkOutOfMemoryError` without making Spark spill again.
The paragraph on Spark exceptions in the unified pools read as if every
exception now reaches the task. A failed `grow` with nothing after it in
the task is still only logged, and an operator that wraps the error
turns it into a `CometNativeException`.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

I found no blocking concerns. The sticky failure in SparkMemory is a sound way to make an exception from Spark reach the task. I checked that check_failure runs at the top of try_acquire, so the greedy pool and the fair pool both fail every later try_grow without asking Spark, and that the fair pool does it under its lock before its own limits so a ResourcesExhausted from the pool cannot hide it. I also read the other try_grow and try_resize call sites under native/ (the Iceberg writer, merge_rows.rs, parquet_writer.rs, delta_dv.rs). None of them takes an arbitrary error for a refusal, so multi_partition.rs was the only caller that needed the is_refusal change. Moving is_refusal into datafusion-comet-common gives the shuffle and the Iceberg writer one definition.

Against branch-1.2 the behavior change is that a task now fails when Spark throws during a native memory request, where the release logged it (in grow) or spilled and carried on (in the shuffle repartitioner and in DataFusion's sort). That matches Spark, where a failed spill fails the task, so it reads as intended. The end-to-end test in CometExecIteratorLifecycleSuite is what shows the exception survives DataFusion's sort retry and reaches the JVM as SparkOutOfMemoryError, and the spills == 1 assertion pins that Spark is not asked again.

The remaining gap is a failed grow with no later try_grow in the task, which the PR describes and memory_management.md now states. Could the end-of-stream check have its own tracking issue so that gap stays visible while the PR is Part of #7244?

@JonSnow1807

Copy link
Copy Markdown
Author

Thanks for the review! I filed #7615 for the end-of-stream check.

@dwsmith1983 dwsmith1983 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the throwable and rethrowing the same object works well, and each new guard has a test that fails without it.

One case falls outside the two that the memory_management.md paragraph describes. DataFusion's sort-merge join takes any try_grow error as a refusal and never returns it: allocate_reservation in materializing_stream.rs matches Err(_), spills the batch and carries on, and bitwise_stream.rs does the same. If those are the only requests left after Spark throws, the task finishes with the exception only in the log, and every later buffered batch now spills. The end-of-stream check in #7615 should cover this too. Could the paragraph name it and link #7615 rather than #7244?

Two smaller points:

  • acquire still calls Spark after a failure is recorded, so a later grow can ask Spark to spill the consumer whose spill just failed. Should it skip the call and book the whole size as overcommit once failure is set?
  • delta_dv.rs maps any try_grow or try_resize error to a GeneralError that says the plan needs more than the execution memory pool. An exception from Spark there fails the task, but loses its class and reads as a size problem. Checking is_refusal first would keep it.

Once a request to Spark had failed, every later `try_grow` failed
without asking Spark, but a later `grow` still called Spark. That call
can make Spark spill the task's other consumers again, including the
one whose spill just failed, for a task that has to fail anyway.

`SparkMemory::acquire` now carries the whole size as overcommit without
asking Spark once a failure is recorded. Releases repay overcommit
before returning anything to Spark, so Spark is still never handed back
more than it granted.
…emory

The deletion vector access plan turned any error from `try_grow` or
`try_resize` into a `GeneralError` saying that the plan needs more than
the execution memory pool. An exception Spark threw there, such as a
failed spill of another consumer, still failed the task, but as a
`CometNativeException` that read as a size problem.

Only a refusal now gets that message, found with `is_refusal`. Any
other error is returned as it is, and a Java exception from Spark stays
`ExecutionError::JavaException`, which the JNI boundary rethrows as
itself.
DataFusion's sort-merge join takes any `try_grow` error for a refusal,
spills the batch and carries on (`allocate_reservation` in
`materializing_stream.rs`, and `bitwise_stream.rs`), so it never returns
an exception Spark threw. If its requests are the only ones left, the
task finishes with the exception only in the log, like a failed `grow`
with no later `try_grow`. Say so, and link apache#7615, which tracks the
end-of-stream check, instead of apache#7244.
@JonSnow1807

Copy link
Copy Markdown
Author

Thanks! All three are in 2e63e3f:

  • Docs: in DataFusion 55.2, allocate_reservation in materializing_stream.rs matches Err(_) and spills, and bitwise_stream.rs does the same, so neither returns the exception. The paragraph now names that case and links A failed native grow is only logged when the task makes no later try_grow #7615 instead.
  • acquire: once a failure is recorded, it books the whole size as overcommit without calling Spark, so a later grow no longer asks Spark, which could make Spark spill that consumer again. Releases still repay overcommit first, and a_failed_acquire_fails_every_later_try_acquire now checks that nothing reaches Spark.
  • delta_dv.rs: the size message is now used only when is_refusal holds, so an exception from Spark stays a JavaException. A new test checks this with a pool that throws.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JonSnow1807, the three follow-ups look right to me. Booking the whole size as overcommit once failure is set means Spark can't be asked to spill the consumer that just failed, and reservation_error keeps the same throwable, which the Arc::ptr_eq check in the new test pins nicely.

The CI (fork) check reports that there's no CI run on your fork for 2e63e3f1c. Could you enable Actions on the fork and push (or rebase on main) so the suites run? Since this touches the memory path every native plan goes through, I'd like to see it green before it merges.

@JonSnow1807

Copy link
Copy Markdown
Author

Thanks! I enabled Actions on the fork and merged main, and CI (fork) passed on 3f70dac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants