Repository navigation
fix: fail the task when Spark throws during a native memory request - #7607
JonSnow1807 wants to merge 10 commits into
Conversation
The multi-partition shuffle repartitioner spilled on any error from `try_grow`, so an exception Spark threw while spilling another consumer made it spill and carry on. It now spills only on a `ResourcesExhausted` refusal, found with `find_root` as the Iceberg writer does, and returns any other error. An infallible `grow` on the Spark-backed pools logged a failed call to Spark and carried the whole size as overcommit, so the exception only reached the log. `SparkMemory` now keeps the first request to Spark that fails, in `grow` or `try_grow`, and every later `try_grow` of the task fails without asking Spark: the first one after a failed `grow` with the exception itself, the others with its message. The fair pool checks this, under its lock, before its own limits, whose refusal would make an operator spill and carry on. An operator that takes any error for a refusal and asks again fails when it asks again.
comphead
left a comment
There was a problem hiding this comment.
The design holds up: check_exception clears the pending Java exception, so the error has to travel back through the pool, and DataFusion's sort retries on any try_grow error, which is why the failure has to stay recorded. The state lives in the per-task SparkMemory, so other tasks are not affected. My questions are about how much of #7244 this closes, which exception the task ends with, and a duplicated helper.
| checks its own limits. A `grow` still records the full amount as overcommit, and when it was the | ||
| first to fail, the first `try_grow` after it is handed the exception itself. The others fail with | ||
| its message. That error is not `ResourcesExhausted`, so an operator that checks for a refusal, | ||
| such as the shuffle repartitioner, fails the task rather than spilling, and one that takes any | ||
| error for a refusal and asks again fails when it asks again. |
There was a problem hiding this comment.
The PR description leaves one case open: a failed grow with no later try_grow in the task is still only logged, which is the second half of the expected behavior in #7244. Could this paragraph say so, or could the gap go under Open problems? As written it reads as if every exception now reaches the task. Also, Closes #7244 will close the issue on merge. If the end-of-stream check goes to a follow-up, Part of #7244 would keep it open.
There was a problem hiding this comment.
Agreed. The doc now says a failed grow with no later try_grow is still only logged, and the PR is now Part of #7244. The end-of-stream check will be a follow-up.
|
|
||
| /// Fails once a request to Spark has failed: the first time after a failed [`Self::acquire`] | ||
| /// with its error, and otherwise with the message of the first request that failed. | ||
| pub(super) fn check_failure(&self) -> CometResult<()> { |
There was a problem hiding this comment.
The original exception goes only to the first try_grow after a failed grow. If that caller is ExternalSorter with batches in memory, it takes the error for a refusal, spills and asks again (reserve_memory_for_batch_and_maybe_spill in DataFusion 55.2). The task then fails with the DataFusionError::Execution message as a generic CometNativeException, where a JVM consumer would get Spark's SparkOutOfMemoryError from TaskMemoryManager.spillConsumer. A failed try_acquire never stores the throwable, so its retries end the same way. Could the throwable be kept and rethrown on every later failure? Global is not Clone in jni 0.22, so that needs an env, and I have not tried it. If that is too much for this PR, could the description's "the task fails with the exception Spark threw" be limited to operators that pass the first error on?
There was a problem hiding this comment.
Done: the throwable is now kept (in an Arc), so every later request fails with the same exception. A new Spark test shows the task now ends with SparkOutOfMemoryError instead of CometNativeException.
| /// by Spark can also fail with an exception Spark threw, such as a failed spill of another | ||
| /// consumer or the interrupt of a killed task. That comes back as another kind of error and has | ||
| /// to fail the task. Spilling would hide it, and the task would carry on as if Spark had refused. | ||
| fn is_refusal(e: &DataFusionError) -> bool { |
There was a problem hiding this comment.
This has the same body as is_refusal in native/core/src/execution/operators/iceberg_write.rs. datafusion-comet and datafusion-comet-shuffle both depend on datafusion-comet-common. One pub fn is_refusal there would give both callers the same definition of a refusal, and its doc comment could explain the Spark exception case once.
There was a problem hiding this comment.
Done: one pub fn is_refusal in datafusion-comet-common, used by both.
| /// each pool type, also where the pool's own limits or the spill replay would have refused it | ||
| /// or recorded it past a refusal. | ||
| #[test] | ||
| fn a_failed_grow_fails_every_later_try_grow_of_the_task() { |
There was a problem hiding this comment.
This test overlaps with a_failed_spark_call_during_the_replay_is_not_recorded in spill_replay.rs. Both run the two pool types, fail merge.grow, check that the replay's try_grow fails without ResourcesExhausted, and check that reserved() does not change. This one adds the fair pool's limits and the repeated calls. Could the older test be removed, or extended with the new assertions, so the case is tested in one place? In the same way, a_failed_acquire_fails_every_later_try_acquire in spark_memory.rs repeats the overcommit and release assertions of failed_acquire_is_all_overcommit.
There was a problem hiding this comment.
Done: removed the older replay test and the repeated overcommit and release checks.
The shuffle repartitioner and the native Iceberg writer each had a private copy of the same predicate, which tells a pool's refusal (`ResourcesExhausted`, under any wrapper) from an exception Spark threw. Move it into `datafusion-comet-common`, which both crates depend on, and explain the Spark exception case once in its doc comment.
Only the first `try_grow` after a failed `grow` was handed the exception Spark threw, and a failed `try_grow` never kept it. Every later request failed with its message as `DataFusionError::Execution`. DataFusion's sort takes any error for a refusal, spills and asks again, so a sort holding batches ended the task with a `CometNativeException` instead of Spark's `SparkOutOfMemoryError`. `CometError::JavaException` now holds its throwable in an `Arc`, since `Global` is not `Clone`. `SparkMemory` keeps the first failure and fails every later request with the same exception, which the JNI boundary rethrows as itself, with no JNI call per request. A call that failed without a Java exception still fails later requests with its message. The pool tests now check that each later error carries the exception thrown first. `a_failed_grow_fails_every_later_try_grow_of_the_task` covers what `a_failed_spark_call_during_the_replay_is_not_recorded` did, which is removed. A new Spark test runs a native sort in a task whose other consumer fails its first spill, and checks that the task fails with `SparkOutOfMemoryError` without making Spark spill again.
The paragraph on Spark exceptions in the unified pools read as if every exception now reaches the task. A failed `grow` with nothing after it in the task is still only logged, and an operator that wraps the error turns it into a `CometNativeException`.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
I found no blocking concerns. The sticky failure in SparkMemory is a sound way to make an exception from Spark reach the task. I checked that check_failure runs at the top of try_acquire, so the greedy pool and the fair pool both fail every later try_grow without asking Spark, and that the fair pool does it under its lock before its own limits so a ResourcesExhausted from the pool cannot hide it. I also read the other try_grow and try_resize call sites under native/ (the Iceberg writer, merge_rows.rs, parquet_writer.rs, delta_dv.rs). None of them takes an arbitrary error for a refusal, so multi_partition.rs was the only caller that needed the is_refusal change. Moving is_refusal into datafusion-comet-common gives the shuffle and the Iceberg writer one definition.
Against branch-1.2 the behavior change is that a task now fails when Spark throws during a native memory request, where the release logged it (in grow) or spilled and carried on (in the shuffle repartitioner and in DataFusion's sort). That matches Spark, where a failed spill fails the task, so it reads as intended. The end-to-end test in CometExecIteratorLifecycleSuite is what shows the exception survives DataFusion's sort retry and reaches the JVM as SparkOutOfMemoryError, and the spills == 1 assertion pins that Spark is not asked again.
The remaining gap is a failed grow with no later try_grow in the task, which the PR describes and memory_management.md now states. Could the end-of-stream check have its own tracking issue so that gap stays visible while the PR is Part of #7244?
|
Thanks for the review! I filed #7615 for the end-of-stream check. |
dwsmith1983
left a comment
There was a problem hiding this comment.
Keeping the throwable and rethrowing the same object works well, and each new guard has a test that fails without it.
One case falls outside the two that the memory_management.md paragraph describes. DataFusion's sort-merge join takes any try_grow error as a refusal and never returns it: allocate_reservation in materializing_stream.rs matches Err(_), spills the batch and carries on, and bitwise_stream.rs does the same. If those are the only requests left after Spark throws, the task finishes with the exception only in the log, and every later buffered batch now spills. The end-of-stream check in #7615 should cover this too. Could the paragraph name it and link #7615 rather than #7244?
Two smaller points:
acquirestill calls Spark after a failure is recorded, so a latergrowcan ask Spark to spill the consumer whose spill just failed. Should it skip the call and book the whole size as overcommit oncefailureis set?delta_dv.rsmaps anytry_growortry_resizeerror to aGeneralErrorthat says the plan needs more than the execution memory pool. An exception from Spark there fails the task, but loses its class and reads as a size problem. Checkingis_refusalfirst would keep it.
Once a request to Spark had failed, every later `try_grow` failed without asking Spark, but a later `grow` still called Spark. That call can make Spark spill the task's other consumers again, including the one whose spill just failed, for a task that has to fail anyway. `SparkMemory::acquire` now carries the whole size as overcommit without asking Spark once a failure is recorded. Releases repay overcommit before returning anything to Spark, so Spark is still never handed back more than it granted.
…emory The deletion vector access plan turned any error from `try_grow` or `try_resize` into a `GeneralError` saying that the plan needs more than the execution memory pool. An exception Spark threw there, such as a failed spill of another consumer, still failed the task, but as a `CometNativeException` that read as a size problem. Only a refusal now gets that message, found with `is_refusal`. Any other error is returned as it is, and a Java exception from Spark stays `ExecutionError::JavaException`, which the JNI boundary rethrows as itself.
DataFusion's sort-merge join takes any `try_grow` error for a refusal, spills the batch and carries on (`allocate_reservation` in `materializing_stream.rs`, and `bitwise_stream.rs`), so it never returns an exception Spark threw. If its requests are the only ones left, the task finishes with the exception only in the log, like a failed `grow` with no later `try_grow`. Say so, and link apache#7615, which tracks the end-of-stream check, instead of apache#7244.
|
Thanks! All three are in 2e63e3f:
|
andygrove
left a comment
There was a problem hiding this comment.
Thanks @JonSnow1807, the three follow-ups look right to me. Booking the whole size as overcommit once failure is set means Spark can't be asked to spill the consumer that just failed, and reservation_error keeps the same throwable, which the Arc::ptr_eq check in the new test pins nicely.
The CI (fork) check reports that there's no CI run on your fork for 2e63e3f1c. Could you enable Actions on the fork and push (or rebase on main) so the suites run? Since this touches the memory path every native plan goes through, I'd like to see it green before it merges.
|
Thanks! I enabled Actions on the fork and merged |
Which issue does this PR close?
Part of #7244.
Rationale for this change
Two native memory paths treat an exception from Spark during a memory request as a refusal:
try_growerror, so an exception (for example a failed spill of another consumer, or a killed task) makes it spill and carry on.growon the Spark-backed pools logs the failed call and books the size as overcommit, so the exception only reaches the log.What changes are included in this PR?
multi_partition.rs: spill only when the error is a refusal (ResourcesExhausted, found withfind_root, the sameis_refusalas the Iceberg writer, now indatafusion-comet-common). Any other error fails the shuffle.SparkMemorykeeps the first request to Spark that fails, ingrowortry_grow, and every latertry_growof the task then fails without asking Spark. Each of them fails with the exception Spark threw, so the task fails with it when the operator passes the error on unchanged.growbooks its whole size as overcommit without asking Spark.growstays infallible.delta_dv.rs: only a refusal gets the size message. An exception from Spark keeps its class.memory_management.mdandnative_shuffle.md.The failure is kept rather than handed out once because DataFusion’s sort and multi-level merge take any
try_growerror as a refusal and ask again, which would drop a one-time error.One gap is left: a failed
growwith no latertry_growin the task is still only logged. Closing it would mean checking for the failure when a plan’s stream ends injni_api.rs, which changes the JNI plan lifecycle, so I left it out. I’ll do that in a follow-up. DataFusion’s sort-merge join also treats anytry_growerror as a refusal, so it is covered by #7615.How are these changes tested?
New unit tests, each failing on main and passing with this change:
pool_errors_other_than_refusal_fail_the_shufflea_failed_acquire_fails_every_later_try_acquirea_failed_try_acquire_fails_every_later_onea_failed_grow_fails_every_later_try_grow_of_the_task(greedy and fair unified pools)attach_access_plans_fails_with_the_exception_spark_threwa_refused_reservation_spillschecks that a refusal, also when wrapped inContextorShared, still spills.cargo test -p datafusion-comet-shuffleandcargo test -p datafusion-comet --libpass, and clippy is clean.