Repository navigation
Conversation
comphead
left a comment
There was a problem hiding this comment.
Converting inside FileContext reuses the per-file identity that ParquetTimestampOverflow already relies on, and file_read_error keeps the classification in one place. One point on the end-to-end check you asked for. Separately, the comment above the fallback in SparkErrorConverter.scala (lines 89 to 92, outside this diff) still says the native error carries a path only for object_store NotFound, which the errors.rs doc change here retires on the native side.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
Naming the failing file in FileContext is the right place for this, and reusing file_read_error keeps the classification in one spot. I found one problem with how the converted error reaches the JVM.
with_file_path turns a DataFusionError::ParquetError into DataFusionError::External(Box<SparkError::CannotReadFile>). In jvm_exception in native/jni-bridge/src/errors.rs, an External SparkError is only recognised when it is the top-level error. parquet_external_spark_error and try_classify_file_read_error look through Context and Shared, but neither handles an External SparkError inside them. Before this change the ParquetError was classified however deep it sat in those wrappers, so a corrupt file raised FAILED_READ_FILE regardless of what the plan wrapped it in. After this change, a wrapped External(CannotReadFile) falls through to generic_exception and surfaces as a plain CometNativeException.
I checked this with a throwaway test in errors.rs that classifies External(CannotReadFile) bare, inside Context, and inside Shared. The bare error gives CometQueryExecutionException with errorType=CannotReadFile. The Context and Shared cases both give CometNativeException with no sparkError payload. DataFusion does produce these wrappers in plans Comet runs natively. OnceFut::try_once in joins/utils.rs returns DataFusionError::Shared for a hash join build side, and RepartitionExec::wait_for_task wraps the input error before sending it to every output partition. So a corrupt file under a native hash join build side or a repartition would have been reported as FAILED_READ_FILE on the release and would now be a generic native exception. The new CometTopKSuite cases would not catch this because the scan feeds the TopK directly.
Could jvm_exception look through Context and Shared for an External SparkError, so a SparkError the scan attaches survives those wrappers? That would also fix the same exposure for ParquetTimestampOverflow. A unit test in errors.rs that wraps an External(CannotReadFile) in Context and Shared and asserts the CannotReadFile payload would pin it. A JVM test with the corrupt file on the build side of a join would show it end to end.
with_file_path now attaches CannotReadFile as DataFusionError::External, but jvm_exception only recognised an External SparkError at the top level. A hash join build side (Shared) or a repartition (Context) wrapping the scan error turned FAILED_READ_FILE into a generic CometNativeException. Look through Context and Shared for an External SparkError or SparkErrorWithContext before the file-read classification.
|
@andygrove thanks, good catch. Fixed in
Local: the (AI-assisted change and reply.) |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @ump45nose, wrapped_external_spark_exception is the fix I was hoping for, and the four cases in the new test pin it well. I'm happy with the unit test instead of a JVM join test.
This also changes errors outside the scan. Any SparkError a native operator wraps in Context or Shared, such as an ANSI overflow on a hash join build side, now reaches the JVM as its Spark exception rather than CometNativeException. That's closer to Spark, so I think it's a good change. Could the description say so? And could the test have one non-file case, for example an ArithmeticOverflow under Shared, so that wider behavior is pinned too?
CI (fork) also reports that Actions aren't enabled on your fork, so no CI has run on any commit here. Could you enable them and push again?
errors.rs: upstream now extracts a Spark payload through Context, Shared and External wrappers (extract_spark_payload), so the local wrapped_external_spark_exception/external_spark_exception pair is redundant and dropped. file_read_error stays (the scan calls it to name the failed file) and now gates on extract_spark_payload instead of the removed parquet_external_spark_error helper. The regression test for a CannotReadFile that a repartition or hash-join build side wraps is kept and passes on top of upstream's implementation.
381c552 to
41e8e6e
Compare
|
Rebased the branch onto current Conflict resolution. Local verification (macOS arm64, rustc 1.100.0, JDK 21.0.13, protoc 36.2):
Not verified locally: the Scala side ( |
Which issue does this PR close?
Part of #7213. This fixes the corrupt-file case. The duplicate-field case from the issue is not changed here.
Rationale for this change
When the native Parquet scan can't read a file (for example a corrupt footer), the error reaches the JVM as
CannotReadFilewith an emptyfile_path. The JVM falls back to the task's file list, so theFAILED_READ_FILE.NO_HINTmessage names no file, or lists every file of the task. Spark names the file that failed.What changes are included in this PR?
FileContextinfile_error_context.rsalready knows which file each planner and stream belongs to, and already fills the path forParquetTimestampOverflow. It now also converts file read failures intoCannotReadFilewith that file's path, both when the file is opened (the planner and its pending planner, where the footer is read) and while its batches are read.file_read_errorinjni-bridge/src/errors.rswrapstry_classify_file_read_errorand returns onlyCannotReadFile. Missing files stayFileNotFound, Spark errors the reader raised itself (for exampleParquetMissingFieldIds) keep their type, and bare I/O errors are still not treated as file reads.SparkErrorConverter.scalanow says so instead of saying the native error carries a path only for object_store NotFound.How are these changes tested?
New unit tests in
file_error_context.rs:read_failures_name_the_failed_file: a corrupt-footerParquetError::Generaland aContext-wrappedEOFbecomeCannotReadFilewith the file's path, and the native message is kept.other_errors_keep_their_type:ParquetMissingFieldIds, a missing file and a bareIoErrorare not turned intoCannotReadFile.CometTopKSuite's corrupt footer and page tests now also run with both files in one scan partition (LEAF_NODE_DEFAULT_PARALLELISM1 and a 128 MBFILES_MAX_PARTITION_BYTES) and assert that the structuredpathnames only the corrupt file. On main the JVM fallback would join both paths there. In that variant the corrupt file is the one with the smallest keys, so the TopK dynamic filter can't prune its row group.I ran the
parquet::lib tests in thedatafusion-cometcrate (330 passed) andcargo clippy -p datafusion-comet -p datafusion-comet-jni-bridge --all-targets -- -D warnings. Thedatafusion-comet-jni-bridgelib tests have the same 11 JVM-backed failures on my machine with and without this change. I could not run the Spark/JVM suites locally, so the newCometTopKSuitecases rely on CI.AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant.