Skip to content

fix: name the failed file in native Parquet read errors - #7228

Open
ump45nose wants to merge 4 commits into
apache:mainfrom
ump45nose:fix/name-failed-parquet-file
Open

ump45nose wants to merge 4 commits into
apache:mainfrom
ump45nose:fix/name-failed-parquet-file

Conversation

@ump45nose

@ump45nose ump45nose commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #7213. This fixes the corrupt-file case. The duplicate-field case from the issue is not changed here.

Rationale for this change

When the native Parquet scan can't read a file (for example a corrupt footer), the error reaches the JVM as CannotReadFile with an empty file_path. The JVM falls back to the task's file list, so the FAILED_READ_FILE.NO_HINT message names no file, or lists every file of the task. Spark names the file that failed.

What changes are included in this PR?

  • FileContext in file_error_context.rs already knows which file each planner and stream belongs to, and already fills the path for ParquetTimestampOverflow. It now also converts file read failures into CannotReadFile with that file's path, both when the file is opened (the planner and its pending planner, where the footer is read) and while its batches are read.
  • The classification is the one the JNI conversion already uses: a new file_read_error in jni-bridge/src/errors.rs wraps try_classify_file_read_error and returns only CannotReadFile. Missing files stay FileNotFound, Spark errors the reader raised itself (for example ParquetMissingFieldIds) keep their type, and bare I/O errors are still not treated as file reads.
  • The JVM fallback to the task's file list is unchanged and still covers read errors raised outside the scan. The comment above it in SparkErrorConverter.scala now says so instead of saying the native error carries a path only for object_store NotFound.

How are these changes tested?

New unit tests in file_error_context.rs:

  • read_failures_name_the_failed_file: a corrupt-footer ParquetError::General and a Context-wrapped EOF become CannotReadFile with the file's path, and the native message is kept.
  • other_errors_keep_their_type: ParquetMissingFieldIds, a missing file and a bare IoError are not turned into CannotReadFile.

CometTopKSuite's corrupt footer and page tests now also run with both files in one scan partition (LEAF_NODE_DEFAULT_PARALLELISM 1 and a 128 MB FILES_MAX_PARTITION_BYTES) and assert that the structured path names only the corrupt file. On main the JVM fallback would join both paths there. In that variant the corrupt file is the one with the smallest keys, so the TopK dynamic filter can't prune its row group.

I ran the parquet:: lib tests in the datafusion-comet crate (330 passed) and cargo clippy -p datafusion-comet -p datafusion-comet-jni-bridge --all-targets -- -D warnings. The datafusion-comet-jni-bridge lib tests have the same 11 JVM-backed failures on my machine with and without this change. I could not run the Spark/JVM suites locally, so the new CometTopKSuite cases rely on CI.

AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant.

@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading area:ffi Arrow FFI / JNI boundary labels Oct 26, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converting inside FileContext reuses the per-file identity that ParquetTimestampOverflow already relies on, and file_read_error keeps the classification in one place. One point on the end-to-end check you asked for. Separately, the comment above the fallback in SparkErrorConverter.scala (lines 89 to 92, outside this diff) still says the native error carries a path only for object_store NotFound, which the errors.rs doc change here retires on the native side.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

Naming the failing file in FileContext is the right place for this, and reusing file_read_error keeps the classification in one spot. I found one problem with how the converted error reaches the JVM.

with_file_path turns a DataFusionError::ParquetError into DataFusionError::External(Box<SparkError::CannotReadFile>). In jvm_exception in native/jni-bridge/src/errors.rs, an External SparkError is only recognised when it is the top-level error. parquet_external_spark_error and try_classify_file_read_error look through Context and Shared, but neither handles an External SparkError inside them. Before this change the ParquetError was classified however deep it sat in those wrappers, so a corrupt file raised FAILED_READ_FILE regardless of what the plan wrapped it in. After this change, a wrapped External(CannotReadFile) falls through to generic_exception and surfaces as a plain CometNativeException.

I checked this with a throwaway test in errors.rs that classifies External(CannotReadFile) bare, inside Context, and inside Shared. The bare error gives CometQueryExecutionException with errorType=CannotReadFile. The Context and Shared cases both give CometNativeException with no sparkError payload. DataFusion does produce these wrappers in plans Comet runs natively. OnceFut::try_once in joins/utils.rs returns DataFusionError::Shared for a hash join build side, and RepartitionExec::wait_for_task wraps the input error before sending it to every output partition. So a corrupt file under a native hash join build side or a repartition would have been reported as FAILED_READ_FILE on the release and would now be a generic native exception. The new CometTopKSuite cases would not catch this because the scan feeds the TopK directly.

Could jvm_exception look through Context and Shared for an External SparkError, so a SparkError the scan attaches survives those wrappers? That would also fix the same exposure for ParquetTimestampOverflow. A unit test in errors.rs that wraps an External(CannotReadFile) in Context and Shared and asserts the CannotReadFile payload would pin it. A JVM test with the corrupt file on the build side of a join would show it end to end.

with_file_path now attaches CannotReadFile as DataFusionError::External,
but jvm_exception only recognised an External SparkError at the top level.
A hash join build side (Shared) or a repartition (Context) wrapping the
scan error turned FAILED_READ_FILE into a generic CometNativeException.
Look through Context and Shared for an External SparkError or
SparkErrorWithContext before the file-read classification.
@ump45nose

Copy link
Copy Markdown
Contributor Author

@andygrove thanks, good catch. Fixed in d6671716.

  • jvm_exception now looks through Context and Shared for an External SparkError (or SparkErrorWithContext) before the file-read classification, via a small wrapped_external_spark_exception helper. A bare External still takes the existing top-level arm. This also covers ParquetTimestampOverflow and any other SparkError the scan attaches.
  • New unit test external_cannot_read_file_survives_context_and_shared in errors.rs: External(CannotReadFile) bare, in Context, in Shared, and in Shared(Context(..)) all classify as a Spark exception whose JSON carries CannotReadFile and the file path. It fails without the change (the wrapped cases fall to the generic exception) and passes with it.

Local: the errors::tests classification tests pass (15) and cargo clippy -p datafusion-comet-jni-bridge --all-targets -- -D warnings and cargo fmt --check are clean. I have not added the JVM join-build-side test yet: there is no JVM on the machine I used for this change, so I could not run it. I can add one if you think the unit test is not enough.

(AI-assisted change and reply.)

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ump45nose, wrapped_external_spark_exception is the fix I was hoping for, and the four cases in the new test pin it well. I'm happy with the unit test instead of a JVM join test.

This also changes errors outside the scan. Any SparkError a native operator wraps in Context or Shared, such as an ANSI overflow on a hash join build side, now reaches the JVM as its Spark exception rather than CometNativeException. That's closer to Spark, so I think it's a good change. Could the description say so? And could the test have one non-file case, for example an ArithmeticOverflow under Shared, so that wider behavior is pinned too?

CI (fork) also reports that Actions aren't enabled on your fork, so no CI has run on any commit here. Could you enable them and push again?

errors.rs: upstream now extracts a Spark payload through Context, Shared and
External wrappers (extract_spark_payload), so the local
wrapped_external_spark_exception/external_spark_exception pair is redundant
and dropped. file_read_error stays (the scan calls it to name the failed
file) and now gates on extract_spark_payload instead of the removed
parquet_external_spark_error helper. The regression test for a CannotReadFile
that a repartition or hash-join build side wraps is kept and passes on top of
upstream's implementation.
@ump45nose
ump45nose force-pushed the fix/name-failed-parquet-file branch from 381c552 to 41e8e6e Compare November 19, 2026 16:04
@ump45nose

Copy link
Copy Markdown
Contributor Author

Rebased the branch onto current main (d187e6823) so the PR is MERGEABLE again; only native/jni-bridge/src/errors.rs conflicted.

Conflict resolution. main now extracts a Spark payload through Context/Shared/External wrappers (extract_spark_payload), which covers what this branch added as wrapped_external_spark_exception/external_spark_exception, so those two helpers are gone. file_read_error (called by the scan to name the failed file) stays and now gates on extract_spark_payload instead of the removed parquet_external_spark_error. The regression test for a CannotReadFile wrapped by a repartition or hash-join build side is kept and passes on top of main's implementation. The file-read classification in the no-payload arm is unchanged.

Local verification (macOS arm64, rustc 1.100.0, JDK 21.0.13, protoc 36.2):

  • cargo test -p datafusion-comet --lib file_error_context — 3 passed (read_failures_name_the_failed_file, file_paths_stay_with_their_streams, other_errors_keep_their_type)
  • cargo test -p datafusion-comet --lib parquet:: — 347 passed, 1 ignored
  • cargo test -p datafusion-comet-jni-bridge --lib — 48 passed, 4 failed; the same 4 (error_from_panic, jint_array_panic_exception, jlong_panic_exception, shuffle_size_limit_survives_datafusion_wrappers_and_jni) fail identically on unmodified main here with assertion failed: env.exception_check(), i.e. they need the CI JVM fixture
  • cargo fmt --all --check and cargo clippy -p datafusion-comet-jni-bridge --lib -- -D warnings clean

Not verified locally: the Scala side (SparkErrorConverter.scala, CometTopKSuite.scala) — I did not run a Maven/Spark build, so those files are covered by CI only. CI on this head is waiting for maintainer approval, same as before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ffi Arrow FFI / JNI boundary area:scan Parquet scan / data reading bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants