Skip to content

fix: use 64-bit row offsets in native columnar-to-row conversion - #7504

Closed
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix/native-c2r-row-offsets
Closed

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix/native-c2r-row-offsets

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #7494.

Rationale for this change

With spark.comet.exec.columnarToRow.native.enabled=true (off by default), the native conversion writes all of a batch's rows into one buffer and returned each row's start as an i32. Once a batch's rows took more than 2 GiB, the offsets wrapped negative without any error, and the JVM read the remaining rows from memory before the buffer. That gives garbage rows, or a SIGSEGV if the memory is unmapped. Every column can be well under Arrow's limit when this happens: two string columns averaging 150 KB per value take about 2.4 GB of rows in an 8192-row batch. Spark converts one row at a time, so it has no such limit.

This is one of the offset overflow sites tracked in #7267.

What changes are included in this PR?

  • ColumnarToRowContext keeps row offsets as i64, and columnarToRowConvert passes them to the JVM as a long[]. NativeColumnarToRowInfo.offsets is now a long[], which NativeRowIterator adds to the buffer address as before.
  • Row lengths stay i32 but are now checked. A row larger than Spark's ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH fails with Row too large for UnsafeRow conversion, the limit DirectColumnarToRowConverter already applies. Spark can't build a row that large either.

How are these changes tested?

  • test_row_offsets_past_2_gib converts three rows of eight string columns that share one array, so the third row starts past i32::MAX, and checks the offsets, the lengths and the third row's contents. It needs about 4 GB of memory, so it is #[ignore]d. It passes locally with --ignored, and fails when the offset is truncated to 32 bits again.
  • test_checked_row_length checks the length limit at and past the maximum, including sizes that a plain as i32 cast would wrap. It fails when the check is removed.
  • The existing columnar_to_row and jni_api tests pass, and CometNativeColumnarToRowSuite (26 tests, default Spark 4.1 profile) passes against a library built from this branch, which exercises the new JNI constructor signature.

The native conversion returned each row's start in the batch buffer as an
i32, so once a batch's rows took more than 2 GiB the offsets wrapped
negative and the JVM read rows from before the buffer. Pass the offsets to
the JVM as a long[], and check each row's length against Spark's
MAX_ROUNDED_ARRAY_LENGTH, the limit DirectColumnarToRowConverter applies.

Closes apache#7494.
@github-actions github-actions Bot added bug Something isn't working area:ffi Arrow FFI / JNI boundary labels Nov 4, 2026
@mbutrovich

Copy link
Copy Markdown
Contributor

Maybe I am beating a dead horse, but I still don't understand why we maintain this code path. It's not the default and it takes resources to maintain for a codepath that I still haven't seen a huge, justifiable win for.

I would love to find a real workload that benefits from the native code that we're keeping around and finding bugs in.

@andygrove

Copy link
Copy Markdown
Member Author

Maybe I am beating a dead horse, but I still don't understand why we maintain this code path. It's not the default and it takes resources to maintain for a codepath that I still haven't seen a huge, justifiable win for.

I would love to find a real workload that benefits from the native code that we're keeping around and finding bugs in.

I agree. I'll take one more look at the implementation and then plan to remove.

@mbutrovich

Copy link
Copy Markdown
Contributor

I agree. I'll take one more look at the implementation and then plan to remove.

Thanks @andygrove. I do think it’s an interesting implementation, I just wish we had a good argument for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ffi Arrow FFI / JNI boundary bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offset overflow - Native columnar-to-row conversion wraps row offsets past 2 GiB

2 participants