Repository navigation
Conversation
The native conversion returned each row's start in the batch buffer as an i32, so once a batch's rows took more than 2 GiB the offsets wrapped negative and the JVM read rows from before the buffer. Pass the offsets to the JVM as a long[], and check each row's length against Spark's MAX_ROUNDED_ARRAY_LENGTH, the limit DirectColumnarToRowConverter applies. Closes apache#7494.
|
Maybe I am beating a dead horse, but I still don't understand why we maintain this code path. It's not the default and it takes resources to maintain for a codepath that I still haven't seen a huge, justifiable win for.
I would love to find a real workload that benefits from the native code that we're keeping around and finding bugs in. |
I agree. I'll take one more look at the implementation and then plan to remove. |
Thanks @andygrove. I do think it’s an interesting implementation, I just wish we had a good argument for it. |
Which issue does this PR close?
Closes #7494.
Rationale for this change
With
spark.comet.exec.columnarToRow.native.enabled=true(off by default), the native conversion writes all of a batch's rows into one buffer and returned each row's start as ani32. Once a batch's rows took more than 2 GiB, the offsets wrapped negative without any error, and the JVM read the remaining rows from memory before the buffer. That gives garbage rows, or a SIGSEGV if the memory is unmapped. Every column can be well under Arrow's limit when this happens: two string columns averaging 150 KB per value take about 2.4 GB of rows in an 8192-row batch. Spark converts one row at a time, so it has no such limit.This is one of the offset overflow sites tracked in #7267.
What changes are included in this PR?
ColumnarToRowContextkeeps row offsets asi64, andcolumnarToRowConvertpasses them to the JVM as along[].NativeColumnarToRowInfo.offsetsis now along[], whichNativeRowIteratoradds to the buffer address as before.i32but are now checked. A row larger than Spark'sByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTHfails withRow too large for UnsafeRow conversion, the limitDirectColumnarToRowConverteralready applies. Spark can't build a row that large either.How are these changes tested?
test_row_offsets_past_2_gibconverts three rows of eight string columns that share one array, so the third row starts pasti32::MAX, and checks the offsets, the lengths and the third row's contents. It needs about 4 GB of memory, so it is#[ignore]d. It passes locally with--ignored, and fails when the offset is truncated to 32 bits again.test_checked_row_lengthchecks the length limit at and past the maximum, including sizes that a plainas i32cast would wrap. It fails when the check is removed.columnar_to_rowandjni_apitests pass, andCometNativeColumnarToRowSuite(26 tests, default Spark 4.1 profile) passes against a library built from this branch, which exercises the new JNI constructor signature.