Description
When Gluten uses Arrow vectors for variable-width columns, large string/binary data can hit Arrow's 2GB limit and fail:
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.shaded.org.apache.arrow.vector.util.OversizedAllocationException: Memory required for vector is (2147483648), which is overflow or more than max allowed (2147483647). You could consider using LargeVarCharVector/LargeVarBinaryVector for large strings/large bytes types
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.checkDataBufferSize(BaseVariableWidthVector.java:435)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.reallocDataBuffer(BaseVariableWidthVector.java:542)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.handleSafe(BaseVariableWidthVector.java:1304)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:1101)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setBytes(ArrowWritableColumnVector.java:1958)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setValueNullSafe(ArrowWritableColumnVector.java:1970)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$ArrowVectorWriter.write(ArrowWritableColumnVector.java:1450)
at org.apache.gluten.vectorized.ArrowWritableColumnVector.write(ArrowWritableColumnVector.java:776)
at org.apache.gluten.vectorized.ArrowColumnarRow.writeRow(ArrowColumnarRow.java:335)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:80)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:33)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3(ColumnarPartialProjectExec.scala:244)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3$adapted(ColumnarPartialProjectExec.scala:242)
Gluten version
No response
Description
When Gluten uses Arrow vectors for variable-width columns, large string/binary data can hit Arrow's 2GB limit and fail:
Gluten version
No response