Skip to content

OversizedAllocationException in building arrow vectors #12447

Description

@wecharyu

Description

When Gluten uses Arrow vectors for variable-width columns, large string/binary data can hit Arrow's 2GB limit and fail:

Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.shaded.org.apache.arrow.vector.util.OversizedAllocationException: Memory required for vector is (2147483648), which is overflow or more than max allowed (2147483647). You could consider using LargeVarCharVector/LargeVarBinaryVector for large strings/large bytes types
	at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.checkDataBufferSize(BaseVariableWidthVector.java:435)
	at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.reallocDataBuffer(BaseVariableWidthVector.java:542)
	at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.handleSafe(BaseVariableWidthVector.java:1304)
	at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:1101)
	at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setBytes(ArrowWritableColumnVector.java:1958)
	at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setValueNullSafe(ArrowWritableColumnVector.java:1970)
	at org.apache.gluten.vectorized.ArrowWritableColumnVector$ArrowVectorWriter.write(ArrowWritableColumnVector.java:1450)
	at org.apache.gluten.vectorized.ArrowWritableColumnVector.write(ArrowWritableColumnVector.java:776)
	at org.apache.gluten.vectorized.ArrowColumnarRow.writeRow(ArrowColumnarRow.java:335)
	at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:80)
	at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:33)
	at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3(ColumnarPartialProjectExec.scala:244)
	at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3$adapted(ColumnarPartialProjectExec.scala:242)

Gluten version

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions