Skip to content

build: upgrade arrow-java to 20.0.0 - #7664

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:arrow-java-20-rc0
Draft

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:arrow-java-20-rc0

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #7362.

Rationale for this change

arrow-java 20.0.0 fixes several bugs Comet works around, listed in #7362. The fix that lets Comet drop code in this PR is apache/arrow-java#1119, which stops the zstd codec writing past its output buffer (#7305).

This is a draft until 20.0.0 is released. CI cannot resolve the 20.0.0 artifacts until then. I tested it locally against artifacts built from the v20.0.0-rc0 tag.

What changes are included in this PR?

  • arrow.version goes from 18.3.0 to 20.0.0.
  • arrow-compression now depends on at.yawk.lz4:lz4-java, a fork of lz4-java with the same net.jpountz classes as the org.lz4:lz4-java that Spark ships. The duplicate-class enforcer rule fails on it, so it is excluded like the other codec libraries Comet takes from Spark. Arrow only calls lz4 APIs that org.lz4 1.8.0 already has.
  • CometZstdCompressionCodec is removed. Arrow 20's ZstdCompressionCodec.doCompress is the same fix, so the cache uses Arrow's codec directly. CometZstdCompressionCodecSuite becomes ArrowZstdCompressionCodecSuite, and its guarded-allocator test now checks Arrow's codec. The test that showed the old defect and the test of the removed error helper are gone.
  • BitVectorHelper.getValidityBufferSize was removed upstream. Comet calls getValidityBufferSizeFromCount, which computes the same size.
  • setValueCount(0) on a large-offset vector now allocates the initial offset. The UtilsSuite test that covers a vector with no offset buffer now uses a freshly constructed vector instead.

User-facing change: a zstd failure while compressing a cached batch now surfaces as Arrow's RuntimeException rather than a Comet internal error. With the corrected capacity, zstd can only fail there by running out of memory.

The VectorLoader.loadBuffers leak in #7362 is still present in 20.0.0, so CachedBatchIpc.decompressed stays. The ArrowImporter cleanup for apache/arrow-java#1239 is left for a follow-up.

How are these changes tested?

Existing tests. With arrow-java built from v20.0.0-rc0 and installed locally, on macOS aarch64 with the default Spark 4.1 profile, these pass: UtilsSuite, the org.apache.spark.sql.comet.execution.arrow suites (including ArrowZstdCompressionCodecSuite), the native and columnar shuffle suites, CometShuffleInputConversionSuite, CometExecSuite, CometArrayExpressionSuite, the in-memory cache suites and the org.apache.comet.parquet suites. The S3 suites were not run because they need Docker. Other Spark profiles have not been run locally.

🤖 Generated with Claude Code

Arrow 20.0.0 ships the zstd capacity fix (apache/arrow-java#1119) that
CometZstdCompressionCodec backported, so the subclass goes and the cache
uses Arrow's ZstdCompressionCodec. Its guarded-allocator test now runs
against Arrow's codec.

arrow-compression now depends on the at.yawk.lz4 fork of lz4-java, which
has the same net.jpountz classes as the org.lz4 jar Spark ships, so it
is excluded like the other codec libraries.

BitVectorHelper.getValidityBufferSize was removed in favour of
getValidityBufferSizeFromCount. setValueCount(0) on a large-offset
vector now allocates the initial offset, so the UtilsSuite test that
covers a vector with no offset buffer uses a fresh vector instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added build Build environment enhancement New feature or request labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Build environment enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant