Repository navigation
Conversation
Arrow 20.0.0 ships the zstd capacity fix (apache/arrow-java#1119) that CometZstdCompressionCodec backported, so the subclass goes and the cache uses Arrow's ZstdCompressionCodec. Its guarded-allocator test now runs against Arrow's codec. arrow-compression now depends on the at.yawk.lz4 fork of lz4-java, which has the same net.jpountz classes as the org.lz4 jar Spark ships, so it is excluded like the other codec libraries. BitVectorHelper.getValidityBufferSize was removed in favour of getValidityBufferSizeFromCount. setValueCount(0) on a large-offset vector now allocates the initial offset, so the UtilsSuite test that covers a vector with no offset buffer uses a fresh vector instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #7362.
Rationale for this change
arrow-java 20.0.0 fixes several bugs Comet works around, listed in #7362. The fix that lets Comet drop code in this PR is apache/arrow-java#1119, which stops the zstd codec writing past its output buffer (#7305).
This is a draft until 20.0.0 is released. CI cannot resolve the
20.0.0artifacts until then. I tested it locally against artifacts built from thev20.0.0-rc0tag.What changes are included in this PR?
arrow.versiongoes from 18.3.0 to 20.0.0.arrow-compressionnow depends onat.yawk.lz4:lz4-java, a fork of lz4-java with the samenet.jpountzclasses as theorg.lz4:lz4-javathat Spark ships. The duplicate-class enforcer rule fails on it, so it is excluded like the other codec libraries Comet takes from Spark. Arrow only calls lz4 APIs thatorg.lz41.8.0 already has.CometZstdCompressionCodecis removed. Arrow 20'sZstdCompressionCodec.doCompressis the same fix, so the cache uses Arrow's codec directly.CometZstdCompressionCodecSuitebecomesArrowZstdCompressionCodecSuite, and its guarded-allocator test now checks Arrow's codec. The test that showed the old defect and the test of the removed error helper are gone.BitVectorHelper.getValidityBufferSizewas removed upstream. Comet callsgetValidityBufferSizeFromCount, which computes the same size.setValueCount(0)on a large-offset vector now allocates the initial offset. TheUtilsSuitetest that covers a vector with no offset buffer now uses a freshly constructed vector instead.User-facing change: a zstd failure while compressing a cached batch now surfaces as Arrow's
RuntimeExceptionrather than a Comet internal error. With the corrected capacity, zstd can only fail there by running out of memory.The
VectorLoader.loadBuffersleak in #7362 is still present in 20.0.0, soCachedBatchIpc.decompressedstays. TheArrowImportercleanup for apache/arrow-java#1239 is left for a follow-up.How are these changes tested?
Existing tests. With arrow-java built from
v20.0.0-rc0and installed locally, on macOS aarch64 with the default Spark 4.1 profile, these pass:UtilsSuite, theorg.apache.spark.sql.comet.execution.arrowsuites (includingArrowZstdCompressionCodecSuite), the native and columnar shuffle suites,CometShuffleInputConversionSuite,CometExecSuite,CometArrayExpressionSuite, the in-memory cache suites and theorg.apache.comet.parquetsuites. The S3 suites were not run because they need Docker. Other Spark profiles have not been run locally.🤖 Generated with Claude Code