You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Performance regression from #3801 (unreleased): heap page reads copy every decompressed page #3967
Describe the bug, including details regarding any error messages, version, and platform.
PR #3801 (unreleased) fixes GH-3150 by copying every decompressed page in ColumnChunkPageReadStore. The copy protects against decompressors that return a view of an internal buffer and overwrite it on the next call, which the BytesInputDecompressor contract allows (the off-heap DirectCodecFactory does this). But on the default heap path, two cases return bytes that nothing overwrites, so for them the copy is pure overhead:
UNCOMPRESSED (NO_OP) returns the page as read from the file, which stays valid until the row group is closed.
ZSTD / LZ4_RAW (HeapBytesDecompressor) already decompress into a fresh array that only the reader holds.
For UNCOMPRESSED the copy also goes through Channels.newChannel, 8 KiB at a time, so each byte is copied twice.
1.18.2 vs 1.18.2 + #3801, page-read stage (readNextRowGroup + readPage + toInputStream, as in Spark), JMH, 10 forks:
Codec
Time
Allocation per scan
UNCOMPRESSED
+28.5% (95% CI +22 to +35)
107 → 214 MB
ZSTD
+9.6%
139 → 242 MB
SNAPPY
no change
no change
A Spark 4.3 scan shows the same +~105 MB per query.
Proposed fix a package-private CodecFactory.BytesDecompressor.returnsStableBytes() (default false), overridden by NO_OP and by HeapBytesDecompressor for ZSTD/LZ4_RAW. The reader skips the copy only for those. DirectCodecFactory and third-party decompressors are still copied. #3801's tests and reproduction pass, and allocation returns to the 1.18.2 baseline.
Worth settling before any 1.18.x backport of GH-3150. PR to follow.
Activity
apacheGH-3967: Skip the page copy for decompressors with stable output