Skip to content

Performance regression from #3801 (unreleased): heap page reads copy every decompressed page #3967

Open

Description

@NicolaeNMV

Describe the bug, including details regarding any error messages, version, and platform.

PR #3801 (unreleased) fixes GH-3150 by copying every decompressed page in ColumnChunkPageReadStore. The copy protects against decompressors that return a view of an internal buffer and overwrite it on the next call, which the BytesInputDecompressor contract allows (the off-heap DirectCodecFactory does this). But on the default heap path, two cases return bytes that nothing overwrites, so for them the copy is pure overhead:

  • UNCOMPRESSED (NO_OP) returns the page as read from the file, which stays valid until the row group is closed.
  • ZSTD / LZ4_RAW (HeapBytesDecompressor) already decompress into a fresh array that only the reader holds.

For UNCOMPRESSED the copy also goes through Channels.newChannel, 8 KiB at a time, so each byte is copied twice.

1.18.2 vs 1.18.2 + #3801, page-read stage (readNextRowGroup + readPage + toInputStream, as in Spark), JMH, 10 forks:

Codec Time Allocation per scan
UNCOMPRESSED +28.5% (95% CI +22 to +35) 107 → 214 MB
ZSTD +9.6% 139 → 242 MB
SNAPPY no change no change

A Spark 4.3 scan shows the same +~105 MB per query.

Proposed fix a package-private CodecFactory.BytesDecompressor.returnsStableBytes() (default false), overridden by NO_OP and by HeapBytesDecompressor for ZSTD/LZ4_RAW. The reader skips the copy only for those. DirectCodecFactory and third-party decompressors are still copied. #3801's tests and reproduction pass, and allocation returns to the 1.18.2 baseline.

Worth settling before any 1.18.x backport of GH-3150. PR to follow.

Component(s)

Core

Activity

  1. added a commit that references this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions