You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Encrypted DataPageV2 pages leave repetition/definition levels in plaintext, breaking interop with Arrow #3946
Describe the bug, including details regarding any error messages, version, and platform.
With Parquet Modular Encryption, parquet-java encrypts only the values of a DataPageV2 page. The repetition and definition levels are written in plaintext in front of the encrypted values (ColumnChunkPageWriteStore.writePageV2, since 1.12.0). Arrow C++ and arrow-rs encrypt the entire page body: levels and values together.
As a result, encrypted V2 pages with non-empty levels can't be read across implementations. The levels also stay unencrypted, and they reveal null positions and list lengths. Results with parquet-java 1.18.1 and PyArrow 18.1.0, for a file with an optional column and a list column:
Arrow writes, parquet-java reads: TagVerificationException: GCM tag check failed (AEADBadTagException: Tag mismatch!)
parquet-java writes, Arrow reads: OSError: Serialized ciphertext length 2495751 is greater than the provided ciphertext buffer length 44
This was reported against the format in apache/parquet-format#636. apache/parquet-format#638 proposes to specify that the whole page body is encrypted as one module, which matches Arrow C++ and arrow-rs. The proposal also documents parquet-java's current layout, so that readers can keep reading existing files.
Activity
raviston commentedon Nov 13, 2026
Duplicate of #3934.