Skip to content

Encrypted DataPageV2 pages leave repetition/definition levels in plaintext, breaking interop with Arrow #3946

Closed as duplicate

Description

@raviston

Describe the bug, including details regarding any error messages, version, and platform.

With Parquet Modular Encryption, parquet-java encrypts only the values of a DataPageV2 page. The repetition and definition levels are written in plaintext in front of the encrypted values (ColumnChunkPageWriteStore.writePageV2, since 1.12.0). Arrow C++ and arrow-rs encrypt the entire page body: levels and values together.

As a result, encrypted V2 pages with non-empty levels can't be read across implementations. The levels also stay unencrypted, and they reveal null positions and list lengths. Results with parquet-java 1.18.1 and PyArrow 18.1.0, for a file with an optional column and a list column:

Arrow writes, parquet-java reads: TagVerificationException: GCM tag check failed (AEADBadTagException: Tag mismatch!)
parquet-java writes, Arrow reads: OSError: Serialized ciphertext length 2495751 is greater than the provided ciphertext buffer length 44

This was reported against the format in apache/parquet-format#636. apache/parquet-format#638 proposes to specify that the whole page body is encrypted as one module, which matches Arrow C++ and arrow-rs. The proposal also documents parquet-java's current layout, so that readers can keep reading existing files.

Component(s)

Core

Activity

  1. raviston commented on Nov 13, 2026

    @raviston
    Author

    Duplicate of #3934.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions