Skip to content

GH-636: Clarify encryption of DataPageV2 page bodies - #638

Closed
raviston wants to merge 1 commit into
apache:masterfrom
raviston:GH-636-pme-datapagev2
Closed

raviston wants to merge 1 commit into
apache:masterfrom
raviston:GH-636-pme-datapagev2

Conversation

@raviston

@raviston raviston commented Nov 11, 2026 •

Copy link
Copy Markdown

Rationale for this change

The Parquet Modular Encryption spec does not say whether the repetition and definition levels of a DataPageV2 are encrypted. Implementations disagree, and cannot read each other's encrypted V2 pages that have non-empty levels:

  • parquet-java (ColumnChunkPageWriteStore.writePageV2, since 1.12.0) writes the levels in plaintext and encrypts only the (compressed) values.
  • Arrow C++ (BuildDataPageV2) and arrow-rs (PageEncryptor::encrypt_compressed_page) encrypt the whole page body: levels and values together.

The current text supports both readings. Encryption.md says "the results of compression of column pages are encrypted", but V2 levels are never compressed. The PageHeader.crc doc describes the body as "levels, then values (optionally compressed, optionally encrypted)".

This PR proposes encrypting the entire page body, for these reasons:

  • The levels expose the structure of the data: where the nulls are and how long the lists are. Leaving them in plaintext weakens column encryption.
  • Two of the three main implementations already do it.
  • V1 pages and dictionary pages already work this way, so the rule is the same for every page type.
  • Few encrypted V2 files exist, because parquet-java and PyArrow write V1 pages by default.

What changes are included in this PR?

  • Encryption.md:
    • Section 3: the entire page body is encrypted as one module.
    • New section 5.1.1, "Column page modules":
      • Defines the plaintext of a page module for dictionary, V1 and V2 pages.
      • V2 levels are not compressed but are encrypted, and writers must not leave them in plaintext.
      • compressed_page_size is the on-disk size of the module, including the 4-byte length and the encryption overhead.
      • uncompressed_page_size and the V2 level byte lengths refer to the decrypted body.
      • The CRC is computed on the encrypted bytes.
    • New "Legacy data page V2 layout" subsection:
      • Describes the layout parquet-java has written. Writers must not produce it; readers may support it.
      • Gives a size check to tell the two layouts apart.
  • parquet.thrift: comment-only updates to DataPageHeaderV2 and PageHeader.crc to match.

Do these changes have PoC implementations?

Arrow C++ and arrow-rs already follow this behaviour. parquet-java needs a matching change to encrypt the levels on write and read the legacy layout on read. I'll open that as a follow-up.

Closes #636

Specify that the entire page body (repetition levels, definition levels
and values) is encrypted as a single page module, including for data
pages V2, where the levels are not compressed. Clarify the meaning of
compressed_page_size, the V2 level byte lengths and the CRC for
encrypted pages, and document the legacy layout written by parquet-java
(plaintext levels) so readers can stay backward compatible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@raviston

raviston commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

The parquet-java implementation is in apache/parquet-java#3947 (tracking issue apache/parquet-java#3946).

  • Writers now encrypt the levels and values of a V2 data page as one page module.
  • Readers detect and still read the legacy layout, as described in the "Legacy data page V2 layout" subsection of this PR.
  • A manual interop check with PyArrow 18.1.0 now passes in both directions. It fails with parquet-java 1.18.1.

@raviston

Copy link
Copy Markdown
Author

Closing in favour of #637, which was opened first and covers the same clarification. If it would help, I can propose the extra details from this PR on #637 instead: compressed_page_size including the encryption overhead, the CRC being computed after encryption, and a note on the legacy parquet-java layout. The parquet-java fix is going ahead in apache/parquet-java#3935.

@raviston raviston closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PME DataPageV2 Interop: Divergent encryption of rep/def levels between parquet-java and Arrow

2 participants