Skip to content

GH-55272: [C++][Parquet][Python] Default to V2 data pages - #55453

Open
Awesome06 wants to merge 10 commits into
apache:mainfrom
Awesome06:gh-55272-parquet-default-v2-pages
Open

Awesome06 wants to merge 10 commits into
apache:mainfrom
Awesome06:gh-55272-parquet-default-v2-pages

Conversation

@Awesome06

@Awesome06 Awesome06 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Parquet V2 data pages have been supported in Apache Arrow for several years, offering key improvements including per-page selective compression (min_compression_savings) and page-level null count headers. This PR switches the default data page format from V1 to V2 across C++ and PyArrow, while retaining full support for writing V1 pages when requested.

What changes are included in this PR?

  • Changes default data_page_version from V1 to V2 in WriterProperties (C++) and PyArrow (parquet.write_table, ParquetWriter, ParquetFileWriteOptions).
  • Refines TypedColumnWriterImpl::WriteArrowDictionary so dictionary is reused when all dictionary entries are referenced even if nulls are present in the chunk.
  • Updates documentation and docstrings in C++ and Python to reflect V2 as the default data page version.
  • Fixes stale comments in type_fwd.h and parquet/core.py describing V2 as an opt-in feature.
  • Updates serialize test assertions and adds regression tests for dictionary statistics handling when nulls are present.
  • Updates TestSerialize size assertions to account for V2 selective compression and small dictionary page behavior across codecs.

Are these changes tested?

Yes. Existing Parquet C++ and PyArrow test suites cover V2 page writing and reading. New unit tests were added:

  • TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength
  • TestArrowWriteDictionary.NestedDictionaryParentNullStatistics
  • test_min_compression_savings_default_v2

Are there any user-facing changes?

Yes. Parquet files written with default writer properties will now emit V2 data pages rather than V1 data pages. Applications requiring V1 data pages for legacy readers can set data_page_version="1.0".

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Copilot AI balanced review requested due to automatic review settings November 9, 2026 12:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #55272 has been automatically assigned in GitHub to PR creator.

Copilot AI balanced review requested due to automatic review settings November 9, 2026 14:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 9, 2026 19:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@adamreeve adamreeve left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @Awesome06! I have a couple of questions

Comment thread cpp/src/parquet/column_writer.cc Resolved
Comment thread cpp/src/parquet/column_writer.cc Resolved
Copilot AI balanced review requested due to automatic review settings November 11, 2026 02:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Awesome06
Awesome06 force-pushed the gh-55272-parquet-default-v2-pages branch from d61267d to fd23951 Compare November 11, 2026 02:45
Copilot AI balanced review requested due to automatic review settings November 11, 2026 02:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 11, 2026 04:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@Awesome06 Please rewrite your PR description as a human writing for humans.

@Awesome06

Copy link
Copy Markdown
Contributor Author

@pitrou Kindly let me know the changes you expect in the PR description. As I'm not quite sure what to reduce or reword in the current description

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@Awesome06 I think my demand was quite self-descriptive. If you are a human, write the PR description yourself instead of pasting AI output.

If you are not a human, please say so and this PR will be closed.

Comment thread cpp/src/parquet/column_writer.cc Resolved
Comment thread cpp/src/parquet/column_writer.cc Resolved
Comment thread cpp/src/parquet/file_serialize_test.cc Resolved
Comment thread python/pyarrow/parquet/core.py Resolved
Copilot AI balanced review requested due to automatic review settings November 16, 2026 10:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Awesome06

Awesome06 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Fixed the description to be simpler. I've updated the comments. Also updated other stale V1 references.

I'm not a bot and thought that the PR described the changes properly

Copilot AI balanced review requested due to automatic review settings November 16, 2026 12:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

One CI failure is related on MinGW and needs fixing: https://github.com/apache/arrow/actions/runs/43867507415/job/133299751333?pr=55453#step:13:4496

Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Awesome06

Copy link
Copy Markdown
Contributor Author

@pitrou fixes made. current failing tests seem to be due to infra and not the pr

@pitrou

pitrou commented Nov 18, 2026

Copy link
Copy Markdown
Member

Thanks @Awesome06 . Indeed, these failures are unrelated.

Awesome06 and others added 10 commits November 18, 2026 10:32
Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow).

V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header.

Closes apache#55272.
… masking and synchronize V2 page defaults

Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.

Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.

Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.

Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
…ion in TestSerialize and synchronize Sphinx doctest sizes

Update TestSerialize in Parquet C++ to allow selective compression leaving small or incompressible pages uncompressed under V2 page defaults. Wrap comment to adhere to 90-char limit for clang-format.

Synchronize column chunk total_compressed_size and total_uncompressed_size in parquet.rst Sphinx doctest for V2 page defaults.
…V2 data pages

Update TestSerialize in Parquet C++ to check positivity of compressed and
uncompressed sizes under V2 data pages rather than asserting strict ordering.
Under V2, selective compression can leave data pages uncompressed, while
unconditional compression of dictionary pages can expand small payloads,
making total_compressed_size less than, equal to, or greater than
total_byte_size.
…atistics with nulls

Add TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength to verify unreferenced dictionary values are not included in statistics when null count causes unique count to equal dictionary length.

Add TestArrowWriteDictionary.NestedDictionaryParentNullStatistics to verify that nested dictionary columns under nullable parents accurately reflect ancestor nulls in page and chunk statistics across both V1 and V2 data pages.
…tionary reuse and selective compression

- Refine dictionary reuse shortcut condition in TypedColumnWriterImpl to handle null indices.

- Fix selective compression comments and assertions in TestSerialize.

- Update min_compression_savings docstring to clarify V1 data pages.

- Synchronize stale comments regarding V2 data page defaults across C++ and Python.
…V2 pages

In Parquet V2, boolean columns do not use dictionary encoding, and small pages remain uncompressed under selective compression, yielding equal compressed and uncompressed sizes. For non-boolean types, unconditional dictionary page compression alters total_compressed_size even when data pages are uncompressed.
…V2 data pages

In Parquet V2 data pages, selective compression leaves small or incompressible data pages uncompressed, while small dictionary pages compressed unconditionally can compress to smaller, larger (due to codec overhead), or identical sizes (e.g. Brotli compressing 200 bytes of floats to exactly 200 bytes). Update TestSerialize to assert positivity of compressed and uncompressed sizes under V2 rather than requiring total_compressed_size != total_byte_size.
@pitrou
pitrou force-pushed the gh-55272-parquet-default-v2-pages branch from e60d1ca to e65fcd8 Compare November 18, 2026 09:35
Copilot AI balanced review requested due to automatic review settings November 18, 2026 09:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@pitrou

pitrou commented Nov 18, 2026

Copy link
Copy Markdown
Member

I'll merge if CI passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Default to V2 data pages

6 participants