Repository navigation
Conversation
|
|
adamreeve
left a comment
There was a problem hiding this comment.
Thanks for this @Awesome06! I have a couple of questions
d61267d to
fd23951
Compare
|
@Awesome06 Please rewrite your PR description as a human writing for humans. |
|
@pitrou Kindly let me know the changes you expect in the PR description. As I'm not quite sure what to reduce or reword in the current description |
|
@Awesome06 I think my demand was quite self-descriptive. If you are a human, write the PR description yourself instead of pasting AI output. If you are not a human, please say so and this PR will be closed. |
|
Fixed the description to be simpler. I've updated the comments. Also updated other stale V1 references. I'm not a bot and thought that the PR described the changes properly |
|
One CI failure is related on MinGW and needs fixing: https://github.com/apache/arrow/actions/runs/43867507415/job/133299751333?pr=55453#step:13:4496 |
|
@pitrou fixes made. current failing tests seem to be due to infra and not the pr |
|
Thanks @Awesome06 . Indeed, these failures are unrelated. |
Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow). V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header. Closes apache#55272.
… masking and synchronize V2 page defaults
Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.
Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.
Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.
Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
…ion in TestSerialize and synchronize Sphinx doctest sizes Update TestSerialize in Parquet C++ to allow selective compression leaving small or incompressible pages uncompressed under V2 page defaults. Wrap comment to adhere to 90-char limit for clang-format. Synchronize column chunk total_compressed_size and total_uncompressed_size in parquet.rst Sphinx doctest for V2 page defaults.
…V2 data pages Update TestSerialize in Parquet C++ to check positivity of compressed and uncompressed sizes under V2 data pages rather than asserting strict ordering. Under V2, selective compression can leave data pages uncompressed, while unconditional compression of dictionary pages can expand small payloads, making total_compressed_size less than, equal to, or greater than total_byte_size.
…atistics with nulls Add TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength to verify unreferenced dictionary values are not included in statistics when null count causes unique count to equal dictionary length. Add TestArrowWriteDictionary.NestedDictionaryParentNullStatistics to verify that nested dictionary columns under nullable parents accurately reflect ancestor nulls in page and chunk statistics across both V1 and V2 data pages.
… nested dictionary test
…tionary reuse and selective compression - Refine dictionary reuse shortcut condition in TypedColumnWriterImpl to handle null indices. - Fix selective compression comments and assertions in TestSerialize. - Update min_compression_savings docstring to clarify V1 data pages. - Synchronize stale comments regarding V2 data page defaults across C++ and Python.
…V2 pages In Parquet V2, boolean columns do not use dictionary encoding, and small pages remain uncompressed under selective compression, yielding equal compressed and uncompressed sizes. For non-boolean types, unconditional dictionary page compression alters total_compressed_size even when data pages are uncompressed.
…V2 data pages In Parquet V2 data pages, selective compression leaves small or incompressible data pages uncompressed, while small dictionary pages compressed unconditionally can compress to smaller, larger (due to codec overhead), or identical sizes (e.g. Brotli compressing 200 bytes of floats to exactly 200 bytes). Update TestSerialize to assert positivity of compressed and uncompressed sizes under V2 rather than requiring total_compressed_size != total_byte_size.
e60d1ca to
e65fcd8
Compare
|
I'll merge if CI passes. |
Rationale for this change
Parquet V2 data pages have been supported in Apache Arrow for several years, offering key improvements including per-page selective compression (
min_compression_savings) and page-level null count headers. This PR switches the default data page format from V1 to V2 across C++ and PyArrow, while retaining full support for writing V1 pages when requested.What changes are included in this PR?
data_page_versionfromV1toV2inWriterProperties(C++) and PyArrow (parquet.write_table,ParquetWriter,ParquetFileWriteOptions).TypedColumnWriterImpl::WriteArrowDictionarysodictionaryis reused when all dictionary entries are referenced even if nulls are present in the chunk.type_fwd.handparquet/core.pydescribing V2 as an opt-in feature.Are these changes tested?
Yes. Existing Parquet C++ and PyArrow test suites cover V2 page writing and reading. New unit tests were added:
TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLengthTestArrowWriteDictionary.NestedDictionaryParentNullStatisticstest_min_compression_savings_default_v2Are there any user-facing changes?
Yes. Parquet files written with default writer properties will now emit V2 data pages rather than V1 data pages. Applications requiring V1 data pages for legacy readers can set
data_page_version="1.0".Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: