Repository navigation
Conversation
rowsPerPage divided the 1 MB page target by the sum of all column widths, and getTypeLength() is 0 for every primitive type except FIXED_LEN_BYTE_ARRAY, so each column was estimated at 32 bytes. On a 19-column schema that gave 1724 rows a page (~7 KB INT32 pages), and ZSTD, which compresses each page on its own, compressed them poorly. Size pages by the widest column, using each primitive type's PLAIN width. On a 19-column TPC-H lineitem-like table (512 MB of Arrow, ZSTD, 16384-row batches) this cuts the write from 2.57 s to 1.73 s and the file from 107.5 MB to 87.5 MB, the same size ExampleParquetWriter produces with dictionary encoding off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A small fix on top of apache#3734, against your branch so you can take it or fold it in.
writeBatchdivides the 1 MB page target by the sum of all column widths, andgetTypeLength()is 0 for every primitive type except FIXED_LEN_BYTE_ARRAY, so each column is estimated at 32 bytes. With 19 columns that gives 1724 rows a page: about 7 KB INT32 pages, and 590 pages in a column chunk whereExampleParquetWriterwrites 49. ZSTD compresses each page on its own, so it gets very little context.This sizes pages by the widest column instead, using each primitive type's PLAIN width (BOOLEAN 1, INT32/FLOAT 4, INT64/DOUBLE 8, INT96 12, FLBA its length, binary 32 as before).
TPC-H lineitem-like table, 19 nullable columns, 16384-row batches, ZSTD, built against parquet 1.18.2 on JDK 21:
The output now matches
ExampleParquetWriterwith dictionary encoding off (21.79 vs 21.74 MB at 128 MB).Tests:
testPageSizeDoesNotShrinkWithColumnCount: 20 INT32 columns x 100K rows should give one page a column. Without the fix it gives 62.testLargeBatchProducesMultiplePagesnow asserts the 2 pages its comment expects. Without the fix there are 16.All 11 tests in
TestArrowParquetWriterpass. Pages still end at eachwriteBatchcall, and a string-heavy schema would still benefit from cutting pages per column by actual bytes. I've left that out to keep this small.