Skip to content

Size ArrowParquetWriter pages by the widest column - #1

Merged
qzyu999 merged 1 commit into
qzyu999:arrow-page-level-writerfrom
mikepapadim:arrow-writer-page-sizing
Nov 14, 2026
Merged

qzyu999 merged 1 commit into
qzyu999:arrow-page-level-writerfrom
mikepapadim:arrow-writer-page-sizing

Conversation

@mikepapadim

Copy link
Copy Markdown

A small fix on top of apache#3734, against your branch so you can take it or fold it in.

writeBatch divides the 1 MB page target by the sum of all column widths, and getTypeLength() is 0 for every primitive type except FIXED_LEN_BYTE_ARRAY, so each column is estimated at 32 bytes. With 19 columns that gives 1724 rows a page: about 7 KB INT32 pages, and 590 pages in a column chunk where ExampleParquetWriter writes 49. ZSTD compresses each page on its own, so it gets very little context.

This sizes pages by the widest column instead, using each primitive type's PLAIN width (BOOLEAN 1, INT32/FLOAT 4, INT64/DOUBLE 8, INT96 12, FLBA its length, binary 32 as before).

TPC-H lineitem-like table, 19 nullable columns, 16384-row batches, ZSTD, built against parquet 1.18.2 on JDK 21:

Arrow input before after
128 MB 700 ms, 26.8 MB 470 ms, 21.8 MB
512 MB 2568 ms, 107.5 MB 1800 ms, 87.5 MB

The output now matches ExampleParquetWriter with dictionary encoding off (21.79 vs 21.74 MB at 128 MB).

Tests:

  • New testPageSizeDoesNotShrinkWithColumnCount: 20 INT32 columns x 100K rows should give one page a column. Without the fix it gives 62.
  • testLargeBatchProducesMultiplePages now asserts the 2 pages its comment expects. Without the fix there are 16.

All 11 tests in TestArrowParquetWriter pass. Pages still end at each writeBatch call, and a string-heavy schema would still benefit from cutting pages per column by actual bytes. I've left that out to keep this small.

rowsPerPage divided the 1 MB page target by the sum of all column widths,
and getTypeLength() is 0 for every primitive type except
FIXED_LEN_BYTE_ARRAY, so each column was estimated at 32 bytes. On a
19-column schema that gave 1724 rows a page (~7 KB INT32 pages), and
ZSTD, which compresses each page on its own, compressed them poorly.

Size pages by the widest column, using each primitive type's PLAIN
width. On a 19-column TPC-H lineitem-like table (512 MB of Arrow, ZSTD,
16384-row batches) this cuts the write from 2.57 s to 1.73 s and the
file from 107.5 MB to 87.5 MB, the same size ExampleParquetWriter
produces with dictionary encoding off.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants