You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++][Parquet] Content-defined chunking of list columns depends on array chunking #55638
Describe the bug, including details regarding any error messages, version, and platform.
With content-defined chunking, list columns written from a multi-chunk ChunkedArray lose the chunk boundaries that fall at the start of an array chunk, so the page layout depends on the in-memory chunking. Flat and struct columns are not affected.
importpyarrowaspaimportpyarrow.parquetaspqdefwrite(table):
sink=pa.BufferOutputStream()
cdc= {"min_chunk_size": 1024, "max_chunk_size": 4096}
pq.write_table(table, sink, use_content_defined_chunking=cdc)
returnsink.getvalue()
rows= [[i, i] foriinrange(100)]
# chunked at row 84table=pa.table({"x": pa.chunked_array([rows[:84], rows[84:]])})
assertwrite(table) ==write(table.combine_chunks())
# chunked at row 85, where a page startstable=pa.table({"x": pa.chunked_array([rows[:85], rows[85:]])})
assertwrite(table) ==write(table.combine_chunks()) # fails
The nested branch in cpp/src/parquet/chunker_internal.cc skips empty chunks (if (levels_to_write > 0)), so a boundary at offset 0 of a WriteArrow() call is dropped instead of closing the previous page.
Activity
GH-55638: [C++][Parquet] Fix CDC to make list page boundaries indepen…
pitrou commentedon Nov 17, 2026
Issue resolved by pull request 55639
#55639