You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support content-defined chunking (CDC) in the Parquet writer #3817
Add content-defined chunking (CDC) to the Parquet writer. It is already available in Parquet C++ (apache/arrow#45750, PyArrow >= 21) and parquet-rs (apache/arrow-rs#9450). With CDC, data page boundaries come from a rolling hash of the column values rather than fixed sizes. Pages then stay stable when rows are inserted or appended, so content-addressable storage (e.g. Hugging Face Xet) can deduplicate them. The output is a regular Parquet file, with no format change.
It would help to use the same parameters as the other implementations:
min_chunk_size (default 256 KiB)
max_chunk_size (default 1 MiB)
norm_level (default 0)
Motivation: Iceberg is adding write.parquet.content-defined-chunking.* table properties (apache/iceberg-rust#2375, apache/iceberg-python#3889). Iceberg Java cannot honor them until parquet-java supports CDC.
Activity
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Decompress empty pages with the direct codecs
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Simplify content defined chunking and align it with Arrow
apacheGH-3817: Simplify content defined chunking and align it with Arrow
apacheGH-3817: Simplify content defined chunking and align it with Arrow
apacheGH-3817: Simplify content defined chunking and align it with Arrow
apacheGH-3817: Simplify content defined chunking and align it with Arrow
apacheGH-3817: Simplify content defined chunking and align it with Arrow
20 remaining items
apacheGH-3817: Assert exactly which pages an edit changes
apacheGH-3817: Chunk only, and leave the page limits as they are
apacheGH-3817: Rename CdcOptions to ContentDefinedChunkingOptions
apacheGH-3817: Test the chunking against a reference of Arrow C++
apacheGH-3817: Hash INT(8) and INT(16) values at the width Arrow hold…
apacheGH-3817: Reject a too narrow chunk size range when the options …
apacheGH-3817: Chunk only, and test as Arrow C++ does
apacheGH-3817: Content defined chunking of data pages
apacheGH-3817: Simplify content defined chunking and align it with Arrow
apacheGH-3817: Chunk in the column writer, as Arrow C++ does
apacheGH-3817: Ask the chunker one question per value
apacheGH-3817: Test that the page size check schedule moves no page b…
apacheGH-3817: Pass the chunker through the existing column writer co…
apacheGH-3817: Pass a missing chunker as null, like a missing bloom f…
apacheGH-3817: Chunk only, and test as Arrow C++ does