Skip to content

[C++][Parquet] Content-defined chunking of list columns depends on array chunking #55638

Closed

Description

@kszucs

Describe the bug, including details regarding any error messages, version, and platform.

With content-defined chunking, list columns written from a multi-chunk ChunkedArray lose the chunk boundaries that fall at the start of an array chunk, so the page layout depends on the in-memory chunking. Flat and struct columns are not affected.

import pyarrow as pa
import pyarrow.parquet as pq


def write(table):
    sink = pa.BufferOutputStream()
    cdc = {"min_chunk_size": 1024, "max_chunk_size": 4096}
    pq.write_table(table, sink, use_content_defined_chunking=cdc)
    return sink.getvalue()


rows = [[i, i] for i in range(100)]

# chunked at row 84
table = pa.table({"x": pa.chunked_array([rows[:84], rows[84:]])})
assert write(table) == write(table.combine_chunks())

# chunked at row 85, where a page starts
table = pa.table({"x": pa.chunked_array([rows[:85], rows[85:]])})
assert write(table) == write(table.combine_chunks())  # fails

The nested branch in cpp/src/parquet/chunker_internal.cc skips empty chunks (if (levels_to_write > 0)), so a boundary at offset 0 of a WriteArrow() call is dropped instead of closing the previous page.

Component(s)

C++, Parquet

Activity

  1. added a commit that references this issue on Nov 17, 2026
  2. added this to the 27.0.0 milestone on Nov 17, 2026
  3. pitrou commented on Nov 17, 2026

    @pitrou
    Member

    Issue resolved by pull request 55639
    #55639

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions