Repository navigation
GH-55638: [C++][Parquet] Fix CDC to make list page boundaries independent of array chunking - #55639
Conversation
…of list columns at array chunk starts The nested branch of the chunker skipped the empty chunk at offset 0 of a WriteArrow() call, which dropped the boundary instead of closing the previous page.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
2 open issues
What changed in this PR
Ensures content-defined chunking of nested/list columns preserves page boundaries across WriteArrow() calls so Parquet output is independent of in-memory array chunking.
Changes:
- Adjust nested (rep/def) chunk boundary emission to retain an empty chunk at offset 0 to close the previous call’s open page.
- Extend internal Parquet inspection to assert that no data pages are empty.
- Add a regression test comparing page boundaries between chunked vs combined-chunk tables.
| File | Description |
|---|---|
| cpp/src/parquet/chunker_internal.cc | Emits a chunk at record boundaries even when the first boundary is empty (offset 0) to preserve page closure semantics across calls. |
| cpp/src/parquet/chunker_internal_test.cc | Adds checks against empty data pages and introduces a test ensuring page boundaries are stable regardless of Arrow array chunking. |
Review effort: Lite
|
|
|
Can you clarify whether this a bugfix, an optimization, or anything else?
Not clear what "the empty chunk" refers to. |
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
|
@pitrou applied your suggestion and tried to clarify that it is a fix with an example. Sadly "chunk" is a fairly overloaded term. |
|
CI failures are unrelated. |
|
Thanks @pitrou! |


Rationale for this change
With content-defined chunking, list columns lose the chunk boundaries that fall at the start of a
WriteArrow()call, so their pages depend on how the data is chunked in memory.What changes are included in this PR?
CDC marks a page boundary at the start of an input array-chunk with an empty CDC-chunk. For list columns this empty CDC-chunk was dropped, so the boundary was lost. Now it is kept, like for non-list columns.
Are these changes tested?
Yes. A new test compares the pages of a chunked table and its combined version for each tested type, and the tests now check that no data page is empty.
Are there any user-facing changes?
No.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: