Skip to content

GH-55638: [C++][Parquet] Fix CDC to make list page boundaries independent of array chunking - #55639

Merged
pitrou merged 3 commits into
apache:mainfrom
kszucs:cdc-list-chunk-boundary
Nov 17, 2026
Merged

pitrou merged 3 commits into
apache:mainfrom
kszucs:cdc-list-chunk-boundary

Conversation

@kszucs

@kszucs kszucs commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

With content-defined chunking, list columns lose the chunk boundaries that fall at the start of a WriteArrow() call, so their pages depend on how the data is chunked in memory.

What changes are included in this PR?

CDC marks a page boundary at the start of an input array-chunk with an empty CDC-chunk. For list columns this empty CDC-chunk was dropped, so the boundary was lost. Now it is kept, like for non-list columns.

content-defined boundary before D

WriteArrow([A B C])   CDC-chunks: [A B C]       page [A B C] stays open
WriteArrow([D E F])
  bug:                CDC-chunks: [D E F]       empty CDC-chunk dropped, D E F join page [A B C]
  fix:                CDC-chunks: [] [D E F]    empty CDC-chunk closes page [A B C]

pages (bug):  [A B C D E F]
pages (fix):  [A B C] [D E F]

Are these changes tested?

Yes. A new test compares the pages of a chunked table and its combined version for each tested type, and the tests now check that no data page is empty.

Are there any user-facing changes?

No.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

…of list columns at array chunk starts

The nested branch of the chunker skipped the empty chunk at offset 0 of a WriteArrow() call, which dropped the boundary instead of closing the previous page.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 21:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
What changed in this PR

Ensures content-defined chunking of nested/list columns preserves page boundaries across WriteArrow() calls so Parquet output is independent of in-memory array chunking.

Changes:

  • Adjust nested (rep/def) chunk boundary emission to retain an empty chunk at offset 0 to close the previous call’s open page.
  • Extend internal Parquet inspection to assert that no data pages are empty.
  • Add a regression test comparing page boundaries between chunked vs combined-chunk tables.
File Description
cpp/​src/​parquet/​chunker_internal.cc Emits a chunk at record boundaries even when the first boundary is empty (offset 0) to preserve page closure semantics across calls.
cpp/​src/​parquet/​chunker_internal_test.cc Adds checks against empty data pages and introduces a test ensuring page boundaries are stable regardless of Arrow array chunking.

Review effort: Lite

Comment thread cpp/src/parquet/chunker_internal.cc Resolved
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55638 has been automatically assigned in GitHub to PR creator.

@kszucs kszucs changed the title GH-55638: [C++][Parquet] Keep content-defined chunk boundaries of list columns at array chunk starts GH-55638: [C++][Parquet] Make CDC of list columns independent of array chunking Nov 17, 2026
Copilot AI balanced review requested due to automatic review settings November 17, 2026 08:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the fix to make Parquet list-column page boundaries independent of in-memory array chunking.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@pitrou

pitrou commented Nov 17, 2026

Copy link
Copy Markdown
Member

Can you clarify whether this a bugfix, an optimization, or anything else?

The nested branch of the chunker keeps the empty chunk at offset 0 of a call,

Not clear what "the empty chunk" refers to.

Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 08:56
@kszucs kszucs changed the title GH-55638: [C++][Parquet] Make CDC of list columns independent of array chunking GH-55638: [C++][Parquet] Fix CDC of list columns depending on array chunking Nov 17, 2026
@kszucs kszucs changed the title GH-55638: [C++][Parquet] Fix CDC of list columns depending on array chunking GH-55638: [C++][Parquet] Fix CDC to make list page boundaries independent of array chunking Nov 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change addressing chunking-dependent Parquet list-column page boundaries.

No open issues 🎉

Review effort: Balanced

@kszucs

kszucs commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

@pitrou applied your suggestion and tried to clarify that it is a fix with an example. Sadly "chunk" is a fairly overloaded term.

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks a lot @kszucs

@pitrou

pitrou commented Nov 17, 2026

Copy link
Copy Markdown
Member

CI failures are unrelated.

@pitrou
pitrou merged commit a092214 into apache:main Nov 17, 2026
54 of 59 checks passed
@kszucs

kszucs commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

Thanks @pitrou!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants