Skip to content

Preserve schema properties when renaming columns - #3

Open
dossett wants to merge 2 commits into
preserve-renamed-column-types-basefrom
preserve-renamed-column-types
Open

dossett wants to merge 2 commits into
preserve-renamed-column-types-basefrom
preserve-renamed-column-types

Conversation

@dossett

@dossett dossett commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Rationale for this change

Renaming a column currently discards parts of its schema definition, including logical annotations, fixed-width lengths, field IDs, and column order. Losing decimal annotations can also change how page statistics are compared during recompression, producing incorrect minimum and maximum values.

Schema types should own the rules for copying their attributes. Keeping those rules beside the fields makes it easier to preserve future attributes without teaching every consumer how to reconstruct a type.

What changes are included in this PR?

Add withName(String) to Type, with covariant implementations in PrimitiveType, GroupType, and MessageType. Copy constructors preserve stored attributes directly, including legacy decimal metadata and message-root semantics. The base method is concrete to avoid adding an abstract requirement for existing subclasses.

Use this API for both the output schema and the types used to interpret statistics in ParquetRewriter. Columns without a rename entry retain their original types.

Are these changes tested?

Docker Maven verification passed: 310 selected unit tests and 17 integration tests, plus Spotless, RAT, and japicmp. Coverage includes schema/parser behavior, metadata conversion, and file rewrites with and without recompression. The two file rewrite regression cases fail against the original implementation. Six API cases cover attribute preservation, legacy metadata, groups, message roots, and null names.

Are there any user-facing changes?

Renaming preserves schema properties and statistics semantics. The schema classes gain a public withName operation.

This draft is for inspection in the fork. Its base branch snapshots upstream master at eae5345e, keeping the diff limited to the two fix commits without updating the fork's older master branch.

Copy complete primitive and group type definitions when renaming columns, preserving logical annotations, fixed widths, field IDs, and column order. Use the preserved types for page-statistics conversion so recompression keeps signed decimal bounds correct. Add focused copy and recompression regressions and clarify the rename contract.
@dossett
dossett marked this pull request as ready for review November 12, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant