Skip to content

GH-3710: Tolerate unrecognized logical/physical type combinations when reading - #3711

Merged
Fokko merged 7 commits into
apache:masterfrom
divjotarora:log-phys-type-combo
Sep 21, 2026
Merged

Fokko merged 7 commits into
apache:masterfrom
divjotarora:log-phys-type-combo

Conversation

@divjotarora

@divjotarora divjotarora commented Aug 6, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

See apache/parquet-format#607 for rationale.

What changes are included in this PR?

This PR modifies parquet-java to gracefully handle unrecognized logical/physical type combinations by dropping the logical type during the read and dropping any associated statistics for the relevant columns. Note that unrecognized logical types are already handled gracefully and no changes were required.

Are these changes tested?

Yes, several unit tests added.

Are there any user-facing changes?

No.

Closes #3710

@emkornfield emkornfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most important question is how we make this transition and an end-to-end test.

@divjotarora divjotarora left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@emkornfield I added an e2e test that reads a golden file with an INT32 column annotated with UUID. We can add this file to parquet-testing as part of this work as well.

@emkornfield emkornfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main concern is on the breadth of the exception cast.

@divjotarora

Copy link
Copy Markdown
Contributor Author

Thanks for the suggestions @RussellSpitzer, I've applied the new code structure now

@RussellSpitzer RussellSpitzer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me now! I left one note on the missing test case.

I'm on board with setup of

We tolerate wrong logical type on physical type except for those few cases we have already explicitly noted are bad (Decimal with the wrong precision)

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one nit, but this looks good to me 👍

@Fokko
Fokko merged commit 61a766b into apache:master Sep 21, 2026
5 checks passed
@divjotarora
divjotarora deleted the log-phys-type-combo branch September 21, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Specify handling for unrecognized logical/physical type combinations

4 participants