Skip to content

Parquet reader does not support unknown physical/logical type combinations per GH-607 #10941

Description

@etseidl

Describe the bug

parquet-format GH-607 adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an INT32 column annotated with UUID). This crate currently checks for unsupported combinations and errors when encountering them.

To Reproduce

Try reading the file added in apache/parquet-testing#122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).

Expected behavior

Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.

It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.

Additional context

We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.

Activity

  1. added theissue type on Sep 1, 2026
  2. etseidl commented on Sep 1, 2026

    @etseidl
    ContributorAuthor
  3. alamb commented on Sep 1, 2026

    @alamb
    Contributor

    . We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.

    this would be my preference -- make an option to control how the reader will handle it

  4. M-Tesla commented on Sep 9, 2026

    @M-Tesla
    Contributor

    Following up from the review on #11017: I will treat incompatible physical/logical combinations as _Unknown with sort order UNDEFINED, keep column statistics, and use the parquet-testing file from apache/parquet-testing#122. The reader option stays, with the default still returning an error.

    Thank you @etseidl for clarifying this, and @alamb for the option guidance earlier.

  5. jhorstmann commented on Sep 10, 2026

    @jhorstmann
    Contributor

    readers should ignore both the logical type annotation and column order for that column.

    I understood this as ignoring the FileMetaData::column_orders property for such columns. As far as I can see, that is actually not used inside arrow-rs itself, but might be used by datafusion or other consumers. We would need to indicate to such consumers that the column order and statistics of a column cannot be trusted, or postprocess the whole FileMetaData struct to update those column orders.

  6. etseidl commented on Sep 11, 2026

    @etseidl
    ContributorAuthor

    readers should ignore both the logical type annotation and column order for that column.

    I understood this as ignoring the FileMetaData::column_orders property for such columns. As far as I can see, that is actually not used inside arrow-rs itself, but might be used by datafusion or other consumers. We would need to indicate to such consumers that the column order and statistics of a column cannot be trusted, or postprocess the whole FileMetaData struct to update those column orders.

    Thanks @jhorstmann, I'd actually forgotten about the column_orders vec in the file metadata. This needs checking, but I think that what happens will depend on what's already in the thrift. If the column order for an unknown logical type is set to TYPE_ORDER, then when decoding the metadata that will get converted to the rust ColumnOrder::TypeDefined(SortOrder::UNDEFINED) variant. But if any of the newer column orders are used, then those will remain unchanged. I think we need to think through what we want to do here. I'm currently inclined to always set the column order to ColumnOrder::UNKNOWN, since that variant seems to be the one that indicates stats should be ignored (although I think UNDEFINED should also result in the same behavior).

    I don't think datafusion currently looks at column_orders. Back when I was trying to fix NaN handling, @adriangb helped me with the plumbing to get the column order pushed down to the predicate handling code. At the time I don't think I found a single read of the column_order anywhere in Datafusion.

    In the linked PR, I think the clearest signal to a user that something is amiss is setting the logical type on the column descriptor to _Unknown.

    TBH I was happier when we could simply error on invalid combinations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions