You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parquet reader does not support unknown physical/logical type combinations per GH-607 #10941
parquet-format GH-607 adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an INT32 column annotated with UUID). This crate currently checks for unsupported combinations and errors when encountering them.
To Reproduce
Try reading the file added in apache/parquet-testing#122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).
Expected behavior
Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.
It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.
Additional context
We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.
Following up from the review on #11017: I will treat incompatible physical/logical combinations as _Unknown with sort order UNDEFINED, keep column statistics, and use the parquet-testing file from apache/parquet-testing#122. The reader option stays, with the default still returning an error.
Thank you @etseidl for clarifying this, and @alamb for the option guidance earlier.
readers should ignore both the logical type annotation and column order for that column.
I understood this as ignoring the FileMetaData::column_orders property for such columns. As far as I can see, that is actually not used inside arrow-rs itself, but might be used by datafusion or other consumers. We would need to indicate to such consumers that the column order and statistics of a column cannot be trusted, or postprocess the whole FileMetaData struct to update those column orders.
readers should ignore both the logical type annotation and column order for that column.
I understood this as ignoring the FileMetaData::column_orders property for such columns. As far as I can see, that is actually not used inside arrow-rs itself, but might be used by datafusion or other consumers. We would need to indicate to such consumers that the column order and statistics of a column cannot be trusted, or postprocess the whole FileMetaData struct to update those column orders.
Thanks @jhorstmann, I'd actually forgotten about the column_orders vec in the file metadata. This needs checking, but I think that what happens will depend on what's already in the thrift. If the column order for an unknown logical type is set to TYPE_ORDER, then when decoding the metadata that will get converted to the rust ColumnOrder::TypeDefined(SortOrder::UNDEFINED) variant. But if any of the newer column orders are used, then those will remain unchanged. I think we need to think through what we want to do here. I'm currently inclined to always set the column order to ColumnOrder::UNKNOWN, since that variant seems to be the one that indicates stats should be ignored (although I think UNDEFINED should also result in the same behavior).
I don't think datafusion currently looks at column_orders. Back when I was trying to fix NaN handling, @adriangb helped me with the plumbing to get the column order pushed down to the predicate handling code. At the time I don't think I found a single read of the column_order anywhere in Datafusion.
In the linked PR, I think the clearest signal to a user that something is amiss is setting the logical type on the column descriptor to _Unknown.
TBH I was happier when we could simply error on invalid combinations.
Activity
etseidl commentedon Sep 1, 2026
cc @alamb @jhorstmann @Jefffrey
alamb commentedon Sep 1, 2026
this would be my preference -- make an option to control how the reader will handle it
M-Tesla commentedon Sep 9, 2026
Following up from the review on #11017: I will treat incompatible physical/logical combinations as
_Unknownwith sort orderUNDEFINED, keep column statistics, and use the parquet-testing file from apache/parquet-testing#122. The reader option stays, with the default still returning an error.Thank you @etseidl for clarifying this, and @alamb for the option guidance earlier.
jhorstmann commentedon Sep 10, 2026
I understood this as ignoring the
FileMetaData::column_ordersproperty for such columns. As far as I can see, that is actually not used inside arrow-rs itself, but might be used by datafusion or other consumers. We would need to indicate to such consumers that the column order and statistics of a column cannot be trusted, or postprocess the wholeFileMetaDatastruct to update those column orders.etseidl commentedon Sep 11, 2026
Thanks @jhorstmann, I'd actually forgotten about the
column_ordersvec in the file metadata. This needs checking, but I think that what happens will depend on what's already in the thrift. If the column order for an unknown logical type is set toTYPE_ORDER, then when decoding the metadata that will get converted to the rustColumnOrder::TypeDefined(SortOrder::UNDEFINED)variant. But if any of the newer column orders are used, then those will remain unchanged. I think we need to think through what we want to do here. I'm currently inclined to always set the column order toColumnOrder::UNKNOWN, since that variant seems to be the one that indicates stats should be ignored (although I think UNDEFINED should also result in the same behavior).I don't think datafusion currently looks at column_orders. Back when I was trying to fix NaN handling, @adriangb helped me with the plumbing to get the column order pushed down to the predicate handling code. At the time I don't think I found a single read of the column_order anywhere in Datafusion.
In the linked PR, I think the clearest signal to a user that something is amiss is setting the logical type on the column descriptor to _Unknown.
TBH I was happier when we could simply error on invalid combinations.