You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Match partition values to the partition spec in deserialize_data_file_from_json #3651
Is your feature request related to a problem or challenge?
When Manifest::parse_avro reads a DataFile, it first matches the partition record's fields to the partition type by name (project_partition_by_name). deserialize_data_file_from_json skips that step. The record conversion then builds the struct in the JSON record's field order, with one value per JSON field. A record whose fields differ in order or number from the partition spec gives a struct that doesn't line up with the spec's fields.
Describe the solution you'd like
Call project_partition_by_name on the JSON path too, so both paths size and order the partition struct by the partition spec. This came out of the #3354 work and is listed in #3262.
Activity