Skip to content

GH-3970: Match the file's list and map structure when converting an Avro requested projection - #3971

Open
shivadarshan-devadiga wants to merge 2 commits into
apache:masterfrom
shivadarshan-devadiga:GH-cli-nonstandard-lists
Open

shivadarshan-devadiga wants to merge 2 commits into
apache:masterfrom
shivadarshan-devadiga:GH-cli-nonstandard-lists

Conversation

@shivadarshan-devadiga

@shivadarshan-devadiga shivadarshan-devadiga commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

With an Avro requested projection, AvroReadSupport converts the projection to Parquet with a single list structure for the whole file: repeated <element> array, or the 3-level list / element names when auto-detection (#3816) applies. Maps always get key_value. A list or map that the file wrote with other repeated or element names is then missing from the projection's point of view, so it reads as null without an error. That covers 2-level lists, single-field groups such as bag or older PyArrow's list / item, standard lists in a file that also has one of those, and maps named map (MAP_KEY_VALUE). parquet-cli always reads with a projection, so cat, head, to-avro and convert print null for these columns.

What changes are included in this PR?

After converting the projection, AvroReadSupport.init walks it together with the file schema and the Avro projection. For each projected list it uses the file's repeated field, and the element inside it for 3-level lists, keeping the projected fields of record elements. For maps it does the same with the file's key-value group. It decides whether the repeated field is the element with AvroRecordConverter.isElementType, which is the check AvroCollectionConverter makes when reading, so the projection and the converters agree. A list or map whose file structure doesn't match the projection's shape (for example, a group element where the projection has a primitive) keeps the converted projection as before.

Only the names come from the file. Each element keeps its projected type, so the requested schema is still checked against the file, and a type that doesn't match fails with incompatible types as before. Its repetition is the one the converted projection gives it: repeated if the file's repeated field is the element, otherwise optional only if the Avro schema allows null. So files whose lists already match the converted projection get the same projection as before, and a non-null Avro element type still fails against optional elements in the file.

Are these changes tested?

Yes. New tests in TestArrayCompatibility read with a requested projection and the matching read schema:

  • testProjectionReadsTwoLevelListOfPrimitives: repeated binary element
  • testProjectionReadsSingleFieldGroupsWithOtherNames: bag / x and list / item, including a null element
  • testProjectionReadsMixedListStructures: a standard list and a list / item list in one file
  • testProjectionPrunesRecordsInNonStandardList: a record element with only one of its fields projected
  • testProjectionReadsMapWithOtherNames: map (MAP_KEY_VALUE)
  • testProjectionReadsNestedNonStandardLists: a list / item list whose elements are 2-level lists
  • testProjectionReadsMapOfNonStandardLists: a map whose values are bag lists, including a null element
  • testProjectionStillChecksElementTypes: an int32 list read with a string element type still fails with incompatible types
  • testProjectionKeepsNonNullElementsRequired: optional list / item elements read with a non-null element type still fail with incompatible types

All of these except testProjectionStillChecksElementTypes fail without this change: the column reads as null, or the expected incompatible types error doesn't happen. testProjectionStillChecksElementTypes guards against losing the type check, which the base already enforces. ReadCliTest.catReadsNonStandardListsWithoutAvroSchema checks parquet cat end to end, with and without --columns, for a 2-level list and a list / item list. It fails without the change too. All other parquet-avro and parquet-cli tests pass.

Are there any user-facing changes?

Yes. Reading with an Avro requested projection now returns the values of lists and maps written with non-standard repeated or element names, instead of null. This includes parquet-cli on such files.

Closes #3970

@shivadarshan-devadiga

shivadarshan-devadiga commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

@wgtmac Could you please approve the workflows and take a look when you have a moment? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parquet-avro: a requested projection reads lists and maps as null unless the file uses the projection's list structure

1 participant