Repository navigation
Read a list or map column referenced by a null filter - #4400
Merged
Merged
Conversation
A scan filter that tests a list or map column for null added the column's own field ID to the projected IDs, and prune_columns rejects a list or map selected by its own ID. Project the column's nested fields instead.
Fokko
reviewed
Nov 17, 2026
| assert repr(result_table.schema) == "id: int32 not null" | ||
|
|
||
|
|
||
| def test_projection_null_filter_on_list_and_map(tmpdir: str) -> None: |
Contributor
There was a problem hiding this comment.
I think this test is hard to read, and it only tests a specific part of the code. How about having something similar as reported in the issue?
Contributor
Author
There was a problem hiding this comment.
Replaced it with a scan-level test that follows the issue: list, map and list-of-struct columns, is null and is not null, through a catalog. 9a74899
Fokko
approved these changes
Nov 17, 2026
Contributor
|
Thanks @gedaliakliger 🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4399
Rationale for this change
A scan whose
row_filtertests a list or map column for null (tags is null,props is not null) fails into_arrow()withValueError: Cannot explicitly project List or Map types.ArrowScan._projected_field_idsleaves list and map IDs out of the projected schema's IDs, becauseprune_columnsrebuilds those columns from their nested fields, but it adds the filter's field IDs as they are. A null test on a list or map column adds the column's own ID, andprune_columnsrejects it. This change projects the nested fields of such a column instead, the same way a selected list or map column is read.Are these changes tested?
Yes.
test_projection_null_filter_on_list_and_mapintests/io/test_pyarrow.pyrunsIsNullandNotNullon a list, a map and a list of structs, with the column selected and with onlyidselected. It fails onmainwith theValueErrorabove and passes with this change.Ran locally:
pytest tests/io/test_pyarrow.py -m unit, and theruff,ruff-formatandmypyhooks on the two changed files. The rest of the suite and the integration tests are left to CI.Are there any user-facing changes?
IS NULLandIS NOT NULLfilters on list and map columns return the matching rows instead of raising.