Repository navigation
Fix row ID scan with a filter when a column is named __iceberg_file_position - #4439
Open
shashank11yadav wants to merge 1 commit into
Open
shashank11yadav wants to merge 1 commit into
shashank11yadav wants to merge 1 commit into
Conversation
…osition When `_row_id` is projected together with a row filter, the reader adds a temporary position column under a fixed name. A data column with that name made the lookup fail with a duplicate field KeyError. Pick a free name for the temporary column instead, as `_determine_partitions` does.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4438
Rationale for this change
When a scan projects
_row_idand has a row filter,_task_to_record_batchesadds a temporary column of row positions to each batch. That column always gets the fixed name_POSITION_COLUMN(__iceberg_file_position). If the data file already has a column with that name, the table ends up with two fields of the same name. The lookup then fails withKeyError: 'Field "__iceberg_file_position" exists 2 times in schema'.The fix picks a free name for the temporary column by putting
_in front until no data column has that name._determine_partitionsalready avoids clashes with its_partition_<name>columns this way. When no data column has that name, the behaviour is the same as before._POSITION_COLUMNitself is unchanged.Are these changes tested?
Yes. The new test is
test_task_to_record_batches_row_ids_with_position_column_nameintests/io/test_pyarrow.py. It writes a file with anidcolumn and a data column named__iceberg_file_position, then reads it with_row_idprojected and the filterid > 1. It checks that:Without the fix, the test fails with the KeyError above. With the fix, it passes.
What was run:
pytest tests/io/test_pyarrow.py -m unit: 256 passed, 1 skipped.ruff check(ruff 0.14.3): pass.ruff format --check(ruff 0.14.3): pass.mypy --config=pyproject.toml pyiceberg tests(mypy 1.18.2): pass.Not run:
make lint(prek): its hooks need network access, which my environment doesn't have.make testsuite.Are there any user-facing changes?
Scans that project
_row_idwith a row filter no longer fail when a data file has a column named__iceberg_file_position. No API changes. The bug only exists onmain, and no released version has it.