Skip to content

Fix row ID scan with a filter when a column is named __iceberg_file_position - #4439

Open
shashank11yadav wants to merge 1 commit into
apache:mainfrom
shashank11yadav:fix/position-column-collision
Open

shashank11yadav wants to merge 1 commit into
apache:mainfrom
shashank11yadav:fix/position-column-collision

Conversation

@shashank11yadav

Copy link
Copy Markdown

Fixes #4438

Rationale for this change

When a scan projects _row_id and has a row filter, _task_to_record_batches adds a temporary column of row positions to each batch. That column always gets the fixed name _POSITION_COLUMN (__iceberg_file_position). If the data file already has a column with that name, the table ends up with two fields of the same name. The lookup then fails with KeyError: 'Field "__iceberg_file_position" exists 2 times in schema'.

The fix picks a free name for the temporary column by putting _ in front until no data column has that name. _determine_partitions already avoids clashes with its _partition_<name> columns this way. When no data column has that name, the behaviour is the same as before. _POSITION_COLUMN itself is unchanged.

Are these changes tested?

Yes. The new test is test_task_to_record_batches_row_ids_with_position_column_name in tests/io/test_pyarrow.py. It writes a file with an id column and a data column named __iceberg_file_position, then reads it with _row_id projected and the filter id > 1. It checks that:

  • the filter is applied,
  • the data column comes back unchanged,
  • the row IDs are 101 and 102.

Without the fix, the test fails with the KeyError above. With the fix, it passes.

What was run:

  • pytest tests/io/test_pyarrow.py -m unit: 256 passed, 1 skipped.
  • ruff check (ruff 0.14.3): pass.
  • ruff format --check (ruff 0.14.3): pass.
  • mypy --config=pyproject.toml pyiceberg tests (mypy 1.18.2): pass.

Not run:

  • make lint (prek): its hooks need network access, which my environment doesn't have.
  • The full make test suite.

Are there any user-facing changes?

Scans that project _row_id with a row filter no longer fail when a data file has a column named __iceberg_file_position. No API changes. The bug only exists on main, and no released version has it.

…osition

When `_row_id` is projected together with a row filter, the reader adds a
temporary position column under a fixed name. A data column with that name
made the lookup fail with a duplicate field KeyError. Pick a free name for
the temporary column instead, as `_determine_partitions` does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scan with _row_id and a row filter raises KeyError when a data column is named __iceberg_file_position

1 participant