Skip to content

Commit 1e7edb4

Browse files
committed
Use the full column name in the dynamic_partition_overwrite delete filter
Closes issue 4196 # Rationale for this change `dynamic_partition_overwrite` built its delete filter from `find_field(source_id).name`, which is only the leaf name. For a partition sourced from the nested column `s.x`, the filter referenced `x`, so: - with a top-level `x`, it deleted rows by the wrong column and kept the old rows of the replaced partition - without one, binding the filter raised `ValueError` The filter now uses `find_column_name`, the full column name, the same way `_determine_partitions` does since PR 2204. ## Are these changes tested? Yes. `test_dynamic_partition_overwrite_by_nested_source_column` in `tests/table/test_init.py` runs the example from the issue against the memory and SQL catalogs: - with this change, only partition `s.x = 5` is replaced: `[(5, 1), (9, 5)]` - without it, the test fails with `[(1, 5), (9, 5)]` ## Are there any user-facing changes? Yes, a bug fix: `dynamic_partition_overwrite` replaces the right partitions on tables partitioned by a nested column. Tables partitioned only by top-level columns behave as before. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change. All submitted content was reviewed by a human before submission. Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
1 parent 09960d5 commit 1e7edb4

2 files changed

Lines changed: 31 additions & 3 deletions

File tree

‎pyiceberg/table/__init__.py‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -643,9 +643,13 @@ def dynamic_partition_overwrite(
643643
)
644644

645645
partitions_to_overwrite = {data_file.partition for data_file in data_files}
646-
partitions_fields = [
647-
self.table_metadata.schema().find_field(field.source_id).name for field in self.table_metadata.spec().fields
648-
]
646+
schema = self.table_metadata.schema()
647+
partitions_fields = []
648+
for field in self.table_metadata.spec().fields:
649+
column_name = schema.find_column_name(field.source_id)
650+
if column_name is None:
651+
raise ValueError(f"Could not find column name for field ID: {field.source_id}")
652+
partitions_fields.append(column_name)
649653
delete_filter = self._build_partition_predicate(
650654
partition_records=partitions_to_overwrite, partition_fields=partitions_fields
651655
)

‎tests/table/test_init.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2489,6 +2489,30 @@ def test_partial_delete_keeps_earlier_deletes_outside_the_filter(catalog: Catalo
24892489
assert table.scan().to_arrow()["k"].to_pylist() == [3]
24902490

24912491

2492+
def test_dynamic_partition_overwrite_by_nested_source_column(catalog: Catalog) -> None:
2493+
"""Overwrite partitions by the nested source column s.x, not the top-level column x with the same name."""
2494+
import pyarrow as pa
2495+
2496+
catalog.create_namespace("default")
2497+
schema = Schema(
2498+
NestedField(1, "x", LongType(), required=False),
2499+
NestedField(2, "s", StructType(NestedField(3, "x", LongType(), required=False)), required=False),
2500+
)
2501+
spec = PartitionSpec(PartitionField(source_id=3, field_id=1000, transform=IdentityTransform(), name="s_x"))
2502+
table = catalog.create_table("default.nested_partition_overwrite", schema=schema, partition_spec=spec)
2503+
2504+
def to_arrow(*rows: tuple[int, int]) -> pa.Table:
2505+
return pa.Table.from_pylist([{"x": x, "s": {"x": s_x}} for x, s_x in rows], schema=schema.as_arrow())
2506+
2507+
# Rows are (x, s.x): one row in partition s.x = 5 and one in s.x = 1
2508+
table.append(to_arrow((1, 5), (5, 1)))
2509+
table.dynamic_partition_overwrite(to_arrow((9, 5)))
2510+
2511+
# Only partition s.x = 5 is replaced, so (1, 5) becomes (9, 5) and (5, 1) is kept
2512+
result = sorted((row["x"], row["s"]["x"]) for row in table.scan().to_arrow().to_pylist())
2513+
assert result == [(5, 1), (9, 5)]
2514+
2515+
24922516
def test_static_table_forwards_location_to_table_file_io(metadata_location: str, monkeypatch: pytest.MonkeyPatch) -> None:
24932517
seen_locations: list[str | None] = []
24942518
real_load_file_io = load_file_io

0 commit comments

Comments
 (0)