Repository navigation
Raise on case-insensitive duplicate names in StructType.field_by_name - #4424
rahulsmahadev wants to merge 1 commit into
Conversation
field_by_name(case_sensitive=False) returned the first field on a struct with ID and id, so union_by_name silently matched a new id column to ID. Raise ValueError on any case-insensitive lookup of such a struct, as Java's caseInsensitiveField does, so union_by_name fails instead. Co-authored-by: Isaac <no-reply@databricks.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rahulsmahadev! Clear fix, and matching Java's error message is a nice touch.
Checked out the branch and ran the new tests locally (tests/test_types.py, plus the union behaviors on the catalog fixtures): all green, and the nested-struct collision case fails the union while leaving the table unchanged.
One non-blocking observation from reading around the change: top-level case-insensitive lookups still go through Schema._lazy_name_to_id_lower (pyiceberg/schema.py:159), which is a plain dict comprehension over _name_to_id, so with an ID/id schema, Schema.find_field(id, case_sensitive=False) and Schema.select(..., case_sensitive=False) still silently resolve to whichever field comes last. Same silent-wrong-column risk in a different path, could be a follow-up to make that index builder raise the same way.
Rationale for this change
Follow-up to #4389.
Bug:
StructType.field_by_name(name, case_sensitive=False)returns the first match when a struct has two fields that differ only by case, likeIDandid. So a case-insensitiveunion_by_namecan silently pick the wrong column:Fix: raise
ValueError: Cannot build lower case index: ID and id collide, the same message as #4389. Java fails here too:Types.StructType.caseInsensitiveFieldbuilds its index withImmutableMap.Builder, which throws on duplicate keys.Remaining differences from Java:
SchemaUpdate.applyChanges). PyIceberg doesn't, so for example a union that adds a new struct with bothIDandidstill succeeds.Are these changes tested?
Yes, added
test_struct_field_by_name_case_insensitive_duplicatesintests/test_types.pyandtest_union_by_name_case_insensitive_duplicatesintests/catalog/test_catalog_behaviors.py, which runs on the memory, sql and sql_without_rowcount catalogs. Both fail on main withDID NOT RAISE ValueErrorand pass with this change.Are there any user-facing changes?
Yes, case-insensitive lookups on such a struct now raise
ValueErrorinstead of returning the first field, and so does a case-insensitiveunion_by_namethat looks up a field in it. Case-sensitive lookups don't change.Please add the
changeloglabel.This pull request and its description were written by Isaac.