Repository navigation
Conversation
_HasIds previously required every column to have an ID. A file with IDs on only some columns, such as a Databricks UniForm table with row-tracking columns, fell back to name mapping and failed when physical names differed from the mapped names. Match Java's ParquetSchemaUtil.hasIds (any) and prune fields without IDs, including incomplete nested types.
There was a problem hiding this comment.
Thanks @linzch3! The read side matches Java's ParquetSchemaUtil.hasIds, and the new tests fail on main and pass here.
On the write-path change you mention in the description, I'd recommend keeping the old behavior, since it can lose data silently:
df = pq.read_table(path_of_a_pyiceberg_data_file).append_column("country", pa.array(["NL"]))
tbl.append(df) # main: ValueError "PyArrow table contains more columns: country"; here: succeeds, country is lostEven after union_by_name, as that error suggests, the append writes None for country, where main writes "NL". One option: appends keep the old rule (match by name or reject), and only reads of data files use the new one.
A couple of smaller suggestions:
- A column without an ID is still type-converted before pruning, so an unsupported type such as
time32[ms]fails the read; pruning first would avoid that. - Removing the element-ID guard in
_ConvertToIceberg.list, or the key-ID check inmap, still passes the tests, so a test for each could help.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @linzch3! Checked out the branch and ran tests/io/test_pyarrow_visitor.py locally: all 63 pass, and the UniForm-style example with physical names that differ from the mapping is nicely covered.
Read side matching Java's ParquetSchemaUtil.hasIds makes sense to me, and I confirmed _ConvertToIcebergWithoutIDs inherits the pruning logic but always returns -1 IDs, so the create-table path is unchanged.
I agree with @rahulsmahadev on the write-path concern: silently dropping a column on append loses data without any signal, so I'd keep the old reject behavior there and only use the prune-on-read rule.
Nit: in _HasIds.map the any(...) means a map where only the key has an ID passes the check and the value is then dropped in _ConvertToIceberg.map. That is consistent with the new policy, just worth keeping in mind when reporting errors on the write path, since the user will not see why a column disappeared.
Review feedback: keep the old behavior on the write path, so an append with an extra column that has no field ID is still rejected instead of silently dropping that column. _HasIds and _ConvertToIceberg are back to their previous behavior. pyarrow_to_schema gets a prune_fields_without_ids flag, used only when reading data files. It drops fields without IDs from the Arrow schema before conversion, so an unsupported type in such a column (e.g. time32[ms]) does not fail the read. Lists without an element ID and maps without a key or value ID are dropped as a whole. Add tests for the list element / map key / map value ID guards, the pre-conversion pruning, a read of a file with partial IDs, and the append rejection.
|
Hi, thx both for the careful review! Agreed on the write path, I kept the old reject behavior for appends and only prune when reading data files. Also prune before type conversion now so unsupported types in ID-less columns don't break reads, and added tests for the list element / map key ID guards. For the map nit, a map with an ID only on its key now gets dropped as a whole on read, and is still rejected on write. Pushed an update. |
|
I tested this branch against the table from #4417, using the PR head ( The table is a Databricks Delta table with UniForm ( I checked each result against a Databricks SQL warehouse reading the same table, comparing the row count and the sum of a double column:
I also read three other tables: a UniForm v3 table, a managed Iceberg v2 table and a managed Iceberg v3 table. Each one reads the same on Local test results:
I used an AI coding assistant to run these checks and draft this comment. I reviewed the results before posting. |
Closes #4417
Rationale for this change
pyarrow_to_schemaonly used field IDs when every column had one (_HasIdsusedall). If any column lacked an ID, it ignored all IDs and fell back to the name mapping. That fails when physical names differ from the mapped names, which is what happens with Databricks UniForm files that attach IDs to data columns but not to row-tracking columns.Java's reader treats a file as having IDs if any column has one (
ParquetSchemaUtil.hasIds) and ignores the columns without one. This PR does the same, but only when reading data files.pyarrow_to_schemagets aprune_fields_without_idsflag (defaultFalse), set only in_task_to_record_batches. When it's on, fields without an ID are dropped from the Arrow schema before it's converted, so an unsupported type in such a column (e.g.time32[ms]) doesn't break the read. A list whose element has no ID, and a map whose key or value has no ID, is dropped as a whole, since it can't be represented without it. If nothing has an ID, it still falls back to the name mapping as before._HasIdsand_ConvertToIcebergare unchanged, so the write path keeps the old rule: an append with an extra column that has no ID is still rejected ("PyArrow table contains more columns") instead of silently dropping it.Are these changes tested?
Yes, in
tests/io/test_pyarrow_visitor.py:time32[ms]column without an ID is pruned before conversionAnd in
tests/io/test_pyarrow.py:test_projection_partial_field_idswrites a Parquet file with one ID'd column and one ID-lesstime32[ms]column and reads it throughArrowScantest_schema_mismatch_additional_field_without_idchecks an append with an extra ID-less column is still rejectedmake lintpassed.pytest tests/ -m unit --ignore=tests/integrationpassed apart from 60 failures that fail the same way onmainhere (missing optional deps likepyiceberg-core,thrift,datafusion).Are there any user-facing changes?
Reading files with IDs on only some columns now works by ID, and columns without IDs are ignored, even if a name mapping is present. Files with IDs on every column, or on none, are read as before. Writes are unchanged.
Limitations
Happy to adjust the approach if you'd prefer something else.