Skip to content

Use field IDs when any Parquet column has one and prune the rest - #4419

Open
linzch3 wants to merge 2 commits into
apache:mainfrom
linzch3:fix/4417-partial-parquet-field-ids
Open

linzch3 wants to merge 2 commits into
apache:mainfrom
linzch3:fix/4417-partial-parquet-field-ids

Conversation

@linzch3

@linzch3 linzch3 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #4417

Rationale for this change

pyarrow_to_schema only used field IDs when every column had one (_HasIds used all). If any column lacked an ID, it ignored all IDs and fell back to the name mapping. That fails when physical names differ from the mapped names, which is what happens with Databricks UniForm files that attach IDs to data columns but not to row-tracking columns.

Java's reader treats a file as having IDs if any column has one (ParquetSchemaUtil.hasIds) and ignores the columns without one. This PR does the same, but only when reading data files.

pyarrow_to_schema gets a prune_fields_without_ids flag (default False), set only in _task_to_record_batches. When it's on, fields without an ID are dropped from the Arrow schema before it's converted, so an unsupported type in such a column (e.g. time32[ms]) doesn't break the read. A list whose element has no ID, and a map whose key or value has no ID, is dropped as a whole, since it can't be represented without it. If nothing has an ID, it still falls back to the name mapping as before.

_HasIds and _ConvertToIceberg are unchanged, so the write path keeps the old rule: an append with an extra column that has no ID is still rejected ("PyArrow table contains more columns") instead of silently dropping it.

Are these changes tested?

Yes, in tests/io/test_pyarrow_visitor.py:

  • the issue's example, with and without a name mapping
  • partial IDs without pruning still fall back to the name mapping (write path behavior)
  • a time32[ms] column without an ID is pruned before conversion
  • nested: a map missing child IDs is dropped, a struct keeps only the children with IDs
  • list element ID guard, and map key-only / value-only ID guards (the whole map is dropped)
  • no IDs at all still uses the name mapping

And in tests/io/test_pyarrow.py:

  • test_projection_partial_field_ids writes a Parquet file with one ID'd column and one ID-less time32[ms] column and reads it through ArrowScan
  • test_schema_mismatch_additional_field_without_id checks an append with an extra ID-less column is still rejected

make lint passed. pytest tests/ -m unit --ignore=tests/integration passed apart from 60 failures that fail the same way on main here (missing optional deps like pyiceberg-core, thrift, datafusion).

Are there any user-facing changes?

Reading files with IDs on only some columns now works by ID, and columns without IDs are ignored, even if a name mapping is present. Files with IDs on every column, or on none, are read as before. Writes are unchanged.

Limitations

  • On read, pruned columns are skipped silently (same as Java). They'd only be visible through the name mapping, which isn't used once the file has IDs.
  • A map with an ID only on its key (or only on its value) is dropped entirely on read rather than partially kept. On write it is still rejected as before.
  • A struct whose children all lack IDs is dropped too, even if the struct itself has one.

Happy to adjust the approach if you'd prefer something else.

_HasIds previously required every column to have an ID. A file with IDs
on only some columns, such as a Databricks UniForm table with
row-tracking columns, fell back to name mapping and failed when
physical names differed from the mapped names.

Match Java's ParquetSchemaUtil.hasIds (any) and prune fields without
IDs, including incomplete nested types.

@rahulsmahadev rahulsmahadev left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @linzch3! The read side matches Java's ParquetSchemaUtil.hasIds, and the new tests fail on main and pass here.

On the write-path change you mention in the description, I'd recommend keeping the old behavior, since it can lose data silently:

df = pq.read_table(path_of_a_pyiceberg_data_file).append_column("country", pa.array(["NL"]))
tbl.append(df)  # main: ValueError "PyArrow table contains more columns: country"; here: succeeds, country is lost

Even after union_by_name, as that error suggests, the append writes None for country, where main writes "NL". One option: appends keep the old rule (match by name or reject), and only reads of data files use the new one.

A couple of smaller suggestions:

  • A column without an ID is still type-converted before pruning, so an unsupported type such as time32[ms] fails the read; pruning first would avoid that.
  • Removing the element-ID guard in _ConvertToIceberg.list, or the key-ID check in map, still passes the tests, so a test for each could help.

@Fokko
Fokko self-requested a review November 18, 2026 22:50

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @linzch3! Checked out the branch and ran tests/io/test_pyarrow_visitor.py locally: all 63 pass, and the UniForm-style example with physical names that differ from the mapping is nicely covered.

Read side matching Java's ParquetSchemaUtil.hasIds makes sense to me, and I confirmed _ConvertToIcebergWithoutIDs inherits the pruning logic but always returns -1 IDs, so the create-table path is unchanged.

I agree with @rahulsmahadev on the write-path concern: silently dropping a column on append loses data without any signal, so I'd keep the old reject behavior there and only use the prune-on-read rule.

Nit: in _HasIds.map the any(...) means a map where only the key has an ID passes the check and the value is then dropped in _ConvertToIceberg.map. That is consistent with the new policy, just worth keeping in mind when reporting errors on the write path, since the user will not see why a column disappeared.

Review feedback: keep the old behavior on the write path, so an append
with an extra column that has no field ID is still rejected instead of
silently dropping that column.

_HasIds and _ConvertToIceberg are back to their previous behavior.
pyarrow_to_schema gets a prune_fields_without_ids flag, used only when
reading data files. It drops fields without IDs from the Arrow schema
before conversion, so an unsupported type in such a column (e.g.
time32[ms]) does not fail the read. Lists without an element ID and maps
without a key or value ID are dropped as a whole.

Add tests for the list element / map key / map value ID guards, the
pre-conversion pruning, a read of a file with partial IDs, and the
append rejection.
@linzch3

linzch3 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Hi, thx both for the careful review! Agreed on the write path, I kept the old reject behavior for appends and only prune when reading data files. Also prune before type conversion now so unsupported types in ID-less columns don't break reads, and added tests for the list element / map key ID guards. For the map nit, a map with an ID only on its key now gets dropped as a whole on read, and is still rejected on write. Pushed an update.

@dgokeeffe

Copy link
Copy Markdown

I tested this branch against the table from #4417, using the PR head (6862dd6) rebased onto main at f26689a. The rebase applied without conflicts.

The table is a Databricks Delta table with UniForm (icebergCompatV2), column mapping name and row tracking. I read it through the Unity Catalog Iceberg REST catalog with vended credentials (Azure, adlfs). The data file footer has six data columns named col-<uuid> with field IDs 1 to 6. It also has two int64 columns with no field ID, _row-id-col-<uuid> and _row-commit-version-col-<uuid>. schema.name-mapping.default maps IDs 1 to 6 to the logical names only.

I checked each result against a Databricks SQL warehouse reading the same table, comparing the row count and the sum of a double column:

Query main This PR
Full scan ValueError: Could not find field with name: col-<uuid> 296,906 rows, matches
Filter on a date column same error 9,905 rows, matches
Three-column projection same error 296,906 rows, matches

I also read three other tables: a UniForm v3 table, a managed Iceberg v2 table and a managed Iceberg v3 table. Each one reads the same on main and on this branch, and all of them match the SQL warehouse. Two have deletion vectors.

Local test results:

  • pytest tests/io/test_pyarrow_visitor.py tests/io/test_pyarrow.py -m unit: 320 passed, 3 skipped.
  • test_projection_partial_field_ids passes on this branch. Against main's pyiceberg/io/pyarrow.py it fails with Parquet file does not have field-ids and the Iceberg table does not have 'schema.name-mapping.default' defined.
  • pytest tests/ -m unit --ignore=tests/integration: main and this branch give the same failures and errors. Most come from optional dependencies I didn't install (flask, s3fs, google-auth, thrift and others). The rest are two error-message assertions in tests/expressions/test_parser.py that fail with pyparsing 3.3.3.

I used an AI coding assistant to run these checks and draft this comment. I reviewed the results before posting.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet file with field IDs on only some columns falls back to name mapping and fails

4 participants