Skip to content

Fixture: Add reference table input for a data file with partial field IDs - #27

Open
slachiewicz wants to merge 2 commits into
apache:mainfrom
slachiewicz:partial-field-ids-table
Open

slachiewicz wants to merge 2 commits into
apache:mainfrom
slachiewicz:partial-field-ids-table

Conversation

@slachiewicz

@slachiewicz slachiewicz commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Adds v2/partial-field-ids-parquet-data: a table with schema.name-mapping.default set and one Parquet data file that carries field IDs on its top-level columns and struct fields but not on its list element, map key or map value. It is the companion to #14 for the case the dev@ thread is discussing.

Following CONTRIBUTING's "Cases the spec has not settled", the PR holds the input only: no expected/scan.json and no index.json entry. The table README names the open question and states, as prose, what the Column Projection section already fixes. It stays a draft until the thread settles.

The first commit lets GenerateTables.java set table properties and write a data file with its own schema through parquet-mr, since Iceberg's writer always writes every field ID.

Current readers differ on this file:

  • Iceberg Java 1.11.0 (IcebergGenerics) reads fields 1, 2, 3, 7, 8 and 6 as the README states, and fails with "is not in the store" on any projection that includes tags or props.
  • iceberg-go main at 62728072 fails on every projection, including id, name, location, added, with "cannot convert element ... missing field_id". Its hasIDs check in table/arrow_utils.go treats the file as fully ID'd because it does not look below a field that has an ID.
  • PyIceberg 0.12.0 (its _HasIds check is unchanged on main at 33b4ce51) fails on every projection with "Could not find field with name: c4.element". It finds a field without an ID, applies the name mapping to the whole file, and the mapping does not list the c* names.
  • DuckDB 2.0.0 with the iceberg extension reads fields 1, 2, 3 and 6 as the README states, reads tags as lists of the right length with every element null, and fails on props because the required key (field 10) "is missing from the file schema".

So the four readers give four different results, and only two of them agree with the values the README says the spec already fixes.

A commit's file-schema is written with parquet-mr so a data file can carry column names, order and field IDs the table schema does not, including none on chosen fields; Iceberg's writer always writes every ID.
@slachiewicz
slachiewicz force-pushed the partial-field-ids-table branch from b79434a to 9542d52 Compare November 19, 2026 20:45
@slachiewicz slachiewicz changed the title Add reference table input for a data file with partial field IDs Fixture: Add reference table input for a data file with partial field IDs Nov 19, 2026
@github-actions github-actions Bot added the fixtures Surfaces, cases, reference tables, and fixture corrections label Nov 19, 2026
@slachiewicz
slachiewicz marked this pull request as ready for review November 19, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fixtures Surfaces, cases, reference tables, and fixture corrections

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant