Skip to content

fix(table): update the table's name mapping on schema changes - #2646

Open
ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:fix/update-schema-name-mapping
Open

ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:fix/update-schema-name-mapping

Conversation

@ShivamSharma43

Copy link
Copy Markdown
Contributor

UpdateSchema never updated the table's schema.name-mapping.default property unless the caller passed a mapping with WithNameMapping, and nothing in the repo does. After AddColumn or RenameColumn the mapping still described the old schema, so files without Parquet field IDs could not be resolved for the new or renamed column. Since AddDataFilesByPath and AddDataFiles set the mapping on first use, any table that iceberg-go has added files to hits this: after adding a column, adding a file that contains it fails with invalid schema: cannot convert category: type=utf8, nullable metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing field_id (see the issue for the reproduction).

Cause

BuildUpdates only ran iceberg.UpdateNameMapping when u.nameMapping was set, and that field is only set by the WithNameMapping option. It never read the mapping stored in the table's properties.

Fix

When no mapping is passed, BuildUpdates now uses the table's own mapping (meta.NameMapping(), read from the transaction's staged metadata) and writes the updated mapping back to schema.name-mapping.default. That matches Java's SchemaUpdate.applyChangesToMetadata, which updates the property on every schema change, and PyIceberg, whose Transaction.update_schema passes table_metadata.name_mapping() to UpdateSchema._commit.

  • Added columns get a new entry, and renamed columns keep their old name and gain the new one, as UpdateNameMapping already does (same as Java's MappingUtil.update).
  • A table without the property is left alone: no mapping is added.
  • An explicit WithNameMapping still takes precedence.
  • If the property can't be parsed, meta.NameMapping() returns nil and the update goes ahead without touching it. Java does the same: it logs a warning and keeps the old value.

Testing

  • New table/update_schema_name_mapping_test.go:

    • TestUpdateSchemaUpdatesTableNameMapping adds a column and renames another on a table with a mapping, then checks the committed mapping (category for the new field, data and payload for the renamed one).
    • TestUpdateSchemaWithoutNameMappingLeavesPropertyUnset checks that a table without the property doesn't get one.
    • TestAddFilesAfterAddColumnReadsNewColumnByName runs the reproduction from the issue end to end: AddDataFilesByPath, AddColumn, AddDataFilesByPath with a file that has the new column and no field IDs, then a scan that reads books from it.

    On main, the first and third fail (the mapping has no entry for field 3, and the second AddDataFilesByPath fails with the missing field_id error above). The second passes on main and with the fix.

  • go vet ./table/ is clean. go test ./table/... passes except for tests that fail the same way on unmodified main on my Windows machine (path handling): TestEmptyStringIdentityPartitionScans, TestScanImportedSignedZeroBounds, TestDeleteImportedSignedZerosWithIn, TestDeleteNormallyWrittenSignedZeroWithIn, TestScanLegacyEmptyStringPartitionDeleteBounds, TestScanWithIDLessNameMappingFields, two TestRewriteDataFiles_PartialProgress* tests and TestWriteMapType. go test ./catalog/... gives the same results as main.

Closes #2645

This PR was prepared with AI assistance (Claude Code).

🤖 Generated with Claude Code

UpdateSchema only updated schema.name-mapping.default when the caller
passed a mapping with WithNameMapping, which nothing in the repo does.
After AddColumn or RenameColumn the property still described the old
schema, so files without field IDs could not be resolved for the new or
renamed column. AddDataFilesByPath sets the mapping on first use, and a
later file carrying an added column failed with "missing field_id".

When no mapping is passed, BuildUpdates now updates the table's own
mapping if it has one, as Java's SchemaUpdate and PyIceberg's
UpdateSchema do. Tables without the property are left unchanged, and an
explicit WithNameMapping still takes precedence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 05:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few comments @ShivamSharma43

Comment thread table/update_schema.go
if err != nil {
return nil, nil, err
}
mappingJSON, err := json.Marshal(updatedNameMapping)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when the mapping is [], ore all entries have empty names (something which is allowed by the spec), a rename or delete gets nil from "UpdateNameMapping" -> so this writes null

go scans and add-files then fail with invalid
'schema.name-mapping.default'

better:

if updatedNameMapping == nil {
	// UpdateNameMapping returns nil when no field keeps a name.
	updatedNameMapping = iceberg.NameMapping{}
}

also, add a test for it

Comment thread table/update_schema.go
// field IDs cannot be resolved for added or renamed columns.
nameMapping := u.nameMapping
if nameMapping == nil {
nameMapping = meta.NameMapping()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this changes what users see: schema updates now rewrite schema.name-mapping.default, and when a new col reuses a name freed by "DeleteColumn" or "RenameColumn" -> older files without field ids show the old values in it

better: add an unreleased compatibility note in releases.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, there is no requirement of creating a new file, we can add the tests in "update_schema_test.go"

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UpdateSchema doesn't update the table's name mapping, so files can't be added after AddColumn

3 participants