Repository navigation
fix(table): update the table's name mapping on schema changes - #2646
ShivamSharma43 wants to merge 1 commit into
Conversation
UpdateSchema only updated schema.name-mapping.default when the caller passed a mapping with WithNameMapping, which nothing in the repo does. After AddColumn or RenameColumn the property still described the old schema, so files without field IDs could not be resolved for the new or renamed column. AddDataFilesByPath sets the mapping on first use, and a later file carrying an added column failed with "missing field_id". When no mapping is passed, BuildUpdates now updates the table's own mapping if it has one, as Java's SchemaUpdate and PyIceberg's UpdateSchema do. Tables without the property are left unchanged, and an explicit WithNameMapping still takes precedence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
badalprasadsingh
left a comment
There was a problem hiding this comment.
few comments @ShivamSharma43
| if err != nil { | ||
| return nil, nil, err | ||
| } | ||
| mappingJSON, err := json.Marshal(updatedNameMapping) |
There was a problem hiding this comment.
when the mapping is [], ore all entries have empty names (something which is allowed by the spec), a rename or delete gets nil from "UpdateNameMapping" -> so this writes null
go scans and add-files then fail with invalid
'schema.name-mapping.default'
better:
if updatedNameMapping == nil {
// UpdateNameMapping returns nil when no field keeps a name.
updatedNameMapping = iceberg.NameMapping{}
}also, add a test for it
| // field IDs cannot be resolved for added or renamed columns. | ||
| nameMapping := u.nameMapping | ||
| if nameMapping == nil { | ||
| nameMapping = meta.NameMapping() |
There was a problem hiding this comment.
this changes what users see: schema updates now rewrite schema.name-mapping.default, and when a new col reuses a name freed by "DeleteColumn" or "RenameColumn" -> older files without field ids show the old values in it
better: add an unreleased compatibility note in releases.md
There was a problem hiding this comment.
also, there is no requirement of creating a new file, we can add the tests in "update_schema_test.go"
UpdateSchemanever updated the table'sschema.name-mapping.defaultproperty unless the caller passed a mapping withWithNameMapping, and nothing in the repo does. AfterAddColumnorRenameColumnthe mapping still described the old schema, so files without Parquet field IDs could not be resolved for the new or renamed column. SinceAddDataFilesByPathandAddDataFilesset the mapping on first use, any table that iceberg-go has added files to hits this: after adding a column, adding a file that contains it fails withinvalid schema: cannot convert category: type=utf8, nullable metadata: ["PARQUET:field_id": "-1"] to Iceberg field, missing field_id(see the issue for the reproduction).Cause
BuildUpdatesonly raniceberg.UpdateNameMappingwhenu.nameMappingwas set, and that field is only set by theWithNameMappingoption. It never read the mapping stored in the table's properties.Fix
When no mapping is passed,
BuildUpdatesnow uses the table's own mapping (meta.NameMapping(), read from the transaction's staged metadata) and writes the updated mapping back toschema.name-mapping.default. That matches Java'sSchemaUpdate.applyChangesToMetadata, which updates the property on every schema change, and PyIceberg, whoseTransaction.update_schemapassestable_metadata.name_mapping()toUpdateSchema._commit.UpdateNameMappingalready does (same as Java'sMappingUtil.update).WithNameMappingstill takes precedence.meta.NameMapping()returns nil and the update goes ahead without touching it. Java does the same: it logs a warning and keeps the old value.Testing
New
table/update_schema_name_mapping_test.go:TestUpdateSchemaUpdatesTableNameMappingadds a column and renames another on a table with a mapping, then checks the committed mapping (categoryfor the new field,dataandpayloadfor the renamed one).TestUpdateSchemaWithoutNameMappingLeavesPropertyUnsetchecks that a table without the property doesn't get one.TestAddFilesAfterAddColumnReadsNewColumnByNameruns the reproduction from the issue end to end:AddDataFilesByPath,AddColumn,AddDataFilesByPathwith a file that has the new column and no field IDs, then a scan that readsbooksfrom it.On main, the first and third fail (the mapping has no entry for field 3, and the second
AddDataFilesByPathfails with themissing field_iderror above). The second passes on main and with the fix.go vet ./table/is clean.go test ./table/...passes except for tests that fail the same way on unmodified main on my Windows machine (path handling):TestEmptyStringIdentityPartitionScans,TestScanImportedSignedZeroBounds,TestDeleteImportedSignedZerosWithIn,TestDeleteNormallyWrittenSignedZeroWithIn,TestScanLegacyEmptyStringPartitionDeleteBounds,TestScanWithIDLessNameMappingFields, twoTestRewriteDataFiles_PartialProgress*tests andTestWriteMapType.go test ./catalog/...gives the same results as main.Closes #2645
This PR was prepared with AI assistance (Claude Code).
🤖 Generated with Claude Code