Skip to content

Core: Drop removed schema and spec ids from TableMetadata.Builder indexes - #18678

Open
LuciferYang wants to merge 2 commits into
apache:mainfrom
LuciferYang:fix/core-builder-stale-schema-spec-index
Open

LuciferYang wants to merge 2 commits into
apache:mainfrom
LuciferYang:fix/core-builder-stale-schema-spec-index

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Closes #18677.

TableMetadata.Builder.removeSchemas and removeSpecs filter schemas and specs but leave the removed ids in schemasById and specsById. CatalogHandlers.commit applies a whole request to one builder, so a transaction that runs expireSnapshots().cleanExpiredMetadata(true) and then adds a column or a partition field can get a freed id and run into the stale entry. For schemas, the server commits metadata whose current-schema-id is not in schemas, and the table can no longer be loaded. For specs, the commit is rejected with Cannot set last added spec: no spec has been added.

This removes the ids from the indexes together with the list entries. setCurrentSchema on a removed id now fails with Cannot set current schema to unknown schema, and a new schema or spec that gets a freed id is added normally.

The same bug was fixed in iceberg-cpp in apache/iceberg-cpp#1012.

New tests:

  • CatalogTests.testRemoveUnusedSchemaThenAddColumnInTransaction and testRemoveUnusedSpecThenAddFieldInTransaction run that transaction against each catalog. Without the fix both fail on TestRESTCatalog; they pass on the in-memory, JDBC and Nessie catalogs either way.
  • TestTableMetadata.testRemoveSchemasThenReuseRemovedId and testRemoveSpecsThenReuseRemovedId run the same sequence on one builder, so the bug shows without a REST server. Without the fix, setCurrentSchema(1) after removeSchemas([1]) succeeds, and the new spec is not added to specs.

AI Disclosure

  • Model: Claude Opus
  • Human Oversight: reviewed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Core: Expiring unused schemas and adding a column in one transaction leaves REST table metadata unloadable

1 participant