Skip to content

Core: Resolve last-added -1 to a re-added schema - #18812

Open
rambleraptor wants to merge 1 commit into
apache:mainfrom
rambleraptor:reuse-last-added-metadata
Open

rambleraptor wants to merge 1 commit into
apache:mainfrom
rambleraptor:reuse-last-added-metadata

Conversation

@rambleraptor

@rambleraptor rambleraptor commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

This was a bug I discovered working on the Terraform provider.

Adding a schema, partition spec, or sort order that already exists in the table and then setting it current or default with -1 failed with "no schema has been added".

The re-added entry is now the last added one. The change log records -1 only when the latest add of that type in the changes has the same ID, so a server replaying the changes selects the same entry.

This was something I discovered working on the Terraform provider.

Adding a schema, partition spec, or sort order that already exists in the
table and then setting it current or default with -1 failed with "no schema
has been added". Clients that send the full desired state, such as
declarative tools that revert a column or a partition spec, hit this on
every revert.

The re-added entry is now the last added one. The change log records -1
only when the latest add of that type in the changes has the same ID, so a
server replaying the changes selects the same entry.
@github-actions github-actions Bot added the core label Nov 6, 2026
return this;
}

Builder removeSchemas(Iterable<Integer> schemaIds) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch on the bug. One remaining issue to flag here: removeSchemas doesn't remove the ID from schemasById (so schemasById becomes stale), so if the same commit removes the highest-ID schema and then adds a new one, the new schema gets the freed ID and addSchemaInternal skips the add because the stale map entry exists. This leads to metadata corruption when it's set current.

The specs seem to have the same issue - which may be a follow-up.

@jiwen624 jiwen624 Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this seems to be caught in #18678 - so we're good here.

}

@Test
void replayedChangesSetCurrentSchemaToReAddedSchema() {

@jiwen624 jiwen624 Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to cover schemas only. How about the specs and sort orders which are also changed?

&& changes(MetadataUpdate.AddSchema.class)
.anyMatch(added -> added.schema().schemaId() == newSchemaId);
this.lastAddedSchemaId = isNewSchema ? newSchemaId : null;
this.lastAddedSchemaId = newSchemaId;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The equivalent reuse path in ViewMetadata still leaves lastAddedSchemaId unset or pointing at a different schema. Should we have a follow-up for views?

@jiwen624

Copy link
Copy Markdown
Contributor

Overall nice catch 👍 It would be great to get more eyes on this since it blocks a routine operation and can silently commit the wrong schema/spec/sort order.

@developer-rpai developer-rpai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. Treating the re-added entry as the last added one, and only writing -1 to the change log when the latest Add of that type in the changes has the same id, looks right to me. replayedChangesSetCurrentSchemaToReAddedSchema is a good test, since the old code would log -1 there and a replay would pick second.

  1. The REST path is the one that actually hits this: a client sends AddSchema(existing) plus SetCurrentSchema(-1) and the server applies them through MetadataUpdate.applyTo. Could a test apply new MetadataUpdate.AddSchema(original) and new MetadataUpdate.SetCurrentSchema(-1) to TableMetadata.buildFrom(updated) that way?
  2. This does shift what -1 means a bit: before it was the last entry added in this set of changes, now it's the last one passed to add, even if it already existed. That seems like the right reading of "last added", but a sentence in the set-current-schema, set-default-spec and set-default-sort-order descriptions in the REST spec would help other implementations line up.
  3. Worth calling out in the description: the old spec and sort order branches compared against lastAddedSpecId / lastAddedOrderId instead of the new id (TableMetadata.java around L1683 and L1730), so they already behaved differently from the schema branch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants