Repository navigation
Core: Resolve last-added -1 to a re-added schema - #18812
rambleraptor wants to merge 1 commit into
Conversation
This was something I discovered working on the Terraform provider. Adding a schema, partition spec, or sort order that already exists in the table and then setting it current or default with -1 failed with "no schema has been added". Clients that send the full desired state, such as declarative tools that revert a column or a partition spec, hit this on every revert. The re-added entry is now the last added one. The change log records -1 only when the latest add of that type in the changes has the same ID, so a server replaying the changes selects the same entry.
| return this; | ||
| } | ||
|
|
||
| Builder removeSchemas(Iterable<Integer> schemaIds) { |
There was a problem hiding this comment.
Nice catch on the bug. One remaining issue to flag here: removeSchemas doesn't remove the ID from schemasById (so schemasById becomes stale), so if the same commit removes the highest-ID schema and then adds a new one, the new schema gets the freed ID and addSchemaInternal skips the add because the stale map entry exists. This leads to metadata corruption when it's set current.
The specs seem to have the same issue - which may be a follow-up.
There was a problem hiding this comment.
Well, this seems to be caught in #18678 - so we're good here.
| } | ||
|
|
||
| @Test | ||
| void replayedChangesSetCurrentSchemaToReAddedSchema() { |
There was a problem hiding this comment.
This seems to cover schemas only. How about the specs and sort orders which are also changed?
| && changes(MetadataUpdate.AddSchema.class) | ||
| .anyMatch(added -> added.schema().schemaId() == newSchemaId); | ||
| this.lastAddedSchemaId = isNewSchema ? newSchemaId : null; | ||
| this.lastAddedSchemaId = newSchemaId; |
There was a problem hiding this comment.
The equivalent reuse path in ViewMetadata still leaves lastAddedSchemaId unset or pointing at a different schema. Should we have a follow-up for views?
|
Overall nice catch 👍 It would be great to get more eyes on this since it blocks a routine operation and can silently commit the wrong schema/spec/sort order. |
developer-rpai
left a comment
There was a problem hiding this comment.
Nice catch. Treating the re-added entry as the last added one, and only writing -1 to the change log when the latest Add of that type in the changes has the same id, looks right to me. replayedChangesSetCurrentSchemaToReAddedSchema is a good test, since the old code would log -1 there and a replay would pick second.
- The REST path is the one that actually hits this: a client sends AddSchema(existing) plus SetCurrentSchema(-1) and the server applies them through MetadataUpdate.applyTo. Could a test apply
new MetadataUpdate.AddSchema(original)andnew MetadataUpdate.SetCurrentSchema(-1)to TableMetadata.buildFrom(updated) that way? - This does shift what -1 means a bit: before it was the last entry added in this set of changes, now it's the last one passed to add, even if it already existed. That seems like the right reading of "last added", but a sentence in the set-current-schema, set-default-spec and set-default-sort-order descriptions in the REST spec would help other implementations line up.
- Worth calling out in the description: the old spec and sort order branches compared against lastAddedSpecId / lastAddedOrderId instead of the new id (TableMetadata.java around L1683 and L1730), so they already behaved differently from the schema branch.
This was a bug I discovered working on the Terraform provider.
Adding a schema, partition spec, or sort order that already exists in the table and then setting it current or default with -1 failed with "no schema has been added".
The re-added entry is now the last added one. The change log records -1 only when the latest add of that type in the changes has the same ID, so a server replaying the changes selects the same entry.