Skip to content

fix(table): assert last-assigned-field-id on UpdateSchema - #2622

Merged
laskoviymishka merged 4 commits into
apache:mainfrom
laskoviymishka:factory/rc-2609-last-assigned-field-id
Nov 18, 2026
Merged

laskoviymishka merged 4 commits into
apache:mainfrom
laskoviymishka:factory/rc-2609-last-assigned-field-id

Conversation

@laskoviymishka

@laskoviymishka laskoviymishka commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UpdateSchema only asserted the current schema ID, so with retries on by default since #2462 a stale writer could replay an AddSchema that reuses a field ID another writer assigned and then dropped. The drop returns the table to its old schema ID, so the assertion still passes after refresh and the new column reads the dropped column's data.

When UpdateSchema adds a schema it now also sends AssertLastAssignedFieldID with the base last-column-id, as Java's UpdateRequirements does. The requirement, its REST encoding, the catalog-side validation and the retry path's check were already on main, so the change is a few lines in BuildUpdates.

The SQL catalog test is Badal's two-writer repro: A stages adding x, B adds y, writes 42 to it and drops it, and A's commit now fails with ErrCommitFailed instead of committing a column that reads 42.

One case is left open. If stored metadata has a last-column-id lower than a field ID in the schema history, the assertion is skipped so those tables stay writable. The first commit writes the corrected counter, so later updates are protected. Re-running the ops against the refreshed base and reassigning IDs, instead of failing, could be a follow-up.

Fixes #2609.

AI-assisted, reviewed by a human.

UpdateSchema only required the current schema ID. With commit retries
enabled by default (apache#2462), a stale writer could replay an AddSchema
whose new column reused a field ID that another writer had assigned and
then dropped. The peer's drop returns the table to the original schema
ID, so the schema assertion still holds after refresh, and the stale
column silently reads the dropped column's data.

When UpdateSchema adds a new schema, it now also sends
AssertLastAssignedFieldID with the transaction base's last-column-id,
as Java's UpdateRequirements does for AddSchema. The requirement type,
its REST encoding and local validation in every catalog already exist,
and the retry path already checks non-rebased requirements against the
refreshed metadata, so a conflicting writer now fails with
ErrCommitFailed instead of committing a reused ID. Using the base value
lets chained schema updates in one transaction dedupe to one
requirement.

If the stored last-column-id is stale (lower than a field ID in schema
history), catalogs compare against the stored counter while the
transaction compares against the builder's raised one. The assertion
is skipped for that case so schema updates on such tables keep
working. The first commit writes the raised counter, so later updates
are protected.

Fixes apache#2609.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
@laskoviymishka
laskoviymishka force-pushed the factory/rc-2609-last-assigned-field-id branch from 3f6eeba to 8b2ab49 Compare November 17, 2026 07:38
@laskoviymishka
laskoviymishka marked this pull request as ready for review November 17, 2026 07:39
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved @laskoviymishka :)

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Asserting the base last-column-id when adding a schema closes the add-then-drop case from #2609 and matches Java's UpdateRequirements. With the requirement disabled, the SQL two-writer test commits instead of failing with ErrCommitFailed, and both BuildUpdates subtests fail. Taking the counter from the transaction's starting table, and the chained-update dedup, both look right.

Comment thread table/update_schema.go
// on the counter and every commit would fail.
base := u.txn.tbl.Metadata()
if base.LastColumnID() == columnIDFloor(base.LastColumnID(), base.Schemas()) {
requirements = append(requirements, AssertLastAssignedFieldID(base.LastColumnID()))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this get an entry under "Unreleased compatibility notes" in website/src/releases.md? Users will see the change: a stale UpdateSchema that adds a column now fails with table.ErrCommitFailed when another writer has assigned field IDs since its base (for example by adding and then dropping a column), where it used to commit silently. Something like:

Schema updates assert the last assigned field ID. When UpdateSchema adds a schema, the commit now also requires the table's last-column-id to match the transaction's base, as Iceberg Java does. A stale writer whose base predates another writer's column additions now fails with table.ErrCommitFailed instead of reusing field IDs. Reload the table and reapply the schema change.

@nssalian nssalian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the skip for tables whose stored last-column-id is lower than a field ID in the schema history. If I force the new assertion on for every schema add (removing the columnIDFloor comparison), the existing TestUpdateSchemaAssignsIDAboveHistoricalSchemas fails. So that skip is already covered by a test.

The dynamodb_integration_test.go reformat in this PR is the same change that landed on main in #2621, and #2620 carries it too. git merge-tree shows this PR merging cleanly with main and with #2620, so the order doesn't matter.

+1 on the release note @zeroshade asked for.

Comment thread catalog/sql/sql_test.go Outdated

current, err := cat.LoadTable(ctx, tblID)
s.Require().NoError(err)
s.Equal(peer.MetadataLocation(), current.MetadataLocation(), "nothing may be committed")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if this check is s.Require().Equal, the loop below that scans every column for "42" can be removed. Once the current metadata location is the peer's, the table schema is just foo (the peer dropped y), so that loop can never find "42" and can't fail on its own.

Also tighten the stale schema update test: require the metadata location to be
unchanged and drop the scan for "42", which could never fail after that check.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
@laskoviymishka
laskoviymishka merged commit 736c387 into apache:main Nov 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UpdateSchema: a stale AddColumn reuses a dropped column's field id on retry and reads its old values

4 participants