Discovered during architecture review of #76 (feat: primary-key upsert/delete + ALTER TABLE schema evolution). Follow-up: correctness hazard in the ALTER path that only surfaces on specific DDL sequences.
Context
LanceCatalog#alterTable(ObjectPath, CatalogBaseTable, boolean) receives
the new CatalogBaseTable only and infers the diff against the existing
Lance dataset schema using SchemaDiff.compute(oldRowType, newRowType).
Renames are reconstructed with a heuristic: if the diff contains exactly
N removed columns and N added columns at matching positions, they are
treated as N RENAME COLUMN operations.
Problems
- Cannot distinguish a true rename from a coincidentally
position-aligned drop+add. If a user's DDL is
DROP COLUMN a; ADD COLUMN b <same type> at the same position, the
heuristic silently keeps a's data under the name b — a correctness
hazard.
- Breaks under Flink's future
ADD COLUMN AFTER col positioning
semantics. Position-based matching is not stable across schema-
evolution features Flink is already adding.
- Loses the user's original DDL wording, degrading error messages.
Proposed change
Override the TableChange-taking variant:
Catalog#alterTable(ObjectPath, CatalogBaseTable, List<TableChange>, boolean)
and dispatch directly on TableChange.AddColumn, DropColumn,
ModifyColumnName, ModifyPhysicalColumnType,
SetOption/ResetOption. Keep SchemaDiff only as a defensive fallback
for the deprecated code path; mark it @Deprecated internally.
Acceptance criteria
ALTER TABLE t RENAME COLUMN a TO b no longer routes through
SchemaDiff.compute on the primary path.- Test:
DROP COLUMN a; ADD COLUMN a <different type> in the same session
verifies the drop actually happened (the heuristic today flags this as
rename-with-type-change and refuses).
- Existing
LanceCatalogTableITCase assertions continue to pass.
Refs
Context
LanceCatalog#alterTable(ObjectPath, CatalogBaseTable, boolean)receivesthe new
CatalogBaseTableonly and infers the diff against the existingLance dataset schema using
SchemaDiff.compute(oldRowType, newRowType).Renames are reconstructed with a heuristic: if the diff contains exactly
Nremoved columns andNadded columns at matching positions, they aretreated as
NRENAME COLUMN operations.Problems
position-aligned drop+add. If a user's DDL is
DROP COLUMN a; ADD COLUMN b <same type>at the same position, theheuristic silently keeps
a's data under the nameb— a correctnesshazard.
ADD COLUMN AFTER colpositioningsemantics. Position-based matching is not stable across schema-
evolution features Flink is already adding.
Proposed change
Override the
TableChange-taking variant:and dispatch directly on
TableChange.AddColumn,DropColumn,ModifyColumnName,ModifyPhysicalColumnType,SetOption/ResetOption. KeepSchemaDiffonly as a defensive fallbackfor the deprecated code path; mark it
@Deprecatedinternally.Acceptance criteria
ALTER TABLE t RENAME COLUMN a TO bno longer routes throughSchemaDiff.computeon the primary path.DROP COLUMN a; ADD COLUMN a <different type>in the same sessionverifies the drop actually happened (the heuristic today flags this as
rename-with-type-change and refuses).
LanceCatalogTableITCaseassertions continue to pass.Refs
TableChange:https://nightlies.apache.org/flink/flink-docs-release-1.20/api/java/org/apache/flink/table/catalog/TableChange.html