feat: add primary-key upsert/delete and ALTER TABLE schema evolution - #76
Conversation
Add docs/src/ following the lance-spark docs template, covering: - Welcome, Install, Config, Performance - Operations: DDL (create-catalog, create-table), DQL (select, vector-search, time-travel), DML (insert-into) Unimplemented capabilities (ALTER TABLE, CREATE INDEX, UPDATE, DELETE) are explicitly noted as unsupported / in progress.
Primary-key aware sink: - declare +I/+U/-D changelog mode and keyBy(PrimaryKeySelector) for ordered upsert - mergeInsert (UpdateAll/InsertAll) for upsert; Dataset.delete(OR-of-AND) for -D - persist the primary key via dataset config Catalog / schema evolution: - CREATE TABLE materializes an empty dataset immediately (matching Spark/Trino) - ALTER TABLE supports ADD/DROP/RENAME COLUMN and SET/UNSET TBLPROPERTIES - type change is rejected: Lance Java SDK 7.0.0 castTo is verified a silent no-op - SchemaDiff detects add/drop/type-change/rename and rejects unsafe drop+add
…currency and replay tests Fix: - LanceUpsertSink.createDataset re-checks dataset existence and falls back to merge-insert when a peer subtask created it first (regression: multi-subtask first write silently dropped rows via Overwrite) Tests: - replay upsert/delete batches are idempotent (checkpoint replay) - two subtasks concurrently write distinct keys to an existing dataset - two subtasks concurrently first-write distinct keys without data loss
…d LanceSink LanceSink.flush re-checks dataset existence BEFORE Fragment.write (which itself creates the dataset data directory), so a peer subtask's earlier first-write is detected and followed by Append instead of a clobbering Overwrite. Regression: two append subtasks concurrently first-writing distinct rows silently dropped one subtask's data.
fightBoxing
left a comment
fightBoxing
left a comment
There was a problem hiding this comment.
Review: 功能方向正确，但 CI 的绿灯覆盖不到本 PR 的核心路径
先说结论：这个 PR 的设计方向我认同，keyBy + mergeInsert 是对的选择，openOrCreate 替代 Overwrite 也解决了真实问题。但不建议在当前状态下合并，原因是 CI 的通过状态并不能证明本 PR 的核心功能可用。
阻塞项：本 PR 新增的 5 个 ITCase 从未在 CI 中执行
pom.xml 里 failsafe 只出现在一句注释中（第 25 行），没有任何 <plugin> 配置或 goal 绑定：
$ git show <head>:pom.xml | grep -A15 failsafe | grep -c "<goal>integration-test</goal>"
0
CI 跑的是 mvn -B -ntp -am -pl lance-flink-${{ matrix.flink }} verify，但 surefire 默认只匹配 *Test/Test*/*Tests/*TestCase，不匹配 *ITCase。所以本 PR 带来的这 5 个文件一次都没跑过：
LanceUpsertSinkITCase（upsert/delete 核心语义 + 并发首写）CompositePkDeleteITCase（复合主键删除）LanceSchemaEvolutionITCase（ALTER TABLE 演进）LanceSinkConcurrencyITCase（多 subtask 并发）LanceConnectorITCase
也就是说，8 个 JDK/Flink 组合全绿，覆盖的是既有单元测试，恰好把这个 PR 最需要验证的路径全漏掉了。我在本地强制用 surefire 跑了一遍（-Dtest='*ITCase'），本 PR head 的 LanceUpsertSinkITCase 8 个用例是通过的，所以代码本身在这批用例上没问题——问题在于这个保护网在 CI 里是断开的，后续任何改动都不会被它拦住。
事实上这个隐患已经发生了：follow-up PR #77 在重构 flush() 时引入了回归，同一批用例 6 个失败，而 #77 的 CI 同样是全绿。详见我在 #77 的评论。
建议在本 PR 或独立 PR 中补上 failsafe 的 <executions> 绑定（integration-test + verify），让这 5 个文件真正进入构建。
实现债（不阻塞合并，但建议记为 follow-up）
-
DELETE 走 SQL 字符串拼接（
LanceUpsertSink#buildDeletePredicate/formatSqlValue，约 306-340 行）。三个具体问题：列名未转义，含空格或保留字的列会被解析成多个 token；formatSqlValue的类型分支不覆盖 DATE/TIME/TIMESTAMP/DECIMAL/VarBinary；谓词长度随删除行数 O(N) 膨胀。更稳的做法是 key-only 的mergeInsert+withMatchedDelete，我验证过该 API 在 lance-core 7.0.0 上可用且会持久化。 -
ALTER TABLE用SchemaDiff反推意图（LanceCatalog约 632-650 行）。Flink 的 planner 已经把用户意图以List<TableChange>形式传进来了，alterTable有对应重载。靠"同尺寸同位置"启发式猜测的后果是DROP score DOUBLE+ADD score STRING会被误判成带类型变更的 rename 而拒绝，实际上这是两个独立的合法操作。 -
RESERVED_OPTION_KEYS硬编码（LanceCatalog第 111 行）与LanceCatalogFactory的ConfigOption声明是两份真相，已经漂移了：硬编码集有 6 个 key，工厂实际声明了 8 个，s3-virtual-hosted-style和s3-allow-http会被当成用户 TBLPROPERTY 写进 dataset config。 -
new RootAllocator(Long.MAX_VALUE)（LanceCatalog第 185 行等多处）无上限、无命名。OOM 时无法从 dump 里区分是哪个组件的分配器。 -
热主键倾斜：
keyBy保序是正确性前提，所以单键吞吐受限于单个 subtask。这是设计的必然代价，建议在文档里写明，并给出复合主键/加盐的缓解手段。
以上 5 项我都已在本地实现并验证，可以在 #77 合并后提交。
1 participant
Summary
Adds primary-key upsert/delete support and ALTER TABLE schema evolution to the directory-backed Lance catalog and table sink.
Changes
Primary-key upsert/delete
Catalog / schema evolution
Tests
Verification