You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CSV connector with exactly-once loading into Raw #9
A spike against the pinned iceberg-rust proved the commit side of this issue.
Position: one Transaction holds the fast_append and an update_table_properties. The position is stored as the table property infostash.position and mirrored in the snapshot summary. Both land in one metadata file. Read it back from the table property.
Crash between read and commit: a rerun loads each row exactly once. The spike's crash test can be reused as is.
Watch out:Transaction::do_commit reloads the table and silently rebases a stale transaction, so two runs from the same position duplicate rows instead of conflicting. We need a guard that fails the commit if the table head moved after the position was read (about 40 lines, wrapping the catalog's compare-and-swap).
An empty fast_append is an error, so a run with nothing new must skip the commit.
Don't use summary keys that start with added- or total-; iceberg-rust overwrites them.
Activity
codecapo commentedon Oct 30, 2026
Spike findings (commit semantics)
A spike against the pinned iceberg-rust proved the commit side of this issue.
Transactionholds thefast_appendand anupdate_table_properties. The position is stored as the table propertyinfostash.positionand mirrored in the snapshot summary. Both land in one metadata file. Read it back from the table property.Transaction::do_commitreloads the table and silently rebases a stale transaction, so two runs from the same position duplicate rows instead of conflicting. We need a guard that fails the commit if the table head moved after the position was read (about 40 lines, wrapping the catalog's compare-and-swap).fast_appendis an error, so a run with nothing new must skip the commit.added-ortotal-; iceberg-rust overwrites them.Points: 5 (unchanged).
docs: supersede ADR 0004 with ordered per-table job commits (#41)