You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
add_files(check_duplicate_files=True) can commit duplicate paths after a conflict retry #4445
0.12.0 (latest release); the same logic is on main
Please describe the bug 🐞
(Found with AI; the script below reproduces it on 0.12.0.)
add_files with check_duplicate_files=True checks the paths once, against the snapshot the transaction started from. When the commit conflicts and is retried, the append is rebuilt on the new snapshot without checking again. Two writers that add the same file both succeed, and its rows are read twice.
importtempfileimportpyarrowaspaimportpyarrow.parquetaspqfrompyiceberg.catalog.sqlimportSqlCatalogd=tempfile.mkdtemp()
cat=SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{d}")
cat.create_namespace("ns")
df=pa.table({"x": [1, 2]})
cat.create_table("ns.t", schema=df.schema)
pq.write_table(df, f"{d}/a.parquet")
# two writers load the table, then both add the same filet1=cat.load_table("ns.t")
t2=cat.load_table("ns.t")
t1.add_files([f"{d}/a.parquet"])
t2.add_files([f"{d}/a.parquet"]) # its first commit conflicts and is retriedt=cat.load_table("ns.t")
print(t.scan().to_arrow()["x"].to_pylist())
# [1, 2, 1, 2], should be [1, 2]: the retry does not check for duplicates again
Expected: the retry checks the paths against the snapshot it commits on, so the second add_files is refused and the table keeps one live reference to the file, as when the second handle is loaded or refreshed after the first commit.
Willingness to contribute
I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time
Activity
GayathriSrividya commentedon Nov 20, 2026
Implemented in PR #4446: #4446. The fix rechecks duplicate file paths after commit retries and adds a regression test.