Skip to content

add_files(check_duplicate_files=True) can commit duplicate paths after a conflict retry #4445

Open

Description

@kdn36

Apache Iceberg version

0.12.0 (latest release); the same logic is on main

Please describe the bug 🐞

(Found with AI; the script below reproduces it on 0.12.0.)

add_files with check_duplicate_files=True checks the paths once, against the snapshot the transaction started from. When the commit conflicts and is retried, the append is rebuilt on the new snapshot without checking again. Two writers that add the same file both succeed, and its rows are read twice.

import tempfile

import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog

d = tempfile.mkdtemp()
cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{d}")
cat.create_namespace("ns")
df = pa.table({"x": [1, 2]})
cat.create_table("ns.t", schema=df.schema)
pq.write_table(df, f"{d}/a.parquet")

# two writers load the table, then both add the same file
t1 = cat.load_table("ns.t")
t2 = cat.load_table("ns.t")
t1.add_files([f"{d}/a.parquet"])
t2.add_files([f"{d}/a.parquet"])  # its first commit conflicts and is retried

t = cat.load_table("ns.t")
print(t.scan().to_arrow()["x"].to_pylist())
# [1, 2, 1, 2], should be [1, 2]: the retry does not check for duplicates again

Expected: the retry checks the paths against the snapshot it commits on, so the second add_files is refused and the table keeps one live reference to the file, as when the second handle is loaded or refreshed after the first commit.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

  1. GayathriSrividya commented on Nov 20, 2026

    @GayathriSrividya
    Contributor

    Implemented in PR #4446: #4446. The fix rechecks duplicate file paths after commit retries and adds a regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions