Skip to content

Core: RewriteManifests can commit stale sequence numbers and row IDs after a transaction retry #19026

Open

Description

@manuzhang

Apache Iceberg version

main (development)

Please describe the bug 🐞

When RewriteManifests explicitly replaces a manifest staged earlier in the same transaction, a concurrent commit can cause the transaction to retry and publish the replacement with metadata from the original attempt.

  1. Stage an append of file A with Transaction.newFastAppend().
  2. Read the staged manifest and write a replacement that copies its entries as EXISTING. In the same transaction, use RewriteManifests to delete the staged manifest and add the replacement.
  3. Append file B outside the transaction, then call commitTransaction().

On retry, the staged manifest keeps its path but can inherit a new sequence number and first row ID from the new manifest list. validateDeletedManifests checks presence by path, so the caller-supplied replacement can still be committed with the original values.

The regression test reports a stale data sequence number for file A in format v2 and overlapping row-ID ranges for files A and B in formats v3 and v4. A stale data sequence number can also affect which equality deletes apply.

The commit should preserve the sequence numbers and row IDs assigned on retry, or reject the replacement if its source manifest's inherited metadata changed.

Activity

  1. shivadarshan-devadiga commented on Nov 18, 2026

    @shivadarshan-devadiga

    @manuzhang I opened #19027 for this. Following apache/iceberg-cpp#1104, it rejects the commit when an explicitly replaced manifest's sequence numbers or first row ID changed on retry, and it includes your regression test from manuzhang#214, changed to expect that failure. Happy to switch to the preserve approach if you'd prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions