Skip to content

Gate each transaction action by supported table format versions #3041

Open

Description

@dannycjones

Is your feature request related to a problem or challenge?

Transaction actions (RowDelta, RewriteFiles, Append) each have logic dependent on the table format version - specifically, their validations. Thus, as we adopt new table format version support we need a way to avoid incorrect behaviors being applied to newer table formats.

Additionally, I'd like to avoid making adopting V3 support a "big bang" change and allow for incremental support.

Describe the solution you'd like

I propose that we add a method to TransactionAction which specifies which table format versions it supports. Thus, one code change can be "update appends to support V3" merged in isolation from, for example, RewriteFiles which needs to check no new manifest delete vectors were applied during its rewrite.

Willingness to contribute

I can contribute to this feature independently

Activity

  1. dannycjones commented on Aug 21, 2026

    @dannycjones
    ContributorAuthor

    Related: #2411, #2269

  2. dannycjones commented on Aug 24, 2026

    @dannycjones
    ContributorAuthor

    I've been giving this a little more thought. It's not necessarily going to be as simple as "do I support V2 and V3" because there may only be partial support.

    For example, V3 introduced delete vectors and row lineage (among other things). Validations for something like RewriteDataFiles will require to validate there were no new deletes (i.e. positional, equality, or vector) as well as ensure that row lineage is updated for the new metadata graph. If only one was implemented, we'd either be unable to use the action as it doesn't 'support V3' or it can mark it as supported but then must be able to reject or avoid committing data it doesn't understand.

    I don't think this is an argument against including a way to specify that an action has been updated for the the next table format version, but this requires more thinking.

  3. mfellner commented on Oct 11, 2026

    @mfellner

    Related to #2411 (v3 tracking). Related to #2418 (v2 tables accept v3-only schema types) and #2580 (RowDelta and deletion-vector writer, which will need this gate).

    Concrete cases a per-action version gate would need to cover, checked against main at 050ddfd. "(run)" means reproduced with unit tests in a scratch copy of main. "(read)" means from code reading only.

    • Position delete files on v3 (read). The spec's Delete Formats section says "Position delete files must not be added to v3 tables, but existing position delete files are valid." The PositionDeleteFileWriter module docs say v3 "forbids adding new position delete files, so callers must not route v3 writes here", and that format-version gating "belongs at the transaction/commit layer" (crates/iceberg/src/writer/base_writer/position_delete_writer.rs:29-32). Its builder takes no table metadata (position_delete_writer.rs:194). No transaction action checks the format version for delete files. FastAppend is the only action that adds files, as Deletion-vector writer + RowDelta action #2580 also notes.
    • Delete content is rejected by content type, not by version (run). fast_append of a PositionDeletes file fails with DataInvalid => Only data content type is allowed for fast append on both a v2 and a v3 test table. Open feat(transaction): add RowDelta transaction action for row-level modi… #2203 returns FeatureUnsupported at commit whenever delete files are added, for any table version (read).
    • v3 row lineage and encryption keys accepted on a v2 table (run). add_snapshot with with_row_range(0, 10) succeeds on a v2 table. The add-snapshot update keeps first-row-id: 0 and added-rows: 10, but the v2 metadata JSON drops both fields (SnapshotV2 in crates/iceberg/src/spec/snapshot.rs has no row range fields). The REST catalog sends the update list as commit updates (crates/catalog/rest/src/catalog.rs:1457, read). add_encryption_key behaves the same way: it emits an add-encryption-key update, while the v2 metadata JSON omits encryption-keys, which the spec lists for v3 only.

    The encryption.key-id on a v2 table case is filed separately as #3402.

    Expected: either the builder rejects v3-only updates on v2 tables, or the update list and the persisted metadata agree. A v3 table should not accept new position delete files.

    Found during an agent-assisted audit of v3 support (see #2411). The reproduction above was run against main at 050ddfd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions