You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Gate each transaction action by supported table format versions #3041
Is your feature request related to a problem or challenge?
Transaction actions (RowDelta, RewriteFiles, Append) each have logic dependent on the table format version - specifically, their validations. Thus, as we adopt new table format version support we need a way to avoid incorrect behaviors being applied to newer table formats.
Additionally, I'd like to avoid making adopting V3 support a "big bang" change and allow for incremental support.
Describe the solution you'd like
I propose that we add a method to TransactionAction which specifies which table format versions it supports. Thus, one code change can be "update appends to support V3" merged in isolation from, for example, RewriteFiles which needs to check no new manifest delete vectors were applied during its rewrite.
I've been giving this a little more thought. It's not necessarily going to be as simple as "do I support V2 and V3" because there may only be partial support.
For example, V3 introduced delete vectors and row lineage (among other things). Validations for something like RewriteDataFiles will require to validate there were no new deletes (i.e. positional, equality, or vector) as well as ensure that row lineage is updated for the new metadata graph. If only one was implemented, we'd either be unable to use the action as it doesn't 'support V3' or it can mark it as supported but then must be able to reject or avoid committing data it doesn't understand.
I don't think this is an argument against including a way to specify that an action has been updated for the the next table format version, but this requires more thinking.
Related to #2411 (v3 tracking). Related to #2418 (v2 tables accept v3-only schema types) and #2580 (RowDelta and deletion-vector writer, which will need this gate).
Concrete cases a per-action version gate would need to cover, checked against main at 050ddfd. "(run)" means reproduced with unit tests in a scratch copy of main. "(read)" means from code reading only.
Position delete files on v3 (read). The spec's Delete Formats section says "Position delete files must not be added to v3 tables, but existing position delete files are valid." The PositionDeleteFileWriter module docs say v3 "forbids adding new position delete files, so callers must not route v3 writes here", and that format-version gating "belongs at the transaction/commit layer" (crates/iceberg/src/writer/base_writer/position_delete_writer.rs:29-32). Its builder takes no table metadata (position_delete_writer.rs:194). No transaction action checks the format version for delete files. FastAppend is the only action that adds files, as Deletion-vector writer + RowDelta action #2580 also notes.
Delete content is rejected by content type, not by version (run).fast_append of a PositionDeletes file fails with DataInvalid => Only data content type is allowed for fast append on both a v2 and a v3 test table. Open feat(transaction): add RowDelta transaction action for row-level modi… #2203 returns FeatureUnsupported at commit whenever delete files are added, for any table version (read).
v3 row lineage and encryption keys accepted on a v2 table (run).add_snapshot with with_row_range(0, 10) succeeds on a v2 table. The add-snapshot update keeps first-row-id: 0 and added-rows: 10, but the v2 metadata JSON drops both fields (SnapshotV2 in crates/iceberg/src/spec/snapshot.rs has no row range fields). The REST catalog sends the update list as commit updates (crates/catalog/rest/src/catalog.rs:1457, read). add_encryption_key behaves the same way: it emits an add-encryption-key update, while the v2 metadata JSON omits encryption-keys, which the spec lists for v3 only.
The encryption.key-id on a v2 table case is filed separately as #3402.
Expected: either the builder rejects v3-only updates on v2 tables, or the update list and the persisted metadata agree. A v3 table should not accept new position delete files.
Found during an agent-assisted audit of v3 support (see #2411). The reproduction above was run against main at 050ddfd.
Activity
dannycjones commentedon Aug 21, 2026
Related: #2411, #2269
dannycjones commentedon Aug 24, 2026
I've been giving this a little more thought. It's not necessarily going to be as simple as "do I support V2 and V3" because there may only be partial support.
For example, V3 introduced delete vectors and row lineage (among other things). Validations for something like
RewriteDataFileswill require to validate there were no new deletes (i.e. positional, equality, or vector) as well as ensure that row lineage is updated for the new metadata graph. If only one was implemented, we'd either be unable to use the action as it doesn't 'support V3' or it can mark it as supported but then must be able to reject or avoid committing data it doesn't understand.I don't think this is an argument against including a way to specify that an action has been updated for the the next table format version, but this requires more thinking.
mfellner commentedon Oct 11, 2026
Related to #2411 (v3 tracking). Related to #2418 (v2 tables accept v3-only schema types) and #2580 (RowDelta and deletion-vector writer, which will need this gate).
Concrete cases a per-action version gate would need to cover, checked against main at 050ddfd. "(run)" means reproduced with unit tests in a scratch copy of main. "(read)" means from code reading only.
PositionDeleteFileWritermodule docs say v3 "forbids adding new position delete files, so callers must not route v3 writes here", and that format-version gating "belongs at the transaction/commit layer" (crates/iceberg/src/writer/base_writer/position_delete_writer.rs:29-32). Its builder takes no table metadata (position_delete_writer.rs:194). No transaction action checks the format version for delete files.FastAppendis the only action that adds files, as Deletion-vector writer + RowDelta action #2580 also notes.fast_appendof aPositionDeletesfile fails withDataInvalid => Only data content type is allowed for fast appendon both a v2 and a v3 test table. Open feat(transaction): add RowDelta transaction action for row-level modi… #2203 returnsFeatureUnsupportedat commit whenever delete files are added, for any table version (read).add_snapshotwithwith_row_range(0, 10)succeeds on a v2 table. Theadd-snapshotupdate keepsfirst-row-id: 0andadded-rows: 10, but the v2 metadata JSON drops both fields (SnapshotV2incrates/iceberg/src/spec/snapshot.rshas no row range fields). The REST catalog sends the update list as commitupdates(crates/catalog/rest/src/catalog.rs:1457, read).add_encryption_keybehaves the same way: it emits anadd-encryption-keyupdate, while the v2 metadata JSON omitsencryption-keys, which the spec lists for v3 only.The
encryption.key-idon a v2 table case is filed separately as #3402.Expected: either the builder rejects v3-only updates on v2 tables, or the update list and the persisted metadata agree. A v3 table should not accept new position delete files.
Found during an agent-assisted audit of v3 support (see #2411). The reproduction above was run against main at 050ddfd.