Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 105 additions & 2 deletions crates/iceberg/src/transaction/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -125,11 +125,36 @@ impl Transaction {
let updated_table = Self::update_table_metadata(table, &updates)?;

existing_updates.extend(updates);
existing_requirements.extend(requirements);
Self::merge_requirements(existing_requirements, requirements);

Ok(updated_table)
}

// Catalogs check against the base, so keep only the first requirement per value.
fn merge_requirements(
existing_requirements: &mut Vec<TableRequirement>,
requirements: Vec<TableRequirement>,
) {
for requirement in requirements {
if !existing_requirements
.iter()
.any(|existing| Self::asserts_same_value(existing, &requirement))
{
existing_requirements.push(requirement);
}
}
}

fn asserts_same_value(left: &TableRequirement, right: &TableRequirement) -> bool {
match (left, right) {
(
TableRequirement::RefSnapshotIdMatch { r#ref: left, .. },
TableRequirement::RefSnapshotIdMatch { r#ref: right, .. },
) => left == right,
_ => std::mem::discriminant(left) == std::mem::discriminant(right),
}
}
Comment on lines +133 to +156

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the first requirement on each value gives the base value only when every action asserts each value it changes. The RTAS replace may not do that. build_replacement removes main. Java's forReplaceTable asserts only the table UUID, and it skips the ref assertion for a replace while still suppressing later ones on that ref.

If the replace action follows Java, the fast append after it derives RefSnapshotIdMatch { ref: "main", snapshot_id: None } from the staged table (the table after the earlier actions in the transaction). That becomes the first requirement on main, and the commit fails. I reproduced this against MemoryCatalog with an action that emits RemoveSnapshotRef { main } and only UuidMatch, followed by a fast append on a table that already has a snapshot:

CatalogCommitConflicts => Requirement failed: Branch or tag `main` already exists

How do you plan to have the replace action emit its requirements? Could we filter against the base in do_commit instead, where self.table is in scope? apply already checks each requirement against the staged table it came from (L121-L123). So a requirement that fails on the base guards a value that an earlier action in the transaction changed, and that action decides how to guard it. For today's actions this keeps the same requirements as the PR. It also handles an action that doesn't assert what it changes, and it removes merge_requirements and asserts_same_value. Line 128 goes back to existing_requirements.extend(requirements), and the commit is built like this:

        // Each action derives its requirements from the staged table, but the catalog checks
        // them against the base. A requirement that fails on the base guards a value an earlier
        // action in this transaction changed, and that action decides how to guard it.
        let mut requirements = Vec::with_capacity(existing_requirements.len());
        for requirement in existing_requirements {
            if requirement.check(Some(self.table.metadata())).is_ok()
                && !requirements.contains(&requirement)
            {
                requirements.push(requirement);
            }
        }

        let table_commit = TableCommit::builder()
            .ident(self.table.identifier().to_owned())
            .updates(existing_updates)
            .requirements(requirements)
            .build();

Whichever rule we keep, could we add the replace-then-append case as a MemoryCatalog test? Here is the action I used:

#[derive(Clone)]
struct RemoveMainAssertingUuidOnly;

#[async_trait]
impl TransactionAction for RemoveMainAssertingUuidOnly {
    type State = ();

    fn new_state(&self) -> Self::State {}

    async fn commit(&self, _state: &mut (), table: &Table) -> Result<ActionCommit> {
        Ok(ActionCommit::new(
            vec![TableUpdate::RemoveSnapshotRef {
                ref_name: MAIN_BRANCH.to_string(),
            }],
            vec![TableRequirement::UuidMatch {
                uuid: table.metadata().uuid(),
            }],
        ))
    }

    async fn cleanup(self, _state: (), _table: &Table, _status: CommitStatus) {}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The replace action will assert main at the base snapshot, so a concurrent commit to main conflicts and the transaction retries against the refreshed table. With that, keeping the first requirement holds for replace-then-append, I'll happily add that test in the next PR if that works? Is it better for my to put that PR up now to see where we're going?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll happily add that test in the next PR if that works? Is it better for my to put that PR up now to see where we're going?

No worries. Not a blocker for me.


/// Sets table to a new version.
pub fn upgrade_table_version(&self) -> UpgradeFormatVersionAction {
UpgradeFormatVersionAction::new()
Expand Down Expand Up @@ -271,7 +296,7 @@ mod tests {
use crate::table::{StaticTable, Table};
use crate::test_utils::{make_encrypted_table, test_runtime};
use crate::transaction::{ApplyTransactionAction, Transaction};
use crate::{Catalog, Error, ErrorKind, TableCreation, TableIdent};
use crate::{Catalog, Error, ErrorKind, TableCreation, TableIdent, TableRequirement};

pub fn make_v1_table() -> Table {
let file = File::open(format!(
Expand Down Expand Up @@ -647,6 +672,84 @@ mod tests {
assert_eq!(summary.get("total-files-size").unwrap(), "300");
}

#[tokio::test]
async fn test_commit_two_fast_appends_in_one_transaction() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says the current schema and sort order ids hit the same failure, but only the fast-append case goes through a catalog. Could we add a MemoryCatalog test for update_schema().add_column(...) followed by replace_sort_order() on the new column? Without the fix it fails with Requirement failed: Current schema id does not match, because replace_sort_order asserts the staged schema id.

let catalog = new_memory_catalog().await;
let table = make_v3_minimal_table_in_catalog(&catalog).await;
let data_file = |path: &str| {
DataFileBuilder::default()
.content(DataContentType::Data)
.file_path(path.to_string())
.file_format(DataFileFormat::Parquet)
.file_size_in_bytes(100)
.record_count(10)
.partition(Struct::from_iter([Some(Literal::long(1))]))
.partition_spec_id(0)
.build()
.unwrap()
};

let tx = Transaction::new(&table);
let tx = tx
.fast_append()
.add_data_files(vec![data_file("test/1.parquet")])
.apply(tx)
.unwrap();
let tx = tx
.fast_append()
.add_data_files(vec![data_file("test/2.parquet")])
.apply(tx)
.unwrap();
let table = tx.commit(&catalog).await.unwrap();

let current = table.metadata().current_snapshot().unwrap();
let parent = table
.metadata()
.snapshot_by_id(current.parent_snapshot_id().unwrap())
.unwrap();
assert_eq!(
(
table.metadata().snapshots().count(),
parent.parent_snapshot_id(),
current.summary().additional_properties["total-data-files"].as_str(),
),
(2, None, "2")
);
}

#[test]
fn test_merge_requirements_keeps_the_first_requirement_on_each_value() {
let main = |snapshot_id| TableRequirement::RefSnapshotIdMatch {
r#ref: "main".to_string(),
snapshot_id,
};
let branch = TableRequirement::RefSnapshotIdMatch {
r#ref: "branch".to_string(),
snapshot_id: Some(2),
};
let mut requirements = vec![main(None), TableRequirement::CurrentSchemaIdMatch {
current_schema_id: 0,
}];

Transaction::merge_requirements(&mut requirements, vec![
main(Some(1)),
branch.clone(),
TableRequirement::CurrentSchemaIdMatch {
current_schema_id: 1,
},
TableRequirement::DefaultSpecIdMatch { default_spec_id: 0 },
]);

assert_eq!(requirements, vec![
main(None),
TableRequirement::CurrentSchemaIdMatch {
current_schema_id: 0
},
branch,
TableRequirement::DefaultSpecIdMatch { default_spec_id: 0 },
]);
}

#[tokio::test]
async fn test_commit_to_encrypted_table() {
let table = make_encrypted_table().await.with_metadata_location(
Expand Down
Loading