Repository navigation
fix(transaction): keep only the first requirement on each value #3701
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -125,11 +125,36 @@ impl Transaction { | |
| let updated_table = Self::update_table_metadata(table, &updates)?; | ||
|
|
||
| existing_updates.extend(updates); | ||
| existing_requirements.extend(requirements); | ||
| Self::merge_requirements(existing_requirements, requirements); | ||
|
|
||
| Ok(updated_table) | ||
| } | ||
|
|
||
| // Catalogs check against the base, so keep only the first requirement per value. | ||
| fn merge_requirements( | ||
| existing_requirements: &mut Vec<TableRequirement>, | ||
| requirements: Vec<TableRequirement>, | ||
| ) { | ||
| for requirement in requirements { | ||
| if !existing_requirements | ||
| .iter() | ||
| .any(|existing| Self::asserts_same_value(existing, &requirement)) | ||
| { | ||
| existing_requirements.push(requirement); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| fn asserts_same_value(left: &TableRequirement, right: &TableRequirement) -> bool { | ||
| match (left, right) { | ||
| ( | ||
| TableRequirement::RefSnapshotIdMatch { r#ref: left, .. }, | ||
| TableRequirement::RefSnapshotIdMatch { r#ref: right, .. }, | ||
| ) => left == right, | ||
| _ => std::mem::discriminant(left) == std::mem::discriminant(right), | ||
| } | ||
| } | ||
|
|
||
| /// Sets table to a new version. | ||
| pub fn upgrade_table_version(&self) -> UpgradeFormatVersionAction { | ||
| UpgradeFormatVersionAction::new() | ||
|
|
@@ -271,7 +296,7 @@ mod tests { | |
| use crate::table::{StaticTable, Table}; | ||
| use crate::test_utils::{make_encrypted_table, test_runtime}; | ||
| use crate::transaction::{ApplyTransactionAction, Transaction}; | ||
| use crate::{Catalog, Error, ErrorKind, TableCreation, TableIdent}; | ||
| use crate::{Catalog, Error, ErrorKind, TableCreation, TableIdent, TableRequirement}; | ||
|
|
||
| pub fn make_v1_table() -> Table { | ||
| let file = File::open(format!( | ||
|
|
@@ -647,6 +672,84 @@ mod tests { | |
| assert_eq!(summary.get("total-files-size").unwrap(), "300"); | ||
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_commit_two_fast_appends_in_one_transaction() { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The description says the current schema and sort order ids hit the same failure, but only the fast-append case goes through a catalog. Could we add a |
||
| let catalog = new_memory_catalog().await; | ||
| let table = make_v3_minimal_table_in_catalog(&catalog).await; | ||
| let data_file = |path: &str| { | ||
| DataFileBuilder::default() | ||
| .content(DataContentType::Data) | ||
| .file_path(path.to_string()) | ||
| .file_format(DataFileFormat::Parquet) | ||
| .file_size_in_bytes(100) | ||
| .record_count(10) | ||
| .partition(Struct::from_iter([Some(Literal::long(1))])) | ||
| .partition_spec_id(0) | ||
| .build() | ||
| .unwrap() | ||
| }; | ||
|
|
||
| let tx = Transaction::new(&table); | ||
| let tx = tx | ||
| .fast_append() | ||
| .add_data_files(vec![data_file("test/1.parquet")]) | ||
| .apply(tx) | ||
| .unwrap(); | ||
| let tx = tx | ||
| .fast_append() | ||
| .add_data_files(vec![data_file("test/2.parquet")]) | ||
| .apply(tx) | ||
| .unwrap(); | ||
| let table = tx.commit(&catalog).await.unwrap(); | ||
|
|
||
| let current = table.metadata().current_snapshot().unwrap(); | ||
| let parent = table | ||
| .metadata() | ||
| .snapshot_by_id(current.parent_snapshot_id().unwrap()) | ||
| .unwrap(); | ||
| assert_eq!( | ||
| ( | ||
| table.metadata().snapshots().count(), | ||
| parent.parent_snapshot_id(), | ||
| current.summary().additional_properties["total-data-files"].as_str(), | ||
| ), | ||
| (2, None, "2") | ||
| ); | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_merge_requirements_keeps_the_first_requirement_on_each_value() { | ||
| let main = |snapshot_id| TableRequirement::RefSnapshotIdMatch { | ||
| r#ref: "main".to_string(), | ||
| snapshot_id, | ||
| }; | ||
| let branch = TableRequirement::RefSnapshotIdMatch { | ||
| r#ref: "branch".to_string(), | ||
| snapshot_id: Some(2), | ||
| }; | ||
| let mut requirements = vec![main(None), TableRequirement::CurrentSchemaIdMatch { | ||
| current_schema_id: 0, | ||
| }]; | ||
|
|
||
| Transaction::merge_requirements(&mut requirements, vec![ | ||
| main(Some(1)), | ||
| branch.clone(), | ||
| TableRequirement::CurrentSchemaIdMatch { | ||
| current_schema_id: 1, | ||
| }, | ||
| TableRequirement::DefaultSpecIdMatch { default_spec_id: 0 }, | ||
| ]); | ||
|
|
||
| assert_eq!(requirements, vec![ | ||
| main(None), | ||
| TableRequirement::CurrentSchemaIdMatch { | ||
| current_schema_id: 0 | ||
| }, | ||
| branch, | ||
| TableRequirement::DefaultSpecIdMatch { default_spec_id: 0 }, | ||
| ]); | ||
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_commit_to_encrypted_table() { | ||
| let table = make_encrypted_table().await.with_metadata_location( | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keeping the first requirement on each value gives the base value only when every action asserts each value it changes. The RTAS replace may not do that.
build_replacementremovesmain. Java'sforReplaceTableasserts only the table UUID, and it skips the ref assertion for a replace while still suppressing later ones on that ref.If the replace action follows Java, the fast append after it derives
RefSnapshotIdMatch { ref: "main", snapshot_id: None }from the staged table (the table after the earlier actions in the transaction). That becomes the first requirement onmain, and the commit fails. I reproduced this againstMemoryCatalogwith an action that emitsRemoveSnapshotRef { main }and onlyUuidMatch, followed by a fast append on a table that already has a snapshot:How do you plan to have the replace action emit its requirements? Could we filter against the base in
do_commitinstead, whereself.tableis in scope?applyalready checks each requirement against the staged table it came from (L121-L123). So a requirement that fails on the base guards a value that an earlier action in the transaction changed, and that action decides how to guard it. For today's actions this keeps the same requirements as the PR. It also handles an action that doesn't assert what it changes, and it removesmerge_requirementsandasserts_same_value. Line 128 goes back toexisting_requirements.extend(requirements), and the commit is built like this:Whichever rule we keep, could we add the replace-then-append case as a
MemoryCatalogtest? Here is the action I used:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The replace action will assert main at the base snapshot, so a concurrent commit to main conflicts and the transaction retries against the refreshed table. With that, keeping the first requirement holds for replace-then-append, I'll happily add that test in the next PR if that works? Is it better for my to put that PR up now to see where we're going?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries. Not a blocker for me.